Compare commits

..
Author SHA1 Message Date
LumpiastyandClaude Opus 5.5 2caae7e113 docs: FORK.md/runbook after the f3397cded deploy - F14 measured (35B TTFT A/B, batch layout, edit cost, disk cache), pointer 16
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 10:48:22 +01:00
LumpiastyandClaude Opus 5.5 f3397cded0 docs: FORK.md F14, user-message batch-break merge (not deployed)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 10:15:30 +01:00
LumpiastyandClaude Opus 5.5 510aa6e090 server: merge user-message batch breaks when the rest of the prompt fits in one ubatch
Upstream (#22929, #24176, #25420) ends a prompt batch at the start of the last
user message (and of earlier ones on a fresh prompt) so that a context
checkpoint lands there. On a hybrid/recurrent model with MoE experts on the
host, that turns a typical chat turn (previous answer + new user message, under
one ubatch) into two partial ubatches, and each one of >= 32 tokens uploads the
host experts again.

When the rest of the prompt from the current batch start fits in one ubatch,
skip the user-start break. The near-end rule still checkpoints the batch start
and the end-4 rule the prompt end, so a later prompt that diverges inside the
last user message re-processes less than one ubatch more, in the same ubatch.
Prompts longer than one ubatch, and batches after a media chunk, keep the
upstream behaviour. LLAMA_USERSTART_MERGE=0 restores it everywhere.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 10:15:30 +01:00
LumpiastyandClaude Opus 5.5 166ec22e2b docs: FORK.md section 5, Inkling vs Qwen3.6-35B quality comparison
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 10:13:29 +01:00
LumpiastyandClaude Opus 5.5 e8386267ec docs: FORK.md/runbook after the b13c41355 deploy - F13 measured (KLD bug-window test, fit and placement guide, quants, profile), Inkling -ot k=4 without the offload knob, nothink fix, pointer 15
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 09:14:29 +01:00
LumpiastyandClaude Opus 5.5 b13c413555 docs: FORK.md F13, per-pass views of host weights in inkling2_loop
What and why (#32755 and the 6.5 GiB of shared expert copies), the knob,
F1/F4/MoE-cache interactions, the extra upload cost, how to drop it, and the
pod validation plan (reference with the loop experts on the GPU, views on,
views off).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 05:40:39 +01:00
LumpiastyandClaude Opus 5.5 312c29c2b1 llama: inkling2_loop - per-pass views of host weights for the later loop passes
inkling2_loop runs the core layers (6..17 on Inkling-2-Mini) three times, and
post_arch_tensors makes every cache slot share the tensors of its physical
layer. With the experts in host memory (--n-cpu-moe, -ot exps=CPU) the
scheduler makes one device copy per (host tensor, backend) per graph, at the
first split that reads it, and keeps it until the last reader. So:

- the expert copies of all 12 loop layers are alive at once (about 6.5 GiB on
  UD-Q4_K_XL), which is most of the ~7 GiB Vulkan0 compute buffer;
- sched_copy_experts runs only for the first reader. When it reads the ids back
  (ubatches of 32-63 tokens: MUL_MAT_ID is offloaded from 32 tokens, the F1
  skip starts at 4*n_expert ids), it uploads only the experts that pass 0
  selected, and passes 1 and 2 read stale VRAM for the others (upstream issue
  #32755, "GGGG" output).

Add a post_load_tensors hook to llama_model (called after the weights are
loaded, and on the no_alloc path after the dummy buffers are set). In
inkling2_loop it replaces, for each slot of a later pass, every tensor in a
host buffer with a view of it (name "<name>.visit<pass>", buffer and data of
the original). The scheduler hashes tensors by pointer, so each pass gets its
own copy, freed after that pass, and the copy callback sees the ids of that
pass. Weights in device buffers and CPU extra buffers (CPU_REPACK, not host)
stay shared. The pointer-keyed weight maps (w_scales, prec_policy, hdmd.rot)
get entries for the views. The views live in a model-owned context outside
ctxs_bufs, so the F4 release/restore, tensor counts and the memory breakdown
do not see them; they point only at host buffers, which F4 never releases.

LLAMA_LOOP_HOST_VIEWS=0 keeps the shared tensors (A/B knob).

Not built or run yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 05:40:39 +01:00
LumpiastyandClaude Opus 5.5 24b605331c docs: FORK.md/runbook after the 3621025e8 deploy - F12 measurements, Inkling ub1024 + ctx 65536 and the offload knob, nothink finding, pointer 14
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 05:38:11 +01:00
LumpiastyandClaude Opus 5.5 3621025e8f docs: FORK.md F12 temporary upstream carries (#32735, #32740), Inkling config; runbook carry procedure
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 02:33:14 +01:00
Piotr WilkinandClaude Opus 5.5 72da62cea0 upstream-carry #32740: ggml: compact bias on the existing FLASH_ATTN_EXT for CUDA, ROCm and Vulkan
Temporary carry of the open upstream PR
https://github.com/ggml-org/llama.cpp/pull/32740 at head 808fd1c4e,
its four commits (1fa93f902, d29f9d489, ff91e4f84, 808fd1c4e) squashed.
Inkling's per-query bias table is applied inside flash attention
instead of a dense [n_kv, n_tokens, n_head] bias tensor, which cuts
Inkling's compute buffers and long-context prefill time on the RX 580.
Drop this commit when the PR is merged upstream (FORK.md, "Temporary
upstream carries").

One conflict, ggml-vulkan.cpp use_mask_opt: kept the fork's GCN
`HSK >= 256 || HSV >= 256` clause and its comment (F2) and upstream's
`!use_sparse`, added the PR's `!bias &&`.

Original commit trailer: Assisted-by: Codex

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 02:30:27 +01:00
RevertronandClaude Opus 5.5 d2396e4e36 upstream-carry #32735: vulkan : support non-contiguous src in SCALE
Temporary carry of the open upstream PR
https://github.com/ggml-org/llama.cpp/pull/32735 (head 0ea4ffd00),
cherry-picked unchanged. The Vulkan backend runs SCALE on a
non-contiguous source instead of rejecting it (CPU fallback). Carried
together with #32740 and evaluated as a pair.
Drop this commit when the PR is merged upstream (FORK.md, "Temporary
upstream carries").

Original commit trailer: Assisted-By: Opus 5.5

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 02:30:27 +01:00
LumpiastyandClaude Sonnet a27217e278 docs: runbook/FORK.md fixes from the 2026-11-21 rebase, validate and deploy cycle
HF_UI_VERSION must use the tag after the '|' in .ui-stamp; bundle path
for agents told not to push; build backup and copy-based rollback;
status, pointers and last-run record for 3bf6b55b3; q35.py build path
note; disk-cache restore test wording; open questions.

Co-Authored-By: Claude Sonnet <noreply@anthropic.com>
2026-11-21 21:02:14 +01:00
LumpiastyandClaude Sonnet 5.5 3bf6b55b39 docs: runbook and FORK.md status for the 2026-11-21 rebase onto e6d624b9a
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-21 19:40:51 +01:00
LumpiastyandClaude Sonnet 5.5 3ab97b345f ggml-vulkan: gate upstream's tiled concat below one tile, honour the fast-path knob
Upstream #30591 added its own tiled-transpose concat pipelines, selected
for any dim-0 concat whose src1 is transposed. Behind the fork's fast path
(which already takes ne0 >= 32 and honours GGML_VK_CONCAT_FASTPATH) that
selector still caught the decode case (ne0 < 32), bringing back the
decode regression the TILE_DIM gate fixed (FORK.md F3), and
GGML_VK_CONCAT_FASTPATH=0 no longer gave the generic kernel.

Apply the same gate and the knob to upstream's selector.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-21 19:39:57 +01:00
LumpiastyandClaude Sonnet 1df5ced65e docs: runbook/FORK.md after the adc2aa3ec deploy
Co-Authored-By: Claude Sonnet <noreply@anthropic.com>
2026-11-21 19:39:57 +01:00
LumpiastyandClaude Opus 5.5 cf217b0318 docs: F11 mmproj incident and fix; runbook - production command line, build targets, detached scripts
FORK.md F11 records the rolled-back deploy (the GGML_ASSERT(!has_mtmd) abort
with the production mmproj), the fix, the new mmproj cases (15 PASS), and
that the earlier 35B numbers were measured with --no-mmproj. The 35B run with
production's exact command line (diskcache-window3.sh) is ready, not run, and
a precondition for deploying --cache-disk-path.

Runbook:
- tests must use production's exact model command line, mmproj included;
- the validated build has all three targets;
- the deploy mtime list includes libllama-server-impl.so and
  libllama-common.so (llama-server is a stub);
- deploy and rollback steps run as one setsid-wrapped script on the pod,
  never as a backgrounded cat && sh -n && setsid chain from kubectl exec.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:57 +01:00
LumpiastyandClaude Opus 5.5 383dbf4e6c scripts: fork-validate.sh --diskcache - the mmproj cases
Production runs the 35B and gemma with an mmproj, and the first deploy of the
disk prompt cache aborted on that path. Every case had used --no-mmproj. Now
--no-mmproj is in DC_MM, separate from DC_ARGS, and two cases run with the
0.8B's mmproj (--no-mmproj-offload, arbiter env, LLAMA_MMPROJ_ONDEMAND=1),
against their own reference:
- restore-mmproj: turn 1 is saved on SIGTERM and restored after the restart,
  prompt_n equal to the reference, output identical;
- mmproj-image: an image request (no restore, answer "red"), then the text
  chat (restored, identical), then an image again, then SIGTERM with media in
  the slot (no state copied, exit 0, the file count unchanged).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:57 +01:00
LumpiastyandClaude Opus 5.5 060e46516e server: disk prompt cache - text-only prompts with an mmproj loaded
The deploy of cb48bf224 with --cache-disk-path was rolled back: the first
request to the production 35B aborted in disk_cache_load().
server_tokens::get_tokens() has GGML_ASSERT(!has_mtmd), and has_mtmd is true
whenever an mmproj is loaded, even for a text-only prompt. Production's 35B
and gemma-26B always load one (-hf pulls it). disk_cache_collect() and
server_prompt_disk_cache::save() made the same call, so the shutdown save
would have aborted too. Every test had used --no-mmproj, which hid it.

disk_cache_text() takes the list from get_text_tokens(). A prompt is
text-only when that list is as long as size(), because media positions hold
LLAMA_TOKEN_NULL. So a text-only prompt is saved and restored with an mmproj
loaded, which is the main case (OpenWebUI chats on the 35B and gemma), and a
prompt with media is never saved or restored. When the slot itself holds
media, its reuse estimate is server_tokens::get_common_prefix(). These four
calls were the only get_tokens()/mtmd-unsafe accesses in the feature.

fork-validate --diskcache with the new mmproj cases, beside production:
15 PASS, 0 FAIL, GPU legs included. With the 0.8B's mmproj loaded, a text
chat is restored: prompt_n 58 of 596, output identical to the reference.
An image request is answered "red" with no restore, the text chat after it
is restored, and a SIGTERM with media in the slot copies no state.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:57 +01:00
LumpiastyandClaude Opus 5.5 35d58d62cd docs: FORK.md F11 - 35B results, the two leg-c anomalies explained
The swapdown window: --diskcache --arbiter 29 PASS. The 35B, 18760 tokens:
83.4 s prefill, turn 2 after a restart 3.0 s with identical output. Warm
save: token free in 328 ms, exit 1952 ms, peak RSS +430 MiB. Cold save:
peak +432 MiB.

Leg c, output != reference: c sent turn 2 again, so the server rolled back
to the checkpoint at the prompt end and processed 4 tokens, while the
reference processed 27 from turn 1. These are different reuse points. A
controlled same-reuse-point check on the 0.8B (X never restarts vs Z after a
restore, a cold save and a restore) is bit-identical, top-3 logprobs
included, on CPU and GPU. The 35B version is ready as diskcache-window2.sh
and has not run.

Leg c, +3055 MiB and token free in 1881 ms: the driver's fake waiter (flock
-s file sleep) leaked the lock into sleep, so the SIGTERM landed in an
ordinary evict_kv release (the full KV shadow, F4). Not the cache; the
driver is fixed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:57 +01:00
LumpiastyandClaude Opus 5.5 da244dc458 docs: FORK.md F11 disk prompt cache, the CPU_REPACK release fix; runbook notes
F11 in the per-feature format: design, invariants, the measured costs (CPU
only) and llama-swap's stop behaviour (it waits for the exit, SIGKILL at
30 s). Also: the recommended config (not applied), the commit map entries, the
F4 incident note for the CPU_REPACK crash, F7's shutdown time with the cache,
the F9 log lines, and the section 7 gaps (GPU leg, 35B and --arbiter not run
yet). Runbook: --diskcache in phase 1, the pod window script, the deploy
notes (every deploy empties the cache, /root free space, a slower stop), a
post-deploy check, and the cache dir.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:56 +01:00
LumpiastyandClaude Opus 5.5 062a025714 scripts: fork-validate.sh --diskcache, the disk prompt cache
Runs beside production. The model is the small hybrid (Qwen3.5-0.8B), CPU only
(-dev none -ngl 0, no VRAM), with chat requests and greedy decoding. A
reference server that never restarts runs turn 1 and turn 2. The candidate
saves turn 1 on SIGTERM and restarts. Turn 2 must then be restored from disk
with the same prompt_n as the reference (only the suffix processed) and the
same answer. Cases:
- restore plain, with the arbiter env warm, and forced cold by a fake waiter
  on want.lock (the save from a released state). Each case records the exit
  time and SIGTERM-to-token-free, also once without the cache;
- no restore for another conversation, or with another n_ctx, KV type or
  model path (other key);
- a truncated file is removed at startup and a byte-flipped one at load,
  never used;
- the size limit evicts the oldest file;
- a write error (RLIMIT_FSIZE 1 MiB, SIGXFSZ ignored: EFBIG) is logged, the
  exit stays 0 with no file left, and the next start serves;
- the same restore on the GPU (-ngl 99, arbiter env, warm and cold) when
  >= 1.5 GiB of VRAM is free, else SKIP.
start_server gains SRV_WRAP, a command prefix that execs the server.

First run (2026-10-25, beside production): 11 PASS, 0 FAIL; the GPU leg
SKIPped (1219 MiB free). The first run found the CPU_REPACK release crash
(exit 139 in the cold case).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:56 +01:00
LumpiastyandClaude Opus 5.5 7a233df171 server: disk prompt cache, save slots on shutdown and restore after a restart
llama-swap SIGTERMs a model when the owner switches models (only one large
model fits in RAM), and switching back re-prefills the whole conversation:
about 150 s for a 30k-token chat on the 35B. With --cache-disk-path DIR,
each slot with a prompt is saved to DIR on shutdown. After the restart, a
request whose prompt starts with a saved prompt is restored from the file,
so only the suffix is processed. Off by default.

Save, in server_context::start_loop() after the loop returns:
- disk_cache_collect() copies the state of each slot with >= 256 text tokens
  to host memory (llama_state_seq_get_data_ext, flags NONE) while the device
  buffers still exist. A cold server reads its released state through
  llama_context_restore_memory_host(), with no device use and no extra copy.
- vram_share_exit() then frees the VRAM and releases the GPU token as before,
  so only the D2H copy delays the token release, never the disk IO.
- disk_cache_flush() writes the files. A slot is skipped if its KV does not
  cover its token list (seq_pos_max + 1 != n_tokens: the last batch was not
  decoded) or if the same token list is already on disk.

File (server_prompt_disk_cache, server-task.cpp):
- A 48-byte header (magic, version, key hash, sizes, payload hash), then a
  payload of 8-byte padded sections: tokens, checkpoint table, state, the
  checkpoint states. The hash is FNV-1a over the 64-bit words of the payload.
- Plain fwrite to <name>.tmp, fdatasync, POSIX_FADV_DONTNEED, rename. The read
  is a plain fread plus DONTNEED. There is no mmap: upstream #28092 livelocked
  the kernel on AMD + Vulkan by DMA-ing into a MAP_SHARED file.
- The index is built at startup from the headers. A bad magic, a size that
  does not match the header, or an inconsistent table means the file is
  removed. A hash mismatch at load also removes it, and nothing is restored.
  Other keys and other versions are kept. A .tmp older than 1 h is removed.
- --cache-disk-size N MiB (default 4096, 0 = no limit): before a write, the
  least recently used *.lpcd files of every model in DIR are removed until the
  new file fits; a restore updates the file mtime. Entries the new prompt
  extends are dropped after the write and do not count against the limit.
  A write also needs its size + 1 GiB free on the file system. A write error
  or a full disk is logged; the file is not saved and the exit stays clean.

Key: the build commit, the model path + size + mtime, LoRA and control
vectors, n_ctx, n_ctx_seq, n_seq_max, n_ubatch, kv_unified, swa_full, the
K/V types, the flash-attn type, and the RoPE/YaRN parameters.

Restore, in get_available_slot() after the RAM cache:
- Only while warm (holding the GPU token), with cache_prompt on and no media
  tokens. The file must reuse at least 256 more tokens than the slot holds.
- Reusable means one of two things. Either the saved token list is an exact
  prefix of the new prompt, or (for a recurrent/SWA model) a saved checkpoint
  at or below the common prefix can be used. A plain-KV model can be cut back
  to any prefix. Exact prefix alone is not enough for OpenWebUI on Qwen: the
  template rewrites the previous assistant turn (the reasoning is dropped), so
  the saved list is almost never a prefix of the next turn. The checkpoints
  are exact snapshots, and update_slots() rolls back to one exactly as it does
  for the RAM cache. They are restored with id_task -1, so one that fails to
  load means a re-prefill, not an abort.
- The slot's current prompt goes to the RAM cache first. A set_data failure
  clears the slot and removes the file.

Not supported with speculative decoding (warning, disabled): the draft state
is not saved. Only the slots are saved, not the RAM cache entries.

Measured on the pod, CPU only, Qwen3.5-0.8B (hybrid, 6 of 24 layers attention):
- 573-token chat: 64.5 MiB file with 2 checkpoints, copy 18 ms, write 171 ms,
  exit 314 ms (125 ms without the cache). After the restart: read 166 ms,
  restore 3 ms, turn 2 prompt_n 58 of 596 in 462 ms against a 2913 ms full
  prefill. The answer is identical to a server that never restarted.
- 12k tokens: 200.7 MiB, copy 110 ms, write 350 ms, exit 641 ms, peak RSS
  +162 MiB during the exit save. After the restart: read 307 ms, restore 18 ms,
  turn 2 prompt_n 35 of 12188 in 1.2 s against a 72 s prefill.
- The /root PVC writes at about 0.66 GB/s with fdatasync. /tmp (the node
  overlay) stalled once: 5.8 s for 64.5 MiB.
llama-swap v239 waits for the old process to exit before it starts the next
model, and SIGKILLs a process that is still there after 30 s. So the exit
save adds to the model switch time: about 1 s per GB on the PVC.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:56 +01:00
LumpiastyandClaude Opus 5.5 7dbe7fdafa llama: read a released KV/recurrent state from its host shadow
After llama_context_release_device(ctx, true) the KV cache and the recurrent
state live only in host shadows, and their tensors have no buffer, so
llama_state_seq_get_data cannot read them. The disk prompt cache must save
a cold server's slots at exit, and a shutting-down server must not restore
to the device (it may not hold the GPU token, F7).

llama_context_restore_memory_host() makes the state readable without the
device. restore_device_buffers(to_host = true) wraps each shadow as a
CPU_Mapped buffer (ggml_backend_cpu_buffer_from_ptr) and points the tensors
into it: there is no copy and no allocation. The first version copied into a
CPU buffer instead, which needs the full KV allocation a second time, about
2.7 GB for the 35B at ctx 131072; the node has had an OOM-controller kill
before. The buffer base is rounded down to the 32-byte alignment that
cpu_buffer_from_ptr asserts (a large std::vector is only 16-byte aligned);
those leading bytes are never accessed. With to_host the shadows are kept,
because the buffers point into them.

It sets device_discarded, so restore_device() aborts afterwards. Exit only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:56 +01:00
LumpiastyandClaude Opus 5.5 2766d7977b llama: do not release CPU extra buffers (CPU_REPACK) on device release
A model with weights on the CPU device (-dev none / -ngl 0) segfaulted in
llama_model::release_device_weights() the first time the VRAM arbiter put
it to sleep: CPU_REPACK buffers are not host buffers, so load_tensors marked
them releasable, and capturing the shadow called their NULL get_tensor.
Reproduced with the production build 2fc9f9c35 (Qwen3.5-0.8B, -dev none
-ngl 0, LLAMA_SLEEP_VRAM_ONLY=1 LLAMA_SLEEP_EVICT_KV=1, force a yield with a
waiter on want.lock): exit 139 at the release, before any SIGTERM. The
faulting call (rip 0) returned into release_device_weights.

Only buffers on a non-CPU device are releasable now. These buffers are in RAM
already, so there is nothing to free. Production is not affected: every
arbiter model there has its weights on the GPU, and a releasable CPU_REPACK
buffer would have crashed it. Found by the --diskcache check, which runs
its arbiter cases CPU only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:51 +01:00
LumpiastyandClaude Opus 5.5 4b29f38b92 docs: FORK.md and runbook after the 2fc9f9c35 deploy
Feedback from the 2026-10-25 deploy, which used only these docs:
- status: production is 2fc9f9c35; F10 deployed and verified on a 35B
  GPU load; --moe-exact and median --perf have run on the GPU; last
  full fork-validate run (37 PASS) and its timings; the first-candidate
  -arm VRAM spike seen twice is an open question.
- runbook: a no-rebase redeploy path; --arbiter beside production only
  if 1536 MiB VRAM is free, else in the swapdown window; full --moecache
  model path; page-cache eviction before and after phase 2; deploy
  steps reordered so the memory check follows stopping llama-swap;
  in-place rebuild and system_fingerprint check; mtime check through the
  versioned symlinks; rollback pointer numbering and one format; disk
  space; nothink alias and a PPM image helper for post-deploy requests;
  power1_cap via sysfs; restart and OOM checks; harmless UI warning.
- fork-validate.sh header: note the --arbiter VRAM requirement.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:51 +01:00
LumpiastyandClaude Opus 5.5 dbae58b893 docs: FORK.md F10 page-cache drop, -lm dio dead end; runbook note
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:51 +01:00
LumpiastyandClaude Opus 5.5 f58520aa4f llama: drop the model file's page cache after a copying load
With -lm none the GGUF is read() into our own buffers, but the kernel keeps
the file in page cache afterwards, charged to the pod's cgroup: about 20 GB
after a 35B load. On 2026-10-24 the Talos OOM controller killed the pod right
after a load under exactly that pressure. Those pages are pure waste once the
data is copied.

llama_file::drop_cache() wraps posix_fadvise(POSIX_FADV_DONTNEED) (Linux only,
errors logged at debug). load_all_data() drops each copied tensor's range right
after reading it, so the cache never grows past about one tensor, and drops the
whole file at the final cleanup to catch the metadata and partial pages at
tensor edges. Both only when the loader holds no mapping of the files
(mappings.empty(): not mmap, not mmap+mlock, no lazy tensors), so nothing that
is mapped is ever advised away. Split GGUFs are covered (every file in files).
clip.cpp always copies the mmproj via ifstream, so it drops the whole file
after loading it.

Measured on the pod, gemma-4-E2B UD-Q4_K_XL (2.5 GiB) + mmproj BF16 (0.9 GiB),
CPU only, -lm none, cold cache, same binary with the drops disabled as the
control (4 rounds, rotated order):
- file pages resident after load (mincore): 2431-2444 / 940 MiB -> 0 / 0
- peak model-file residency during load: 2.5 GiB -> 1.2-1.5 GiB (largest tensor)
- pod cgroup file after load: +3.37 GiB -> +0; host MemFree after load
  0.6 GB -> 2.5-2.8 GB (MemAvailable counts clean cache, so it cannot show this)
- load time: 5.87 s -> 6.82 s median (+0.95 s); a post-load-only drop is
  6.56 s (+0.68 s). The cost is synchronous page freeing that reclaim would
  otherwise do later; -lm dio (direct IO) costs the same (+0.9 s) and does not
  cover metadata or the mmproj.
- greedy output byte-identical; -lm mmap keeps the model file fully resident
  and mapped (2498 MiB, no refaults); -lzm on skips the drop (file mapped).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:51 +01:00
LumpiastyandClaude Opus 5.5 263d75cba3 docs: runbook - new fork-validate checks, gdb gone, scratch lost on restart
Phase 1 now covers the VRAM release/restore check and the handoff latency
A/B against --ref (which must be the production build); phase 2 adds
--moe-exact on the 35B and keeps its scratch dir. Explains median gating and
--perf-report, that a pod restart empties /tmp, that the manual checks are
now automated, and that gdb is not in the pod image (lost in the 2026-10-24
restart). States the commit trailer convention and what llama-swap does with
a child that exits vs one that wedges.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:51 +01:00
LumpiastyandClaude Opus 5.5 5f753f7a0c docs(readme): short fork summary, FORK.md is the source of truth
The README fork section still said "the two code changes below are auto-on"
(the fork has three prefill changes plus the VRAM sharing and server fixes)
and recommended --n-cpu-moe 28 on the 35B, which FORK.md section 4 records as
worse at depth and unable to hold a long context. Replace it with a short
accurate summary that points to FORK.md, the runbook and the scripts. The
Laguna time profile, the iSWA KV note and Laguna's --parallel 1 saving were
only in the README and move to FORK.md sections 4 and 5.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:51 +01:00
LumpiastyandClaude Opus 5.5 4e455b79df docs: FORK.md - review fixes, new checks, llama-swap restart behaviour
- status: no hardcoded fork-commit count (the list includes docs and
  scripts; count git log origin/master..HEAD); what is after the production
  commit; the commit-trailer convention of this private fork vs upstream's
  AGENTS.md rules
- commit map: the commits after the production commit
- F1: validated by the new --moe-exact check and the LLAMA_MOE_IDS_SKIP knob
- F3: every graph that builds concat(state, transpose(x), 0) (bailingmoe3,
  clef, kimi-linear, kimi-k3, glm5-next, mamba-base, plamo2, lfm-base,
  qwen4exp besides delta-net); only the delta-net path is measured
- F4/F6: the new v-vram-* and a-handoff-latency checks with first results
- F5: encode_mtmd_chunk was split out of process_mtmd_chunk, not renamed
- F6: the vram_cold ordering rule restated from the code (the measured
  failure was under the doorbell design; with the inline probe should_sleep
  cannot run inside the restore; keep it, and A/B handoff latency before any
  change) and why vram_acquire_for_load sets the flag early
- F7: what llama-swap does when a child dies, verified with a scratch
  llama-swap: "upstream exited unexpectedly", restart on the next request
- F9: new flags, phases, median gating, 2026-10-24 reference values, the
  /tmp and gdb loss on the pod restart, new grep targets and gaps
- section 7: stale comments fixed, what is not yet run on the GPU

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:51 +01:00
LumpiastyandClaude Opus 5.5 9537f885e0 scripts: fork-validate.sh - --moe-exact, greedy identity for the readback skip
F1 (the routing-ids readback skip in sched_copy_experts) had no automated
check. --moe-exact MODEL reads expert_count / expert_used_count from the GGUF
header, computes the skip threshold ceil(4 * n_expert / n_expert_used) tokens
per ubatch (128 on the 35B), and runs one greedy request with a prompt of
about 2x that in a single ubatch, every expert on the host, so the prefill
takes the skip and each decode step takes the readback. The candidate's 64
tokens must equal the reference's byte for byte; with the LLAMA_MOE_IDS_SKIP
knob in the candidate, also the candidate with the skip forced off (=0), which
proves the skip exact even when the reference carries it, and =log must show
skips, all at >= 4 * n_expert ids. Without the knob skip-engaged SKIPs with
the batch-size argument. It needs the GPU to itself (same guard as --perf,
now shared as gpu_exclusive); FV_TEST_SHARED_GPU=1 bypasses the guard for
script tests against stub binaries only.

Also unset inherited LLAMA_MOE* variables.

Not run on the GPU yet (needs a swapdown window). Dry-run on the pod
against stub servers and the real 35B GGUF header: threshold 128, all-PASS
when outputs match, FAIL on a differing reference, SKIP without the knob,
FAIL for a dense model and while llama-swap runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:51 +01:00
LumpiastyandClaude Opus 5.5 12a2add2c5 scripts: fork-validate.sh - check that a cold server gives its VRAM back
Nothing checked that a release actually frees VRAM. --arbiter now takes a
baseline of mem_info_vram_used before any scratch server starts, then with A
warm and alone on the scratch arena makes it release through a fake waiter
on want.lock. v-vram-release requires the card-wide delta to drop by at
least 70% of what A added warm; v-vram-restore requires it back (and the
output equal to the reference) after the next request. Production's own
share is re-read at every point from the DRM fdinfo of every process the
script did not start (deduplicated by drm-client-id); if it moved by more
than 64 MiB the check SKIPs instead of blaming the candidate.

Run beside production on the pod (reb1024 vs itself): warm +546 MiB, cold
+4 MiB, restored +546 MiB, production steady at 5053 MiB; 16 PASS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:51 +01:00
LumpiastyandClaude Opus 5.5 deb09fb305 scripts: fork-validate.sh - A/B ordinary handoff latency against --ref
--arbiter only reported the candidate's handoff latency, while the fork's
process rule (two production deploys lost) is to A/B the ordinary handoff,
not only the pathological cases. After the arbiter cases, a-handoff-latency
starts fresh A/B server pairs of each build on the scratch arena and runs
blocks ref, cand, cand, ref of 16 alternating handoffs each. It FAILs if the
candidate median exceeds the reference median by more than max(25%, 200 ms)
and reports both medians, maxima and per-block medians.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:51 +01:00
LumpiastyandClaude Opus 5.5 3b2c7188fb scripts: fork-validate.sh - gate --perf on medians, not means
--perf averaged the two per-arm llama-bench means of each build, so one
catastrophic sample (CPU contention from the ~60 other pods on the node) moved
the result, against the fork's own rule of medians, never means. llama-bench
now writes -o jsonl (every repetition sample) next to the markdown table, and
the gates use the median of all samples of a build pooled over its two arms
(4 per prefill cell, 6 for decode); the table shows min-max and n.

--perf-report TSV re-aggregates a saved <scratch>/perf.tsv without a GPU.

Checked offline on the pod: the 2026-09 A/B samples in results.txt give
pp2048 reference 250.24 (old mean) vs 247.76 (new median), candidate 280.15
vs 276.93; a synthetic 2026-10-24-like set with one 180 t/s candidate sample
read -9.3% (FAIL) the old way and -0.2% (PASS) with medians.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:51 +01:00
LumpiastyandClaude Opus 5.5 3141ad2a9e scripts: rx580-bench - decode at tg256 depth 4096, not tg32
The llama-bench sweep measured decode as tg32 at the default depth and the
README called it the continuity metric, while tg32 at n_ctx 32 is launch /
barrier bound on this card and is not a valid decode metric (it once read
-36% between builds at parity at tg256). lbsweep.sh now runs prefill
(-p $LB_P -n 0, depth 0) and decode (-p 0 -n 256 -d 4096, LB_N / LB_D) as
two invocations and labels depth cells (tg256@d4096). summarize.py marks
old depth-less tg rows as invalid-metric history. The README examples and
the reference values are updated to the 2026-10-24 production numbers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:51 +01:00
LumpiastyandClaude Opus 5.5 0ecd1a43cc llama: env knob for the routing-ids readback skip
LLAMA_MOE_IDS_SKIP=0 forces the ids readback in sched_copy_experts (the
upstream path) and =log reports each skip on stderr, so validation can show
that the skip engages at prefill and that output is byte-identical with and
without it in one build. Read once (function-local statics); unset changes
nothing. Not set in production.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:51 +01:00
LumpiastyandClaude Opus 5.5 bf3bda9f65 server: fix stale VRAM-arbiter comments and the sleep log line
Comments only, plus one log string:
- the arbiter block comment said the KV cache is never evicted; with
  LLAMA_SLEEP_EVICT_KV it is (and the scheduler and scratch with it)
- vram_go_cold said vram_ensure_warm() rebuilds the encoder; only
  encode_mtmd_chunk() restores it, and only for an encode
- vram_ensure_warm and the mtmd init block named process_mtmd_chunk; the
  restore lives in encode_mtmd_chunk
- handle_sleeping_state said waking happens lazily before decode; it
  restores eagerly on the wake transition, before update_slots()
- the vram_cold ordering note now says why it still holds with the inline
  want-lock probe
- the sleep log line said "keeping KV cache" even when LLAMA_SLEEP_EVICT_KV
  evicts it; it now names what actually happens (the prefix
  "server entering sleeping state" that fork-validate.sh counts is kept)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:51 +01:00
LumpiastyandClaude Opus 5.5 bdf2b81b81 docs: FORK.md and rebase runbook for the RX 580 fork
FORK.md documents every fork feature (what and why with measured numbers,
where, how it works, invariants, past failures, rebase hazards, validation),
the dead ends measured on this hardware, the production configuration and
the measurement traps. docs/fork-rebase-runbook.md is the rebase, validation,
deploy, rollback and GPU-recovery procedure plus an operations reference.
The README fork section links both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:51 +01:00
LumpiastyandClaude Opus 5.5 a213514855 scripts: add fork-validate.sh, post-rebase validation of the fork
Checks the op kernels, the concat fast path, the VRAM arbiter incl. every
shutdown path that has caused a production incident, and prefill/decode
against a reference build. Prints PASS/FAIL per check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:51 +01:00
LumpiastyandClaude Opus 5.5 8742446dba ggml-vulkan: env knob for the concat transpose fast path
GGML_VK_CONCAT_FASTPATH=0 forces the generic path and =log reports each
fast-path dispatch, so validation can prove the path fires at prefill and
not at decode.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:51 +01:00
LumpiastyandClaude Opus 5.5 79e6ade8b3 server: release the GPU token promptly on shutdown
Two ways a shutting-down server kept the VRAM arbiter's token, stalling every
other model:

- SIGTERM with a request in flight: terminate() stopped only the task queue.
  The HTTP handler kept waiting for results the loop would never send, the main
  thread blocked joining it, and the token was released only in the destructor
  after that join - i.e. never, until the client went away. Terminate the
  result queue too (readers now get "server is shutting down": a 503, or an SSE
  error event), and let wait_until_no_sleep give up once stopping.

- SIGTERM while waiting for the token: the loop thread sat in a blocking flock
  inside the wake callback, holding mutex_tasks, so terminate() waited for the
  current holder to finish and then for a pointless restore of our weights.
  Poll the lock non-blocking and abort the wait on shutdown; the aborted wake
  stays cold, skips the restore, and the loop exits before update_slots.

Also release the token right after start_loop() returns rather than in the
destructor, and only after the device buffers are actually freed: the old
order unlocked before destroy(), so the next model could start uploading into
VRAM that was still occupied. Freeing at exit uses a new
llama_context_discard_device(), which drops weights, KV, MoE cache and scratch
without capturing host shadows first.

Off production: SIGTERM mid-generation or mid-wait now exits in ~0.2 s with
exit 0 and frees the token within 5-70 ms (previously never / after the
holder's whole generation). Handoff latency and greedy output are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:51 +01:00
LumpiastyandClaude Opus 5.5 3b068bdb3a server: run the shutdown handler off the signal-interrupted thread
A SIGTERM that lands while the start_loop() thread is inside a sleeping
callback deadlocks the server. The callbacks run under
server_queue::mutex_tasks, and the kernel delivers the signal to that same
thread; the handler calls terminate(), which locks mutex_tasks again, and the
thread waits on itself forever.

With the VRAM arbiter that window is wide - a whole device release, or a
blocking wait for the GPU token - and llama-swap sends SIGTERM to evict
models, so it hit production: an evicted model wedged mid-release, never
released token.lock, and every other model queued behind it. SIGTERM could
not stop it either; only SIGKILL did.

Make the handler only write the signal number to a self-pipe, which is
async-signal-safe, and run shutdown_handler on a dedicated thread. terminate()
there simply waits until the loop thread drops mutex_tasks, after which the
loop exits normally. The thread is joined before the objects the handler uses
are destroyed. If pipe() fails the old direct call is kept.

Reproduced 6/6 off production (SIGTERM during release and during the token
wait); with the fix 8/8 clean exits, and handoffs, image requests and greedy
output are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:51 +01:00
LumpiastyandClaude Opus 5.5 3d4e7bc065 scripts: rx580-bench - use -lm none, document the MoE cache option
Upstream #28334 removed --mmap/--no-mmap/--mlock and llama-bench's -mmp, so the
harness flags no longer parse: the server would refuse to start and llama-bench
would silently produce no rows. Switch both harnesses to -lm none, the same
load mode, and note --moe-cache-mib as a server-only option worth measuring.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-21 19:39:51 +01:00
LumpiastyandClaude Opus 5.5 a620add3b7 llama: release/restore the MoE cache with the device weights
The MoE cache (--moe-cache-mib, #29887) keeps the most recently used host
experts in device banks. The on-demand VRAM path (release_device /
restore_device, used by the server VRAM arbiter and the on-demand mmproj path)
did not know about it, so its banks were never freed on handoff. Restoring them
by reallocation alone would also be wrong: the banks come back zeroed while the
slot maps still mark experts as cached, so the model would silently read zero
weights.

Free the banks on release and empty the cache instead of shadowing it: re-init
every LRU, re-point its slot-map pointers at the host slot maps and clear those
to -1. The banks are reallocated (which also re-inits their views) and cleared
on restore, before the scheduler is rebuilt, and the cache refills from the
host experts as before. Release runs in both modes, weights-only and evict_kv.

device_resident() now covers the cache too, for the early returns in
release/restore, the auto-restore in decode() and
llama_context_weights_resident().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-21 19:39:51 +01:00
LumpiastyandClaude Opus 5 9c042626b3 llama: skip the compute-buffer size check when the context is cold
release_device(evict_kv) drops the scheduler, so a context evicted by the VRAM
arbiter and then destroyed reaches the destructor with a null sched. The upstream
"compute buffer size matches expectation" loop calls
ggml_backend_sched_get_buffer_size() unconditionally, whose GGML_ASSERT(sched)
then aborts: the server exits 134 instead of 0 on every shutdown taken while
cold, which under llama-swap makes an ordinary stop look like a crashed child and
leaves a ggml backtrace in the log each time. It also skipped the rest of the
destructor, so the cold teardown path had never actually run to completion.

Guard at the call site rather than relaxing the assert - the assert is right, and
every other caller reserves the scheduler first. Same shape as the null guards in
synchronize(), memory_breakdown() and the released-buffer iteration.

The loop is a diagnostic size comparison, already reported by sched_reserve() at
load, so skipping it when cold loses nothing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-21 19:39:51 +01:00
LumpiastyandClaude Opus 5 2c6aa3b6e7 server: replace the VRAM doorbell with a want-lock
The doorbell was edge-triggered: a waiter wrote <arena>/doorbell/<pid> and the
holder's warden turned the inotify event into a queue flag. A ring carried no
notion of "still wanted", so it was lost whenever the holder was not already warm
and listening - during its model load (the watch does not exist yet, and the
kernel does not queue events for a watch that is not there) or inside
restore_device - and any ring that survived past the point its sender had been
satisfied caused a spurious release. Lost rings wedged the waiter permanently,
because flock(LOCK_EX) never times out and, with idle-sleep disabled, the holder
had no other reason to release. The wait blocks under mutex_tasks, so the whole
server stopped answering while /health still returned 200.

Express the request as kernel state instead. A waiter holds <arena>/want.lock
shared while it waits and drops it once it owns the token; the sleep decision
probes that lock non-blocking and releases the GPU while anyone is waiting. The
probe needs its own fd - flock treats two open file descriptions of one file
independently, so probing on the waiter's fd would convert our own lock rather
than conflict with it. Nothing can be missed, nothing goes stale, and a waiter
that dies is cleaned up by the kernel.

Probe from should_sleep() on the loop thread rather than from a warden thread.
Routing it through a flag is what made the first attempts fail: start_loop()
holds mutex_tasks from should_sleep() through the callbacks to the wait, so a
warden's request_yield() blocks on that mutex and is admitted only after the flag
has been consumed, latching a release for the next wake. Reading live state where
the decision is made has no edge to latch, and drops the warden's poll latency.

Two sleep-path bugs this exposed: wait_until_no_sleep() waited on !sleeping but
the loop clears req_stop_sleeping on the way in, so a loop that slept again
before the waiter ran stranded it forever - re-ask on every wake. And a task
queued after the waiter saw us awake could not wake us by itself, so sleep now
also breaks on a non-empty queue. Hold the GPU for 100 ms after a wake: the
request that woke us is not queued yet, and yielding at once only sends it round
again.

Measured on the RX 580 pod, two servers contending, 60 alternating handoffs:
0 stranded, median 0.309 s against the doorbell's 0.314 s.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-21 19:39:51 +01:00
LumpiastyandClaude Opus 5 c61db474d8 scripts: add the RX 580 benchmark harness
Encodes the production config, fixed corpus slices, repeat/median discipline
and the noise floor, so the measurement method does not have to be
rediscovered each time. Runs the corpus prefill test through llama-server and
the llama-bench sweep as a controlled cross-check, with interleaved A/B.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-21 19:39:51 +01:00
LumpiastyandClaude Opus 5 7194d47311 ggml-vulkan: skip the concat transpose fast path below one tile
The tiled-transpose fast path in ggml_vk_concat is a large prefill win but a
loss at decode. On a hybrid model like Qwen3.6-35B (gated delta-net on 30 of
40 layers) build_conv_state emits a transposed concat per recurrent layer, so
the fast path runs 30 times per decoded token.

At one token the transposed source is a single column, so there is no
uncoalesced stride left to fix, but the path still costs two dispatches and an
unconditional ggml_vk_sync_buffers - a full pipeline barrier that serializes
the command stream. Measured -17% tg256 on RX 580; the barrier, not the
half-empty transpose dispatch, is nearly all of it.

Require the transposed source to be at least one TILE_DIM wide. That source is
qkv_mixed transposed, so ne[0] is the ubatch token count: prefill keeps the
fast path, decode and small speculative drafts take the generic one.

pp8192 +11% retained, tg256 back to parity, generation byte-identical.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-21 19:39:51 +01:00
Lumpiasty 04552507fe docs(readme): Polaris MoE profile - GCN mask_opt, the ids-readback win, tuning and dead ends 2026-11-21 19:39:51 +01:00
Lumpiasty d773ab0ffc server: on-demand mmproj - free encoder VRAM on the text path
The vision encoder sits idle in VRAM on every text request. With
LLAMA_MMPROJ_ONDEMAND=1 the server keeps the encoder in RAM (released
after load) and brings it into VRAM just-in-time before mtmd_batch_encode,
releasing it again afterwards - so the encoder's VRAM is free for KV /
expert cache on the common text-only path.

When the encoder does not fit (that VRAM has been claimed), evict the LLM
backbone weights first: they are not needed while the encoder runs (the
decode that uses them happens afterwards), their host shadow is read-only
(cheap free, no D2H), and the KV / prompt cache is left untouched. Order
on the way back matters: free the encoder before re-uploading the weights
so the peak stays within VRAM.

Also wire the encoder release/restore into the VRAM arbiter: vram_go_cold
releases it; the just-in-time encode-path restore replaces the previous
restore in vram_ensure_warm.

Assisted-by: Claude
2026-11-21 19:39:51 +01:00
Lumpiasty 58e693b794 mtmd: release/restore the encoder weights from VRAM on demand
Add clip_release_device/clip_restore_device (and mtmd_release_device/
mtmd_restore_device wrappers over the vision + audio contexts) that free
the multimodal encoder's device weight buffer to a read-only host shadow
and rebuild it on demand, using the same shadow/free/reallocate pattern
as llama_model weights. No-op for a CPU-backed encoder. This lets the
server drop the ~hundreds-of-MiB vision encoder from VRAM when it is not
encoding an image.

Assisted-by: Claude
2026-11-21 19:39:51 +01:00
Lumpiasty b7cb284e54 llama: free backend compute scratch on cold; guard memory_breakdown
On device release with KV eviction, also call ggml_backend_free_scratch()
on each backend so a cold model drops its Vulkan compute preallocations
(reallocated lazily on the next compute via restore).

Also guard llama_context::memory_breakdown() against a freed scheduler:
evict_kv release resets sched to null, so a cold model that is then torn
down (e.g. terminated by the process manager) hit GGML_ASSERT(sched) in
ggml_backend_sched_get_buffer_type. Skip the compute-buffer accounting
when sched is null.

Assisted-by: Claude
2026-11-21 19:39:51 +01:00
Lumpiasty fa24747925 ggml: add ggml_backend_free_scratch to drop Vulkan compute prealloc
Add an optional backend interface method free_scratch (with a public
ggml_backend_free_scratch wrapper) that frees transient/scratch device
memory a backend holds outside of any allocated buffer, keeping the
backend usable - the scratch is reallocated lazily on the next compute.

Implement it for the Vulkan backend (ggml_backend_vk_free_scratch): free
the prealloc_x/y/split_k/add_rms_partials and sync_staging device buffers
and reset their sizes, so an idle/cold model does not hold the vision or
matmul compute preallocations in VRAM. All other backends leave the hook
null (no-op).

Assisted-by: Claude
2026-11-21 19:39:51 +01:00
Lumpiasty 98c7e109b7 llama: evict recurrent/SSM state on device release
The recurrent (SSM/conv) state of hybrid models (e.g. Qwen3.5) was left
resident when a model's device buffers were released for on-demand VRAM
sharing - llama_memory_recurrent::release_device_buffers() was a no-op
default. Implement it (and restore_device_buffers) with the same
capture-host-shadow / free / reallocate pattern as llama_kv_cache, so
llama_memory_hybrid now evicts both its attention KV and its recurrent
state. The state is read-write, so its shadow is recaptured on every
release.

Assisted-by: Claude
2026-11-21 19:39:51 +01:00
Lumpiasty bc55fb9990 server: also free the compute-graph scheduler on KV eviction
Under LLAMA_SLEEP_EVICT_KV a cold model still held the scheduler's worst-case
compute buffer (hundreds of MiB, e.g. ~700 MiB for gemma-26B) plus the resident
experts. release_device(evict_kv) now also frees the sched (sched.reset() +
sched_need_reserve), and restore_device() rebuilds it via sched_reserve(), so a
fully-evicted model holds essentially no VRAM (gemma cold: 7147 -> 51 MiB).
Trades a heavier re-warm (weights+KV H2D + sched reserve) for the freed VRAM;
weights-only mode is unchanged (fast switch, keeps KV+compute).

Assisted-by: Claude
2026-11-21 19:39:46 +01:00
Lumpiasty 25c12e56ca server: restore weights/KV on wake, not at decode, so KV eviction is safe
With LLAMA_SLEEP_EVICT_KV the KV cache device buffers are freed when a model
goes cold. update_slots() touches the KV before the decode-time vram_ensure_warm
(e.g. SWA models create a checkpoint that reads the KV via ggml_backend_tensor_get),
so the KV must already be resident by then. Move the restore to the sleep-wake
handler (handle_sleeping_state(false)), which runs before update_slots, fixing a
GGML_ASSERT(buffer) crash on iSWA models (gemma) when KV eviction is enabled.

Assisted-by: Claude
2026-11-21 19:39:46 +01:00
Lumpiasty 8e575c9e54 llama: guard buffer iteration against released device buffers
release_device_weights()/release_device_buffers() leave a null buffer in
ctxs_bufs while the device memory is released for on-demand VRAM sharing.
The memory-breakdown / total_size / clear paths iterated these and called
ggml_backend_buffer_get_size()/get_type()/clear() on the null buffer, tripping
GGML_ASSERT(buffer) and aborting on shutdown of a cold model. Skip null buffers
in llama_model::memory_breakdown and llama_kv_cache::{memory_breakdown,
total_size,clear}.

Assisted-by: Claude
2026-11-21 19:39:46 +01:00
Lumpiasty 1ee9806b3d server: coordinate model load with the VRAM arbiter to avoid load-time OOM
Before uploading a model's weights, acquire the shared VRAM token (ring the
doorbell so any resident model releases first). Previously load uploaded weights
to VRAM before the arbiter was active, so loading a large model while another
(e.g. the warm 4B task model) held VRAM could exceed the budget and OOM.

- add vram_arena_open() (idempotent flock/doorbell setup) and
  vram_acquire_for_load(), called from load_model() before
  common_init_from_params().
- a coordinated load now stays warm holding the token and serves its first
  request without a re-warm (drop the init-time go_cold cold-start).

Assisted-by: Claude
2026-11-21 19:39:46 +01:00
Lumpiasty 558f7f65a2 server: keep the sleep-wake path active when the VRAM arbiter is enabled
The request handler only calls wait_until_no_sleep() (which wakes a server out
of its sleeping state) when sleep_idle_seconds >= 0. But the VRAM arbiter's
cross-process doorbell can put a server to sleep even when idle-sleep is
disabled, so without this a doorbell-slept server would never wake and requests
to it would hang until timeout.

Do not bypass the wake path when LLAMA_SLEEP_VRAM_ONLY is set, so the arbiter no
longer depends on --sleep-idle-seconds being configured.

Assisted-by: Claude
2026-11-21 19:39:46 +01:00
Lumpiasty cb471f9b2f server: optionally evict the KV cache too (Phase 2 of VRAM sharing)
Extend on-demand device residency to the KV cache so that when a model's KV
plus another model would not fit in VRAM, the KV can also be evicted to a host
shadow (D2H on release, H2D on restore) instead of only the weights.

- llama_memory_i: add release_device_buffers()/restore_device_buffers()
  (default no-op). Implemented in llama_kv_cache (D2H shadow of the live
  ctxs_bufs, freed and reallocated like the weights); llama_memory_hybrid and
  llama_kv_cache_iswa delegate to their child caches.
- llama_context::release_device(evict_kv): also evict the memory's device
  buffers when requested; restore_device() rebuilds them. Public API
  llama_context_release_device gains an evict_kv flag.
- server: LLAMA_SLEEP_EVICT_KV=1 enables it. Off by default (weights-only),
  since the KV shadow adds a D2H/H2D copy of the live cache each cycle.

Validated on RX 580 (Vulkan), 4B @ 32k ctx: weights-only cold VRAM 1750 MB
(KV stays); weights+KV cold VRAM 726 MB (KV freed, ~1 GB reclaimed). KV
survives the round-trip: prompt cache reused after the cycle (prompt_n 4 vs
42), correct output.

Assisted-by: Claude
2026-11-21 19:39:46 +01:00
Lumpiasty 56a1010bbc server: on-demand VRAM sharing to time-share one GPU between models
Add release/restore of a model's GPU weight buffers (keeping a host shadow
and the KV cache) so several always-loaded llama-server processes can
time-share a single GPU without reloading or losing the prompt cache.

- llama-model: release_device_weights()/restore_device_weights() capture a
  compact host shadow (stable iteration order, view-skipping) and free then
  realloc the device weight buffers; weights_resident() query.
- llama-context: release_device()/restore_device() wrappers; decode() auto-
  restores; public C API llama_context_release_device/restore_device.
- server: LLAMA_SLEEP_VRAM_ONLY makes idle-sleep release only the VRAM weights
  (not a full unload/reload). A cross-process flock token in LLAMA_VRAM_ARENA
  enforces "resident iff holds token"; an inotify doorbell forces the holder
  to release on contention. The warden thread only touches the task queue, so
  releases run on the loop thread and never race a decode.

Validated on RX 580 (Vulkan): two models share 8GB, never both resident,
correct output under contention, KV cache preserved (no re-prefill).

Assisted-by: Claude
2026-11-21 19:32:30 +01:00
Lumpiasty cfda297cbb ggml-vulkan: tiled transpose fast-path for concat with transposed source
The generic concat shader reads a transposed source (nb[1]==type_size) with an
uncoalesced stride, which is catastrophically slow on discrete GPUs (~5.6ms for
a 16MB concat on the RX 580 vs ~130us of memory bandwidth), a ~9% prefill
hotspot on Qwen3.5-4B (delta-net state concat).

Add a fast path for concat along dim 0 where one source is stored transposed and
the other source + dst are contiguous along dim 0: copy the contiguous source
with copy.comp and transpose the other source into the matching dst sub-region
with the existing tiled copy_transpose shader (shared-memory 32x32 transpose,
coalesced read+write). Reuses pipeline_cpy_* / pipeline_cpy_transpose_* with
custom push constants + doffset, no new shader. Falls back to the generic path
otherwise (gated on type/shape/contiguity + 16-bit doffset bound).

Assisted-by: opencode
2026-11-21 19:32:13 +01:00
LumpiastyandClaude Opus 5 e1bc1f7398 vulkan: record why flash-attn shmem staging stays off for GCN
Staging K/V through shared memory looks like an obvious win on GCN: without it
each rowgroup re-reads the whole K/V block from global memory, and with row_split
4 that pulls one 16KB block through a 16KB vector L1 four times.

Measured, it loses: -6.7% pp2048 at depth 16k and -7.4% at 32k on Polaris at head
size 128. The kvsh stride of D/4+1 dwords is 4 mod 32, which costs an 8-way LDS
bank conflict on wave64 - the +1 padding is tuned for warp32 - and the extra
shared memory eats occupancy this shader is already short of.

Comment only, no behaviour change. Leaving a note so the next person does not
spend a GPU on it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-11-21 19:32:13 +01:00
Lumpiasty b5085151f2 ggml-vulkan: enable flash-attn mask_opt for GCN large head sizes
mask_opt was disabled on AMD GCN for head sizes <= 256, but it is
beneficial there in high-context prefill: on fully-visible causal
blocks it skips the per-block mask load+add, and it skips fully-masked
blocks entirely. The attention op on GCN is compute-bound on the
softmax path (no matrix cores), so this cuts real work.

Verified lossless (perplexity bit-identical with it on vs off) and a
measured prefill win on Qwen3.5-35B-A3B (head_dim 256) on an RX 580:
pp2048 unchanged at short context, +8% @ 16k, +12% @ 32k, growing with
depth. Enable for GCN when HSK/HSV >= 256; the existing large-mask
conditions keep it off for decode.

Assisted-by: Claude
2026-11-21 19:32:13 +01:00
LumpiastyandClaude Opus 5.5 ab7e03ec6c llama: skip the routing-ids readback in sched_copy_experts when the batch uses every expert
Offloaded expert weights are uploaded per split, and to decide which experts to
upload the copy callback reads the routing ids back to the host. The ids are
produced on the same device we are about to upload to, so the readback forces a
full pipeline flush - 47 of them per eval on a 48-layer MoE.

Once the batch draws enough experts the readback stops telling us anything. At
2048 tokens x 10 experts over 256 experts every expert comes back used, so the
used set is all true and the copy is a single whole-tensor range anyway.

Skip the readback when ggml_nelements(ids) >= 4 * n_expert and mark every expert
used. Uploading an expert that no id selects cannot change the result, since
mul_mat_id only reads the rows the ids point at, so this stays exact. With the
MoE cache on, the all-used set still goes through moe_cache->copy_experts, which
copies the cached experts device-to-device from the host slot maps and uploads
the rest - still exact, and no readback is needed for it. Decode is unaffected:
n_ids there is the number of experts per token, far below the threshold. The
small-batch MoE cache slot path has its own readback and is not touched.

The st.ids/st.used caching is unchanged: a repeat call with the same ids tensor
reuses the all-true set, which is what the threshold would decide again.

Port of dbc449afd ("sched: skip the MoE routing-ids readback when the batch
uses every expert"), whose target block in ggml_backend_sched_compute_splits was
removed upstream (#29943) in favour of this copy callback.

Measured (original, before the port) on an RX 580 (Vulkan, Polaris) running
Laguna-S-2.1 118B IQ2_M with all experts host-resident: pp2048 +4.7% at depth
32k, +0.9% at 16k, tg32 +3%.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-21 19:32:13 +01:00
saiph e6d624b9a4 mtmd : merge Parakeet rel-pos views to avoid OOB (#32722)
Assisted-by: Grok Bot
2026-11-21 19:05:02 +01:00
Xuan-Son Nguyen 19bc41dbb4 model: apply .scale tensors automatically + fix correctness (#32669)
* model: apply .scale tensors automatically

* rm build_moe_ffn "_s" overload

* use llama_prec_policy

* fix ci

* build_lora_mm_id: rm w_s
2026-11-21 16:30:01 +01:00
Frosty40andTitaniumtown e50c254a22 sycl: bound CONV_TRANSPOSE_1D input iteration (#31109)
Used angelX to optimize SYCL/drama 3 coordination, results are as follows

Each output reads only inputs with i*s0 <= idx < i*s0 + K, in the same ascending order as before.

Arc Pro B70, oneAPI 2026.1, test-backend-ops perf, base 9b56ccf00 vs patch, median of 3 same-boot A/B/A rounds, A/A 1.000-1.001:

| input [ne0, ne1] | kernel [K, OC, IC] | s0 | base us | patch us | speedup |
|---|---|---|---|---|---|
| [16, 1536] | [16, 768, 1536] | 8 | 3910.57 | 1348.59 | 2.90x |
| [128, 768] | [16, 384, 768] | 8 | 37735.00 | 1665.26 | 22.66x |
| [1024, 384] | [8, 192, 384] | 4 | 270371.75 | 1471.74 | 183.71x |
| [4096, 192] | [4, 96, 192] | 2 | 534614.63 | 752.74 | 710.23x |

Shapes are the drama 3 codec decoder upsampling stack, added as perf cases. Chained 4-layer stack at 1/4/24 input frames: base 66.3/847.4/28638.5 ms, patch 2.3/6.7/30.7 ms. Kernel numbers, not end-to-end TTS latency.

Correctness: test-backend-ops -o CONV_TRANSPOSE_1D 116/116 on base and patch, and 116/116 rebased on 3f03b1772. Seeded output dump (7.46 MB) byte-identical to base.

Full test-backend-ops aborts on base and patch alike at the non-F32 SWIGLU_OAI assert (element_wise.cpp:1187, see #30857). With only those cases excluded: 20145/20173 on base and patch, and every patch-run failure also fails on base.

Co-authored-by: Titaniumtown <titaniumtown@proton.me>
2026-11-21 16:20:29 +02:00
Kazutoshi, Akao 660be5506b ggml : improvements to the memory access pattern in ggml_compute_forward_mul_mat_one_chunk (#30846) 2026-11-21 16:16:40 +02:00
Xuan-Son NguyenandPascal c622f623fc mtmd: support kyutai/hibiki realtime full-duplex (#32491)
* mtmd: support hibiki realtime duplex

* fix and clean up

* fix long session quality

* fix ci

* tests : add the moshi fusion baseline for Metal

The Metal fusion check found 5 RMS_NORM+MUL fusions in the new moshi
arch and no baseline for them.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-21 14:25:19 +01:00
Titaniumtown c661033349 sycl: add IQ4_NL and IQ4_XS weight reorder (#31998) 2026-11-21 15:11:14 +02:00
Titaniumtown 44c20b1c0a sycl: column-blocked gated_delta_net kernel (#32456) 2026-11-21 14:38:00 +02:00
Aleksander Grygier 80325823d4 ui: open the context gauge panel on click as a popover (#32685)
* ui: open the context gauge panel on click as a popover

Assisted-by: Claude (Opus 5.5 Medium)

* ui: give the context gauge trigger the ghost button hover

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-21 13:21:47 +01:00
Clemens Wasser a818a46f3d sycl: add fused ADD, RMS_NORM, MUL and vectorize rms norm (#31753)
* sycl: vectorize contiguous rms norm

* sycl: add fused ADD, RMS_NORM, MUL

* tests: test fused ADD, RMS_NORM, MUL at decode widths
2026-11-21 14:20:22 +02:00
Johannes Gäßler b019be0c86 CUDA: abort on UB in fastdiv (#32698) 2026-11-21 13:46:59 +02:00
Ruben Ortlam 362ac92974 vulkan: unroll FA coopmat1 array due to a driver bug on AMD Windows (#32466)
* vulkan: unroll FA coopmat1 array due to a driver bug on AMD Windows

* simplify code to make it clearer
2026-11-21 13:46:20 +02:00
KaiFelixBennettandRuben Ortlam fb7090df56 vulkan: improve RDNA4 MMVQ performance for Q3_K, Q5_K and IQ4_XS (#31804)
* vulkan: keep default MMVQ rows for Q3_K, Q5_K, IQ4_XS on RDNA4 AMD proprietary driver

Assisted-by: Claude Opus 5.5

* clean up mmvq shader load logic

---------

Co-authored-by: Ruben Ortlam <rortlam@redhat.com>
2026-11-21 13:41:55 +02:00
Daniel Han 690b87d881 model : inkling2_loop share the loop slots after the scale pass (#32704)
The loop slots copied the physical layers at the end of load_arch_tensors,
before the generic pass attaches the optional .scale tensors, so the later
passes through the core ran NVFP4 experts without their per-expert scales.
Add an optional post_arch_tensors hook that runs after the scale pass and
share the slots there.
2026-11-21 12:10:38 +01:00
Sigbjørn Skjæret 838d942438 model : additional shortcut checks for OOB in hrm_text (#32720)
[no ci]
2026-11-21 12:10:04 +01:00
Aaron Teo 3da11de1f1 ggml-zdnn: fix ztensor load race condition (#32279)
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
2026-11-21 13:05:23 +02:00
shalinib-ibm 93cb37ea7b ggml-cpu: fix nc aligment in tinyBLAS_PPC_Q0 (#32659)
Fixes https://github.com/ggml-org/llama.cpp/issues/31680

The matmul_tiled() implementation in tinyBLAS_Q0_PPC requires nc to be a multiple of 8 to perform matrix multiplication correctly.

The previous alignment logic contained a special case for n == 4, which allowed n_aligned to be 4. Consequently, the tiled path could be selected with nc = 4, violating the kernel's tile-width requirement.

This caused incorrect matrix multiplication results, observed as garbled inference output when running  Flint Q4_K_M and Gemma 3 27B Instruct.
2026-11-21 12:58:49 +02:00
Aaron Teo 1fdda89a4d ci: switch ppc64le runners to ubuntu 26.04 (#32679)
* ci: switch ppc64le runners to ubuntu 26.04

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* ci: rm problem matchers experiment

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

---------

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
2026-11-21 10:54:00 +01:00
Georgi Gerganov e412c6f468 llama : remove fused lightning indexer gating (#32650)
The fused lightning indexer is always used now, so drop the cparams flags, the auto probe and the non-fused fallbacks.

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
2026-11-21 10:26:55 +02:00
Xuan-Son Nguyen 40b7a546de ci: temporary disable no realloc case for mtmd (#32708) 2026-11-21 10:02:26 +02:00
Daniel Kuts 46eb1742db llama-grammar: Fix integer overflow (#32551)
* llama-grammar: Fix integer overflow

* Check incorrect values at parsing and bail out nicely

* fix style
2026-11-21 07:00:30 +01:00
Aman Gupta 6792507b7f llama : score only the k-pools of the ubatch sequences in QSA (#32519) 2026-11-21 13:22:36 +08:00
cwriter 418fe97b5f sycl: Improve Grouped XMX GEMM by avoiding SLM staging (#31428) 2026-11-20 22:37:40 -05:00
Ashutosh Singh 7476e47261 docs: openvino backend details, fix SYCL docs (#30967) 2026-11-20 22:08:53 -05:00
Pedro Cuenca 29b52d398b model : Inkling 2 Mini (#32694)
* Conversion

* Separate model_type

* Add gate_rank to allow building from metadata while testing

* Base model

* Loop graph

* Support the new is_varloop attribute

* Adopt naming scheme from https://github.com/ggml-org/llama.cpp/pull/32588

* Rename to varying
2026-11-20 23:01:07 +01:00
Pelayo Garcia BárcenaandJohannes Gäßler e3c3f666a4 hip: add RDNA1 MMVQ parameter table (1 warp per block for ncols=1) (#32389)
* hip: add RDNA1 MMVQ parameter table (1 warp per block for ncols=1)

* hip: write the RDNA1 MMVQ values explicitly instead of recursing into the generic table

* hip: list the RDNA1 MMVQ table before RDNA2 in the enum

* Update ggml/src/ggml-cuda/mmvq.cu

---------

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
2026-11-20 22:52:19 +01:00
Ravi Panchumarthy b2e659ec24 ggml-openvino: update openvino CI runs triggers (#32593)
* ci : run the OpenVINO GPU job only for OpenVINO changes

The self-hosted OpenVINO GPU job ran for most PRs and every master push
on a shared runner, so it queued for hours.

- ci-self-hosted-openvino.yml (GPU): runs only for changes to
  ggml/src/ggml-openvino.
- build-openvino.yml: keeps its triggers (OpenVINO changes on PRs, every
  source change pushed to master). The Linux job runs on a GitHub-hosted
  runner and tests the OpenVINO CPU device only.

* address review comments

* ci : limit the OpenVINO PR CMake trigger to ggml

* ci : drop the OpenVINO Ubuntu job

* ci : trigger OpenVINO PR runs on tests/CMakeLists.txt
2026-11-20 22:51:49 +01:00
Xuan-Son NguyenandPascal 9197945004 convert: qwen3-asr: put "language ..." output into thinking block (#32571)
* convert: qwen3-asr: put "language ..." output into thinking block

* convert: qwen3-asr: read the chat template from models/templates

The converter embedded its own copy of Qwen3-ASR.jinja. Read the
bundled file instead, as the Kimi-K3 and Kokoro converters do, so
the GGUF and test-chat share a single template.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-20 20:54:57 +01:00
Tarek Dakhran a83d68b562 metal : allow the norm chain in either operand of the fused MUL/ADD (#32300)
Assisted-by: Claude Opus 5.5
2026-11-20 19:55:44 +01:00
Aleksander Grygier e46069c409 ui: fix the flaky reset check of the MermaidPreview Flowchart story (#32675)
Assisted-by: Claude (Opus 5.5 Medium)
2026-11-20 19:43:24 +01:00
Aleksander Grygier 427a836734 ui: stage Llama Voice's sources for CMake and Docker UI builds (#32680)
Llama Chat compiles @ggml-org/llama-voice from ui/lib/voice/src, but the
CMake stage and the Docker UI stages copied only ui/lib/ui, so their npm
build could not resolve the voice imports. Stage ui/lib/voice/src next to
Llama UI and track it as an input of the CMake build.

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-20 19:42:49 +01:00
Daniel Bevenius 374d90a9fc ci : disable musa.Dockerfile from docker.yml (#32674)
* ci : disable musa.Dockerfile from docker.yml

This commit disables musa in docker.yml due dl.mthreads.com is sometime
unavailable.

Refs: https://github.com/ggml-org/llama.cpp/pull/32672#issuecomment-7042886858

* remove the SKIP_MUSA option
2026-11-20 18:58:11 +01:00
Xuan-Son Nguyen 98709fd82b server: add more decision tests (#32578)
* server: add more decision tests

* fix ci

* also test audio
2026-11-20 18:45:31 +01:00
Georgi Gerganov 7716ec8cbc ggml : major version bump reminders + deprecate ggml_ftype (#32647) 2026-11-20 19:36:42 +02:00
FeitengandXuan Son Nguyen 34ff2e047c mtmd: drama3 voice cloning (#32586)
* mtmd: drama3 voice cloning

The Fish S2 codec encoder (fish_s2_spkenc) now also serves drama3:
- conversion: drama3 writes the codec encoder and the audio encoder KV, shared with fish_s2
- fish_s2_spkenc: outputs the codes of the clip (gen-side reference) and, in a drama3_gen mmproj,
  builds the drama3 backbone rows (E_lm[semantic] + E_cb[1][code 1]) / sqrt(2)
- drama3_gen tower: teacher forcing when the caller gives the codes of the next frame
- drama3 pipeline: reference-bracket-v1 prompt with one reference (vLLM-Omni fish_omni), the
  reference frames go through the backbone as codec rows and through the tower with their codes
- mtmd_gen_state_get_ref_codes(): read-only access to the reference codes of a gen state

Claude-Session: https://claude.ai/code/session_01PpX3DmaxNSmJbxGaxnUBrX

* fix correctness

* clean up

* rm redundant code

* fish_s2: rm unused import

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-11-20 18:34:52 +01:00
Ruben Ortlam 6f5c9ad217 vulkan: add support for COL2IM_1D misalign bytes and split workgroups (#32673) 2026-11-20 19:11:59 +02:00
Meet Amin 184862801a gemma4: Optimized prompt processing for multi-seq queries (#32342) 2026-11-20 23:41:02 +08:00
Aleksander GrygierandNicolas Mowen 79e81b0bd4 ui: voice messages and dictation with live waveforms and transcripts (#32531)
* ui: add a voice engine for capture, peaks and transcription

Voice input needs a live waveform, a clip that can be sent and drawn,
and a transcript, whether the model streams text while the user speaks
or not. This adds the engine for it in src/lib/voice, with no imports
of app stores or components, so it can move into a package later.

- VoiceCapture: one microphone start gives live levels (RMS on a dB
  curve, fixed history, newest last), PCM16 mono chunks at 16 kHz from
  an inline AudioWorklet (ScriptProcessor fallback), and on stop a
  16-bit mono WAV clip with its duration and peaks. Tracks, nodes and
  the context are released on stop, cancel, a failed start and page hide.
- computePeaks / computePeaksFromAudio: normalised peaks of samples, a
  buffer, or a stored base64 or Blob file.
- transcribeClip: POST /v1/audio/transcriptions, with the server's own
  error message on failure.
- RealtimeTranscriber: the /v1/realtime?intent=transcription socket,
  queueing audio until session.created, a final commit, and a typed
  RealtimeUnsupportedError when the model cannot stream or the socket
  is refused (a browser websocket cannot send the API key header).
- startVoiceSession and VoiceRecorder: try realtime, fall back to one
  HTTP transcription of the clip, and expose levels and live text.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: add voice waveform and transcript components to Llama UI

Voice input needs three visuals that the app can feed with its own
recording and transcription logic: the live waveform while recording,
a playable waveform of a recorded clip, and a clip with its transcript
that opens on hover, keyboard focus or a tap.

VoiceWaveformLive draws input levels as many bars as fit the width,
newest on the right, and holds still with reduced motion.
VoiceWaveform plays a clip through its own audio element, fills the
played part in the primary colour, seeks by press, drag and arrow keys
(a slider) and pauses any other clip that was playing.
VoiceTranscriptReveal keeps the transcript in the DOM for screen
readers while it is closed, and shows transcribing and error states.

The waveform math (resampling peaks, bar capacity, seek and time
formatting) is in src/lib/voice with unit tests. The components are
exported from the package root and from @ggml-org/llama-ui/voice, which
Llama Chat resolves to the sources like the other entry points.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: add stories and tests for the voice components

Storybook gets UI/Blocks/Voice: the live waveform fed by a fake
microphone meter in a chat form row, the clip waveform in its sizes and
states, and the transcript reveal in each state and on touch. The demo
audio is a WAV synthesised in the story, so no binary file is added.

The play functions check seeking by keys and by press, the reveal on
hover, keyboard focus and a tap, and a clip without audio. Pressing play
needs a real click before a browser plays audio, so a client test covers
play and pause and one clip pausing another.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: store voice messages with their transcript and send it to models that cannot hear

A voice message recorded in the chat form is an audio attachment like an
uploaded file, plus what the app needs to draw and read it. The audio
attachment gets four optional fields: voice (recorded in the app),
transcript, peaks and durationMs. Stored rows, export and import carry
them as they are, so no migration is needed and older messages keep
working.

The transcript is for reading. A model with audio input still gets the
audio alone; a model without it (the user switched models later) gets the
transcript as a labelled text part in place of the audio. When the
model's modalities are not known yet, or there is no transcript, the
audio goes as before.

A transcript often comes back after the message was sent. The new
voiceTranscriptsStore follows each pending transcript by the clip's name,
hands it to a send that happens after it arrived, and otherwise writes it
into the stored attachment of the message it went out with, in the open
chat or, if the user left it, in the database.

Copying a message now includes its voice transcript.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: add Voice settings and resolve what the microphone does per model

The user should not have to think about which kind of voice input is in
use, so the app decides it from the conversation's model and two
settings. resolveVoiceMode is a small pure function: with Voice input on
Automatic and a chat model that has audio input, a recording is a voice
message, transcribed by that same model so a router swaps nothing.
Otherwise it is dictation, with the chat model when it can hear, else,
in router mode, with the chosen transcription model. When no model can
transcribe, the microphone is unavailable and says why.

Settings get a Voice section of their own rather than more rows in
General, which is already long: these four belong together and one of
them lists the server's models.

- Voice input: Automatic (default) or Always dictate.
- Transcription model: Same as chat model (default) or, in router mode,
  any model with audio input; on a single-model server it only says
  whether that model can transcribe. It notes that dictating with
  another model loads it beside the chat model.
- Transcribe voice messages: on by default.
- Transcription language: empty for automatic.

The keys live in the settings registry like the others, so defaults,
storage, reset and export need nothing new. The model picker is drawn by
SettingsChatVoiceTab, the way Extensions and Tools draw their sections.

Assisted-by: Claude (Opus 5.5 Medium)
Co-authored-by: Nicolas Mowen <14866235+NickM-27@users.noreply.github.com>

* ui: record voice messages and dictation from the chat form

The microphone now does what the conversation's model allows, without
the user choosing a mode: its label says "Send a voice message" or
"Dictate", and when no model can take audio it is disabled and says why
(shown in router mode only when some model there has audio input, so the
reason can be acted on; a single-model server without audio keeps the
send button as before). The microphone errors keep their warning state
and toast, now also for the voice engine's own capture errors.

Recording goes through the voice engine's VoiceRecorder instead of the
MediaRecorder flow. While it records, the actions row becomes a
recording row: cancel, the live waveform, the elapsed time and stop.
Escape or cancel drops the recording. The form keeps its two rows while
recording, so the message stays visible on desktop and the row fits the
phone layouts and the expanded sheet.

- Dictation writes the live transcript into the message at the caret
  as it streams. DictationSpan replaces only the dictated stretch, moves
  it when the user types before it, keeps what is typed after it, and
  hands text the user edits over to them. When nothing streams, the row
  shows Transcribing... after stop and the text goes in once it is back.
  Cancel takes the dictated text out again. No audio is attached.
- A voice message streams its live text, if any, under the waveform,
  not into the message. On stop the clip is attached as a voice file
  with its peaks and length, and its transcript is followed by the voice
  transcripts store (unless Transcribe voice messages is off).

Realtime transcription is skipped for servers reached through a tunnel
or with an API key, which a browser websocket cannot carry. The first
dictation with a separate router model shows a one-time note that it
loads beside the chat model.

The form takes an optional voiceRecorder, so tests and stories can run
it without a microphone.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: draw audio attachments as playable waveforms with their transcript

A voice message should read as a clip, not as a file chip. Audio
attachments now render with Llama UI's VoiceWaveform inside
VoiceTranscriptReveal, through a direct branch in the core attachment
item (voice is core, so it does not go through the extension renderers):
the removable sm chip in the chat form, and the md clip in a sent
message, on the user bubble's surface.

- The audio plays from an object URL made from the stored base64 (or the
  uploaded file's data URL), revoked when the clip leaves the screen.
- A clip stored without peaks, an older voice message or an uploaded
  audio file, gets them computed from its audio once per session, in
  memory only; uploaded files keep their name under the waveform.
- The transcript opens under the clip on hover, keyboard focus or a tap,
  and shows Transcribing... while the voice transcripts store waits for
  it, Transcription unavailable when it failed, and nothing when none
  was asked for.
- The full-screen preview keeps its player and shows the transcript.

Tests run the chat form with a fake recorder: the microphone per mode,
the recording row, dictation around typed text, the Transcribing...
state, cancel, and a voice message chip whose transcript arrives later
and goes with the send. Stories show the form recording each kind, a
recorded chip, and sent voice messages with the transcript, pending,
and computed peaks.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: drop the old audio recorder and proxy realtime transcription in dev

The voice engine replaced AudioRecorder, so it goes. The dev server now
proxies WebSockets on /v1, so realtime transcription works under vite dev.
The VoiceCapture test waits for the tone to be recorded instead of a fixed
700 ms, which a loaded test run could miss.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: dictate with a model that is already loaded when the chat model cannot hear

The Transcription model setting's Automatic choice now falls back to a
router model with audio input that is already loaded, so dictating with a
text-only chat model works without picking a model and loads nothing.
A picked model still wins, and the empty value is a named constant.
Idea from upstream PR 27567, by NickM-27.

Assisted-by: Claude (Opus 5.5 Medium)
Co-authored-by: Nicolas Mowen <14866235+NickM-27@users.noreply.github.com>

* ui: fill the recording waveform, and open voice message transcripts

- The live waveform fills the row from the start: a recording shorter
  than the row is stretched over all its bars, a longer one scrolls with
  the newest level on the right. No dotted placeholder.
- A voice message's transcript is open under its clip, in the chat form
  while it comes in and in the sent message, with the toggle beside it
  to fold it.
- Settings > Voice > Collapse transcripts (off by default) keeps them
  folded instead, opening on hover, keyboard focus or a tap.

Assisted-by: Claude (Opus 5.5 Medium)
(cherry picked from commit 2a93f2f948ed976e40f693ce5d3acf1168284b05)

* ui: record voice without ducking other audio, from the first word

- Voice input asks for the raw microphone signal: no echo cancellation,
  noise suppression or auto gain. Echo cancellation made the system treat
  a recording as a call, which lowers or interrupts other playback, and
  auto gain faded the first words in.
- One AudioContext serves every recording, with the capture worklet
  loaded once, suspended in between and, where the browser allows it,
  without an output device, so recording never touches the speakers.
- The chat form prepares it when it gets focus (no microphone, so no
  recording indicator), and a mouse starts recording on press.
- A recording starts counting once audio flows: until then the bar says
  Starting... instead of running the timer. Development builds log how
  long each step of a start took.
- Settings > Voice > Microphone: Automatic (the system default, or the
  built-in microphone when the default is a Bluetooth headset, which
  would switch to its low-quality call mode), System default, or a
  listed microphone.

Assisted-by: Claude (Opus 5.5 Medium)
(cherry picked from commit 76643e7210d21617d734fdfde677e84af7273153)

* ui: show voice text live in the message box, and follow transcripts as spoken

- A model that hears but has no live ASR (e.g. LFM3-Omni) now gives text
  while you record: the realtime client commits the audio piece by piece,
  at the pauses the levels show, one piece at a time, and the final
  commit takes only what is left.
- A voice message's live text shows as the empty message box's
  placeholder. Clicking into the box makes it text to edit there, and
  what is said next follows it. On stop the text leaves the box and goes
  with the clip; when it was edited, the edit is the clip's transcript.
- Llama UI gains VoiceSpokenText and spokenLength(): spoken words in the
  full foreground colour, which they keep, the rest muted. A clip's
  transcript follows its playback (estimated over the parts with speech,
  as there are no word timings), and live text brightens as it arrives.
- The recording bar has a send button that stops and sends in one go.

Assisted-by: Claude (Opus 5.5 Medium)
(cherry picked from commit ce773017f902f8a5690ab9ecca368cecb6427422)

* ui: let the capture test start with silence

A capture starts with its first audio block, which can come before a
stream's first sound, so the steady tone is checked from where it begins.

Assisted-by: Claude (Opus 5.5 Medium)
(cherry picked from commit a41ee5337b83fef685a5cbaff176925f1647ce9d)

* ui: show live voice text sooner, and keep silence from becoming "yeah"

- A model that transcribes per commit (LFM3-Omni) gets shorter pieces:
  cut at a 0.35 s pause after speech, or at the next quiet moment once a
  piece reaches 3 s, so the text follows the speaker within a few
  seconds in dictation, the message box and voice mode.
- When the recording stops, the audio after the last piece is committed
  only if speech was heard in it. LFM3-Omni transcribes a commit of pure
  silence as "Yeah." (or "はい。"), which ended up at the end of the text.
  A recording whose only piece is that rest still commits it.

Assisted-by: Claude (Opus 5.5 Medium)
(cherry picked from commit 43e52cbab0fba9d8e84b9491890a406e3c6c90f3)

* ui: transcribe a recording cut at pauses once more as a whole

Live text comes piece by piece, cut at the speaker's short pauses, and a
model reads each piece as a sentence of its own: "I went. To the store."
When the recording ends, one pass over the whole clip gives the final
text, punctuated by meaning, in dictation and in voice mode alike. The
pieces' text stays the live text until then, and is kept when that pass
fails or the recording is longer than two minutes.

Assisted-by: Claude (Opus 5.5 Medium)
(cherry picked from commit 359dd9632e6c88000a1d23c2575bd1e6fcdd9ec1)

* ui: give live text over HTTP for models without a realtime session

A model that hears but cannot hold a realtime session, such as
Qwen3-ASR, was transcribed only after the recording, so dictation
showed no text while speaking. The recording is now cut at the
speaker's pauses as with a realtime session, and each piece sent to
/v1/audio/transcriptions on its own, one at a time, its text joined
live. The audio is kept from the start, so what is said while the
socket answers goes with the first piece. The whole clip is still
transcribed once more at the end, and the pieces' text is kept if
that fails. A server a websocket cannot reach (an API key, a remote
server) gets live text the same way.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: say when the browser processes the microphone despite the raw request

The browser keeps one processing setting per microphone: while a call,
another tab or voice mode has it open with echo cancellation, a
dictation that asked for the raw signal is processed too, sounds
distant, and the system may lower other sound. The capture now warns
when it gets processing it did not ask for, and its development log
names the microphone, its rate and what processing is on.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: keep the microphone beside the stop and send buttons

While a reply streams, or once the message has text, the microphone no longer
gives way to the main button: it stays beside it as a quieter, secondary
button, so more can be dictated without clearing the message or waiting for
the reply.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: show the voice components' stories with Llama UI

VoiceWaveform and VoiceTranscriptReveal are Llama UI components, so their
stories move to the library's Storybook, under Llama UI/Voice, with the demo
audio they play. Llama Chat's Storybook shows them there too, and its own
voice stories take the demo audio from the library. VoiceWaveformLive stays
with Llama Chat, as its demo records through the chat form's microphone.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: add a noise suppression setting for voice input

Recordings stay raw by default. Settings > Voice can now switch the
browser's noise suppression on; echo cancellation and automatic gain stay
off either way, as they interrupt other sound and fade the first words in.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: keep live voice text coming when the room is never quiet

Pieces were cut only at a quiet moment, so with music or talk in the room
none was ever cut and no text showed until the recording ended. A piece half
as long again as its longest length is now cut where it is.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: add echo cancellation and automatic gain settings for voice input

Settings > Voice now has a switch for each kind of microphone processing the
browser offers. All three stay off by default, so recordings are raw; each
one says what switching it on costs.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: put the microphone after the stop button, filled like it

While a reply streams, the microphone is the last button of the row and
looks like the stop and skip buttons beside it, instead of a bare icon
before them.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: say when dictated text is being refined, and send it refined

After a recording with live text, the whole clip is transcribed once more
so pauses do not end sentences. The form now says so while that pass is
out, and a send asked for meanwhile waits for it, so the message goes out
with the refined text rather than the pieces.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: keep the send button in place and put the microphone beside it

The send button no longer gives way to the microphone on an empty
message. The microphone stands before it as a quieter, tertiary button,
with the same tooltip, warning and recording states as before.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: let a recording be sent while a reply streams, once it has words

While a reply streams, the recording row now shows its send button as
soon as the first transcribed words are in the message, so the
recording no longer has to be stopped first. The microphone also stays
beside the send button once a voice message is attached.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: raise quiet speech for transcription, and say when it is too quiet

A quiet microphone gave the model near silence to transcribe. A new
Voice setting, Boost quiet speech (on by default), raises speech to a
level speech models hear well, up to eight times louder: the audio
streamed for live text as it goes, and the whole recording in one pass
when it stops, so its first words get the same gain as the rest. Only
speech is measured, never the silence around it, loud speech is left
as it is, and a soft limiter keeps raised peaks from clipping. When
speech stays quiet even at the highest gain, the recording row says so.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: keep the voice engine free of app code

The engine in lib/voice imported the microphone messages from the app's
utils and the WAV type from its enums, the only two things keeping it
from moving into a package of its own. The microphone checks and
messages move into the engine (voice/support.ts), next to the capture
that needs them; the uploaded-file helpers stay in the app's utils.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: start voice sessions from one voice service

How a recording reaches the server (routes, API key, whether the
realtime socket can be used) and how the Voice settings become capture
and transcribe options were built inside the chat form's voice hook,
and copied into every other feature that records. VoiceService builds
them in one place: a feature asks it for its session options and adds
only its own callbacks.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: tell speech from the room in one place

The pause detector, the speech gain and (in voice mode) the listener for
speech over a reply each followed a noise floor with their own numbers.
A SpeechDetector now does it for all of them, in dB, with one set of
settings in VOICE_SPEECH.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: let any text field take dictation

Writing dictated text into a field (at the caret, around the user's
typing, replaced by the whole recording's text when it comes) moves out
of the chat form's hook into a plain createDictation(). The hook keeps
what is the chat form's own: voice messages, toasts and the hint.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: move the recording row into Llama UI

VoiceRecordingBar draws the row shown while recording (cancel, waveform
and time, send, and a caption or warning under it) from data alone, so
voice mode or a notes page can show the same row. The chat form passes
its labels and its microphone button as the stop button.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: keep the voice engine's Svelte recorder behind its own entry

VoiceRecorder, the one part of lib/voice written with runes, moves to
lib/voice/svelte, so the rest of the engine is plain TypeScript any app
can use without Svelte.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: move voice input into its own package, Llama Voice

The voice engine (capture, the speech signal, transcription over the
realtime socket or HTTP) and dictation into a text field move from the
app to ui/lib/voice as @ggml-org/llama-voice, so another app, such as a
notes app, can use them. The Svelte recorder is its ./svelte entry.
Llama Chat compiles the sources, as it does Llama UI's, and keeps its
server and settings wiring. The package has its own type check, tests
and build, and a CI job for them.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: show which model does what in voice input

The model picker and the Models manager mark each model that hears:
Dictation on the one that transcribes dictation, Speech to text on
transcription models such as Qwen3-ASR, and Voice on chat models that
take voice messages. A loaded model kept only for dictation sits apart
from the chat models, a model can be made the dictation model from its
row, and Settings > Voice groups the transcription models apart from
the chat models with audio input. A voice store now resolves what the
microphone does, for the chat form and the model lists alike.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: warn of a quiet microphone inside the recording row

The too-quiet line under the waveform made the form grow and jump. The
warning now stays inside the row: the waveform turns the warning colour,
an icon sits by the time, and the advice opens above the row in a
tooltip for a few seconds, then on hover or focus. Screen readers still
hear it when it appears.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: open the microphone while the pointer is on its button

The first seconds of a recording could come out quiet while the
microphone settled, and the too-quiet warning showed for them. With the
pointer or the focus on the microphone button, the microphone now opens
ahead of the press (only when the page may already use it, so nothing
asks), and the recording takes it at once; it is let go shortly after
the pointer leaves without a press. The warning also waits until the
speech has stayed too quiet for two seconds.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: tidy voice input after review

Fix a microphone left open when a recording is cancelled while it starts
or the pointer leaves while it opens, a capture that is still letting go
silencing the next one, and live text stopping after a realtime error.
Share the dictation model choice, the transcript lookup, base64 decoding,
PCM joining and stopping a stream instead of repeating them, and export
only Llama Voice's public API.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: pin Llama Voice's dev dependencies to Llama UI's versions

Assisted-by: Claude (Opus 5.5 Medium)

* ui: let demos speak into a synthetic microphone

Assisted-by: Claude (Opus 5.5 Medium)

* ui: keep the microphone for voice input, off the audio files entry

The attach menu's Audio Files entry used a microphone, which now sits
beside the form's own microphone button. The microphone stands for
recording and dictation only; a model's audio modality is headphones, and
the audio files entry is a file with headphones.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: set voice input on the slimmed design tokens

Voice input's components use the type roles, the one corner scale and
shadcn's utility names for the neutral colours, as the rest of Llama UI
and Llama Chat do since the design tokens were slimmed down.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: name voice states, platform strings, patterns and tunings

Addresses the review on the voice PR:

- Llama Voice: VoiceCaptureState, VoiceRecorderStatus, RealtimeState and
  RealtimeUnsupportedReason are enums (src/enums.ts, exported); DOMException
  names, AudioContext states, transcription form fields, socket protocols,
  device kinds, worklet messages, error messages, regexes and tuning values
  are named constants.
- Llama UI: VoiceTranscriptStatus, VoiceWaveformSize and
  VoiceRecordingBarStatus are enums; KeyboardKey, MouseButton and PointerType
  join its UI enums; the voice components' sizes, labels, timings, selectors
  and patterns move to constants/voice.constants.ts. The voice unit tests
  move from src to tests/unit.
- App: ChatFormVoiceState is an enum, the recording bar's status comes from a
  map, and the voice code uses the enums above, BooleanString, HEADERS and
  new voice constants in place of inline strings, regexes and numbers.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: keep all of Llama UI's tests under tests

The Model UI formatting spec moves to tests/unit and the button spec to
tests/client, next to the voice specs; the test projects now read only
tests/unit and tests/client.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: show a transcript's words without the model's own format

Qwen3-ASR names the language before its text (language French<asr_text>...) and
VibeVoice-ASR answers with timed segments as JSON, which the server passes on as they
are. The voice engine now takes both off, over HTTP and over the realtime socket, also
while segments still stream, so the message box and the clip show the words alone.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: dictate when a speech-to-text model is picked for the conversation

A speech-to-text model has audio input, so on Automatic the microphone recorded a voice
message for it, which such a model has nothing to answer with. It now dictates, as the
model's badge already says it is for.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: say why a live-only model cannot transcribe behind an API key

A browser websocket cannot send the API key, so the realtime socket is left out on a
server that needs one, and a model that only transcribes over it (Voxtral Realtime)
answers the HTTP route with a 501. The clip and the dictation toast now say so instead of
only that transcription is unavailable.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: pick the dictation model with a microphone on its row

The model that transcribes dictation was marked with a filled Dictation badge. It is now
marked by a red microphone among the row's actions in the model picker, beside the heart,
and every other model that hears shows the same microphone on hover: a click makes it the
transcription model, and a click on the chosen one hands the choice back to Automatic. The
badge keeps saying what kind of hearing model a row is, and the Models manager marks the
dictation model with a small microphone beside its name.

Assisted-by: Claude (Opus 5.5 Medium)

---------

Co-authored-by: Nicolas Mowen <14866235+NickM-27@users.noreply.github.com>
2026-11-20 16:34:52 +01:00
Aleksander Grygier 059cac2848 ui: search message content and Discover models from the search dialog (#32670)
* ui: search message content and Discover models from the search dialog

The search dialog matched chat titles only. It now also scans what was
said in each chat (content, reasoning, tool calls and results, attachment
names and text), newest chat first, stopping at the first match per chat
and after 20 chats, and shows a snippet around the match. Ids, timestamps
and base64 payloads are not searched.

With Discover on, it also lists matching catalog models and, once typing
settles, Hugging Face repos; picking one opens Discover on that repo.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: keep the search dialog one width whatever it lists

The shared dialog sizes itself to its content from md up, so the search
was narrow while empty and widened as results came in. It now takes the
full max-w-2xl there, less a margin on narrower windows.

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-20 16:20:00 +01:00
Aleksander Grygier cf4153a9ba ui: publish llama-ui as an alias of @ggml-org/llama-ui (#32667)
The unscoped llama-ui on npm is a 0.0.1 placeholder. After the scoped
publish, the workflow renames the same build and publishes it again at
the same version, with a readme note pointing at the scoped package.

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-20 15:46:18 +01:00
yoch 8e7862411d tts : forward -t to the mtmd context (#32664)
llama-tts built the mtmd context from mtmd_context_params_default(), which hardcodes 4 threads, and never copied n_threads over, so -t never reach the mtmd side.
Copy params.cpuparams.n_threads into mtmd_params.n_threads, as mtmd-cli and the server do; with no -t this now follows the resolved default.

Verified: -t 1 now runs single-threaded, -t 4 is unchanged, and the produced WAV is identical.

Assisted-by: Claude
2026-11-20 15:34:58 +01:00
Xuejun ZhaiandSigbjørn Skjæret e626db7caa server-bench : fix TypeError in histogram generation (#32518)
* fix(server-bench): prevent float bin-count TypeError in histogram generation

Symptom: benchmark crashes while generating gen_rate histogram.

Root cause: np.ceil(...) returns numpy.float64, and that value was used where an integer count/shape argument is required.

Fix: cast ceil result to int before +1, ensuring bin count is an integer.

Result: histogram path completes and gen_rate.png is generated reliably.

* Update scripts/server-bench.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-11-20 15:16:52 +01:00
Aleksander Grygier c6a25fc622 ui: load highlight.js languages on demand (#32661)
* ui: load highlight.js on first use

highlight.js and every lowlight language (~900 KB raw) moved out of
Llama Chat's first load into a chunk that loads with the first code block.

- @ggml-org/llama-ui/code: highlightCode stays synchronous. Until
  highlight.js has loaded it returns the code escaped as plain text and
  starts the load; a $derived or $effect that called it runs again once
  it has loaded. New loadHighlighter() and isHighlighterLoaded().
- Markdown: rehype-highlight runs through a lazy rehype step, like
  KaTeX's: the first message with a code block waits for it, every
  other one renders in the same pass.

First-load JS of the chat page: 3.47 -> 2.56 MB raw, 1.05 -> 0.75 MB
gzip, 0.84 -> 0.60 MB brotli.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: load each highlight.js language on first use

highlight.js now loads one language at a time instead of all ~190 at
once with the first code block: every language is its own chunk, and
the core loads with the first one.

- @ggml-org/llama-ui/code: highlightCode looks a language or alias up
  in a generated table (scripts/generate-highlight-languages.mjs), loads
  it with the languages it embeds (CSS and JavaScript inside HTML, say)
  and returns plain text until then; a $derived or $effect that called
  it runs again once it has loaded. Auto-detection picks among lowlight's
  37 common languages, loaded the first time something auto-detects.
  New loadHighlightLanguages(), isHighlightLanguageLoaded(),
  resolveHighlightLanguage() and getLoadedHighlightLanguages() replace
  loadHighlighter() and isHighlighterLoaded().
- Markdown: rehypeLazyHighlight highlights with lowlight's core and the
  languages a message uses, waiting only for one it is the first to use.
  rehype-highlight is no longer in the pipeline, since it bundles
  lowlight's common languages.

First-load JS of the chat page stays at 2.58 MB raw, 0.76 MB gzip,
0.61 MB brotli: no highlight.js code, only the 14 KB language table.
The first code block now loads its language (1-12 KB for most) and the
20 KB core instead of 900 KB.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: name the code-block class names and shell language as constants

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-20 15:06:10 +01:00
Xuan-Son Nguyen 5f837145d8 server: add more audio tests (#32579) 2026-11-20 14:21:58 +01:00
Xuan-Son Nguyen 36cd3aff40 server: add --ui-mcp-proxy-allow-private-urls (#32633) 2026-11-20 14:10:06 +01:00
Aleksander Grygier 28de970cc1 ui : focus the chat form when typing with nothing focused (#32570)
A printable key pressed on the chat screen while no field, editor,
dialog or menu has focus now moves focus to the chat form, and the
character lands there.

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-20 14:01:50 +01:00
Georgi Gerganov 4b69c79cca cuda : update GDN supports_op check (#32660) 2026-11-20 14:35:20 +02:00
Daniel HanandSigbjørn Skjæret a1f4222608 model : load ffn_gate_up_exps scales (#32657)
* model: load `ffn_gate_up_exps` scales

The generic scale pass loads per-expert scales for ffn_gate_exps,
ffn_up_exps and ffn_down_exps but not for the merged ffn_gate_up_exps.
Load ~ffn_gate_up_exps_s there~ and expose it as ffn_up_exps_s ~when the layer
has no separate up tensor,~ since build_moe_ffn applies up_exps_s to the
merged gate_up output. Inkling 2 now passes the expert scales to
build_moe_ffn like the other MoE models.

* Update src/llama-model.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update src/llama-model.h

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-11-20 13:11:28 +01:00
Aman Gupta 4783b1145c llama : move the indexer key rotation choice into the model classes (#32653)
* llama : move the indexer key rotation choice into the model classes

The KV cache forced a Hadamard rotation for the lightning indexer keys
based on a list of architectures. The virtual llama_model::indexer_k_rot()
now returns true for the same models. DeepSeek4.1 inherits from DeepSeek4
but was not in the list, so it overrides it back to false.

* llama : rename indexer_k_rot() to lid_k_hdmd()
2026-11-20 18:49:16 +08:00
Sigbjørn Skjæret 6abcbe5d35 remove ui-lib publish verification step (#32658) 2026-11-20 11:45:46 +01:00
Aleksander Grygier b81aa847c2 ui: slim the design tokens and set all text through type roles (#32615)
* ui: drop component tokens that only alias a semantic token

A component token now exists only for what a component decides itself:
a value no semantic token has, or one that only this component changes
between themes. Components that take a semantic token as it is read it
directly, so 88 of the 118 component tokens go, with no change to what
is drawn.

Also drops semantic tokens that repeat another one (status-* is fill-*,
elevation-tooltip is elevation-surface), the unused info lamp tokens and
14 primitives nothing reads, and adds fg-soft for the label of an option
that is not the picked one (tabs, toggle group).

Assisted-by: Claude (Opus 5.5 Medium)

* ui: add a short summary on top of DESIGN.md

The rules most changes need, in one screen: components, colour, type,
shape, sizes, surfaces, tokens and no arbitrary values. The sections
below keep the detail.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: derive design tokens by rule and drop the alias layers

Semantic tokens are either a pick (a primitive per theme) or a derivation,
mixed from another semantic token by a fixed rule and declared once: hover
fills, status surfaces and lines, dark shadows (--elevation-strength), the
middle of a matte edge, a lamp's rim and glyph.

Primitives keep colours only. shadcn's colour names stay as Tailwind
utilities and no longer exist as variables; neutral semantic utilities that
duplicated them are gone.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: set all text through type roles and move off-scale values onto the scale

Tailwind's text sizes and bracketed sizes become the type roles, with a new
text-caption-sm (10px) for tags and counts. Body and caption roles no longer
set a weight, and line heights are written as Tailwind writes its own, so the
swap from text-sm and text-xs changes no pixel. A unit test keeps old sizes
from coming back.

Arbitrary values with a scale equivalent use it (z-20, max-w-4/5, min-w-32),
and the two easing curves repeated across the app are ease-out-quint and
ease-sheet. DESIGN.md is updated and regenerated.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: bump @ggml-org/llama-ui to 0.5.0

Removed and renamed tokens are breaking for an app that reads them.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: add a display type role for the biggest headline

The greeting on a new chat keeps its 30px from md up, now as text-display
instead of a Tailwind size.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: one corner scale, one set of shadows and three more derived tokens

Six corner roles named values Tailwind's rounded-md to rounded-4xl
already have, and components used the Tailwind names nine times out of
ten. The roles are gone; the whole scale is computed from --radius.

Tailwind's shadow sizes were restated next to the elevation tokens and
did not follow the theme, so they were five times fainter in dark. The
five uses take elevation tokens and a unit test keeps both from coming
back.

A button group's seam is the button's own text colour at 25%, the dark
status text is the dark status fill, and the pressed shadow scales with
--elevation-strength like the inset one.

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-20 11:26:26 +01:00
Aleksander Grygier 9d6f04fa99 ui: show vision as glasses and audio as headphones (#32617)
The model modality icons match the ones the Llama apps use: glasses for
vision, headphones for audio. The attach menu's list of what the model
takes reads the same icons instead of its own microphone and picture.

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-20 10:33:34 +01:00
Aman Gupta 652d559edf llama : move the graph node budget into the model classes (#32648)
graph_max_nodes() picked the nodes-per-tensor factor from lists of
architectures. Each model class now gives it through a new virtual,
llama_model::graph_nodes_per_tensor(), which defaults to 8. The
architectures that need 32 or 64 override it in models.h, and DFlash
keeps its hparams-based rule in dflash.cpp.

LoRA adapter nodes are now added for every architecture, not only for
the ones on the default factor.
2026-11-20 17:19:14 +08:00
Aleksander Grygier 6aee50801d ui: set the reasoning effort from the model picker (#32632)
* ui: move the reasoning effort into the model picker

The chat form's row no longer holds the reasoning button: the model
picker ends with a Reasoning row, in place of Manage models, that shows
the level in use and opens the levels under it. A phone's model drawer
has the same row. A desktop server with a single model, whose selector
opens the manager instead of a picker, keeps the button beside it.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: show a reasoning effort that is not the default on the model selector

Next to the model on the closed selector, a set level shows as its word,
and reasoning that is off as a small crossed-out lightbulb. At the
default, which leaves it to the model, nothing more shows.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: keep the model picker's footer in place while it opens in Safari

WebKit pins a bottom-sticky element against the scaled box of a parent
that zooms in, so the Reasoning row rode up over the model list until
the opening animation ended. The desktop picker now scrolls its search
and list in a box of their own, with the footer under it.

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-20 09:58:11 +01:00
Aleksander Grygier 55b8d8d898 ui: full-screen diagram preview that zooms around what is on screen, inert diagrams (#32591)
* ui: keep Pele diagrams inert

Pele turns a node with `click ... href` into a live link, where Mermaid's
strict security level left diagrams without links. Unwrap every link in
Pele's output, keeping its children, so the inline block, the live-preview
editor, the preview dialog and the downloaded svg stay inert. The Diagrams
story's flowchart now carries click directives and its play test checks
that no link or href is left.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: open diagrams full screen, zoom around what is on screen

The diagram preview dialog fills the window on every device. The preview
places the diagram from its top left corner, so a zoom keeps a screen point
fixed: the wheel and a trackpad pinch zoom around the pointer, the buttons
and +/- keys around the middle of the screen, wherever the diagram has been
panned to. Two-finger pinch, double-click and arrow keys work too. The
diagram opens fitted and keeps fitting until panned or zoomed; Reset view
fits it again.

Adds MermaidPreview and DialogMermaidPreview stories with play tests and a
unit test for the zoom math.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: add a diagrams trace to the demo fixtures

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-20 09:57:44 +01:00
Aman Gupta 4c4a0f0a42 llama : reserve graph nodes for the MoE cache (#32636)
The MoE cache adds a slot lookup to every MoE layer, and with
--moe-cache-cpu-share a second expert path, but graph_max_nodes() did
not account for it. Models on the default 8*n_tensors budget with few
tensors per layer (cohere2moe: 443 tensors, 49 layers) overflow the
graph context when the cache is combined with a quantized KV cache
(K/V rotation), aborting in ggml_new_tensor with GGML_ASSERT(obj_new).

Fixes #32584

Assisted-by: Claude Opus 5.5
2026-11-20 16:13:00 +08:00
nicholas-mainardi-domyn 55d323507d model: add Domyn Edge 4B Model (#32559)
* feat: add domyn-edge model

* Generate tokenizer hash for conversion + fix tests

* Address changes according to code-review skill

* catch ValueError in convert_hf_to_gguf_update.py

* renegerate fusion test baseline + fix model naming style
2026-11-20 09:51:41 +02:00
Aaron Teo bed8ec2f8d llamafile(s390x): add QK data type support (#32600)
* llamafile: add QK support for s390x

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* llamafile: clean up for s390x

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

---------

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
2026-11-20 09:47:41 +02:00
sys2008 52e8db6550 kleidiai : allow OHOS (HarmonyOS) AArch64 target (#32265)
* kleidiai : allow OHOS (HarmonyOS) AArch64 target

* kleidiai : allow OHOS AArch64 target

Assisted-by: GLM
2026-11-20 09:46:59 +02:00
Nikhil Jain d7d1b632da webgpu : fix 2D indexing for copy kernel (#32576)
* Only run webgpu CI on my fork

* Add webgpu only workflow

* resolve merge conflicts

* Add 2d indexing to copy kernel

* move test
2026-11-20 09:45:44 +02:00
Teng Mei ea84d70903 CUDA: optimize Orin single-column MMVQ (#31268)
* CUDA: optimize Orin single-column MMVQ

* Create a separate Orin branch in `calc_rows_per_block`
2026-11-20 09:44:04 +02:00
Chinmay ShrivastavaandJonSnow1807 9fbe30c2a0 CUDA: keep the MUL_MAT_ID precision in the per-expert fallback (#31345)
* tests: add CUDA-only MUL_MAT/MUL_MAT_ID cases with results above the fp16 range

IQ1_M weights have no MMQ kernel, so once the batch is bigger than the MMVQ
limit the path is dequantize + cuBLAS. These cases produce results around
1.3e5, past the fp16 max of 65504, so an fp16 result buffer turns them into inf
while the CPU reference stays finite. They are only added for the CUDA backend.
Vulkan deliberately accumulates quantized matmuls in fp16 at default precision
and would fail them by design.

* CUDA: use an fp32 result for quantized src0 in the cuBLAS fallback

On the F16 compute path, quantized weights went through cuBLAS with fp16
accumulation and an fp16 result buffer that was converted to fp32 afterwards.
Anything above 65504 became inf and the next layer turned into NaN (see #30951:
gpt-oss-120b with IQ1_M experts at -ub 512). Set prefer_f32_output for
quantized src0 so the GEMM still uses fp16 operands but accumulates and stores
in fp32, the same mode already used on Volta, RDNA4 and CDNA. Every other
quantized path already accumulates in fp32.

Diagnosis and fix by denizyuret in #30951.

* CUDA: keep the MUL_MAT_ID precision in the per-expert fallback

The per-expert dst_slice kept op = GGML_OP_NONE, so ggml_prec_get_acc returned UNDEFINED and a requested precision was ignored. Set the op to GGML_OP_MUL_MAT. Also revert the fp32 result change and remove the tests.

---------

Co-authored-by: JonSnow1807 <JonSnow1807@users.noreply.github.com>
2026-11-20 09:42:16 +02:00
Amadeus dyw c0b97be0b2 vulkan: optimize RMS_NORM_MUL_ROPE loop control (#31149) 2026-11-20 09:38:43 +02:00
Ruben OrtlamandJeff Bolz 16f80bd0c0 vulkan: chunked GATED_DELTA_NET implementation (#30207)
* vulkan: add cm2 chunked GATED_DELTA_NET shader

* fix shmem bank conflict bottleneck

* optimize further

* add coopmat maintenance1 variant

* optimize for RDNA4

* add gmem version RDNA4

* optimize for Ampere

* fix nvidia shmem overuse

* fix cm2 validation errors

* improve cm2 perf

* improve cm1 perf

* clean up

* enable by default

* show cm maintenance1 marker in device info

* properly check for cm_m1 features

* add tests

* add v32 mode, improve chunked condition

* allow other subgroup sizes

* vulkan: specialize the GDN CM2 workgroup size

Assisted-by: Codex

* vulkan: coopmat2 GDN implementation

* work around Nvidia shader compiler crash

* remove shared shader cm2 code

---------

Co-authored-by: Jeff Bolz <jbolz@nvidia.com>
2026-11-20 09:38:14 +02:00
Leon 2d8d003937 webgpu : add COL2IM_1D support (#31708) 2026-11-20 09:36:58 +02:00
7e960f40fb opencl: fold GQA head for f16 FA for better efficiency (#32631)
Assisted-by: Codex

Co-authored-by: Li He <lih@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
2026-11-19 22:26:28 -08:00
Hongqiang Wang 8bf88ba936 vulkan: disable subgroup arithmetic on Qualcomm's driver (#30826)
On Adreno, subgroupAdd() after divergent code returns wrong sums. The
fused ADD+RMS_NORM partial-sum path hits this, so the decode-step norm
is garbage and models emit stray tokens after long prompts.

Clear subgroup_arithmetic for the Qualcomm proprietary driver, which
also turns off add_rms_fusion and the other subgroupAdd paths.

test-backend-ops on Adreno 850 and X2-90: ADD_RMS_NORM 20/25 -> 25/25,
RMS_NORM_MUL_ADD 33/45 -> 45/45. llama 3B Q4_0 on Adreno 850: pp512
and tg128 unchanged.
2026-11-20 05:58:46 +01:00
Johannes Gäßler b94959a64d CUDA: implement support for duplicate expert IDs (#32505)
implement support for duplicate expert IDs
2026-11-19 23:21:24 +01:00
Imdadullah Mohammed 50518b64c6 model : hrm : do not compute slot count as uint32_t (#32601)
load_arch_hparams multiplies n_hrm_layers_per_stack, n_hrm_h_cycles and
(n_hrm_l_cycles + 1) into a uint32_t, then asserts the product equals
block_count.

Any factors whose true product is block_count + k*2^32 still pass the
assert; 1073741825 * 2 * 2 = 2^32 + 4 compares equal to block_count = 4.

load_arch_tensors iterates the same three values in int and writes past
the end of the block_count-sized layers vector.
2026-11-19 23:21:03 +01:00
c406f4367e opencl: support dp4a for KQ mm in decomposed FA (#32577)
* opencl: support int8 KQ for DK = 512

Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
Assisted-by: Codex

* opencl: allow query chunk up to 96 for int8 KQ

Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
Assisted-by: Codex

* opencl: extend `build_program_from_source` to allow non-fatal failure

Assisted-by: Codex

* opencl: add int8 KQ for decomposed FA with DK <= 256

Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
Assisted-by: Codex

---------

Co-authored-by: Li He <lih@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
2026-11-19 11:30:19 -08:00
anujj 7fce5439fe ggml : add BF16 support to ROPE and SET_ROWS (CPU, CUDA) (#32387)
* ggml : add BF16 support to ROPE and SET_ROWS (CPU, CUDA)

* cuda : SET_ROWS from BF16 into quantized types and F8_E4M3

* cuda : fuse BF16 rope into a BF16 cache

* tests : trim the BF16 rope cases to one per code path
2026-11-19 23:43:55 +05:30
JamePeng 53ecea05a0 mtmd : fix YuE2 projector loading (#32606)
- Allow cls.output.weight to be absent from the mmproj. It is an optional
Qwen3 reranking head that the YuE2 NAR decoder never uses.

- Without this fix, mtmd_init_from_file() fails with
"tensor cls.output.weight is missing from the mmproj", preventing
audio generation from starting.
2026-11-19 19:11:11 +01:00
9a080f44b5 ggml-openvino: fix OpenVINO CI, minor fixes and optimizations (#32590)
* ggml-openvino: round the GPU KV length up to a power of two

Review notes (follow-ups, not blocking):
- convert_ggml_input_to_ov pads the KQ mask into a std::vector and then
  copies it into the ov::Tensor. Writing the padded rows straight into
  padded.data() avoids the second allocation and copy (up to 16 MB per
  call at a 16K context).
- The padding is skipped when the mask is not contiguous, and the KV
  inputs are then sliced to more rows than the mask has columns. Masks
  are always contiguous today; a GGML_ASSERT would make that explicit.
- compute_llm_params reads GGML_OPENVINO_STATEFUL_EXECUTION directly;
  is_stateful_enabled() is local to ggml-openvino.cpp.

* ggml-openvino: Optimize model recompiles of prefill stage

Optimize the model recompiles of prefill stage when attended size
changed between prompts. This can save model recompile time for
llama-bench and llama-server.

* ggml-openvino: Optimize model recompiles of prefill stage

Optimize the model recompiles of prefill stage when attended size
changed between prompts. This can save model recompile time for
llama-bench and llama-server.

* ggml-openvino: Remove solution C and keep A as default

* ggml-openvino: match the output count before reusing a compiled NPU prefill

The static prefill model has a fixed inp_out_ids shape [1,1,1,output_len].
The new reuse check compared only the attention sizes, so llama-perplexity,
which alternates 1 and 512 outputs per ubatch, reused the model compiled
for 512 outputs on a 1-output ubatch and failed with "got [1,1,1,1]
expecting [1,1,1,512]". Store the output count of the cached prefill
model and require it to match.

* ggml-openvino: Fix LFM2 context create on GPU and DSpark speculative decoding

* ggml-openvino: remove leftovers from merging dev into the LFM2/DSpark fix

- compute_model_outputs exported OUTPUT-flagged nodes twice: dev's block
  ran before the new one and still exported result_norm, so the
  result_norm exclusion had no effect. Keep only the new block.
- compute_node_dynamic_dims checked is_activation_input twice in a row.
  Remove the second copy.
- process_view_input_new created the dynamic Split on split_dim. With
  stateful execution the input drops the leading size-1 axis, so use
  split_axis, which already accounts for the rank difference.

* ggml-openvino: hash the output flags into the model-cache fingerprint

The fingerprint hashed the ops, names and weights but not the outputs, so
a blob compiled with another output set was imported. With the result_norm
change, an NPU blob written before it gave wrong output, and enabling
llama_set_embeddings_layer_inp() reused a blob without the layer-input
outputs. Hash each node's output flag and bump the cache version to 8.

* ggml-openvino: support f16 SCALE and f16/bf16 TRI

test-backend-ops now runs SCALE on f16 and TRI on f16 and bf16. Both
translators built their constants as f32, and Multiply, Add and Select
need all inputs in one element type, so building the graph threw for
these cases.

SCALE now converts a non-f32 input to f32, applies the scale and bias,
and converts back once, as the CPU backend computes it. TRI creates its
zero constant in the input's element type. f32 graphs are unchanged.

* ggml-openvino: [Qwen3.6 A3B]fix static-shape ids view of the last MoE
 layer

The same-stride single-dim Slice in process_view_input_new was followed by a Reshape
whose target baked the static chunk size into the token axis. After the output-ids
GET_ROWS the last layer runs on n_outputs tokens (1 in the prefill graph), so the
Reshape failed shape validation for qwen35moe with FORCE_STATIC or on the NPU.
Skip the Reshape in static graphs when the slice already keeps the rank;
the dynamic graphs are unchanged.

* enable mamba based models on OV CPU-GPU

* revert changes in tests

* docs : add Falcon-H1 to the OpenVINO validated models

Falcon-H1 runs on CPU and GPU since Mamba SSM support was added. Add its
row and the known limitations: multi-sequence batches fail, GPU accuracy
is slightly lower than CPU, and the NPU fails with an element type
mismatch.

* ggml-openvino: simplify the SSM_SCAN build check and clean up

- Drop the SelectiveSSM configure check and its guards: OpenVINO 2026.4.1
  always provides the op. Drop the no-op LANGUAGE CXX source property.
- Drop the semicolon after OPENVINO_OP("SelectiveSSM"). GCC 15 reports it
  with -Wextra-semi, which fails builds with fatal warnings.
- Add ggml_openvino_is_static() for the repeated NPU or
  GGML_OPENVINO_FORCE_STATIC check.
- Share the f16 and f32 KQ mask padding code.
- Drop the redundant inline on in-class static functions.

---------

Co-authored-by: Suhas Somashekar <suhas.somashekar@intel.com>
Co-authored-by: yzhou51 <yaowei.zhou@intel.com>
Co-authored-by: Ananth, Swathi <swathi.ananth@intel.com>
Co-authored-by: Beheshti <nazanin.beheshti@intel.com>
2026-11-19 07:22:26 -08:00
Bhargav Krish c7468107ef cuda: Scatter kernel for set_n (#32489)
* cuda: add scatter for batched tensor writes

* update test case to mirror get_n test case

* save restore timing logs

* refactor code into batch-copy.cu file
2026-11-19 20:14:47 +05:30
ac830a9cbb ggml : add backend scheduler tests (#27258)
* Adds initial scheduler stress tests

* Adds new sleep op to simplify scheduler testing

* Adds testcase addressing https://github.com/ggml-org/llama.cpp/issues/23321

* Adds experimental vulkan GGML_OP_SLEEP

* Improves structure and naming of previous tests. Adds new tests,
including one which detects the nkvo-bug correctly, and validates the
proposed hotfix.

* Cleanup: Removes CUDA specific comments and nvtx-markers

* Fix: forgot to add sleep specific test case

* Fix oversights identified by CI

* Drops default prints so that it behaves like other ctests. Adds -v flag
to toggle this behavior

* Fix: Early exit in CPU-only settings (cause for graviton CI fails)

* Improves stress_test_linked_list by removing overlap with
test_chain_all_backends and test_pair_user_inputs. The latter ones test
either activations or user inputs in isolation and just once, whereas
the stress test tests all combinations multiple rounds

* Exclude ACCEL devices from new tests due to incompatibility

* Fixes vulkan test failure in llvmpipe

* Adds asserts, improves comments and some checks

* Fixes nitpicks in white-space and variable naming

* Adds comments that certain vk drivers could time OP_SLEEP incorrectly

* Drops limitiation to CPU/GPU in test-sleep

* Aligns OP_SLEEP guards in cuda with vk guards (input shape and
contiguity)

* Implements feedback for OP_SLEEP to work in place (no copy)

* Apply non-code suggestions  from JohannesGaesslers code review

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

* Removes struct backend_supports; enables tests to return skip, instead
of only pass/fail

* Make clang-tidy happy

* Removes sched_backend_caps, improves create_test_scheduler according to
feedback (L149)

* Removes "name" field from struct sched_check in favor of
ggml_set_name/ggml_get_name

* Drops unnecessary shader_clock.comp test and guards

* Replaces sched_ prefixes with helper_ prefix

* Moves test suite to new RAII types for better memory management

* Adds sleep option to test_chain_all_backends

* Adds sleep option to first-run arm of the y-shaped test

* Fix switchcase order for legibility in stress_test_dag

* Simplfies stress_test_linked_list via double for-loop

* Adds pseudo-random input to stress_test_linked_list, and runs all
transitions both with and without CPU input (next to activation copy as
input)

* Fix RPC Proto patch due to GGML_OPS bump

* Adds ggml-cpp.h treatment to test-sleep.cpp

* Adds TODO pointers for subsequent additional tests.

* Apply suggestion to ggml.h

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

* Apply suggestion to tests/test-backend-sched.cpp

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

* Apply suggestion to tests/test-backend-sched.cpp

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

* Fixes node compatibility check to not take buffer locations into account.
This led to skipped tests due to incorrectly rejected ops due to wrong buffer locations in multi-GPU (which is one of the scheduler behaviors under test)

* Apply batched suggestions from gg code review

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* Completes callsite change of gg-suggestion

* Switch test-sleep from std::chrono::steady_clock::now() to ggml_time_us
to avoid mac clock drift and spurious test failures.
This makes the test a bit nonsensical on CPU (same clock checks itself),
but the test still has merit on the other backends where
independent device clocks are used.

---------

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-11-19 16:40:13 +02:00
Piotr Wilkin (ilintar) 8af68d486a llama: fixes for Inkling 2 attention mask (#32429)
* llama : give Inkling2 its own relative-position attention input

The Inkling2 graph built its flash attention mask from the hybrid iswa
input and passed it to build_attn through a new kq_mask argument. That
argument let a caller pair its own dense mask with the kv_rows of
llm_graph_input_attn_kv_iswa, whose mask follows the per-sequence cell
lists whenever kv_rows is enabled.

Add llm_graph_input_attn_rel_pos instead. It owns dense KQ masks and the
query-to-cell distances, fills both from the same cells, and its
build_attn overload adds the per-head bias to its own mask and never
passes kv_rows. The iswa build_attn is back to its previous signature;
its body moves to build_attn_iswa_impl, which both overloads call.

Inkling2 now folds the per-query sink into the bias with and without
flash attention, so build_attn_mha no longer needs the per-query sink
path.

Assisted-By: Claude Opus 5.5 <noreply@anthropic.com>

* inkling2 : make the expert weight scale and final logit softcap optional

Load both keys as optional and skip them when they are zero, as other
models do, instead of rejecting zero at load.

Assisted-By: Claude Opus 5.5 <noreply@anthropic.com>

* llama : build kv_rows masks for the hybrid iswa memory

llm_graph_input_mem_hybrid_iswa checked its masks with
can_reuse_attn_mask, which expects kv_rows whenever they are enabled,
but built them dense. With a unified cache and flash attention the
check always failed, and hybrid iswa models (LFM2, LFM3, Inkling2)
never reused a graph. Build the masks with build_attn_inp_attn_mask,
like the plain iswa input.

Assisted-By: Claude Opus 5.5 <noreply@anthropic.com>

* llama : make the relative-position input a hybrid iswa input

Address review: llm_graph_input_mem_hybrid_iswa_rel derives from
llm_graph_input_mem_hybrid_iswa and owns the attention and recurrent
inputs, so Inkling2 builds one memory input instead of two. Its
attention input is built with dense masks, can_reuse checks them as
dense, and its build_attn asserts that the mask has no kv_rows (TODO:
a kv_rows variant). build_attn_iswa_impl is now a static helper.

Assisted-By: Claude Opus 5.5 <noreply@anthropic.com>

* llama : compose the hybrid iswa rel input, format inkling2

Address review: llm_graph_input_mem_hybrid_iswa_rel owns its attention
and recurrent inputs like the other composite inputs instead of deriving
from llm_graph_input_mem_hybrid_iswa.

Format src/models/inkling2.cpp like the other model sources: grouped
and aligned key and tensor loading, descriptive per-layer names, blank
lines and section comments between the steps, and cb() names for the
intermediate tensors. The graph is unchanged.

Assisted-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 14:36:30 +01:00
Alexey Kopytko f8caa5ea30 metal : fix the second argument of NSMakeRange (#32602) 2026-11-19 13:49:22 +02:00
SS1618 4db8b7638f webgpu: add POOL_2D support (#32250)
* Shader, pipeline, and launcher

* Update docs and add newline to wgsl

* Formatting fixes
2026-11-19 13:35:10 +02:00
Georgi Gerganov 01592dc58b llama : remove redundant fields from llm_graph_params (#32596)
llm_graph_params no longer carries hparams, prec_policy and hdmd, since the model reference already provides them. llm_graph_context sources them from the model and keeps prec_policy as a reference.

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
2026-11-19 13:31:50 +02:00
Georgi Gerganov 8bf2132c3f readme : fix release badge (#32597) 2026-11-19 13:20:47 +02:00
Adrien Gallouët 5812201a81 ci : fix build-opencl.yml (#32594)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-11-19 11:24:10 +01:00
PascalandGeorgi Gerganov 41d8297fe4 metal : CONV_2D tensor API for M5, split-K for all chips (#32542)
* metal: add a tensor API conv_2d kernel and split long reductions

The conv_2d matmul kernel runs on simdgroup matrices only, and gives
each threadgroup the whole reduction of its output tile. On M5 this
leaves the tensor cores unused, and convolutions with few output
positions and a long reduction (IC * KW * KH in the thousands) run a
handful of threadgroups on an otherwise idle GPU.

Add kernel_conv_2d_mm_tensor for devices with the tensor API: like
kernel_mul_mm, it gathers the patches of 64 output positions into
threadgroup memory as the left operand and reads the kernel rows
from device memory as the right operand. Devices without the tensor
API keep the simdgroup kernel.

Both kernels can now split the reduction until there are enough
threadgroups: each split writes its partial sums to a buffer placed
after the output, as flash attention does, and a small kernel adds
them in a fixed order, so the result stays deterministic. CONV_2D is
marked as an op whose allocation may exceed its output size.

test-backend-ops -o CONV_2D passes, with and without
GGML_METAL_TENSOR_DISABLE.

CONV_2D, F16 kernel, us/run on an M5 Max:
  [19,19,256,16] k [4,4,256,4096]     19008 -> 5235
  [16,16,128,8] k [3,3,128,512]         333 -> 124
  [1536,1536,32,1] k [3,3,32,64]      12086 -> 6512
  [33,1,1280,1] k [3,1,1280,1280] s2    659 -> 33
  [2054,1,768,1] k [7,1,768,768]       3161 -> 699

* metal: walk the conv_2d patches incrementally, size splits by output

The patch gather of both conv_2d matmul kernels spent most of its
time on address math: 64-bit multiplies, divisions and four bound
checks per value. Walk the reduction index with 32-bit byte offsets
inside the image, updated by additions, with one unsigned compare per
bound; the host checks that an image fits. The tensor kernel no
longer gathers the reduction indices past the end of its split.

The number of splits now targets about 1M partial sums, with at least
3 reduction steps per split, instead of counting the tiles of one
kernel.

CONV_2D, F16 kernel, us/run on an M5 Max:
  [19,19,256,16] k [4,4,256,4096]     4998 -> 4512
  [1536,1536,32,1] k [3,3,32,64]      6721 -> 5354
  [4096,1,1024,1] k [7,1,1024,1024]   2701 -> 2236
  [2054,1,768,1] k [7,1,768,768]      1050 -> 871
  with GGML_METAL_TENSOR_DISABLE:
  [4096,1,1024,1] k [7,1,1024,1024]   9047 -> 8283

* nit: share the CONV_2D simdgroup count between host and kernel

kernel_conv_2d_mm and its pipeline both hardcoded 4 simdgroups per
threadgroup. N_SG_CONV_2D in ggml-metal-impl.h now feeds the kernel
constant and res.nsg, so the dispatch and the kernel cannot drift apart.

* metal: fall back to the generic conv_2d kernel for images past 4 GiB

The matmul conv_2d kernels walk the patches with 32-bit byte offsets
inside an image. Instead of asserting on nb13, compute the largest
offset actually read from ne/nb, which also covers strided views of
the input, and route images that do not fit to the generic kernel.

* cont : comments

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-11-19 11:48:44 +02:00
saiph 51a77c4f24 qwen3vl : set t_layer_inp for EAGLE-3 (#32589)
Assisted-by: Grok Bot
2026-11-19 10:13:02 +01:00
Bartowski 76f186df71 ggml : IQ2 creation speedup through caching (#31910)
* IQ2 quant creation speedup from caching

* Rename variables for clarity

* Explicit casts

* Initialize all values, add comment to tolerance
2026-11-19 10:04:58 +02:00
Adrien Gallouët 95b09b62be llama : remove PATH_MAX (#32557)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-11-19 08:57:57 +01:00
Adrien Gallouët b5dbca1e44 ci : use LLVM 22 (#32565)
* ci : fix build-vulkan.yml

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* ci : fix behavior-check.yml

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

---------

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-11-19 08:54:04 +01:00
Aleksander Grygier 80437488a4 ui: cut Llama Chat's first-load JavaScript by 63%, load diagrams, math and PDFs on demand (#32555)
* ui: import Lucide icons by path in modules that run in Node

Assisted-by: Claude (Opus 5.5 Medium)

* ui: register highlight.js languages once, from lowlight's modules

Assisted-by: Claude (Opus 5.5 Medium)

* ui: load the pdf.js worker and nerdamer as files of their own

Assisted-by: Claude (Opus 5.5 Medium)

* ui: draw Mermaid diagrams with Pele

Assisted-by: Claude (Opus 5.5 Medium)

* ui: pin dependency versions and use one Svelte across the UI packages

Assisted-by: Claude (Opus 5.5 Medium)

* ui: build Llama Chat as split chunks

Assisted-by: Claude (Opus 5.5 Medium)

* ui: load KaTeX and pdf.js on first use

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-19 08:37:35 +01:00
Adrien Gallouët 39039aa27f llama : fall back to buffered I/O on EIO (#32545)
* llama : fall back to buffered I/O on EIO

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Preserve bytes already read, resumes at the current file offset

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

---------

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-11-19 07:50:33 +01:00
Birk Skyum 7a4117eef9 vulkan : enable MMQ with mixed signedness integer dots (#31045)
* vulkan : enable MMQ with mixed signedness integer dots

* vulkan: specialize MMQ shaders for mixed-signedness dot products

* vulkan: track integer dot product support with signed, mixed and unsigned flags

* vulkan: move signed dot check to the MMVQ call sites
2026-11-19 08:15:09 +02:00
Jeff Bolz 573aa89de6 vulkan: Implement stream-k MUL_MAT (#28528)
* vulkan: Implement stream-k MUL_MAT

Stream-k tries to split chunks of 256 K elements evenly across all SMs.
Workgroups that finish a tile write to dst, others write to scratch, and
there's a resolve shader similar to split-k.

The shader code is implemented for scalar+cm1+cm2, but only currently enabled
for cm2. The heuristic for enabling on coopmat2 is pretty simple due to the
large tile size (one workgroup per SM), but would be more complex for the tile
sizes being used for cm1, and it doesn't make sense for me to tune that on
NVIDIA devices.

Assisted-by: Codex

* address review feedback
2026-11-19 08:14:14 +02:00
Daniil Sterkhov ecafc67145 vulkan: use floating-point zero for storage-only FP16 in DIAG/TRI (#32379)
* vulkan: Change integer zero to floating-point zero for storage-only FP16 in DIAG/TRI.

Assisted-by: OpenAI Codex

* delete narrow test
2026-11-19 08:13:06 +02:00
Aman Gupta d4f5c26049 CUDA: use 64-bit dst offsets in mul_mat_q (#32374)
The MMQ kernels computed the dst tile offset (jt*J*stride_col_dst) in
32-bit int. For a dst with more than 2^31 elements, column tiles that
start past INT_MAX/nrows wrap around and write out of bounds.

This happens with an all-logits output head, e.g. llama-perplexity
-c 32768 -b 32768 -ub 16384 on a 248320-vocab model: the q8_0 output
matmul has a 248320 x 16384 dst (4.07e9 elements) and aborts with an
illegal memory access.

Widen the offsets in mul_mat_q and the stream-k fixup to int64_t, and
add a (disabled by default, ~9 GB) test-backend-ops case.

Assisted-by: Claude Opus 5.5
2026-11-19 08:11:44 +02:00
lhez 08c252e221 opencl: allow adjusting KV padding in decomposed FA (#32522)
Assisted-by: Codex
2026-11-18 20:11:46 -08:00
Xuan-Son Nguyen e4b620af19 mtmd: support voxcpm2 (#32443)
* mtmd: support voxcpm

* speed up a bit

* various fixes
2026-11-19 02:39:05 +01:00
En YaoandXuan Son Nguyen b001d7bb22 ggml : WASM Relaxed SIMD Enhancement (#19590)
* add relaxed simd

* fix indentation

* remove comments and whitespace

* wasm: fix relaxed SIMD kernels and add q3_K/q6_K support

* rm trailing space

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-11-19 01:36:17 +01:00
mahounairoandAlexey Kopytko 0a76757469 llama : fix sticky NaN in recurrent state (#31913)
Co-authored-by: Alexey Kopytko <alexey@kopytko.com>
2026-11-19 08:08:12 +09:00
Xuan-Son Nguyen b044890408 server: fix sleep state for v1/realtime (#32567) 2026-11-18 23:48:02 +01:00
parthod0x 78e05298f0 model : limite: reject mudd configs that index past mudd_tap_idx on load (#32562)
n_mudd_taps comes from limite.mudd.tap_count, a scalar read separately
from the length-checked tap_indices array. mudd_tap_idx is a fixed
std::array<int32_t, 512> indexed at [i*n_taps] in both the tensor load
and the graph build, so a crafted GGUF with n_layer * n_mudd_taps > 512
reads past the array.

Reject that at hparams load; the one check covers both index sites.
2026-11-18 23:09:55 +01:00
Chenyang Li 785f6bac00 ui : keep streaming reasoning box inside the viewport when zoomed (#31966) 2026-11-18 22:56:47 +01:00
Adrien Gallouët edec462cc0 ggml, ci : use llvm 22.1.8 (#32548)
* ggml, ci : use llvm 22.1.8

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Update build-cpu.yml too

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Fix clang 22

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Fix release.yml (arm64 with vulkan)

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Fix server.yml

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

---------

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-11-18 21:10:23 +01:00
Alessandro de Oliveira Faria (A.K.A.CABELO)andAlessandro de Oliveira Faria 4483b2b987 vendor : update cpp-httplib-0.67.0 [no ci] (#32563)
Co-authored-by: Alessandro de Oliveira Faria (A.K.A.CABELO) <cabelo@multicortex.ai>
2026-11-18 21:00:31 +01:00
Tarek Dakhran bafa0420c1 server : enable packing for d1-2B and d1-Omni-2B models (#32553)
* llama-bench : size the batch to the prompt for models without memory

Assisted-by: Claude Opus 5.5

* server : read up to 16 lfm3-d1 questions in one pass

Assisted-by: Claude Opus 5.5
2026-11-18 20:15:43 +01:00
mmalpe-nvandJohannes Gäßler 2cbfc3cf34 [CUDA] : Add SKU-aware FP16/BF16 accumulation precision gating for NVIDIA GPUs (#30672)
* cuda: select cuBLAS precision by GPU SKU

Assisted-by: Codex

* cuda: fix remaining GB10 macro reference

* cuda: review comment addressed, store accumulation preference in device info

* Apply batched suggestions from code review

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

* addressed Oliver's comment regarding initialize NVML once and install headers in Windows CI

* move precision selection helper into init

---------

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
2026-11-18 18:29:56 +01:00
Masashi Yoshimura 6f311d7ad6 webgpu: (fix CI) allow the FA mask to be broadcast over the Q batch (#32535)
* allow the mask batch to be broadcast over the Q batch

* use Q_ne3 for blk_batch_count of get_alloc_size function
2026-11-19 01:08:02 +09:00
Aman Gupta ff6b8d1761 ggml-cpu : use all threads for large GET_ROWS (#32514) 2026-11-18 23:39:30 +08:00
varunandvarun 98d67c8d53 model:add Zaya1 8B (Zyphra) (#32071)
* model:add Zaya1 8B (Zyphra)
- Added the support for Zaya.
- Fixed the SSM_CONV case, so the model runs on cuda without abort.

 Please enter the commit message for your changes. Lines starting

* Made the following changes based on the bot reviewer feedback:
1) Added runtime asserts when loading wrong KV head size, unsupported conv kernel size.
2) Removed the static cast usage.
3) Fixed double load of tokenizer.
4) Updated res_scale_in to be input tensor.

* - Moved the res_scale_in back to layer 0 to fix the cuda bug

* i)  Removed the EOS addition.Team agreed that updating Tokenizer is the right move.
ii) Fixed formatting

* Fixed the comment to make it clear

---------

Co-authored-by: varun <varun_subramanian@zyphra.com>
2026-11-18 15:46:49 +01:00
Aleksander Grygier 387395aa8d ui: install @ggml-org/llama-brand from npm (#32544)
* ui: install @ggml-org/llama-brand from npm

Both packages take llama-brand 0.1.0 from the registry instead of a
codeload tarball, so npm 12 installs them without allow-remote, which
leaves Llama Chat's .npmrc.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: pin @ggml-org/llama-brand to 0.1.0

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-18 15:31:49 +01:00
Aleksander Grygier a693a5f66f ui: llama-chat publish drops the registry check, bin paths without ./ (#32543)
* ui: llama-chat publish waits for the registry, bin paths without ./

The first publish of @ggml-org/llama-chat took about four minutes to show
up on the registry, so the 60 seconds the check after publishing waited
failed a release that had gone through. It now waits up to ten minutes.

npm normalizes "./bin/cli.js" to "bin/cli.js" on publish and warns that
the bin was "invalid and removed", which reads as if the command was
dropped. Both package.json files now give the paths as npm writes them.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: drop the registry check after llama-chat publishes

npm publish already fails when the registry refuses a package, and the
check after it only failed a release that had gone through, while npm
was still processing it. Nothing waits on the registry any more.

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-18 15:29:54 +01:00
Jai Patel b069e526de gguf-py : fix tensor offset overflow in GGUFReader (#32495)
* gguf-py : fix tensor offset overflow in GGUFReader

* gguf-py : cast alignment padding to int
2026-11-18 15:28:47 +01:00
Aleksander Grygier ffc87c3a79 ui: soften the action card border and show the split-button divider in dark mode (#32538)
The tool permission card took the fields' border (white 30% in dark), which
outweighed everything else on the page; it now has its own token, white 18%
in dark, the focused chat form's edge. The split-button divider had the
keys' own rim colour (white 12% over a white 9% fill) and vanished between
them; it now has its own token, white 20% in dark. Light mode is unchanged.

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-18 13:46:25 +01:00
Emanuil Rusev 10f173b253 ui: hide Models and MCP behind settings, off by default (#32527)
* ui: hide Models and MCP behind settings, off by default

Two new settings in General, "Show Models" and "Show MCP", both off by default, like "Enable Discover Models". The WebUI is often the first thing people new to local models see, and most of them don't need a models manager or know what MCP is.

"Show MCP" only hides the way to manage MCP; servers already added keep working. Turning MCP off for real would mean gating every place that reads the server list (health checks, tool groups, prompts, avatars). Anyone who already has servers saved gets MCP shown, so they don't lose the way to manage them.

With Models hidden, every entry point to the manager goes: the sidebar, search, the shortcut and "Manage models" in the selector -- it opens the same full manager. The /models page still opens by its URL, which apps link to. On a single-model server the selector chip used to open the manager; with it hidden, the chip is a plain label.

* ui: keep Models shown while Discover Models is on

Discover opens only from the Models page (its header button and the manager's row actions), so with Models hidden by default, checking Discover showed nothing. Checking Discover now shows Models too and locks its checkbox, the same way it already locks the Hugging Face Hub API one; unchecking Discover leaves Models as it is. Saved configs with Discover on get Models turned on when they load, so people who use Discover don't lose it.
2026-11-18 13:26:31 +01:00
Aleksander Grygier 97069ff6c4 ui: run Llama Chat with npx @ggml-org/llama-chat (#32524)
* ui: run Llama Chat with npx @ggml-org/llama-chat

Makes @ggml-org/llama-chat publishable: drops private, adds bin,
files, exports, repository and a prepack build. Its CLI serves the
prebuilt dist on localhost (3000 by default) and forwards every other
request to a llama-server (--server, default http://127.0.0.1:8080),
with the headers llama-server sends for the UI. A thin unscoped
llama-chat package in alias/ forwards to it, so npx llama-chat works.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: llama-chat revalidates service-worker.js and asks for a build before packing

SvelteKit 3 adds service-worker.js next to sw.js, and llama-server
revalidates it, so the CLI does too. The folder's .npmrc sets
ignore-scripts, so prepack never ran: it goes, and the publishing steps
build dist explicitly and check the packed file list.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: llama-chat keeps llama-server's DNS rebinding and cross-origin guards

The CLI called llama-server with its own Host and dropped every Origin,
so a rebound host name or another page got past the checks llama-server
makes when --cors-origins is not *. It now drops only its own page's
Origin, passes other Origins and Sec-Fetch-Site through, and, when it
listens on loopback, answers only to localhost and loopback addresses.
`start` on Windows gets its empty title verbatim.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: llama-chat build writes the app to a folder for any static server

`npx @ggml-org/llama-chat build [--out <dir>] [--force]` copies the
prebuilt app, without source maps, to ./llama-chat or the given folder,
so a plain file server can host it next to the CLI serving it. It
refuses a folder that is not empty unless forced, and the serving flags
and build flags refuse each other's command. The README shows serving
the folder with http-server forwarding the API to llama-server.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: publish Llama Chat to npm from CI when its version is bumped

ui-chat-publish.yml publishes @ggml-org/llama-chat and its llama-chat
alias the way ui-lib-publish.yml publishes @ggml-org/llama-ui: the
version is set by hand, and a commit that reaches master with a version
npm does not have builds dist, checks the packed files and publishes both
with trusted publishing. PRs that change the packages get a dry run.
npm-version-check.mjs takes the package folder, so both workflows share
it. ui/README.md says how releases work, and Chat's AGENTS.md says not
to bump versions unless asked to release.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: call the app Llama Chat in its README

The README's title and a few lines still called the app llama-ui or
Llama UI, the name of the library it is built on. They now say Llama
Chat, and the intro points to Llama UI in ui/lib/ui.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: llama-chat connects to llama-server's default port, 9931

The CLI's --server default and its examples used 8080, which is not
where llama-server listens. They now use 9931, its default port, so
npx llama-chat reaches a llama-server started without --port.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: a shorter Llama Chat README that matches the code

The README grew to nearly 900 lines, and its tables of components,
hooks, stores and services had drifted from the code. It now keeps what
a reader needs: running it with npx and self-hosting, a short feature
list, development commands, the architecture (the line between Llama UI
and the app, a table of the layers and where they live, a flowchart and
one sequence diagram for sending a message), a few patterns worth
knowing, cross-origin isolation, how the build is embedded in
llama-server, the project tree and publishing. The folders now carry the
detail the tables used to. ASCII replaces the unicode arrows.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: llama-chat serves only static files, without the API proxy

The CLI forwarded every request it had no file for to llama-server, from
loopback. With --host 0.0.0.0 that let any LAN client reach a llama-server
started on 127.0.0.1 as a local request, tools included, which llama-server
on its own refuses without an API key. The browser now talks to each
llama-server directly, so llama-server's own rules apply. --server is gone,
and anything that is not a file in dist is a 404, or a 405 for other methods.

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-18 12:58:16 +01:00
Gaurav Garg bba8408821 Add LLAMA_MOE_RANDOM_ROUTING=1 for benchmark-only random expert selection (#29539)
* Add LLAMA_MOE_RANDOM_ROUTING=1 for benchmark-only random expert selection

* Handle expert caching
2026-11-18 17:04:40 +05:30
Xuan-Son Nguyen 48a32194ee server: fix transcription prompt for v1/realtime (#32507) 2026-11-18 12:09:22 +01:00
Aleksander Grygier 6da36139e8 ui: run the Storybook deploy on HF Jobs, with a token that can write the Space (#32525)
The deploy failed on master with a 403: HF_TOKEN_UI_STATIC_OUTPUT is
scoped to the UI bucket and cannot write ggml-org/llama-ui-storybook, so
hf upload tried to create the Space. The upload now uses
HF_TOKEN_UI_STORYBOOK when that secret exists, and the job runs on
hf-jobs-cpu-upgrade instead of a GitHub-hosted runner. That image gets
python3-venv from apt for the hf CLI, as the other HF Jobs workflows do.

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-18 12:07:23 +01:00
Daniel Bevenius 61f182fcde unicode : add continuation check lambda (#31070)
This commit adds a lambda to check if a specified byte is not a valid
continuation byte in a multibyte UTF8 sequence.

The motivation for this is to improve readability and avoid some code
duplication.
2026-11-18 11:44:28 +01:00
losewayy 03ec5b4326 cuda : fuse NORM + MUL (+ ADD) (#32454)
Assisted-by: Codex
2026-11-18 12:43:52 +02:00
Daniel Bevenius 7144124a5b vocab : extract bpe_rank loading into function (#30962)
* vocab : extract bpe_rank loading into function

This commit extracts the BPE rank loading logic into a separate function
named `load_bpe_ranks`.

The motivation for this is that the came code is currently used by both
gpt2/hybriddna/whitespace tokenizer models and gemma4 and this avoids
the duplication of code to improve readability and maintainability.

* change signature of load_bpe_ranks
2026-11-18 11:33:12 +01:00
Ruben Ortlam 43bd7a2a42 vulkan: add bfloat16 support to various ops (#32202)
* vulkan: expand bf16 support for unary and binary ops and rms_norm

* cpu: add bf16 support for scale, clamp, leaky_relu, swiglu_oai, swiglu_clamp, tri and diag

* fix sycl fill supports_op condition

* restore CUDA op order
2026-11-18 11:26:03 +01:00
Aleksander Grygier 65758d01f5 ui: Storybook follow-ups: generated DESIGN.md tokens, shared helpers and one Llama Mono (#32523)
* ui: generate DESIGN.md's token sections from the CSS

Add scripts/design-md.js in ui/apps/chat (npm run design:tokens, and npm
run check:design in CI), which writes the colors, typography and rounded
sections of DESIGN.md's front matter from llama-brand's colors.css and
Llama UI's primitives, semantic tokens and font stacks. The front matter
had drifted: it gave Matte's surfaces as the light base and system-ui for
every text role. Twins are now written wherever a token resolves
differently in dark mode (-dark) or in Matte (-matte).

Describe the floating panel's rim as the CSS draws it, on the border,
and say which parts of DESIGN.md are generated and which are kept by hand.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: say the Storybooks show accessibility findings without enforcing them

Both Storybooks keep a11y test 'todo': under 'error' 98 of the stories
fail, most on colour contrast of the muted foreground and on nested
interactive controls, which need design changes. Reword the Uses tile
and record the reason next to the setting.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: list every Llama Chat group in the Introduction's Sections figure

The figure said it was in sidebar order but left out Forms, Actions,
Badges, Misc, Servers and PWA. Its list moves to sections.ts, and a unit
test checks it against storySort in .storybook/preview.ts, which has to
stay inline there because Storybook parses it as written.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: load the package's Llama Mono in both Storybooks

The library's Storybook declared Llama Mono a second time from
tests/stories/brand/fonts, a 72276-byte file that differs from the
76896-byte one the package ships, so its pages showed a Llama Mono that
does not ship. Drop both Storybooks' copies: theme.css already loads the
package's font.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: take Llama Chat's shared Storybook helpers from Llama UI

Llama Chat kept copies of the Editorial, Chapters, StateGallery,
StateGalleryItem, svx.d.ts and docs.css that live with the library's
stories. Its stories now import them as $llama-ui-stories/..., an alias
its Storybook config and tsconfig.json map onto ui/lib/ui/tests/stories,
the way it already compiles the library from source; docs.css and
svx.d.ts are referenced by path. The library's Editorial takes Llama
Chat's newer list numbering. An empty svelte.config.js next to the
stories keeps Llama Chat's svelte-check from loading the library's
vite.config.ts, as src/svelte.config.js does for the sources.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: pin the library's Storybook to Llama Chat's versions

ui/lib/ui used ^10.6.2 with @storybook/addon-mcp at 10.6.2, while
ui/apps/chat pins 10.6.6, so one tree had two Storybooks. Pin the
library's Storybook packages exactly to Llama Chat's: 10.6.6,
addon-svelte-csf 5.1.6 and @chromatic-com/storybook 5.5.1.

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-18 09:45:36 +01:00
Ryn Wang a300a9c334 cuda : increase Pascal DP4A MMQ I tile for J=64 (#31352) 2026-11-18 09:08:35 +01:00
+4 4577932fab ggml-openvino: expand model support, optimize NPU,GPU and memory usage (#32172)
* ggml-openvino: fix SWA mask detection

When the SWA cache size matches the full cache size, identify SWA
layers by the number of layers using each mask instead of cache size;
the larger group is SWA, and a tie leaves the layers unclassified.

* ggml-openvino: fix SWA KV cache write indices

Give SWA KV cache index tensors unique input names (a _swa suffix, like
the masks) to prevent them from being merged with the full-attention
cache indices.

* ggml-openvino: classify SWA layers only when the mask shows the window

Two attention masks do not imply SWA: DeepSeek sparse attention uses two
full-attention masks with the same name. When cache extents match,
classify the larger mask group as SWA only once its mask has fewer keys
per row than the other mask. Keep the classification per KV cache buffer
so new sequences reuse the same graph, and include swa_layers when
checking graph reuse.

* ggml-openvino: classify SWA layers from the first graph

Waiting for the window to appear in the mask caused a mid-sequence graph
recompile (~9 s for Gemma-4 at -c 1024 after position 512).

SWA classification already gives each mask group its own mask, KV write
indices, and attention size, which is correct whether or not the mask is
a sliding window. Stateful mask rebuild recovers the window from the
mask, or the causal mask for full attention. Classify the larger of two
mask groups from the first graph and drop the per-KV-buffer bookkeeping.

* ggml-openvino: mark SWA fallback as a model-level heuristic

Add a TODO above the equal-extent fallback to document that the larger
mask group is currently assumed to be the SWA group. Since stateless
graphs only require the two groups to remain distinct, this heuristic
may only be necessary for stateful execution and could potentially be
restricted to that path.

* ggml-openvino: derive attention sizes from the KQ mask shape

Dynamic CPU/GPU graphs previously passed n_seq_active and
attention_size(_swa) as separate inputs. Derive these values directly
from the KQ mask shape using ShapeOf + Gather inside the model. This
allows the GPU plugin to infer the Q reshape and KV slice shapes from
shape inference rather than runtime input values. Q's head size also
becomes static, enabling the fused SDPA kernel.

GGML_OPENVINO_DISABLE_SHAPE_FROM_MASK=1 restores the previous
separate-input behavior.

The fused SDPA path also requires two supporting changes:

- Clamp the GPU mask to -30000. A key block that is entirely -inf for
  a row produces NaNs in the fused kernel.
- Restate the head count and head size after the KV slice. The plugin
  loses all slice dimensions when the end index is dynamic, causing
  SDPA to fall back to the reference kernel.

Additional graph and quantization improvements:

- Skip the identity reshape of rows in SET_ROWS.
- Share a single Split for one-element view slices in dynamic GPU
  graphs (e.g. Gemma-4 per-layer embeddings), replacing one
  StridedSlice per layer.
- Add GGML_OPENVINO_REQUANT_EMBD=native (off by default) to preserve
  the file's native quantization for token_embd/output instead of
  requantizing to Q8_0_C.
- Include both environment settings in the graph and model cache keys
  and document them.

* ggml-openvino: only reuse Split outputs for exact chunk-aligned views

The dynamic GPU path reused a Split output for any single-element view
by deriving the chunk index from offset / stride. This produced
incorrect results for views that were not chunk-aligned or whose
remaining dimensions did not preserve the source strides, as seen in
test-backend-ops CONT with use_view_slice.

Restrict Split reuse to cases where the view exactly matches a split
chunk, meaning the offset is aligned to the split stride and the
remaining non-unit dimensions preserve the original source strides.
Gemma-4 per-layer embedding slices continue to use the shared Split
path.

* ggml-openvino: initialize ModelExtraInputInfo shape fields

Initialize the newly added `shape_source` and `shape_axis` fields in
the `ModelExtraInputInfo` brace initializer used by
`add_extra_inputs()`.

Without initializers for these fields, GCC emits
`-Wmissing-field-initializers`, causing CI to fail with
`LLAMA_FATAL_WARNINGS`.

* ggml-openvino: remove GGML_OPENVINO_REQUANT_EMBD

The option identifies embedding and output weights by their tensor
names, which are defined by core llama.cpp. Remove it from this PR;
preserving the file quantization of these weights will be addressed in
a separate change that identifies them by their roles in the graph.

* ggml-openvino: keep manual GQA attention 4-D

The manual GQA attention path reshaped Q, K, and V to 5-D so that
query heads sharing a KV head could broadcast against it. The GPU
plugin cannot fuse this graph: it materializes the attention scores
and copies the transposed K and V caches for every layer.

Instead, stack the query heads for each KV head along the sequence
axis (Q [B, H, S, D] -> [B, H_kv, factor * S, D]) and tile the mask
accordingly. This keeps the graph 4-D from the MatMuls through the
Softmax, allowing the GPU plugin to fuse the attention into its SDPA
kernel and fold the K and V transposes into the fused operation.

Clamp the mask to -30000, matching the SDPA path, because the fused
kernel can return NaN when an entire block is masked with -inf.

Models where the K or V head size differs from the Q head size now
use the SDPA path, since the manual path assumes one head size.

* ggml-openvino: use a 4-D stateless KV cache on GPU

In stateless GPU mode, bind the entire KV cache as a
`[1, ctx, n_heads_kv, head_size]` tensor, with one token per row,
instead of binding the first `n_kv` rows as a
`[1, 1, ctx, n_heads_kv * head_size]` tensor.

Write cache updates with `ScatterUpdate` along the token axis, and
read the cache through `VariadicSplit` and `Transpose`. This allows
the GPU plugin's `StatelessKVFusion` to replace cache reads and writes
with in-place `StatelessKV` nodes, avoiding full-cache copies on every
decode step and repeated memory pool allocations as the bound cache
size grows.

Set `GGML_OPENVINO_DISABLE_KV_4D=1` to retain the previous layout.

* make embedding to have dynamic shape for OV GPU

* ggml-openvino: make the 4-D stateless KV cache opt-in

* add support for Q5_0 data type in OV GPU backend

* resolve CI test error

* ggml-openvino: fix -fa off and optimize grouped-query attention

With -fa off, llama.cpp stores the V cache transposed, with value
elements as rows and tokens along each row. The KV slicing code assumes
one token per row and binds the wrong data when n_kv < kv_size. Detect
transposed caches with GgmlOvDecoder::is_kvcache_token_rows() and skip
slicing for them.

For grouped-query attention, avoid repeating K and V for each query
head. Stack the query heads for each KV head along the row dimension
and let SOFT_MAX apply the scale, mask, and softmax so the GPU plugin
can fuse the operation with the MatMuls.

Read only the current layer's V-cache rows when layers use different
cache sizes. Reject -fa off in stateful mode with a clear error and
document that it requires stateless mode.

* ggml-openvino: fall back to stateless execution when -fa off

With `-fa off`, the graph writes the V cache in transposed form, which is
not supported by the stateful KV cache. Detect this pattern in
`compute_llm_params()` as a `SET_ROWS` into a KV cache through a `[1, n]`
reshape.

When stateful execution is enabled, log a warning and fall back to
stateless execution instead of failing graph conversion.

Update the `-fa off` note in `OPENVINO.md` to document this behavior.

Co-authored-by: Suhas Somashekar <suhas.somashekar@intel.com>

* ggml-openvino: preserve flat KV cache layout when -fa is off

With -fa off, the 4-D KV cache layout introduced in #344 breaks MUL_MAT
attention for Phi-3.5 mini and Phi-4 mini, causing oneDNN to fail with
"could not construct a memory descriptor using strides".

Preserve the [1, 1, ctx, n_heads_kv * head_size] layout when the graph
writes a transposed V cache.

* ggml-openvino: extract Q5_0 weights when requantization is off

Q5_0 weights are requantized to Q4_0, but test-backend-ops builds naive
graphs, where requantization is skipped. Q5_0 then reached the normal
weight extraction, which had no Q5_0 case, so every Q5_0 MUL_MAT and
GET_ROWS op test failed with "Unsupported quantized type: q5_0".

Extract Q5_0 as symmetric i8 weights (q - 16) with one f16 scale per 32
weights, the same layout as Q8_0. Models are unchanged: they still
requantize Q5_0 to Q4_0.

* ggml-openvino: Optimize GDN output pre-scale for the NPU static path

The NPU GatedDeltaNet kernel writes its output in FP16 and flushes values below 6.1e-5 to zero.

* Refine the code as review comments

1. Add envs desc into doc
2. Refine the code with removing duplicated env

* ggml-openvino: reuse graph bindings between decode calls

Reduce host overhead outside `infer()` from 0.5–1.1 ms to 0.14–0.38 ms
per token on Panther Lake.

- Reuse KV cache tensors and skip redundant port bindings.
- Use persistent USM host tensors for small GPU inputs and outputs.
- Defer graph-wide consumer lookup until after shape checks.
- Return model and compute params by const reference.
- Preserve tensor extras when memory, shape and type are unchanged.

* ggml-openvino: Add npuw dump envs for profiling

* Add the comments of new npu dump envs into doc

* ggml-openvino : ensure all leaves have unique names, remove input_srcs from graph key

* openvino: slice NPU prefill KV cache by active length

Add GGML_OPENVINO_NPU_KV_SLICE as an opt-in NPU prefill optimization. When enabled, static prefill keeps the graph's attention dimension at the active prompt/KV length instead of expanding it to the full context capacity, and the KV-cache input/output tensors are bound with a shortened context axis over the same backing storage.

This reduces the amount of KV cache and attention-mask data imported by the NPU plugin and lets the prefill graph plan smaller attention buffers for early prompt chunks. For example, with a 4096-token context and a 256-token prefill chunk, the first chunk can expose KV/mask dimensions at 256 instead of 4096, avoiding work over padded cache rows.

The trade-off is graph stability: NPU static shapes now depend on the aligned active attention length for prefill, so a cached prefill model is reused only when the attention sizes match. Decode keeps a fixed full-context graph to avoid recompiling on every generated token, and SWA/ring-cache cases still fall back to full-size binding when the live prefix is not a simple leading slice.

* openvino: use L0 host tensors for NPU buffers

Add GGML_OPENVINO_NPU_L0_HOST_TENSORS as an opt-in NPU buffer allocation mode. When enabled, non-remote OpenVINO backend buffers on NPU are allocated through the NPU default context with create_host_tensor(), and ggml tensor data points directly at that OpenVINO-owned Level Zero host allocation.

The optimization targets the data path rather than the graph math: KV-cache and compute tensors can be passed back to the NPU plugin as OpenVINO tensors backed by importable Level Zero host memory. That avoids an extra ordinary-host-memory-to-plugin-staging copy on each infer boundary, which is especially useful for prefill where large KV/cache buffers dominate input and output traffic.

If Level Zero host allocation fails, the backend logs a warning and falls back to the existing allocation path. The destructor tracks OpenVINO-owned storage so it does not free memory owned by the host tensor. The trade-off is that this is NPU-specific, consumes Level Zero host allocation resources, and improves transfer/import overhead without changing attention compute complexity or tensor shapes.

* openvino: fill NPU prefill masks directly

Add GGML_OPENVINO_NPU_FAST_MASK as an opt-in NPU static prefill input-packing optimization. When enabled, the prefill attention mask is built directly in the OpenVINO input tensor instead of first materializing a padded std::vector and then copying that whole buffer into the tensor.

The new fill_prefill_mask() helper writes each row once: it copies the valid mask prefix, fills padding columns with -inf, handles padded rows, and restores the causal diagonal to zero. This keeps the final mask numerically identical to the existing pad_input() plus set_zero_diagonal() path while removing one intermediate allocation and one chunk_size-by-context_size host copy per prefill chunk.

The optimization targets CPU-side staging overhead, not attention math. It composes with GGML_OPENVINO_NPU_KV_SLICE: KV slice reduces the mask dimensions, while FAST_MASK reduces the work needed to construct whatever mask shape remains. The trade-off is a second mask construction path to maintain, with limited benefit for very small prompts or contexts.

* openvino: make NPU requant layout configurable

Add GGML_OPENVINO_NPU_REQUANT_POLICY as the NPU-side selector for the default weight requantization layout. The existing NPU behavior remains the default: unset or group-128 maps quantized weights to Q4_0_128, while channel-wise selects the experimental Q4_0_C layout. Unknown policy values fail fast during model loading instead of silently compiling a graph with unintended weight packing.

The optimization target is NPU weight bandwidth and compressed-weight code generation. Q4_0_128 keeps weights in a 4-bit symmetric layout with one scale per 128 values, reducing both payload bytes and scale metadata versus smaller groups while preserving a more conservative quantization granularity than channel-wise. Q4_0_C uses one scale over the innermost dimension, which can further shrink metadata and simplify dequant broadcast, but may cost accuracy because the quantization block is much larger.

Include the selected NPU requant policy in the OpenVINO model cache discriminator. The same cgraph compiled with group-128 and channel-wise has different weight constants and dequant shapes, so reusing a cached decoder across policies would be incorrect. This keeps cache reuse stable within a policy while forcing recompilation when the layout policy changes.

Trade-off: policy selection gives a performance/accuracy knob for NPU experiments, but all options still pay the load/compile-time requantization cost. The default remains group-128 for a safer balance of bandwidth, metadata, and accuracy; channel-wise is intentionally documented as experimental.

* openvino: centralize NPU plugin config overrides

Add GGML_OPENVINO_NPU_CONFIG as the recommended catch-all override for OpenVINO NPU plugin and compiler properties. The value is parsed as comma-separated KEY=VALUE pairs and applied after the built-in defaults and named NPU aliases, so advanced users can override any exposed property without adding a new llama.cpp environment variable for every plugin tuning knob.

Keep the common knobs as named aliases for discoverability and compatibility: GGML_OPENVINO_NPU_COMPILER_TYPE maps to NPU_COMPILER_TYPE, GGML_OPENVINO_NPUW_FUNCALL_FOR_ALL maps to NPUW_FUNCALL_FOR_ALL, GGML_OPENVINO_NPUW_UNFOLD_IREQS maps to NPUW_UNFOLD_IREQS, and GGML_OPENVINO_COMPILATION_NUM_THREADS maps to COMPILATION_NUM_THREADS. GGML_OPENVINO_NPU_COMPILE_CONFIG is retained as a legacy alias for NPU_COMPILATION_MODE_PARAMS, while the documentation now recommends expressing it through GGML_OPENVINO_NPU_CONFIG.

This keeps the public tuning surface compact while preserving high-signal aliases for settings with known performance or stability implications. NPU_COMPILER_TYPE can select the driver compiler, which currently generates faster prefill kernels on the tested NPU stack. NPUW_FUNCALL_FOR_ALL and NPUW_UNFOLD_IREQS expose NPUW graph lowering trade-offs around dispatch overhead, memory use, and long-context stability. COMPILATION_NUM_THREADS gives a way to cap compiler worker parallelism to reduce peak host memory during large graph compilation.

Trade-off: a free-form KEY=VALUE string is less type-safe than one env var per option and invalid plugin properties will only be diagnosed by the OpenVINO stack. Applying it last is intentional: it gives one escape hatch for bisecting driver/compiler behavior and for testing new plugin options without rebuilding llama.cpp, while the named aliases continue to document the most important performance knobs.

* openvino: add scatter-elements KV update option

Add GGML_OPENVINO_KV_SCATTER_ELEMENTS as an opt-in lowering for non-stateful single-row KV cache writes. When the flag is enabled and the set_rows indices are the simple decode case, translate_set_rows uses ScatterElementsUpdate with broadcasted rank-4 indices instead of ScatterUpdate.

The motivation is NPU decode performance. The default ScatterUpdate path updates one KV row but the NPU plugin does not perform that update in place; it effectively copies the full destination tensor to write a small slice. As context length grows, that full-tensor movement dominates the per-token KV update cost. ScatterElementsUpdate expresses the same single-row write at element granularity and is measurably cheaper for this shape.

Keep the optimization guarded. Stateful execution keeps its existing concat/update path, and multidimensional indices already use the broader ScatterElementsUpdate lowering. The new flag only changes the common non-stateful decode write, so the default behavior and graph shape remain unchanged unless the user opts in.

Trade-off: ScatterElementsUpdate introduces extra ShapeOf, Reshape, and Broadcast nodes to build indices matching the update tensor. That overhead is worthwhile only when it avoids the NPU plugin's full-destination ScatterUpdate copy, so the path is controlled by an environment variable rather than becoming the unconditional lowering.

* openvino: gather token embeddings from compressed rows

Add NPU token-embedding controls for models whose token_embd.weight would otherwise be widened from Q6_K to FP16. GGML_OPENVINO_TOKEN_EMBD_I8 keeps the embedding table on the normal int8 compressed-weight path, while GGML_OPENVINO_TOKEN_EMBD_I4 requantizes it to the group-128 int4 layout used by the NPU weight path.

The performance target is tied embeddings and large vocabulary tables. When token_embd.weight is also consumed by the lm_head matmul, a normal Gather from the dequantized table gives the dequantization subgraph a second consumer and can force the full table to be materialized. The new gather_compressed_rows helper recognizes the dequantization chain emitted for int4/int8 constants, gathers only the selected packed rows from the leaf constants, and rebuilds the convert/subtract/multiply/reshape chain on those rows.

This reduces memory pressure and avoids dequantizing or materializing a full embedding table when inference only needs the token rows for the current input. The helper is deliberately conservative: it only rewrites static 2D tables above a size threshold, walks only the known compressed-weight node patterns, limits recursion depth, and falls back to the original Gather when the subgraph is not recognized.

Trade-off: int8 and especially int4 token embeddings can change accuracy and may not be faster for small untied embedding tables, because the rewrite adds extra Gather/Reshape nodes and only pays off when it avoids a large full-table dequantization. The default behavior is unchanged; the NPU-specific I8/I4 paths are opt-in through environment variables.

* openvino: stream u4 requantization chunks

Extend the GGML_OPENVINO_REDUCE_COMPILE_MEM streaming requantization path to Q4_0/u4 targets. The previous streaming path intentionally skipped u4 because Q4_0 packs two weights per byte and the zero-point tensor stores two 4-bit values per byte, so the original quantize_q4_0 helper wrote as if every call started at block zero.

Teach quantize_q4_0 to accept a destination block offset and use the absolute block index for weights, scales, and zero-point writes. Streaming requantization emits whole-row chunks in increasing order, and the target block size is required to divide the row width, so no target block straddles a chunk boundary. That keeps packed weight bytes aligned and preserves the even-block assignment before the odd-block zero-point OR for each shared zero-point byte.

With that indexing in place, reduce-compile-memory mode can dequantize a chunk of rows into the scratch buffer and immediately quantize it into the final u4 buffers instead of materializing the entire tensor as temporary F32. This extends the peak-memory reduction already used by Q8/F16 targets to the NPU group-128/channel u4 layouts used for compressed weights.

Trade-off: the u4 path is more sensitive to block alignment than Q8/F16 because of nibble packing, so streaming remains guarded by the same row-divisibility check and only runs when GGML_OPENVINO_REDUCE_COMPILE_MEM is enabled. When the flag is off, the full-materialization path and quantized output layout remain unchanged.

* openvino: fail fast on static recompile after weight release

Extend the GGML_OPENVINO_RELEASE_WEIGHTS safety check to the static graph cache-miss path. Once host weight buffers have been released, any later compile would read invalid/zeroed host pages and bake the wrong constants into a new OpenVINO model.

The dynamic path already rejected this situation. Static execution can still miss the decoder cache when graph shapes or model parameters change, so it needs the same guard before erasing cached infer requests and rebuilding prefill/decode models.

This is a correctness and diagnosability change rather than a throughput optimization. It preserves the memory-saving release-weights mode for stable graph shapes, but turns unsupported recompilation into an explicit abort instead of silently producing a corrupted compiled model.

* openvino: add experimental NPU Q4_0 keep option

Add GGML_OPENVINO_NPU_KEEP_Q4_0 as an opt-in escape hatch for NPU weight handling. When enabled for native Q4_0 tensors, the NPU requant path returns nullopt and keeps the original Q4_0 extraction instead of regrouping the tensor through the selected NPU requant policy.

The motivation is to avoid unnecessary work for weights that are already stored as 4-bit values. The default NPU policy maps quantized weights to layouts such as Q4_0_128, which can reduce scale metadata but requires round-tripping Q4_0 through the requantization path. Keeping Q4_0 native can save that conversion work and preserve the original group-32 packing.

Keep the option experimental and disabled by default. Current NPU driver/plugin stacks can compile the resulting group-32 graph, but inference may hang or fail with ZE_RESULT_ERROR_DEVICE_LOST. The default behavior therefore continues to use the safer NPU requant policy; this flag is mainly useful for debugging driver behavior, measuring the true cost of regrouping, and testing future NPU stacks.

Trade-off: native Q4_0 avoids regrouping overhead and keeps the original quantization blocks, but it gives up the more regular group-128 layout expected by the default NPU compressed-weight path and may be unstable on current hardware/software combinations.

* openvino: unfold NPUW only for decode graph

Compile the static prefill and decode models with separate OpenVINO config maps so the decode graph can opt into NPUW_UNFOLD_IREQS without forcing the same lowering on prefill. The build_static_model helper now receives the config map explicitly, and the decode path gets a copy of the base config with NPUW_UNFOLD_IREQS=YES when the user has not already set it.

The performance target is NPU token generation. Decode executes a long stream of single-token infers, where folded NPUW function-call dispatch overhead becomes visible in every token. Unfolding the NPUW calls into separate infer requests removes that repeated dispatch cost and measured about +20% token-generation throughput on the tested stack, matching the behavior seen in OpenVINO GenAI.

Prefill keeps the original folded config. Prompt processing is a batched workload dominated by larger matmuls, and the folded form is faster there while also using less memory. Splitting the config lets prefill and decode use the lowering that matches their workload shape instead of treating NPUW_UNFOLD_IREQS as a single global compromise.

Trade-off: unfolding can increase memory and request-management overhead, so the automatic default is limited to the NPU decode model. If a user explicitly provides NPUW_UNFOLD_IREQS through the named env alias or GGML_OPENVINO_NPU_CONFIG, that value is respected because the decode override is only applied when the property is absent.

* openvino: tidy NPU tuning helper ordering

Collect the remaining small cleanups from the NPU tuning series. This reorders the NPU KV-slice documentation and environment-variable cache entry next to the other NPU opt-in toggles, adds a short header comment for the centralized NPU helper declarations, and makes the static prefill attention-size assignment in the decoder a little more explicit.

There is no intended behavior change in this commit. The purpose is to make the final source layout match the split commits that introduced the NPU tuning knobs, so future readers can find the related flags and helpers in one place without mixing that mechanical cleanup into the functional performance commits.

* openvino: normalize NPU compile env overrides

Move the default NPU_COMPILER_TYPE=DRIVER setting into the NPU compile_config initializer so the default NPU plugin options are declared in one place.

Route GGML_OPENVINO_NPU_COMPILE_CONFIG and GGML_OPENVINO_NPU_COMPILER_TYPE through the same set_compile_option_from_env helper used by the other named NPU compile option aliases. This keeps the legacy NPU_COMPILE_CONFIG mapping to NPU_COMPILATION_MODE_PARAMS intact while making all optional env-to-compile_config overrides follow the same non-empty-value rule.

GGML_OPENVINO_NPU_CONFIG is still parsed last, so explicit comma-separated KEY=VALUE overrides continue to have the highest precedence over both defaults and named aliases.

* openvino: stream NPU weight requant by default to cut compile peak

Default the streaming requantization path on for NPU so the compile-time F32 dequant transient (~1-2 GB for token_embd) is capped without an env flag. Output is byte-identical to full materialization; GGML_OPENVINO_REDUCE_COMPILE_MEM / MEMORY_OPTIMIZE still override (set to 0 to force off). Kept the g_nonov_weight_cache opt-in to avoid holding token_embd resident (which regressed decode steady memory). pp1024_d0: PrefillPeakPrivWS 5974->4664 MiB (-22%), decode steady and pp/tg throughput unchanged.

* openvino: add self-contained NPU model cache

Add an opt-in embedded-weight cache mode for the static NPU backend. GGML_OPENVINO_SELF_CONTAINED_BLOB=1 exports separate prefill and decode compiled models with ENABLE_WEIGHTLESS=false, uses isolated NPUW weight banks, validates raw GGUF weight manifests, binds imported ports by deterministic frontend order, and serializes the large NPUW compile/export/import operations.

Preserve raw model bytes during cold cache construction by requantizing into separately owned OV tensors instead of rewriting the backend model buffer. This keeps later graph fingerprints and manifests stable when llama-bench creates independent prompt and generation contexts.

Add strict GGML_OPENVINO_SELF_CONTAINED_MMAP=1 warm-import mode. The backend exposes a non-owning buffer_from_host_ptr wrapper over the GGUF mmap, uses raw tensor allocation sizes, avoids allocating/copying the private OpenVINO model-weight buffer, and stages the few mmap-backed runtime constants into small importable NPU host tensors. Invalid or missing cache entries fail clearly instead of trying to requantize into read-only mapped storage.

The normal path and existing weightless MODEL_PTR cache remain unchanged when the new flags are unset. Verified with phi-4-mini: chunk512 pp3072/tg512 cold 1123.10/27.60 t/s and warm mmap 1124.72/27.16 t/s; warm mmap removes roughly 3.9 GiB peak commit in the measured prefill workload. Self-contained blobs consume about 8.34 GiB for the separate pp and tg contexts.

* openvino: add lazy import for self-contained mmap cache

Add GGML_OPENVINO_SELF_CONTAINED_LAZY_IMPORT to make the low-working-set strict mmap path opt in instead of changing the default warm mmap behavior.

When the flag is set with SELF_CONTAINED_BLOB=1 and SELF_CONTAINED_MMAP=1, the static NPU path imports only the current phase blob: prefill imports the prefill blob, decode imports the decode blob later. This avoids the pair-shaped shared compiled-model cache for the lazy path and caches only the current phase infer request.

The regular warm mmap path remains eager and continues to prepare both prefill and decode compiled models. Non-mmap self-contained builds remain serial, and the regular non-self-contained path keeps the existing parallel build behavior.

Also register the new environment variable, document the memory/latency tradeoff, and add profiling-only import timing around model-cache imports.

Validated with ReleaseOV build: cmake --build build/ReleaseOV --target llama-bench llama-simple --config Release. Lazy warm mmap run measured PrefillPeakWS around 5759 MiB versus the eager warm mmap path around 7897 MiB.

* openvino: release GGUF mmap resident pages before blob import

Add GGML_OPENVINO_SELF_CONTAINED_RELEASE_MMAP_PAGES as an explicit opt-in for self-contained compiled-model cache hits.

Before importing a valid self-contained blob, recursively walk the graph to find unique external GGUF mmap-backed weight buffers and drop their resident pages while preserving the mapping address and tensor pointers.

On Linux use page-aligned madvise(MADV_DONTNEED). On Windows, where there is no reliable per-range mapped-file equivalent, call VirtualUnlock for the identified range and trim the current process working set with SetProcessWorkingSetSize(-1, -1). Pages required later can fault back on demand.

Keep the behavior disabled by default and document the physical-memory versus commit-memory tradeoff. Profiling reports the submitted GGUF range and model import duration.

Validated with ReleaseOV llama-bench/llama-simple build and pp3072+tg512 warm mmap lazy import. With page release enabled, PrefillPeakWS dropped from about 5758 MiB to 3053 MiB while throughput remained comparable; PeakCommit stayed about 3331 MiB.

* openvino: add GGML_OPENVINO_NPU_SHARED_BANK to share prefill/decode weights bank

When set, prefill and decode self-contained blobs use one shared NPUW_WEIGHTS_BANK name (ggml-sc-shared-<pfp>-<dfp>) instead of separate per-phase banks. Since both decoders are built from the same model_weights map, NPUW dedups them into a single in-process bank; on warm import the identical (uid,tensor) pairs are content-verified and reused (patched NPUW weights_bank). Cuts resident compiled-weight footprint ~2GB (pp3072 warm: In-Use 7954->5871 MiB, WS 7891->5765) with prefill throughput unchanged.

* openvino: drop SELF_CONTAINED_LAZY_IMPORT and RELEASE_MMAP_PAGES

The shared weights bank (GGML_OPENVINO_NPU_SHARED_BANK) supersedes both: it deduplicates the prefill/decode compiled-weight copy directly, so lazy per-phase import and GGUF-page release no longer reduce steady memory. Remove the two env vars, the self-contained lazy-import branches (defaulting to the eager path that imports both blobs), the release_external_weight_pages helper, and the now-unused ggml_openvino_buffer_is_external / _release_external_pages buffer APIs.

* openvino: demand-page large NPU compute buffer in strict warm-mmap mode

In strict self-contained warm-mmap mode weights come via buffer_from_host_ptr,
so every alloc_buffer is graph scratch/KV. The large prefill compute buffer is
never touched on the host by the on-device fused OV model, yet the NPU L0 host
tensor (create_host_tensor + memset) makes all ~2.4 GiB resident. For buffers
>= 1 GiB in this mode, reserve demand-zero pages (no L0, no memset) so they stay
non-resident; KV/small buffers keep the L0 host tensor for zero-copy import.

pp3072 warm mmap + shared bank: prefill peak WS 5765 -> 3314 MiB (-43%),
In-Use delta ~5940 -> ~3232 MiB; prefill 1256.7 -> 1252.9 t/s and decode
28.27 -> 27.96 t/s (both flat). Commit charge unchanged (demand-paged).

* perf improvement success

* openvino: size static NPU sliding-window masks to the window cache

The static graph built the sliding-window KQ mask at the full context length,
while the windowed layers bind their (smaller) window KV cache. With gemma-4 at
-c 4096 -ub 1024 the window cache is 1536 cells, so SDPA shape inference failed
([1,1,1024,4096] mask vs [1,8,1536,256] K/V) and every prompt errored out.

With GGML_OPENVINO_NPU_KV_SLICE the prefill mask was meant to use
attention_size_swa, but the check matched "swa" in the ggml tensor name, and
llama.cpp names both masks "attn_inp_kq_mask"; the windowed layers got the
full-attention sliced length instead ([1,1,1024,256] vs [1,8,1024,256]).

Identify the sliding-window mask by tensor identity (is_swa_mask) and size it to
ctx_per_seq_swa, both in the graph input shape and in the runtime mask padding.

* ggml-openvino: create the model cache dir under a Windows drive root

make_dirs() mkdir'ed every path prefix, including the drive root "D:", which
fails with EACCES (not EEXIST) on Windows. Any absolute cache path therefore
logged "could not create model cache dir ... caching disabled" and silently
ran without GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR. Skip the drive root.

* ggml-openvino: split device-specific passes and NPU config out of shared code

Mechanical refactor, no behavior change (step 1 of the NPU/MoE architecture plan):

* openvino/pipelines/: TranslateSession::apply_transformations() now registers
  common passes, then the device's passes, then the execution-mode passes
  (stateful KV, static shapes), selected by a per-graph GraphTraits. The GPU MoE
  fusions and the GPU eltwise-rank workaround move to pipelines/gpu.cpp; CPU and
  NPU pipelines are empty placeholders for device lowerings. Pass order is
  unchanged.
* npu/npu_config.*: the NPU (NPUW) compile config built at backend init, the
  NPUW strip for cacheless graphs and the per-graph static prefill/decode
  variants (HFA prefill, decode NPUW_UNFOLD_IREQS) move out of
  ggml-openvino-extra.cpp and utils.cpp.
* npu/npu_quant_policy.*: the NPU requantization rules (requant policy,
  token_embd Q6_K, keep-Q4_0) move out of ggml_openvino_get_requant_type().

Verified identical output before/after on OV-CPU, GPU and NPU (gemma-4 E4B).

* ggml-openvino: one KV-slice switch for dynamic and static graphs

KV-cache slicing (bind the KV cache, and size the KQ masks, at the attended
length instead of the full context) existed twice: on by default for dynamic
graphs (GGML_OPENVINO_DISABLE_KV_SLICE to turn it off) and opt-in for static
prefill (GGML_OPENVINO_NPU_KV_SLICE). The static mode is not NPU-specific; it
applies to any static graph, including CPU/GPU with GGML_OPENVINO_FORCE_STATIC.

Describe it as one mechanism with two modes behind GGML_OPENVINO_KV_SLICE
(unset = each mode's default, 0 = off, 1 = on) and two predicates,
ggml_openvino_{dynamic,static}_kv_slice_enabled(), used by the KV binding,
the static graph shapes, the mask padding and the cache keys. The two old
variables remain as deprecated aliases.

* ggml-openvino: initialize all ModelExtraInputInfo fields for the HFA inputs

dev_backend_openvino added shape_source/shape_axis to ModelExtraInputInfo; the HFA past-KV and mask
initializers still listed four fields, which GCC rejects under -Werror=missing-field-initializers.

* ggml-openvino: make new NPU defaults opt-in and fix static model conversion

- Convert the static prefill and decode models one after the other, both
  before either is compiled. The decoders share the weight nodes;
  converting on two threads crashed intermittently inside libopenvino,
  and converting decode after the prefill compile gave wrong output on
  Bonsai-8B.
- Compile prefill and decode in parallel only with
  GGML_OPENVINO_NPU_PARALLEL_COMPILE=1, since it raises the host memory
  peak.
- Unfold NPUW function calls on the decode graph only with
  GGML_OPENVINO_NPU_DECODE_UNFOLD=1. The unfold default gave wrong output
  on gemma-4-E4B and slowed decode on several small models.
- Stream NPU weight requantization only with
  GGML_OPENVINO_REDUCE_COMPILE_MEM or GGML_OPENVINO_MEMORY_OPTIMIZE, as on
  the other devices.
- Fix a -Werror=missing-field-initializers build error in utils.cpp.
- Declare ggml_openvino_get_npu_requant_type() only in
  npu/npu_quant_policy.h.
- Docs: document the new switches, deprecate stateful execution, show
  stateless-only results in the validated models table, and update the
  validation system.

* ggml-openvino: honour the -1 skip id in MUL_MAT_ID

Upstream made an id of -1 in MUL_MAT_ID skip the slot and zero its row of
the result. The translator passed -1 straight to GatherMatmul or Gather,
so 169 of the 743 MUL_MAT_ID op tests on CPU (all skip=1) failed.

A skipped slot now looks up a valid expert, and the result is multiplied
by a 0/1 mask of the kept slots. For the packed MXFP4 path (Gather and
MatMul), clamping the id to expert 0 is enough. GatherMatmul computes a
wrong row when a token names the same expert twice, so the general path
gives each skipped slot an expert that no other slot of the token uses.

TopK indices are never -1, and model graphs take the ids from the
router's TopK. Those graphs skip the handling and stay unchanged, so the
GPU MoE fusion still matches them.

* ggml-openvino: Gather rows before dequantizing a quantized embedding table

1. Optimize gather to fetch row before dequant
2. add NPU token_embd quantization option

* ggml-openvino: select the NPU token_embd type with GGML_OPENVINO_NPU_TOKEN_EMBD_QUANT

GET_ROWS had two rewrites that gather the rows of a quantized table before
dequantizing them: gather_compressed_rows, which runs on every device, and
gather_before_dequant, enabled on NPU by GGML_OPENVINO_GATHER_BEFORE_DEQUANT.
Both match the int4 and int8 chains built by make_int4_weights() and
make_int8_weights(). Keep gather_compressed_rows and drop the second rewrite
and its switch.

The NPU type of a Q6_K token_embd.weight was set by two booleans,
GGML_OPENVINO_TOKEN_EMBD_I8 and GGML_OPENVINO_TOKEN_EMBD_I4, and by
GGML_OPENVINO_NPU_TOKEN_EMBD_QUANT. Keep only the string variable, with the
values f16 (default), q8_0_c, q4_0_128 and q4_0_c, so the int8 and group-128
int4 settings stay available. An unknown value aborts, as for
GGML_OPENVINO_NPU_REQUANT_POLICY, instead of silently using f16. The variable
replaces the two booleans in the NPU compiled-model cache key.

* align PAD pads to input rank in stateful mode

* Gate the changes under is_stateful check to address review comment

* ggml-openvino: lift Q tensors for non-RoPE layers to rank 4 in stateful mode

* ggml-openvino: unify KV slicing under GGML_OPENVINO_KV_SLICE, on by default

Replace ggml_openvino_dynamic_kv_slice_enabled() and ggml_openvino_static_kv_slice_enabled() with a single ggml_openvino_kv_slice_enabled(). KV slicing binds the KV cache, and sizes the KQ masks, at the attended length instead of the full context.

GGML_OPENVINO_KV_SLICE now defaults to 1 on CPU, GPU and NPU; set it to 0 to disable. Dynamic graphs (CPU/GPU) bind a view over the cache without recompiling. Static graphs (NPU or GGML_OPENVINO_FORCE_STATIC) compile prefill at the aligned attended length and recompile when it changes; decode keeps its full-context graph.

Remove the deprecated aliases GGML_OPENVINO_DISABLE_KV_SLICE and GGML_OPENVINO_NPU_KV_SLICE from the env var list, the model-cache settings, docs/backend/OPENVINO.md and the backend README.

Behavior change: static KV slicing is now on by default on NPU, so prefill compiles one graph per distinct n_kv (multiples of 256).

* ggml-openvino: keep stateful execution with -fa off

Keep the V cache token-major in stateful mode instead of falling back to
stateless execution. SET_ROWS appends to the state, and PERMUTE reads it
back in the layout attention needs, so MUL_MAT does not transpose it.
Dropping a reshape in grouped attention lets OpenVINO fuse it into SDPA.

For padded V caches, slice after the transpose so the cache is read in
place, without a copy that is recompiled for every KV length.

This removes the stateless fallback for -fa off.

* ggml-openvino: Add the stateful check to make the gate safer

* ggml-openvino: support gemma-2 soft-capped attention

Gemma-2 caps its attention scores as softcap * tanh(scores / softcap)
before the softmax. FLASH_ATTN_EXT now applies the cap on the manual
attention path, as SDPA has no soft-capping, and the decoder looks
through the SCALE, TANH, SCALE chain of -fa off when it matches
attention.

One helper finds the K cache of an attention for both the
sliding-window pre-pass and the main pass. The pre-pass copy did not
see through the chain, so the sliding-window layers used the
full-attention sizes once the context exceeded n_swa + n_ubatch.

Soft-capped attention keeps the flat KV cache layout: with the 4-D
layout the GPU plugin fails in its oneDNN path.

* ggml-openvino: enable NPU Level Zero host tensors by default

Back NPU buffers with Level Zero host memory unless GGML_OPENVINO_NPU_L0_HOST_TENSORS=0.
The plugin imports these buffers without a host-to-device copy each infer; on phi-4-mini
Q4_0 (pp3072+tg512) decode goes from 7.2 to 10.6-10.8 t/s. Allocation failures still fall
back to plain host memory.

* ggml-openvino: enable NPU fast prefill mask by default

GGML_OPENVINO_NPU_FAST_MASK now defaults to 1; set it to 0 to fall back to pad_input() + set_zero_diagonal().

The fast path builds each static prefill mask chunk directly in the OpenVINO input tensor (row memcpy of the valid prefix, -inf padding, zero diagonal), avoiding a temporary chunk_size x context_size vector and a full-tensor copy per chunk. The result is bit-identical to the fallback path.

* ggml-openvino: add opt-in exact zero points for asymmetric weights

Asymmetric quantized weights (Q4_K, Q4_1, Q5_1, and non-requantized
Q5_K) represent each group as `w = d * q - m`. The backend currently
converts these to u4/u8 weights with an f16 scale and an integer zero
point `round(m / d)`, which can shift all weights in a group by up to
half a quantization step.

Add `GGML_OPENVINO_EXACT_ZERO_POINT=1` to preserve the zero point in f16
instead, using the same path already used for MoE expert weights.

This is disabled by default because, with current OpenVINO releases, an
f16 zero point moves many GPU fully-connected layers off oneDNN,
resulting in slower prompt processing.

* ggml-openvino: rename npu/ to device/ and openvino/pipelines/ to openvino/pass_registry/

npu/ holds device-specific backend behaviour (NPU compile config and
requant policy), and pipelines/ only registers passes per device; the new
names describe what the folders contain. No functional change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: document NPU self-contained model cache and shared weights bank

Add GGML_OPENVINO_SELF_CONTAINED_BLOB, GGML_OPENVINO_SELF_CONTAINED_MMAP and GGML_OPENVINO_NPU_SHARED_BANK to the OpenVINO environment variable table, and a note describing the export/import workflow and the requirement to keep NPU settings identical between runs.

* ggml-openvino: remove experimental GGML_OPENVINO_NPU_KEEP_Q4_0

Keeping native Q4_0 (group-32) on NPU compiles but hangs the device at inference (ZE_RESULT_ERROR_DEVICE_LOST), so the option was never usable. NPU Q4_0 weights now always go through GGML_OPENVINO_NPU_REQUANT_POLICY. Drop the env var, the requant bypass, its self-contained blob fingerprint term and the docs entry.

* ggml-openvino: Update PERMUTE and CONCAT dynamic dim axis selection, Keep fp32 precision for sum-of-squares

All changes of https://github.com/ravi9/llama.cpp/pull/358 as one commit. The PR branch had
dev_backend_openvino merged into it, so its commits could not be replayed one by one when
dev_backend_openvino was rebased onto upstream master.

Commits of the PR:
- Update fix to just maintain fp32 precision instead of modifying the graph
- Limit the fp32 precision spreading in the graph by fencing it with Convert. Prevents perf degrade.
- Update OPENVINO.md
- Update flash_attn_ext.cpp
- Update utils.cpp

* enable flash_attn with sink input

* ggml-openvino: fix FLASH_ATTN_EXT input detection and sink scale

translate_flash_attn_ext read get_input_names()[3] and [4] on every
FLASH_ATTN_EXT node, also on nodes with only 3 or 4 inputs. The
out-of-bounds read crashed test-thread-safety and
test-speculative-rejection-sampling. The inputs are q, k, v, then the
optional mask and sinks, and supports_op rejects sinks without a mask,
so the input count tells which inputs are present.

With sinks, the scale was replaced with 1.0. llama.cpp passes the
attention scale in op_params[0] and adds the sinks separately, so keep
op_params[0]. The "scale" input branch is removed: ggml's src[4] is
only ever the sinks.

Only SDPA applies the sinks. The manual GQA path and the NPU HFA
prefill path drop them, so attention with sinks now always takes the
SDPA path. SDPA has no logit soft-capping, so supports_op rejects sinks
together with soft-capping.

* ggml-openvino: accept flash attention on layers without RoPE

On layers without RoPE (SmolLM3: every 4th layer), Q reaches
FLASH_ATTN_EXT through PERMUTE -> VIEW -> RESHAPE. The support check
rejected this pattern, causing these layers to fall back to CPU and the
split graph to fail with -fa on/auto.

* ggml-openvino: keep only weight-divisor DIVs on CPU

The GPU DIV guard for per-channel scale vectors (`MPT`'s `act_scales`) currently checks only the divisor's shape. During single-token Qwen2-MoE decode, the shared-expert gate `silu(x) / x` has a `[1,1,1,1]` divisor that matches this shape, causing its DIVs to be kept on the CPU.

As a result, every decode step incurred CPU execution and Qwen1.5-MoE decoded at only ~2 tok/s.

Apply the guard only when the divisor is a model weight, so runtime tensors such as the Qwen2-MoE gate divisor can remain on the GPU.

* ggml-openvino: Qwen MTP and multi-seq optimization with new OV ops

All changes of https://github.com/ravi9/llama.cpp/pull/370 as one commit. The PR branch had
dev_backend_openvino merged into it, so its commits could not be replayed one by one when
dev_backend_openvino was rebased onto upstream master.

Commits of the PR:
- openvino : support Qwen MTP decoding
- openvino: Optimize TopK
- openvino: remove single-slot recurrent cache shortcuts
- openvino: fuse generic recurrent caches and remove shape waits
- openvino: include dense Q8 weights in all INT4 requantization
- openvino: use two GDN outputs for snapshots
- docs: explain native OpenVINO recurrent cache contracts
- docs: clarify Qwen3.5 purpose of native OpenVINO cache ops
- llama: skip unallocated hybrid graph inputs
- quantize: preserve Qwen hybrid layer metadata when pruning
- openvino: emit fused recurrent computation and rotate device caches
- openvino: limit KV slicing to attention caches and share compiled blobs
- tools: add natural-text server benchmark and OpenVINO profile summary
- docs: record fused-cache branch split and UD validation
- server-bench: preserve native token counts and save request timings
- openvino: document MTP regression and host USM alias recovery
- openvino: record large-context allocation diagnosis and default-batch validation
- server-bench: count buffered native tokens and reject stream errors
- openvino: document UD split and unified server benchmarks
- openvino: record Vulkan long-context baseline and GPU-fit controls
- openvino: finalize UD benchmark report and MTP comparison
- openvino: use stable in-place recurrent caches and optional separate convolution
- openvino: accept matching empty axes in DIV repeat translation
- openvino: remove the obsolete cache rotation environment setting
- openvino: validate imported KV results in their four-dimensional layout
- docs: record the bounded in-place KV and convolution benchmark rerun
- docs: complete default-batch OV unified-KV 32K benchmarks
- openvino: remove the separate recurrent convolution path
- openvino: handle zero-logits batches and changing sampler sequences
- docs: compare default and large Qwen3.6 benchmark batches
- docs: use the historical Vulkan default-batch reference consistently
- server-bench: use an integer histogram bin count
- benchmarks: summarize current OV unified MMLU timings
- docs: record cache-only unified KV MMLU results
- openvino: preserve the TopK index port in GET_ROWS
- openvino: reuse the MTP sampling model across active slots
- openvino: expose selection flags and offsets in graph dumps
- docs: consolidate Qwen OpenVINO defaults and performance evidence
- docs: retain continuous-request MTP benchmark results
- docs: compare stable MTP with the non-MTP serving baseline
- openvino: isolate GDN snapshots and skip naive cache fusion
- ggml: document Qwen3.6 GenAI benchmarks
- ggml-openvino: keep PR changes inside the backend folder

* ggml-openvino: drop the extra semicolons after OPENVINO_OP

OPENVINO_OP expands to member function definitions that end in a closing
brace, so a semicolon after it is an empty declaration inside the class.
GCC 15.2 reports it with -Wextra-semi, which ggml enables for GCC 8.1 and
newer, so a build with LLAMA_FATAL_WARNINGS=ON (the default in ci/run.sh)
fails in gated_delta_net.hpp. The CI compilers do not report it.

recurrent_cache_update.hpp has the same pattern twice. It is compiled only
when the OpenVINO package provides the recurrent cache-update ops.

* ggml-openvino: make q8_0_c the default NPU token_embd requant type

Default of `GGML_OPENVINO_NPU_TOKEN_EMBD_QUANT` changed from `f16` to `q8_0_c`.
Applies only to NPU and only to a Q6_K `token_embd.weight`; other devices and
quantization types are unchanged.

* ggml-openvino: do not treat the recurrent state gather as out_ids

The output_len scan now runs for every model and takes the first GET_ROWS
whose index is a graph input. On Qwen3.5 that is the recurrent state gather
of build_rs, which indexes the inp->s_copy leaf directly, so output_len
became 1. In static (NPU) mode the prompt model then returned one logits
row per chunk, and llama-perplexity read stale rows (PPL about 240000 on
Qwen3.5-2B). The same predicate also gave the s_copy input the out_ids
shape and data in static mode.

is_output_idx now skips GET_ROWS that read a recurrent state cache. The old
predicate stays as is_get_rows_input_idx for the dynamic-dim bookkeeping,
so the s_copy leaf keeps its dynamic length there.

* ggml-openvino: prefer the size-preserving dim when a RESHAPE repeats a stride

The RESHAPE dynamic-dim rule matches the source's dynamic stride against
the result's strides. A size-1 dim repeats the next dim's stride: Gemma 4
E2B has one KV head, so [head_size, n_tokens] -> [head_size, 1, n_tokens]
has the same stride on dims 1 and 2. The first match picked the size-1 head
dim, so the K/V views lost their dynamic token dim and became static
reshapes with the first graph's token count. Every later graph with another
token count failed.

Prefer the stride match that keeps the dynamic size, and fall back to the
first stride match for reshapes that change it.

* ggml-openvino: correct the RESHAPE dynamic-dim comment for size-1 dims

The comment above the RESHAPE stride match in compute_node_dynamic_dims()
still said the match is unambiguous because result strides are strictly
monotone. A size-1 dim has the same stride as the dim after it
([head_dim, T] -> [head_dim, 1, T] with one KV head), so it is not.

Explain the size-1 case in the block comment, drop the duplicate comment
inside the loop, and name the source dim's size once.

* examples: add llama-chat-replay

Replays JSONL prompts like OpenVINO GenAI chat.py (-pf, -i chat history, greedy, -o, -pc metrics). The first prompt is prefilled plus one decode step during load (Warmup time) and the memory is cleared, so compile/import is counted in Load time as in GenAI.

* ggml-openvino: make the NPU lazy host threshold configurable, default 2 GiB

With a large context the KV cache (~1 GiB at ctx 8K) crossed the 1 GiB threshold, lost its Level Zero host tensor in strict warm-mmap mode and made decode copy the KV every token. Default to 2 GiB so only the larger prefill compute buffer stays demand-zero, and allow overriding it with GGML_OPENVINO_LAZY_HOST_THRESHOLD_MB.

* ggml-openvino: fix NPU HFA prefill with ordinal-suffixed KV input names

KV cache inputs are now named with a graph-ordinal suffix (cache_k_l0#l12), but flash_attn_ext looks the HFA past input up as ggml_hfa_past_cache_k_l<layer>, so GGML_OPENVINO_NPU_HFA_PREFILL failed with 'ggml_hfa_past_cache_k_l0 not found in tensor map'. Name the past inputs after the bare cache name and resolve the source cache by that name when binding.

* ggml-openvino: keep the NPU prefill model on cache hits after decode

With KV slicing, decode calls overwrite the shared decoder's attention size with the full context, so the next prefill compared that against its sliced size, missed the cache and rebuilt the decoder and both infer requests (plus a cold first chunk). Remember the attention sizes the prefill model was built with and compare against those.

* ggml-openvino: bind NPU HFA past KV zero-copy in the cache layout

The HFA prefill past K/V inputs were [1, heads, past, head_size] copies of the cache, relaid out per chunk on the host and allocated per decoder (256 MiB on phi-4-mini at 3k). Declare them in the ggml cache layout [1, past, heads, head_size], bind the first past rows of the cache directly and transpose in the graph; NPUW HFA reads the tiles from the BSHD past (needs the matching NPUW HFA support). HFA input ABI v4.

* ggml-openvino: size the NPU decode KV window by GGML_OPENVINO_NPU_DECODE_BUCKET

Static decode attended the whole context, so every token read and wrote the full KV cache. With a bucket base set, decode attends the smallest base*2^k rows covering the next position (clamped to ctx): the KV inputs and masks are sliced to the window, the compiled pair's decode graph is built at the window the prefill reaches, and other windows are built or imported from the self-contained cache on first use with the pair's settings and bank. With self-contained blobs, keep the token_embd weight node for the later window models so they reuse the pair's bank entry instead of embedding a second vocab copy. The static build/convert lambdas become functions shared with the window path.

* ggml-openvino: allocate the NPU KV cache on demand with GGML_OPENVINO_NPU_KV_ON_DEMAND

The NPU KV buffer was committed for the whole context up front. With the option set, the KV buffer is swapped for demand-zero memory at its first KV tensor and every KV tensor gets its own storage in a registry, grown in base*2^k rows (base = decode bucket or 1024) as the static graphs reach further. Host accessors, KV inputs and outputs, sliced KV views and the HFA past bind that storage; when it moves, idle requests bound to the old storage are recreated so it is released.

With decode buckets and KV slicing the storage also takes the layout GenAI uses on NPU: K is [heads][rows][head_size] and V is [heads][head_size][rows], each bound whole to prefill and decode. SET_ROWS scatters new K rows on axis 2 and V rows on axis 3, the attention permute only cuts the window, the HFA past is the storage itself, and attention multiplies the probabilities by V^T with transpose_b instead of SDPA. A chunk-sized scatter on the innermost axis is slow on NPU, so the HFA prefill returns each chunk's V^T rows and the host copies them into the cache. Host reads, writes and fills go through a layout-aware copy, storage growth re-packs each head, and the layout is part of the model cache fingerprint.

* ggml-openvino: NPU weightless blob cache with one shared weights file

GGML_OPENVINO_NPU_WEIGHTLESS_BANK=1 (with GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR) exports weightless prefill/decode NPUW blobs plus one <prefill>-<decode>.weights.bin. Weight Constants are laid out once by data pointer and their WeightlessCacheAttribute is retagged with the real offsets. Warm import uses ov::weights_path, and NPUW dedups both models into one bank through the LazyTensor hashes, so the OpenVINO weights_bank duplicate-uid patches are not needed. Decode KV-window models resolve the shared weights through the pair's file, or get their own file when the pair was imported. Also drop the old blob and manifest before publishing a re-export: std::rename does not replace an existing file on Windows.

Based on anikulk/llama.cpp 4ab270aa9.

Co-authored-by: Anisha Kulkarni <anisha.dattatraya.kulkarni@intel.com>

* ggml-openvino: prebuild all NPU decode KV windows with GGML_OPENVINO_NPU_DECODE_PREBUILD

Decode windows were built lazily when the sequence first grew into them, which stalled generation for a compile without a cache and made a strict-mmap import fail for windows the export run never reached. GGML_OPENVINO_NPU_DECODE_PREBUILD replaces GGML_OPENVINO_NPU_DECODE_BUCKET: 0 (default) keeps decode over the whole context; 1 sizes decode by 1024*2^k KV windows and the call that builds or imports the prefill/decode pair also builds or imports the decode graphs of every other window up to the context, from the pair's graph and compute params. Requests are still created on first use. Window building is split out of the request lookup so both paths share it, and window ports are checked against the pair's decode ports.

* Revert "examples: add llama-chat-replay"

This reverts commit 4eded2c52c2df9fa1251673a523f8c7cded20d7c.

* ggml-openvino: initialize the new compiled-graph and build-args fields

ov_compiled_graph and static_build_args gained members, but three brace
initializations still listed only the old ones. GCC reports
-Wmissing-field-initializers there, and CI builds with warnings as errors.
Set the fields by name, and pass {} for weights_path, which the window
build sets afterwards.

* ggml-openvino: fix gemma-4 when only the output rows reach the KV-shared layers

With one sequence, gemma-4 now drops the rows that are not output before
its first KV-shared layer. Those layers take their positions through
GET_ROWS(inp_pos, out_ids), and the backend mishandled three things:

- The positions are reshaped from [n] to [1, n]. That matched RESHAPE
  case 1, which marks the wrong dim dynamic. Use case 3, which keeps
  the source dim as dim 1.
- ROPE shared one sin/cos table per frequency setting, so the KV-shared
  layers reused the table built from the full inp_pos. Add the position
  input to the cache key.
- compute_llm_params took input_len from the last attention op. That is
  now a KV-shared layer with only the output rows, so the static (NPU)
  path padded prefill to the wrong length. Take the largest query length.

Stateful execution still fails on the per-layer embedding view and is
not covered here.

* ggml-openvino: keep the inp_pos layout for gemma-4's gathered positions

A batch can have no output rows, for example a prompt chunk that
llama-cli processes before its last tokens. gemma-4 then reshapes an
empty position gather for the ROPEs of its KV-shared layers. The size
checks of the RESHAPE cases cannot place a dim of size 0, so the
positions got the wrong layout and the ROPE multiply failed on CPU and
GPU. A RESHAPE that feeds the ROPE positions now always takes the
inp_pos layout [1, 1, 1, n].

* ggml-openvino: enable stateless GPU execution for MLA models

The per-head MLA weights (`attn_k_b` / `attn_v_b`) are 3D quantized tensors. Their `MUL_MAT` operations were falling back to the CPU, splitting the graph at every layer and causing GPU execution to fail.
Accept these tensors in the GPU support check and address the following issues exposed by enabling GPU execution:

* Quantization: Use exact zero points only for MoE expert weights. U4 weights with F16 zero points fail in batched GPU MatMul.
* Dynamic VIEW dimensions: Select the correct dimension when a size-1 dimension shares its stride (e.g. `k_pe`), and match KV cache views by stride.
* KV cache V view: Translate the V view of the K cache as a reshape while preserving the dynamic cell axis.
* Flash Attention: Accept the MLA Q (`CONCAT`) pattern and V views derived from another view.

Limitations: The GPU plugin fails on batched compressed weights with static shapes. Keep these unsupported in op tests and `GGML_OPENVINO_FORCE_STATIC`.

* openvino : use writable recurrent cache inputs

Assisted-by: Codex

* ggml-openvino: fix decode crash for Qwen3.5 4B

When the dynamic size is 1, the code now keeps the innermost match. When the size is greater than 1, it still uses the MLA rule, so the MLA k_pe fix still works for multi-token graphs.

* ggml-openvino: keep the GDN state in the OpenVINO layout

Adds an opt-in switch, GGML_OPENVINO_GDN_NATIVE_STATE. When it's on, the GATED_DELTA_NET recurrent state stays in the layout the OpenVINO GatedDeltaNet op uses, [B, H_v, key_dim, value_dim], instead of the ggml layout [B, H_v, value_dim, key_dim]. That removes the two state transposes the translator adds on every call.

* ggml-openvino: support Ling-mini-2.0 group-routed MoE

Ling-mini-2.0 selects experts by group, but the group-selection graph
used ops that were not supported by the backend. This left 370 nodes on
the CPU and reduced decode to ~1.5 tok/s.

* Translate the batched 4D `GET_ROWS` gather used for group scores.
* Support `SET_ROWS` for compute tensors, not just KV caches.
* Preserve the dynamic token dimension in `FILL`.
* Skip the `DIV` repeat when dimensions already match.
* Use the tracked token dimension for `RESHAPE` / `GET_ROWS` op cases,
  so a graph compiled during one-row warmup also works for ubatches
  with more output rows.

* ggml-openvino: disable output staging

The backend stages small outputs (up to 4 MB) in a USM host buffer
and binds larger outputs directly to the ggml buffer. When the number
of outputs changes between calls to the same graph, as in
llama-perplexity, an output port can switch from USM host memory to
plain host memory.

The OpenVINO GPU plugin mishandles this switch for some dynamic
outputs, causing out-of-bounds writes on subsequent calls. This
results in `CL_OUT_OF_RESOURCES`. For example,
`llama-perplexity -c 2048 -ub 512` fails on the fourth ubatch with
Gemma-2, Gemma-4, Phi-3.5, Phi-4-mini, and Qwen3.5.

Disable output staging until the plugin issue is fixed, so each output
is always bound the same way. Input staging remains enabled.

* ggml-openvino: match attention masks by identity

Gemma-4 prunes non-output rows before its KV-shared layers. These
layers select the corresponding mask rows with GET_ROWS(mask, out_ids).

Mask parameters were named by their consumers, but GET_ROWS is not
recognized as a mask consumer. As a result, the sliding-window layers
selected their rows from the full-attention mask instead of the
sliding-window mask. This either caused the mask add to fail
broadcasting beyond the sliding-window cache (d1024 and above with
-ub 512) or allowed attention beyond the sliding window at shorter
contexts.

Track graph-input attention masks by identity and assign them the
same parameter name for every consumer. The SWA pre-pass now keeps
each layer's own attention mask, as the KV-shared layers read the
same cache with mask rows derived from it.

* ggml-openvino: preserve input mask names and window sizes

KV-shared layers appear at the end of the graph, and their derived
masks are not graph inputs. Processing these masks overwrote the
names and sliding-window sizes recorded from the original input
masks with empty values.

Preserve the names and window sizes of graph-input masks so the
attention dimensions continue to be derived from the original
mask shapes.

* ggml-openvino: accept empty tensors in stride checks

In Gemma-4 batches without output rows, the attention tensors of
KV-shared layers are empty. Their strides beyond the empty
dimension are zero, causing has_strides_on_element_grid() to reject
them.

This sent SOFT_MAX, the KQV MUL_MAT, and CONT to the CPU with
-fa off. It also caused the per-layer input crossing the graph
split to be bound with the wrong layout.

Accept empty tensors in the stride check. Their contents are never
read, so the remaining strides do not need to satisfy the element
grid constraints.

* ggml-openvino: classify transposed V caches using v_trans

With -fa off, the V cache is transposed. When the attended length
is the whole cache, as with a prompt that fits in one ubatch, its V
view is contiguous and was mistaken for the K cache layout.

The K layout path does not keep only the rows of the layer's own V
size. Gemma-4 allocates 512 V rows in every layer, the size of its
widest layer, and its sliding-window layers use the first 256. These
layers then attended with all 512 rows, and prompts fitting in a
single ubatch failed with a broadcast error.

Use v_trans to identify the V cache layout instead of relying on
contiguity.

* ggml-openvino: use per-layer Split in stateful execution

Stateful execution was excluded from the per-layer Split path
used for Gemma-4's per-layer embedding slices. The generic
re-slicing path therefore cut the token axis instead of the layer
axis.

Allow stateful execution to use the per-layer Split and adjust
its axis by the rank dropped during stateful graph construction.

* ggml-openvino: centralize stateful KQ mask lookup

Stateful execution rebuilds the KQ masks from positions. The
existing replacement logic applied to every mask input of
FLASH_ATTN_EXT and SOFT_MAX, including masks derived through
GET_ROWS.

For Gemma-4's KV-shared layers, this replaced the row-selected mask
with the full rebuilt mask, losing the output-row selection. As a
derived mask's name does not name the sliding window, these layers
also got the full-attention mask instead of the sliding-window one.

Add get_kq_mask_input() to replace only the actual mask parameters.
Use it for FLASH_ATTN_EXT, SOFT_MAX, and GET_ROWS, so the derived
masks select their rows from the rebuilt masks, and the lookup is no
longer duplicated.

* ggml-openvino: Removed duplicated if clause.

* fix flash_attn with sink conversion to OV SDPA

* ggml-openvino: drop the unused sink-without-mask path in FLASH_ATTN_EXT

supports_op already rejects FLASH_ATTN_EXT with sinks but no mask, so the
translator never sees a 4-input node whose fourth input is the sink; the
name check could only reach the assert after it. Restore the input-count
detection, fetch the sink where it is used, and format the sink reshape.

* ggml-openvino: fix dimensions bug in stateful recurrent-state remainder writeback

* ggml-openvino: move scalar scales before weight MatMuls on GPU and NPU

GPU and NPU run the graph in f16. Granite multiplies the attention and FFN
outputs by residual_scale right after their output projections. For some
tokens the ffn_down output is far above the f16 maximum (65504), so it
becomes inf and every later layer turns NaN: granite-4.0-350m and
granite-4.0-1b answer with repeated tokens on GPU, and llama-perplexity
reports NaN.

Add MoveScaleBeforeMatMul. It rewrites Multiply(MatMul(x, W), s), with a
scalar constant |s| < 1 and a constant W, as MatMul(Multiply(x, s), W). The
result is the same up to rounding and the MatMul output stays in range.
Register it for GPU and NPU; CPU runs in f32.

* ggml-openvino: translate convolution rollback snapshots

Assisted-by: Codex

* ggml-openvino: run GDN rollback snapshots in OpenVINO on CPU

Speculative decoding that rolls back recurrent state (MTP, EAGLE3, DFlash)
builds GATED_DELTA_NET with K = n_rs_seq + 1 state snapshots and copies them
into the recurrent cache. supports_op accepted the snapshots only on GPU, so
on CPU the GDN ran in ggml while the copy, whose destination is the OpenVINO
cache buffer, ran in OpenVINO. The copy read the packed ggml GDN output
wrongly and stored wrong states without an error: test-llama-context
rollback failed and MTP accepted no drafts.

translate_gdn_snapshots builds plain GatedDeltaNet ops on token prefixes and
does not depend on the device. Accept the snapshots on every device with
dynamic shapes and keep rejecting them only for static execution (NPU and
GGML_OPENVINO_FORCE_STATIC). Update the two doc sentences that said the
snapshots need the GPU.

* ggml-openvino: abort when GDN snapshots would run in ggml

The snapshot copy writes into the recurrent cache in the OpenVINO buffer, so
it always runs in OpenVINO. When OpenVINO rejects the GATED_DELTA_NET that
produces the snapshots (per-key gating as in Ling-3.0, a permuted v, static
execution), the GDN runs in ggml and the OpenVINO side reads its packed
output wrongly. Depending on the shapes, this silently corrupts the cache or
fails inference with a reshape error. Abort when the graph is scheduled,
with the reason.

* ggml-openvino: share one weights file and NPUW bank across all NPU prefill/decode pairs

With GGML_OPENVINO_NPU_WEIGHTLESS_BANK, every prefill/decode pair (one per KV window, e.g. llama-server's startup probe and a long prompt) wrote its own weights file and used its own NPUW bank, so import kept the weights resident once per pair (+2.1 GiB for phi-4-mini). Pairs now use a model-level <wfp>-<wfp>.weights.bin and bank; the weights file records a layout/content hash and is reused when identical, with a per-pair file and bank as fallback. Old caches are invalidated.

* ggml-openvino: export and preload NPU prefill blobs for every KV window

With head-major KV the prefill also runs at a KV window picked by prompt length, so an import-only run failed for prompts whose window was never exported. After a pair is exported, also export the prefill model of every other 1024*2^k window (weights resolved through the pair's shared weights file), and on import preload them into a process-wide cache of imported blobs that pair and decode-window builds reuse. Cache the source fingerprint of weights outside OpenVINO buffers, which made each model fingerprint cost ~9 ms. A window switch now costs ~60 ms instead of ~250 ms.

Rename GGML_OPENVINO_NPU_DECODE_PREBUILD to GGML_OPENVINO_NPU_BLOB_PREBUILD since it now covers prefill as well as decode windows.

* ggml-openvino: drop invalid prefill blobs and clear blob caches on backend free

The prefill preload imported each KV window's cached blob without a try, so
one blob that failed to import failed the graph compute, and the blob stayed
in the cache for every later run. Remove the blob and its manifest and log a
warning, as the pair import does; the window is built when a prompt first
needs it.

The imported-blob map and the fingerprints of weights outside OpenVINO
buffers lived until the process exited. A process that loads models one
after another kept every compiled model resident, and a later model whose
weight landed at a freed weight's address reused that weight's fingerprint.
Clear both with the shared compiled-model cache when the last backend is
freed.

Also apply clang-format to the changed lines.

* ggml-openvino: enable NPU KV windows by default

With GGML_OPENVINO_NPU_BLOB_PREBUILD unset, every NPU decode step ran a
static graph that attends the whole context, so decode was slow for any
sequence much shorter than -c. On a Lunar Lake NPU, Llama-3.2-1B decodes at
0.79 t/s with -c 4096 and 0.20 t/s with -c 16384, against 11.1 and 12.0 t/s
with KV windows. Each window adds one decode graph to compile at load: with
-c 4096, load goes from 11.1 s to 14.6 s for Llama-3.2-1B and from 61.2 s to
81.6 s for Llama-3.1-8B, whose decode goes from 0.20 to 0.82 t/s.

Default GGML_OPENVINO_NPU_BLOB_PREBUILD to 1; 0 keeps the full-context
decode graph. Update the env table and the Boolean-flag note, which said an
unset flag is always disabled.

* ggml-openvino: add POOL_1D and allow F32 EXP on GPU

* skip empty tensors

* ggml-openvino: detect the gated LFM2 conv input as a recurrent sequence

A reshape whose output reaches SSM_CONV through concat(conv_state,
transpose(x)) gets op_case 6, which keeps the active sequence count apart
from the token count, as the conv state does. The LFM2 and LFM3 short
convolutions gate x first, so the transposed tensor is
mul(view(x), view(x)) and the check missed it. The reshape got op_case 5
instead, and:

- in stateful mode, which drops the leading dim for op_case 5, TRANSPOSE
  got a rank-3 input with a 4-axis order and graph conversion failed;
- with more than one sequence, the concat with the conv state failed.

Accept the gated input too. The output for a single stateless sequence
does not change.

* ggml-openvino: always stream weight requantization

Requantizing Q6_K/Q5_K weights to 8-bit (or F16 for some tensors) previously converted the entire tensor to f32 first. For large output or embedding matrices, this required 2–2.5 GB of temporary memory.

The chunked conversion path already processes 256 rows at a time and produces identical output. It was previously enabled by `GGML_OPENVINO_REDUCE_COMPILE_MEM` or `GGML_OPENVINO_MEMORY_OPTIMIZE`; use it by default whenever the target block size divides a row. Q4_1_64, which has no chunked quantizer, still converts the whole tensor.

The variable retains its other effects.

* ggml-openvino: allocate host buffers as untouched OS memory

Host buffers (weights, KV cache and compute) were allocated with `ggml_aligned_malloc` and explicitly zeroed with `memset`. This made the entire weight buffer resident before any tensors were loaded.

Allocate host buffers using `mmap(MAP_ANONYMOUS)` on Linux and `VirtualAlloc` on Windows instead. Both provide zero-initialized memory without touching the pages, allowing the OS to bring them into memory as tensors are written.

This preserves the zero-initialization required by ggml-org/llama.cpp#20857 while avoiding unnecessary memory residency during model loading.

Remove the cache-only allocation branches, the corresponding `ggml_aligned_free` path from the destructor and the `data_owned_by_ov` member it used, since they are no longer needed.

* ggml-openvino: release GGUF file pages once a weight is converted

Once `set_tensor` converts a weight into the backend buffer, its source pages in the mapped GGUF file are no longer needed for that buffer. Release these clean pages using `madvise(MADV_DONTNEED)` on Linux and `VirtualUnlock` on Windows. The mapping itself remains intact, and pages can be loaded again from the page cache or file if needed.

Only release whole pages within tensor ranges belonging to read-only file mappings. Skip buffers that wrap the mapping itself.

Keep locked pages resident. On Linux, `MADV_DONTNEED` fails on mlocked memory. On Windows, check page lock status with `QueryWorkingSetEx` and skip ranges containing locked pages, since `VirtualUnlock` would otherwise unlock them.

* ggml-openvino: key weight constants by name for the NPU model cache

The NPU compiled-model cache (GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR without the
weightless bank) imports a blob with the freshly converted model as ov::hint::model,
and NPUW finds each constant by its WeightlessCacheAttribute bin_offset. The keys came
from a per-model counter, and a constant that already had a key kept it. Weight nodes
are shared between models: between the NPU prefill and decode models, and across the
process through the tensor extras. So a weight kept the key from whichever model
converted it first.

The decode model for a KV window is built both as the decode half of the
prefill/decode pair and as a standalone window model, under one fingerprint, and the
two builds got different keys. Importing the blob of the other build failed with
"Couldn't find Constant in cache!", and the decode model was recompiled and exported
again on every run. With GGML_OPENVINO_NPU_BLOB_PREBUILD on by default, this happens
whenever a prompt reaches a larger KV window.

Key weight constants by a hash of the weight name and their position in the weight's
subgraph, above the range of the counter. Other constants are created per model and
keep the counter.

* ggml-openvino: drop CACHE_MODE=OPTIMIZE_SIZE from the NPU plugin cache

With GGML_OPENVINO_CACHE_DIR set, the NPU compile config added
CACHE_MODE=OPTIMIZE_SIZE next to NPUW_CACHE_DIR. With it, the NPU plugin
compiles the NPUW submodels with weights separation, which fails for them:
with NPUW folding, "No WeightlessCacheAttribute has been found in any of the
model's Constant nodes"; without folding, a driver compile error. So every
NPU run with GGML_OPENVINO_CACHE_DIR failed on its first compile.

Keep the default cache mode. NPUW_CACHE_DIR still caches the compiled
submodels.

* ggml-openvino: load each self-contained NPU blob once per process

A self-contained blob (GGML_OPENVINO_SELF_CONTAINED_BLOB) embeds its NPUW
weights bank, and NPUW cannot load a blob into a bank that already holds
those weights: Bank::read_and_add_tensor asserts. Two cases did this on
every run that imported the cache:

- Pairs that differ only in the prefill model share the decode model and
  its bank. A long prompt builds such pairs, and each one imported the
  decode blob again. The import failed, so the decode model was compiled
  and exported again for every pair, until the run ran out of memory.
- The decode model of each KV window uses the bank of the pair's decode
  model, so the import of a window blob failed and stopped the run.

Keep each self-contained model in the process once it is imported or
compiled, and reuse it for every pair with the same fingerprint. Turn the
KV windows off for self-contained blobs; the weightless bank keeps them.

* ggml-openvino: update the Docker image GPU drivers to 26.40.40196.8

The OpenVINO image pins the Intel GPU drivers at compute-runtime
26.35.39758.10. With that driver, gemma-4-E4B hangs on the Lunar Lake
GPU. With 26.40.40196.8 it runs.

Take the GPU driver pins from the compute-runtime 26.40.40196.8 release:
IGC v2.42.6, compute-runtime 26.40.40196.8 and gmmlib 22.10.2.

* ggml-openvino: fix long NPU prompts failing with KV slicing

With GGML_OPENVINO_KV_SLICE, on by default, a static prefill is compiled
at the attended KV length that llama reports for the ubatch. That length
grows with every ubatch of a long prompt, and each new length compiled
another prefill/decode pair, with GGML_OPENVINO_NPU_BLOB_PREBUILD (on by
default) also its decode KV windows. The shared compiled cache keeps
every pair until the backend is freed.

Each loaded prefill model keeps NPU memory for its inferences. On a Lunar
Lake NPU, a prefill model loaded next to two or three others cannot get
it: DRM_IOCTL_IVPU_BO_CREATE fails with "Buffer size is too big", and
pfnAppendGraphExecute returns ZE_RESULT_ERROR_UNINITIALIZED. With -c 4096
and the default -ub 512, a 2200-token prompt failed on smollm2-1.7B,
Phi-4-mini, Ministral-3b and gemma-3-4b.

- Round the attended length of a sliced static prefill up to a power of
  two, clamped to the context. A growing sequence now compiles a prefill
  model when its attended length doubles, not at every ubatch. The added
  rows are masked, so the output does not change.
- When only the prefill attention size changes, the new pair takes over
  the decode models (base and KV windows) of the entry's current pair,
  because they do not depend on that size, and the current pair is
  dropped before the new prefill model compiles. A pair that another
  backend may use stays cached. Self-contained pairs are exported as one
  blob pair, so they keep their own decode models.
- The taken-over KV window that covers the next decode position becomes
  the base decode model, as in a pair built at that length, so the first
  decode step does not have to create an infer request for it.

* ggml-openvino: pre-scale the GDN q input by default on the NPU

The NPU GatedDeltaNet kernel writes its output in FP16 and flushes
values below 6.1e-5 to zero, and the RMS norm after it amplifies the
loss. GGML_OPENVINO_GDN_Q_SCALE lifts the output above that floor, but
it was off by default. On a Panther Lake NPU with the compiler bundled
in OpenVINO, Qwen3.5-2B scores a perplexity of 14.77 without it and
13.00 with it (12.93 for the same weights on CPU), and its greedy chat
no longer loops while thinking.

- Enable the pre-scale by default with the factor 64.
  GGML_OPENVINO_GDN_Q_SCALE=0 turns it off.
- Scale the eps of the norm only when the GDN that feeds it is
  pre-scaled. A norm that reads the GDN output directly got the scaled
  eps on every device, while q is scaled only on the NPU. With the
  variable set, Qwen3.5 gave wrong output on CPU and GPU.
- Add the factor to the model cache fingerprint of NPU graphs with a
  GATED_DELTA_NET, so a cached model is not reused across settings.
- docs: update the GGML_OPENVINO_GDN_Q_SCALE row and note.

* openvino-ggml: Decompose the GDN for LNL

1. Automatically decompose LNL GDN prefill into chunked MatMuls; retain the native op for decode.
2. Isolate lowering selection, fix positive-gate handling, and add overflow guards.
3. Update cache identity and add chunk-boundary regression cases.

* ggml-openvino: fix GDN decomposition in decode and on long prompts

Two problems remained with the GATED_DELTA_NET decomposition on a Lunar
Lake NPU.

The decomposition covered only multi-token (prefill) graphs, and decode
kept the GatedDeltaNet op. Under Linux the decode kernel loses the
device as well: after a successful prefill, the first generated token
failed with ZE_RESULT_ERROR_DEVICE_LOST in llama-completion, llama-bench
and llama-server.

The chunk masks were new constants in every converted model. The NPUW
weights bank is shared by all compiled models (NPUW_WEIGHTS_BANK=shared)
and identifies a constant by its data pointer, also after the model
that owned it is gone. When a longer sequence needs a second prefill
model, its masks can get the addresses of freed masks of the first one
and are then read as those. The Qwen3.5-2B perplexity over 512 tokens
changed between 12.99 and 56.94 from run to run, and a 1900-token
prompt gave garbage.

- Lower a single-token GATED_DELTA_NET to one recurrence step when the
  decomposition is on: the chunk formulas for a chunk of one token,
  where P = 1 and the decay is 1.
- Keep one instance of each chunk mask for the whole process.
- test-backend-ops: add a single-token case with two sequences, repeated
  heads and a positive gate.
- docs: GGML_OPENVINO_GDN_DECOMPOSE covers single-token graphs.

* openvino-ggml: Enhance the safe env parsing

* ggml-openvino: keep the GDN q pre-scaling on when its variable is unset

ggml_openvino_gdn_prescale_enabled() returned false when
GGML_OPENVINO_GDN_Q_SCALE is unset, which switched the default-on
pre-scaling off. Qwen3.5-2B on a Panther Lake NPU went from a perplexity
of 13.00 back to 14.77.

- Treat an unset variable as enabled. Zero, negative and non-numeric
  values still turn the pre-scaling off.
- Comments: ggml_openvino_getenv_int() no longer goes through std::atoi.
- docs: a boolean flag with a non-integer value is no longer always
  disabled; a flag whose default is 1 stays enabled.

* ggml-openvino: use the PLUGIN compiler for GDN graphs on the NPU

The NPU driver compiler, the default, gives wrong output for graphs
with GATED_DELTA_NET (Qwen3.5). Qwen3.5-2B on a Panther Lake NPU answers
with unrelated text and has a perplexity of 16.77, against 13.00 with
the compiler bundled with OpenVINO. Other models stay on the driver
compiler, which builds faster prefill kernels.

- Compile static graphs with GATED_DELTA_NET with
  NPU_COMPILER_TYPE=PLUGIN, unless GGML_OPENVINO_NPU_COMPILER_TYPE or
  NPU_COMPILER_TYPE in GGML_OPENVINO_NPU_CONFIG sets the compiler type.
- Add the choice to the model cache key of these graphs, so blobs from
  the driver compiler are not imported.
- docs: describe it in the GGML_OPENVINO_NPU_COMPILER_TYPE row.

* ggml-openvino: fix gemma-4 E2B on the Lunar Lake NPU at small contexts

The NPU 4000 compiler computes ScaledDotProductAttention wrongly when K
has one KV head and is square, that is when the attended rows equal the
head size. gemma-4 E2B has one KV head, 512-wide full-attention heads
and 256-wide sliding-window heads, so a context of 512 tokens or less,
and a prefill that attends 256 or 512 rows, gave wrong output.

For that shape on NPU 4000, build the attention from MatMul, Softmax and
MatMul, as the transposed-V path already does. Other shapes and devices
keep the op.

* ggml-openvino: support multiple GPUs with -sm layer

* ggml-openvino: check GGML_OPENVINO_* variables and reorganize the docs

The backend ignored a misspelled GGML_OPENVINO_* name and a value that
was not valid, and the docs listed every variable in one table.

Code:
- One table holds the known variables and their types. A value that is
  not valid stops the backend with a message that names the variable and
  the accepted values. An unknown GGML_OPENVINO_* name gives a warning.
- With log verbosity 4, one line shows the variables in effect.
- GGML_OPENVINO_GPU_CONFIG and GGML_OPENVINO_CPU_CONFIG pass plugin
  properties to the GPU and the CPU, as GGML_OPENVINO_NPU_CONFIG does
  for the NPU. If the plugin rejects a property, the error names the
  variable.
- GGML_OPENVINO_STATEFUL_EXECUTION gives a deprecation warning.
- The key of the compiled model cache covers the two new variables,
  GGML_OPENVINO_KV_SCATTER_ELEMENTS and
  GGML_OPENVINO_DISABLE_ELTWISE_RANK_ALIGN.
- GGML_OPENVINO_ENABLE_CACHE, which no code read, is no longer
  registered.

Docs:
- docs/backend/OPENVINO.md: Quick Start, collapsible blocks, runtime
  configuration in groups (basic, plugin properties, advanced tuning,
  compiled model cache, deprecated), updated validated models and known
  limitations, corrected stale entries.
- dev_doc/OpenVINO-ENV-VARS.md: the variables for development and the
  NPU properties that the backend sets.

* docs : add LFM2.5 and Ling-mini-2.0 to the OpenVINO validated models

* ggml-openvino: rebuild converted weights in get_tensor

set_tensor converts a quantized weight in place into the extracted or
requantized layout, so get_tensor returned bytes that are not in the ggml
format. A reader such as the CPU reference in test-backend-ops then
computed on garbage and produced NaN (MUL_MAT cases with weights=1).

get_tensor now rebuilds the ggml bytes for such a weight: it evaluates the
weight node on the host and quantizes the values back to the tensor type.
The result is close to the original data, not bit-exact. Other reads are
unchanged, and the inference path does not call get_tensor on weights.

* ggml-openvino: drop the GDN test cases from test-backend-ops

Restore tests/test-backend-ops.cpp to the upstream version.

* docs : improve the OpenVINO backend docs

* docs : restore the OpenVINO intro and the development note

* ggml-openvino: fix slow performance of MoE models with requantize and workaround to use fused MOE Compressed

* ggml-openvino: document the fused GPU MoE op limits

Converting K-quant MoE experts to Q4_1_64 on GPU lets the plugin use its
fused MoE op, which runs in f16 only. granitemoe models give wrong output
with it, and an f32 inference precision hint fails to compile. Document
GGML_OPENVINO_MOE_OP=0 as the way to turn the op off, and that
llama-perplexity with MoE models needs -b equal to -c.

Also split the long TODO comment over two lines.

* ggml-openvino : FILL for in-place clear of recurrent cache, sticky NaN fix

* restore this part

* ggml-openvino: bump the model-cache version for the recurrent-cache reset change

Recurrent-cache resets now clear slots with Select instead of Multiply,
so a NaN in a cleared slot no longer survives. Compiled models cached
before this change keep the old graph, so bump the fingerprint version
to rebuild them. Also note how SCALE and FILL store their parameters.

* ggml-openvino: read rope_factors_short/long as weight constants

create_weight_nodes() registers the ROPE frequency factors as
"rope_freqs.weight", but set_input_output() and compute_model_inputs()
kept the tensor's own name. For models whose factors are named
rope_factors_short or rope_factors_long (Phi-3.5, Phi-4, MiniCPM with
LongRoPE) the factors became a Parameter fed from the host weight buffer.
That buffer is empty in cache-only mode and after the host weights are
released, so these models gave garbage output with
GGML_OPENVINO_COMPILED_MODEL_CACHE_ONLY, GGML_OPENVINO_RELEASE_WEIGHTS
and GGML_OPENVINO_MEMORY_OPTIMIZE.

Use the same name in all three places, so ROPE always reads the weight
Constant. The graph changes, so bump the model-cache version to 7.

* ggml-openvino: import cached models from a memory-mapped tensor

A stream import reads the whole compiled blob into a heap buffer while
the plugin builds its own copy of the weights. On CPU this held the blob
twice at the peak. Import from ov::read_tensor_data() instead, which maps
the file. This lowers the CPU cache-hit and cache-only peak memory; GPU
memory is unchanged.

* ggml-openvino: do not write CPU model-cache blobs for MoE models

The CPU plugin exports its MoE op (GatherMatmulCompressed) but cannot
import it again with OpenVINO 2026.4 and 2026.5 ("unsupported opset").
Every later run then failed the import, compiled again and rewrote the
blob. Skip the blob for such models on CPU and print a warning.

In cache-only mode no blob exists for them, so the generic advice to run
once without cache-only cannot help. Print an error that says cache-only
does not work for MoE models on CPU instead.

* ggml-openvino: fix GGML_OPENVINO_DEBUG_INPUT on GPU

GPU inputs can be device (USM) tensors, whose data() throws, so every
GPU compute failed with GGML_OPENVINO_DEBUG_INPUT=1. Print a host copy
of remote tensors instead.

* docs : add OpenVINO memory optimization tips and MiniCPM-V-4.5

Add a "Tips for Memory Optimization" section to OpenVINO-ENV-VARS.md
with examples for the spill directory, the compiled-model cache, host
weight release on GPU, CPU MoE compile memory and the NPU weightless
cache, and link it from OPENVINO.md. Add MiniCPM-V-4.5 to the validated
models.

* ggml-openvino: run MoE models with F16 and BF16 weights on GPU

On GPU, supports_op rejected MUL_MAT_ID with non-quantized expert weights
and GET_ROWS with BF16 data, so the scheduler ran those ops on the CPU.
For F16 MoE models the CPU MUL_MAT_ID then read the strided top-k view as
expert ids and crashed. For BF16 models the token embedding came from the
CPU and reached the OpenVINO graph as a static-shaped input, which failed
once the token count changed. Both ops translate correctly on GPU, so
remove the two rejections.

* ggml-openvino: fix BERT embedding models on CPU and GPU

BERT models (bge-m3, nomic-embed-text-v1.5) failed on every device:

- A VIEW of a weight with a GGML_OP_NONE source was marked as having a
  dynamic token dimension, a rule meant for KV cache views. The token type
  row view of BERT was then never sliced, and the whole two-row table was
  added to the activations. Skip the rule for model weights.
- A row range view of a 2D weight now becomes a Slice of the weight rows,
  as GET_ROWS already does.
- Attention without a KV cache pads K and V up to the mask width, which
  depends on the token count. PAD baked the amount from the first graph,
  so later token counts gave mismatched shapes. Compute the end padding
  from the mask width at run time.

The NPU still fails for these models.

* docs : add bge-m3 and nomic-embed-text-v1.5 to the OpenVINO validated models

---------

Co-authored-by: Sabnis, Omkar <omkar.sabnis@intel.com>
Co-authored-by: Beheshti <nazanin.beheshti@intel.com>
Co-authored-by: Mustafa Cavus <mustafa.cavus@intel.com>
Co-authored-by: Suhas Somashekar <suhas.somashekar@intel.com>
Co-authored-by: yzhou51 <yaowei.zhou@intel.com>
Co-authored-by: Mostafa Faheem <mostafaaafaheem@gmail.com>
Co-authored-by: zhaixuejun1993 <xuejun.zhai@intel.com>
Co-authored-by: Anisha Kulkarni <anisha.dattatraya.kulkarni@intel.com>
Co-authored-by: Ananth, Swathi <swathi.ananth@intel.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Yu, Zijun <zijun.yu@intel.com>
Co-authored-by: Proshunin, Nikolay <nikolay.proshunin@intel.com>
Co-authored-by: Alexey Kopytko <alexey@kopytko.com>
2026-11-18 09:56:20 +02:00
Johannes Gäßler 398f00d657 CUDA: use MMQ for MoE on V100 (#32504) 2026-11-18 08:46:19 +01:00
Aleksander Grygier 9e93cd8001 ui: a Storybook for Llama UI and Llama Chat (#32485)
* ui: Llama UI's own Storybook in ui/lib/ui

The library gets its Storybook (`npm run storybook`, `npm run
build-storybook`) with a story for everything it holds, under Llama UI:
the foundations, the 29 base components (as actions, forms, display,
overlays and navigation), the app components (ActionIcon, SearchInput,
SyntaxHighlightedCode), the Model UI and the native app mockups. The
stories import the package by its published names, aliased onto src/lib,
run as tests under `npm test` (the storybook project in vite.config.ts),
and are type-checked with the rest of the package. `npm run dev` starts
Storybook next to the SvelteKit check page; the toolbar switches the
color mode and the surface theme, and pages that are mostly prose are
.svx, through mdsvex.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: a Storybook for Llama Chat

Llama Chat's Storybook gets an introduction, Agents › Design guidelines
(DESIGN.md), Llama UI's stories as the library has them, and Llama Chat:
its own foundations (safe areas, the splash screens), then one section per
src/lib/components/app/<area> (Chat with the chat screen, form, messages,
attachments and tabs; Content, Models, MCP, Navigation, Settings, Dialogs,
Forms, Actions, Badges, Misc, Servers and PWA), each accessibility story
next to its component's. It replaces the flat pages it had (chat message,
chat form, markdown content, the PWA refresh alert and their fixtures,
and the ChatForm single-row exploration).

The stories run on the new tokens and surfaces with the color mode and
surface theme in the toolbar (by mode-watcher's key, the ColorMode and
SurfaceTheme enums and applySurfaceTheme), and pages that are mostly
prose are .svx in an editorial frame (AGENTS.md says how to write them
and where a story goes). The story tests run Llama Chat's stories and
leave the library's to the library. The brand and communications pages
are in ggml-org/llama-brand; the logo files come from @ggml-org/llama-brand.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: let the ChatScreen story store its reply before the next one starts

The ContinueConversation story ended while its reply was still being stored,
and the next story reset the stores under it. It now waits for the reply to
finish.

Assisted-by: Claude (Fable 5.2)

* ui: publish the Storybook to its Hugging Face Space on every UI change

A new UI Storybook workflow builds ui/apps/chat's Storybook whenever
ui/apps/chat or ui/lib/ui changes on master (or on demand) and uploads it
to the static Space ggml-org/Llama-UI-storybook, with a Space card naming
the commit it was built from. Files the new build no longer has are
removed; the Space's .gitattributes stays. It runs only in
ggml-org/llama.cpp, takes the UI's Hugging Face token
(HF_TOKEN_UI_STATIC_OUTPUT), and the Space can be changed with the
HF_SPACE_UI_STORYBOOK variable.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: run the library's tests, its stories among them, in CI

The Library checks job type-checked and packed ui/lib/ui but never ran its
tests, so the library's stories, which now hold every library component's,
went untested in CI. It installs Chromium and runs `npm test`.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: draw a floating panel's rim on its border, so its edge is one line

Menus, popovers, dialogs and the other matte-float panels carry a 1px border that the
surface made transparent, and drew their inner highlight as an inset shadow. An inset
shadow sits inside the border, so the 1px of fill between it and the outer ring showed
as a second edge. The border now draws the highlight, right inside the ring.

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-18 08:27:44 +01:00
Pascal aaa807c568 cuda: decode the conv2d patch indices with fastdiv (#32492)
The implicit GEMM conv2d kernel recovers the channel and the kernel
position of each reduction index with three integer divisions by
runtime divisors, once per loaded row of every tile. Outside the 3x3
and 1x1 specializations this integer division bounds the kernel:
removing it halves the time on long reductions.

Divide with the precomputed magic numbers of fastdiv instead. The 3x3
and 1x1 specializations keep their compile time divisors.

test-backend-ops -o CONV_2D: 2006/2006 on CUDA0.

CONV_2D, F16 kernel, us/run on an RTX PRO 6000:
  [2054,1,768] k [7,1,768,768]       221 -> 138
  [16390,1,384] k [7,1,384,384]      433 -> 266
  [65542,1,192] k [7,1,192,192]      420 -> 271
  [8240,1,256] k [11,1,256,256]      165 -> 108
  [19,19,256,16] k [4,4,256,4096]   1672 -> 1076
2026-11-18 07:45:14 +01:00
Hongqiang Wang 401563f10e vulkan: run rms_norm with a runtime loop count on Qualcomm (#30824) 2026-11-18 05:51:31 +01:00
Xuan-Son NguyenandPascal 52975342ff server: slot_sched: fix lora and recurrent (#32434)
* server: fix lora and recurrent

* tests: run the hybrid SWA checkpoint test with one token per ubatch

With and without the cache, the prompt was not split into the same
ubatches, and the generated tokens are decoded one at a time while the
uncached run processes them in a batch. On CUDA this moves the top
logprobs of the tiny model by up to 0.03 and swaps near ties, so the
test failed while the cache behaved as expected.

With one token per ubatch both runs go through the same kernels and
the output is compared exactly.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-18 00:12:20 +01:00
Xuan-Son Nguyen 3574df84b7 mtmd: support tiny-voca (#32427)
* mtmd: support tiny-voca

* improve G2P
2026-11-18 00:04:15 +01:00
thelittlefireman 2e53a0bab7 ggml-cuda: hip: Tune stream_k algo and mmq configs to use stream_k on AMD GCN (#30022)
* ggml-cuda: hip: enable stream_k on GCN

* ggml-cuda: Test on CDNA done no improvement, remove TODO

* ggml-cuda: HIP: GCN cleanup
2026-11-17 23:37:14 +01:00
Aleksander GrygierandSigbjørn Skjæret 744c68cd14 ui: publish @ggml-org/llama-ui from CI, and release 0.4.0 (#32480)
* ui: publish @ggml-org/llama-ui from CI when its version moves

ui/lib/ui reaches npm only when someone remembers to publish it by hand, which let 0.3.0
sit a day behind master. The version in its package.json becomes the release: a commit
landing on master with a version npm does not have gets built, linted and published with a
provenance attestation, through OIDC rather than a stored token, as release-publish.yml
already does for GitHub releases. A library change carrying no bump publishes nothing and
says so, on the PR and on the run.

Assisted-by: Llama Chat (unsloth/Qwen-3.8-Flash-Next:IQ4_XS)

* ui: bump @ggml-org/llama-ui to 0.4.0

The library on npm, published before the design system PR merged, holds no entry points for
the Model UI, the app components or the shared constants and enums, and no dependency on
@ggml-org/llama-brand, whose colours the primitives now build on. Consumers import the new
subpaths and get ERR_PACKAGE_PATH_NOT_EXPORTED.

Assisted-by: Llama Chat (unsloth/Qwen-3.8-Flash-Next:IQ4_XS)

* ui: name no environment for the Llama UI publish job

Creating a repository environment needs admin rights the UI maintainers do not hold, so a job
that names one cannot be set up by them. The job works without it: npm's trusted publisher
matches on the repository and the workflow file, and its environment field stays blank to
match. Nothing else in the job depends on an environment.

Assisted-by: Llama Chat (unsloth/Qwen-3.8-Flash-Next:IQ4_XS)

* ui: address review on the Llama UI publish job

- the advisory version check leaves ui.yml for the publish workflow, on a pull_request trigger
  whose path filter keeps it off PRs that touch only Llama Chat: a chat-only PR was told to
  bump a library it did not change
- publish and verify run on master only. The trusted publisher matches on the repository and
  the workflow file, not the branch, so a dispatch from a PR branch - anything with write
  access can start one - could publish code no one reviewed. With no environment, this was the
  only branch protection in place.
- the registry read after a publish retries six times rather than sleeping once, so a release
  the registry is slow to serve is not reported as a failure
- a build metadata suffix (1.2.3+build) is not a prerelease, and versions order by semver's
  prerelease rules, so 0.4.0-beta.1 behind a released 0.4.0 is refused instead of shipped to next
- the summary step passes its values through env like the others, actions: read is gone, and the
  concurrency comment claims only what the setting does
- npm older than 11.5.1 is refused up front, as it cannot publish with an OIDC token

Assisted-by: Llama Chat (unsloth/Qwen-3.8-Flash-Next:IQ4_XS)

* ui: note the two trusted-publishing traps in the Llama UI README

A publisher names a workflow file that must already be in .github/workflows, so it is added
after this workflow lands rather than before, and one created now publishes only through
staged publishing until npm publish is permitted for it. Provenance comes free with either,
and only on a GitHub-hosted runner.

Assisted-by: Llama Chat (unsloth/Qwen-3.8-Flash-Next:IQ4_XS)

* Update .github/workflows/ui-lib-publish.yml

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-11-17 23:19:05 +01:00
Xuan-Son Nguyen 307e3f1551 mtmd: add kokoro tts support (#32413)
* jinja: impl scope-chain key lookup

* add test for call_statement lambda

* chat: add tts_dict for kokoro G2P template

* more testing

* value_float_t range check

* no json-in-meta, add llama_model_meta_arr_str/_n

* mtmd: add kokoro tts support

* detailed guide about G2P and text norm port

* refine G2P

* clean up

* migrate batch api

* nits
2026-11-17 22:25:44 +01:00
Pascal 9048bad8d6 cuda: use the implicit GEMM conv2d path for F32 kernels (#32482)
The fast CONV_2D path (implicit GEMM with MMA, or cuBLAS for
pointwise convolutions) required an F16 kernel. F32 kernels fell
back to the naive direct convolution, which is much slower than
im2col + mul_mat on large channel counts.

Convert F32 kernels to F16 in a pool buffer, as is already done for
the input, and take the same path with F32 accumulation. The Metal
implicit GEMM kernel already stages both operands as F16.
2026-11-17 20:43:11 +01:00
shaofeiqi a9d43ecc30 opencl: add A7 Q5_K non-MoE binary kernel (#31002) 2026-11-17 11:01:15 -08:00
James PrevettandXuan Son Nguyen 8c16babba5 mtmd: Add support for Qwen3-TTS' ICL/reference mode (#32438)
* Added support for Qwen3-TTS ICL/reference mode

Assisted-By: Claude Opus 5.5

* fix correctness

* attach transcription to bitmap

* nits

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-11-17 18:59:11 +01:00
Aman Gupta ad9d9e8f50 qwen4exp : fix graph reallocation from the n-gram table scale (#32483)
* qwen4exp : apply the n-gram table scale with ggml_scale

* read the scale once after load

* remove comment
2026-11-18 00:35:54 +08:00
eddiechen 955c4e8ee5 server : return 400 when grammar and ignore_eos are on (#32377)
* sampling : fix crash when both ignore_eos and grammar are on

* server : rework, throw 400 when grammar and ignore_eos are on

* sampling : move the check into the sampler (review suggestion)

* tests : remove redundant tests
2026-11-17 17:03:21 +01:00
karan 10152bc4a0 ggml-cuda: disable quantized matvec fusion on all Pascal GPUs (#32239) 2026-11-17 23:34:04 +08:00
anujj 8777c0c833 Title: CUDA: few-row MMQ for MUL_MAT_ID (#32065)
* CUDA: few-row MMQ for MUL_MAT_ID

* CUDA: few-row MMQ for MUL_MAT_ID from 2 tokens on DGX Spark

* CUDA: few-row MUL_MAT_ID: one launcher, tighter pair count, more tests

* CUDA: few-row MUL_MAT_ID: dispatch order MMVQ, few-row, MMQ and review fixes
2026-11-17 21:01:40 +05:30
Emanuil Rusev 918076a635 ui: turn the live markdown preview off by default (#32469)
As discussed in #31955, the live preview is now opt-in. With it on, the editor renders blocks as widgets, and moving the cursor down from the first line of a prompt with a few blocks (a heading, a list, a code block) jumps to the last line; with it off, the cursor moves one line at a time. Users with a saved config keep their current value, since the full config is stored in localStorage.

The message input also gets spellcheck, autocorrect and autocapitalize back. CodeMirror turns them off by default, so they were off with the preview on or off; the former textarea and contenteditable inputs had the browser's defaults.
2026-11-17 16:17:46 +01:00
Giorgio Sarno ebc9fb933e CUDA: use block reductions for whole-row pool_1d (#31855)
* CUDA: use block reductions for whole-row pool_1d

Assisted-by: Claude Code

* CUDA: template pool1d row kernel by ggml_op_pool
2026-11-17 15:51:15 +01:00
Aman Gupta 8b8ca4c98e ggml : decode F8_E4M3 to fp32 without ldexpf (#32477) 2026-11-17 22:40:53 +08:00
08a9b35893 llama : keep the k-pool layout prefix on sequence edits (#32137)
* llama : keep the k-pool layout prefix on sequence edits

Assisted-by: Claude Code

* llama : address review of the k-pool layout update

* Append and edit share one branch, using stale > sq.pos_min.
* LLAMA_KPOOL_VERIFY=1 checks every update against a full rebuild. The scan moved into kpool_scan() so the check can reuse it.
* The test turns the check on for its rollback context, skips -sm tensor, and drops kimi-k31.
* Two comments that were no longer true are fixed.

Assisted-by: Claude Code

* llama : check the k-pool layout update in debug builds instead of LLAMA_KPOOL_VERIFY

* llama : remove the k-pool layout verification

* llama : inline the k-pool scan again

* remove dangling new line

* do not search twice

---------

Co-authored-by: Aman Gupta <amangupta052@gmail.com>
Co-authored-by: Alexey Kopytko <alexey@kopytko.com>
2026-11-17 22:37:25 +08:00
Hiroyuki Ikegami acad5e405c vocab : speed up PLaMo-2/3 tokenizer table build (#32249)
Assisted-by: Claude Code (Claude Opus 5.5)
2026-11-17 15:36:15 +01:00
Aman Gupta d3064fddce convert : fix FP8 qwen4exp checkpoints (n-gram table scale, block-scaled FP8) (#32476)
* convert : fix FP8 n-gram table export for qwen4exp

* convert : dequantize block-scaled FP8 weight_scale_inv in ModelOpt checkpoints
2026-11-17 22:25:01 +08:00
Aleksander Grygier 7f106b5459 ui: Llama UI design system: tokens, surfaces and fonts (#32468)
* ui: bring @ggml-org/llama-ui into ui/lib/ui

The design system package (tokens, fonts, surfaces, components and Model UI), so far in the temporary ggml-org/llama-ui
repository at edaf518, moves to its home in ui/lib/ui, built with @sveltejs/package and published as @ggml-org/llama-ui
from here. ui/lib holds the packages, one folder each. Its references to tools/ui now name ui/apps/chat, its
package.json points at this repository, and the UI workflow type-checks and packs it. ui/README.md describes the
layout.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: Llama UI design system: tokens, surfaces and brand

The design system from allozaur/docs/storybook-overhaul: three tiers of
tokens, matte glass surfaces, Llama Sans and Llama Mono, and the base
components on the new tokens (Tabs and GlassKey among them). It also
restyles the stack's features with it, replacing the plain stand-ins they
used until now. DESIGN.md is generated from the tokens. The Storybook that
documents it is the next PR.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: have Llama Chat use @ggml-org/llama-ui for its base components, tokens and fonts

Llama Chat kept its own copy of the design system: 29 base components, the
three tiers of tokens, the surfaces, the theme and the fonts, next to the 12
components ui/lib/ui held. The copy goes, and the library gets what it lacked.

ui/lib/ui
- the 17 base components the app had and the library did not (alert,
  alert-dialog, button-group, checkbox, collapsible, context-menu, dialog,
  drawer, dropdown-menu, empty, hover-card, radio-group, scroll-area,
  skeleton, table, tabs, textarea), with vaul-svelte for the drawer
- the inset-group and selected-row tokens and surfaces
- sources import each other by relative path instead of $lib, which in an
  app points at the app's own src/lib
- theme.css no longer imports tw-animate-css: it is a peer dependency that
  the consumer imports before the theme, as it does Tailwind
- 0.3.0

ui/apps/chat
- imports the base components as @ggml-org/llama-ui/ui/<name> and cn/tv as
  @ggml-org/llama-ui/utils; app.css imports the library's theme.css
- compiles the library from source, with no install or build of it:
  scripts/vite-plugin-llama-ui.ts resolves its entry points and what it
  imports, and svelte.config.js gives TypeScript the same paths
- src/lib/components/ui, styles/tokens, styles/surfaces.css and assets/fonts
  are removed; components.json moves to the library

Builds
- CMake stages ui/lib/ui next to the app, keeping the layout of ui/, and
  rebuilds when a library source changes
- the Dockerfiles copy the library's sources; the Nix derivation takes ui/
  as its source

The built CSS has the same rules in the same order, apart from one
zero-specificity rule and the @font-face rules moving, plus ten utilities
and two tokens that only the library's own components use.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: AGENTS.md places Llama UI in ui/lib/ui

Llama Chat's AGENTS.md says where each part lives: Llama UI is
ui/lib/ui, and Chat imports its tokens, fonts and base components
from there.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: Llama Sans and Llama Mono from llama-brand's latest, and shell commands marked in bash code

The fonts in ui/lib/ui come from llama-brand's latest, with optical sizes
from 10 to 36. Bash code blocks mark shell commands and flags.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: open submenus outside the menu they come from

A submenu was rendered inside its parent menu, so the menu's scroll box
clipped it: the glass menus' backdrop filter, and the filter and transform
of the open animation, make the menu the containing block even for the
fixed-position submenu. Dropdown and context submenus now render in a
portal, like the menus themselves.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: draw submenus like the menu they open from

Submenus had smaller corners than their menus, and context menus had none
of the dropdown's glass. Context-menu parts now match the dropdown's, and
both submenus take the menu's corners and height cap, with their first row
level with the row that opens them, 4px past the menu's edge. Sub-triggers
take the items' radius and pointer.

The submenus' height cap and zoom origin named the dropdown's CSS variables,
which bits-ui does not set on submenus; they now use its --bits-menu-* ones.
The chat group submenu takes the actions menu's z-[999999] layer, as it no
longer renders inside that menu.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: keep a late current-node write off a closed or switched conversation

updateCurrentNode checked the active conversation, awaited the database write, then wrote currNode to whatever was active after it. When the conversation closed during the write it threw, and the reply ended in a streaming error dialog. When another conversation opened, that one got the wrong node.

Assisted-by: Claude (Fable 5.2)

* ui: have Llama Chat use the library's Model UI and app components

Llama Chat still kept its own copies of what the library holds above the
base components. It now takes them from @ggml-org/llama-ui too, and its
copies go.

ui/lib/ui
- ModelId gains what Chat's had: the model's own draft kind, draft
  sidecars with their quant (ModelDraftSidecars, new), aliases, hideName
  and a tooltip, in the same badge order
- entry points for them, like ui/<name>: @ggml-org/llama-ui/app/<group>
  (actions, content, forms, misc) and @ggml-org/llama-ui/models

ui/apps/chat
- ActionIcon, SearchInput, SyntaxHighlightedCode, Logo,
  ModelCapabilityIcons, ModelDownloadProgressBar and ModelDraftSidecars
  come from the library; Chat's copies and the two constants only they
  used are removed
- ModelId and ModelOrgAvatar stay as thin wrappers: ModelId parses the raw
  id and applies the user's display settings, ModelOrgAvatar hides the
  avatar while Hub metadata is off
- the svelte/* TypeScript path falls back to Svelte's types/index.d.ts:
  svelte/reactivity has no file of its own, so a library source importing
  it got its types from ui/lib/ui's own install when there was one

Assisted-by: Claude (Opus 5.5 Medium)

* ui: Llama UI takes the brand's colours and logo from llama-brand

@ggml-org/llama-brand is now a dependency of the library, by the same GitHub
tarball Llama Chat already installs, which now names it @ggml-org/llama-brand
too.

- primitives.css drops the 25 tokens it held copies of (the neutral
  greys, sand, the skies and cinder) and builds on llama-brand's
  css/colors.css instead; the skies' gradients point at llama-brand's.
  Apps import that file before theme.css, as they do Tailwind and
  tw-animate-css: the library's stylesheets import no package, so an app
  that compiles them from source resolves it from its own node_modules
- Logo draws llama-brand's icon, in currentColor, and assets/logo.svg goes
- Llama Chat's TypeScript paths resolve llama-brand from its own
  node_modules, like the library's other dependencies

The fonts stay as the library's WOFF2 copies until llama-brand ships
WOFF2 builds of them.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: have Llama Chat take the library's constants, enums and helpers

Llama Chat redefined most of what the library exports next to its
components. Where the two were the same, Chat now re-exports the library's
from the files that held them, so its own imports stay as they are.

- the library gets entry points for them: @ggml-org/llama-ui/constants,
  @ggml-org/llama-ui/enums and @ggml-org/llama-ui/code, and
  @ggml-org/llama-ui/models carries the model data with the Model UI
- from the library now: the shared CSS classes and code-block constants,
  the byte units, the memory-fit constants and minMemoryTierGb, the bit
  depth labels, the capability and modality icons, labels and order,
  ColorMode, TooltipSide, BooleanString, the model enums, ModelModalities,
  formatParameters, highlightCode and trimCodePadding
- the library's modality order follows Chat's: vision, video, audio
- Chat keeps its own where they differ: UI_DATA_ATTRS and
  ModelDownloadProgress are supersets of the library's (the latter extends
  it now), and formatFileSize counts in 1024s

Assisted-by: Claude (Opus 5.5 Medium)

* ui: give the chat form's add key and model selector a tighter shadow

In light, the two keys inside the chat form, white on its white glass, took
the control elevation, whose soft spread blended into the form. They take a
new elevation-key instead: a 0.5px hairline all round and a 1px drop. In
dark they keep the control elevation. Chosen from the options in
Scrapbook › UI › Composer keys.

- semantic tokens: --elevation-key (light: the hairline and the drop; dark:
  the control elevation)
- component token: --chat-form-key-shadow, for the add key and every model
  selector trigger
- DESIGN.md names it next to the other elevations

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-17 14:39:03 +01:00
Xuan-Son Nguyen d92ea9bb12 mtmd: support Minimax-Music3 (#32435)
* mtmd: support Minimax-Music3

* fix and clean up
2026-11-17 12:53:19 +01:00
Xuan-Son Nguyen 8d1599fca1 chat: add tts_dict for kokoro G2P template (#32397)
* chat: add tts_dict for kokoro G2P template

* more testing

* value_float_t range check

* no json-in-meta, add llama_model_meta_arr_str/_n
2026-11-17 12:52:50 +01:00
Gaurav Garg 054a1babf3 Rotate K/V read order for better L2 cache hit rates (#32407) 2026-11-17 16:49:28 +05:30
Emanuil Rusev 513ad7ae07 ui: turn conversation tabs off by default (#32402)
Browsers already have tabs, so a second tab bar inside the page adds a layer most people don't need. The setting stays, so anyone who wants tabs can turn them on. Users with a saved config keep their current value, since the full config is stored in localStorage.
2026-11-17 11:44:13 +01:00
Aleksander Grygier 67cf8967eb ui: move Llama Chat to SvelteKit 3 and its built-in service worker (#32463)
* ui: replace deprecated packages and fix npm audit findings

Swap the mdast stub for @types/mdast, move the katex override to 0.19.0,
and override glob, source-map and html-encoding-sniffer under
workbox-build and http-server so installs no longer pull deprecated
packages. Apply the non-breaking npm audit fixes to the lockfile.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: migrate Llama Chat to SvelteKit 3 and its built-in service worker

Move to SvelteKit 3, Vite 8, TypeScript 6 and Storybook 10.6. The
SvelteKit options move from svelte.config.js into sveltekit() in
vite.config.ts, $lib imports become #lib subpath imports from
package.json, and $app/environment becomes $app/env. Modules named
server.* or living under a server/ folder are server-only in SvelteKit 3,
so the llama-server components, store and enums move to llama-server
names.

@vite-pwa/sveltekit and Workbox give way to SvelteKit's own service
worker in src/service-worker, registered by usePwa(), which precaches the
app files and leaves API requests to the network. A small plugin emits
manifest.webmanifest. static/sw.js removes the worker of older builds, and
llama-server serves a self-removing service-worker.js as well as sw.js
when the built-in UI is off or replaced.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: use SvelteKit 3's update detection and Svelte 5's current patterns

A page the service worker doesn't control now learns of a new build
from SvelteKit itself: `updated` from $app/state checks
_app/version.json on navigation, on focus and every hour. It replaces
the version Llama Chat kept in localStorage and compared by hand.
Reload on the update prompt pulls in the new worker and activates it
before reloading, so one click lands on the new build, whether or not
a worker controlled the page.

Following Svelte's current guidance:
- usePwa keeps its state in $state rather than a store, and no effect
  checks `browser`
- nearViewport, autoResizeTextarea and the custom CSS injection are
  attachments ({@attach}) rather than actions (use:)
- class lists use arrays rather than class: directives
- the chat contexts come from createContext, typed, in place of
  setContext/getContext with string keys
- ActionIconCopyToClipboard takes $props() rather than `export let`

Assisted-by: Claude (Opus 5.5 Medium)

* ui: move to ESLint 10

ESLint 9 is out of support. Every plugin the app uses supports 10:
typescript-eslint, eslint-plugin-svelte, perfectionist,
simple-import-sort, eslint-plugin-storybook and eslint-config-prettier.
@eslint/js and @eslint/compat follow.

ESLint 10's recommended rules ask for two things the code now does:
errors rethrown from a catch keep the caught error as their `cause`,
and no variable starts with a value that every path overwrites.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: override minimatch to clear npm audit's last finding

ESLint 10.14, @eslint/config-array 0.23.5 and typescript-eslint 8.72
depend on minimatch 10, which GHSA-g3hv-75wg-m249 (denial of service
via globparts amplification) affects below 11.0.1. Their fixed releases
move to minimatch 11 but are all younger than the repo's 7-day
min-release-age, so an override pins minimatch 11.0.5, the newest the
rule allows. The three only call new Minimatch(), match() and
minimatch(), which minimatch 11 keeps. npm audit now finds nothing.

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-17 11:43:36 +01:00
cherry77-cloud 4df3f2959c qwen3next : apply output projection weight scale (#32393)
Assisted-by: Codex
2026-11-17 11:32:30 +01:00
Daniel Bevenius a49db739fb cuda : raise shared mem limit only for the launched kernel (#32392)
This commit removes a call of the CUDA_SET_SHARED_MEMORY_LIMIT macro as
the fallback decision was moved out of launch_mul_mat_q.
launch_mul_mat_q now has the fallback template parameter which was added
in Commit 6eddde06a4 ("CUDA: refactor MMQ
kernel configuration (#24127)").

This code was then refactored in Commit
c49edf0963 ("CUDA: fuse FFN gate/up
matmuls and GLU in MMQ with interleaved gate/up rows (#29948)") which
added the has_fusion template parameter but it kept both macro calls.

launch_mul_mat_q only launches one type of kernel now (this is now
decided earlier in the templated call chain) and each specialization
will set the limit itself.

The motivation for this is to simplify the code and improve readabilty.
2026-11-17 10:53:08 +01:00
Titaniumtown eada89c315 nix: expose a SYCL package (#31066) 2026-11-17 11:04:12 +02:00
Ruben Ortlam 8f647f1fdb vulkan: disable flip Flash Attention shader on AMD Windows due to a driver bug (#32415) 2026-11-17 10:55:28 +02:00
Neo Zhang dde9de82cb sycl : Fix FA mkl issue: a mask broadcast over heads or batches (#31536)
* fix the SYCL fa with mkl error of a mask broadcast over heads or batches; add argument cache-ram to start-svr.sh

* add argument cache-ram to win-start-svr.bat

* fix format issue, remove --cache-ram in script: move to another PR
2026-11-17 10:50:32 +02:00
Souleyman e0a9118fe9 metal : add implicit GEMM kernel for CONV_2D (#28069) 2026-11-17 09:46:24 +01:00
Nikhil Jain 140a279874 webgpu : Implement timestep embedding (#31695)
* Only run webgpu CI on my fork

* Add webgpu only workflow

* resolve merge conflicts

* save state

* save kernel

* add kernel back in

* fix merge issues with kernels

* updates

* update tests

* change invocation count and workgroup to be u32

* clang format

* clang format

* update docs

* format

* remove extra file
2026-11-17 10:41:37 +02:00
Takamichi Ohara c145db2728 webgpu : use f32 while dequantizing to avoid FTZ (#32140) 2026-11-17 10:40:23 +02:00
lhez ede6106780 opencl: support f16 and non-contig f32 silu (#32319)
* opencl: support non-contig silu

Assisted-by: Claude Opus 5.5

* opencl: support f16 silu

Assisted-by: Claude Opus 5.5
2026-11-17 10:38:07 +02:00
9214bdb554 sycl: use fast division in bin-bcast kernels (#32213)
Replace the per-element modulo of i0 with fast_div_modulo in the ravel and
unravel bin-bcast kernels, drop the now-unused ne10/ne20 kernel arguments,
and reject divisors that do not fit in 32 bits when the magic values are
computed, as the CUDA implementation does.

Co-authored-by: zhaoyshine <zhaoyshine@users.noreply.github.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
2026-11-17 15:17:47 +08:00
shaofeiqi d3c3e242c1 opencl: add A7 Q8_0 non-MoE binary kernel (#31003) 2026-11-16 20:15:56 -08:00
Sidhartha Reddy Potu 9581154a22 mtmd : fix Muse Glimmer vision embedding order (#32433)
Assisted-by: Claude
2026-11-17 00:29:02 +01:00
Gabe GoodhartandSigbjørn Skjæret 1ef8c5d684 convert : respect max_position_embeddings for granite hybrid (#32436)
* fix(conversion): Respect max_position_embeddings for granite hybrid

Branch: GraniteHybridContextLengthConversion
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* style: shorter comment w/ HF reference

Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-11-16 23:49:57 +01:00
Pascal e8fbcad855 tests: print the exception message in test-llama-context (#32428)
A case that throws only reported "unhandled exception", which hides
the cause of the intermittent failures on the Vulkan Apple runner.
2026-11-16 21:53:32 +01:00
Xuan-Son Nguyen 95e837aaf1 skills: explicitly specify paths to README-dev.md (#32431) 2026-11-16 21:52:12 +01:00
Bartowski d998c676ff convert: Fix inkling 2 mmproj conversion (#32430) 2026-11-16 20:39:32 +01:00
Xuan-Son NguyenandPascal 92192d31e4 mtmd: support YuE2-3B (#32369)
* mtmd: support YuE2-3B

* improve speed

* add CFG

* fix streaming

* add cfg control from api

* check ctx size

* fix n_predict

* add flow_steps config

* nits

* fix batch api

* nits

* adapt

* mtmd: yue2 samples from its own saved logits

The context logits belong to the last decode of any slot, so with
several server slots the score and the unguided codes were sampled
from another sequence. Keep the logits of each own decode in
save_logits(), which also does the guidance mix.

* mtmd: let the yue2 snake chain fuse

The beta reshape landed between sqr and mul, so the mul, sin, sqr,
mul, add chain was not consecutive and the backends never fused it.
Expand both reshapes first, like qwen3tts does.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-16 20:22:23 +01:00
shaofeiqi ad5cbfa911 opencl: add A7 Q4_K non-MoE binary kernel (#30948) 2026-11-16 10:58:29 -08:00
Hongqiang Wang d261e49e7e opencl: fix three defects of the bin flash-attention prefill path (#32110)
* opencl: release the WMM flash-attention mask fallback image

The fallback image is created on every call of the bin flash-attention
kernel over a sub-buffer of the K cache and was never released, so each
call kept a K-cache sub-buffer alive. Every call with more than one query
row takes this path, including each speculative-decoding verify step, so
the objects pile up and the server slows with every request.

* opencl: keep q8_0 K/V at head size 512 off the WMM flash-attention kernel

With q8_0 K/V at DK=512 the bin flash-attention kernel gives wrong results
whenever V holds different values from K (test-backend-ops FLASH_ATTN_EXT
hsk=512 q8_0, max error 2.04 on X2-90); f16 K/V at DK=512 is correct. The
backend's own flash-attention kernels already decline q8_0 at DK=512, so the
shape is now handled the same way with or without the kernel library.

* opencl: decline WMM flash-attention prefill beyond the device's 3D image limits

The WMM path repacks Q/K/V into 3D images; V is n_kv wide, so past the device's
maximum 3D image width (about 16k tokens of context on X2-90) clCreateImage
failed and the call asserted. Such shapes now take the regular flash-attention
path.
2026-11-16 10:57:48 -08:00
Aldehir Rojas 4ddf2b9470 chat : implement inkling parser (#32381) 2026-11-16 19:48:24 +01:00
Xuan-Son Nguyen 4303cd94ae contributing: mark agents.md and code-review as required (#32416) 2026-11-16 19:26:10 +01:00
Beinsezii c7518dbec5 MMVQ: Tune ne11 Gate for RDNA3_5 (#32308) 2026-11-17 00:56:20 +08:00
Xuan-Son Nguyen a68c6136bb mtmd: support fish audio s2 (#32330)
* add fisk-audio-s2 support

* add voice cloning

* cont merge master

* nits

* nits

* clean up

* nits

* use new batch api
2026-11-16 17:32:27 +01:00
Aleksander Grygier 3a7660371d ui: move the web UI to ui/apps/chat and call it Llama Chat (#32032)
* ui: move tools/ui to ui/apps/chat

The chat app moves to its own root folder, ui/apps/chat, as the first step of splitting the UI into one library
(ui/lib) and the apps built on it (ui/apps). The code is unchanged; every path that pointed at tools/ui now points at
ui/apps/chat: CMake adds it from tools/, and the Dockerfiles, the Nix package, the workflows, the labeler, CODEOWNERS,
.editorconfig, the git hooks, the dev and demo scripts and the docs follow it.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: name Llama Chat's package @ggml-org/llama-chat

It was still called llama-ui, the name of the library that now sits beside it
in ui/lib (@ggml-org/llama-ui). The package stays private; the app's display name
and the llama-ui.zip build artifact are unchanged.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: call the built-in web UI Llama Chat in docs, help text and comments

The app in tools/ui is Llama Chat; Llama UI is the component library it is
built from. Prose that said "Web UI", "web UI" or "WebUI" for the app now
names it: the READMEs, SECURITY.md, the --ui help text, the CMake option
description, one server log line, code comments and the Storybook intro.

Names that others depend on are unchanged: the deprecated --webui flag
aliases, the LLAMA_*_WEBUI CMake and environment names, the webui_settings
field, the storage migration keys and the Nix attributes.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: name tools/ui as a former path of Llama Chat in the code-review skill

Changes under the old tools/ui path, as well as tools/server/webui, still
get the Llama Chat checklist.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: name Llama Chat's path in the security policy's exclusions

Llama Chat no longer lives under tools/server/, so the exclusion listed
under it now names ui/apps/chat. What is covered stays the same.

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-16 17:23:36 +01:00
Shubham RoyandDeveloper e8698ac163 fix:- Misc. bug: Settings/Tools is lying about tool picks inside a ch… (#32223)
* fix:- Misc. bug: Settings/Tools is lying about tool picks inside a chat #32214

* checks run lint error

---------

Co-authored-by: Developer <dev@example.com>
2026-11-16 17:15:16 +01:00
Rudransh Shekhar 7804be0634 arg : allow up to llama_max_devices() configs for --tensor-split and --fit-target (#13293) (#31435) 2026-11-16 17:07:45 +01:00
Xuan-Son Nguyen ef6bc6f367 jinja: impl scope-chain key lookup (#32396)
* jinja: impl scope-chain key lookup

* add test for call_statement lambda
2026-11-16 15:48:46 +01:00
Niklas Wenzel a06044b934 llama : deprecate old batch API (#32306) 2026-11-16 15:16:13 +01:00
Niklas Wenzel 62a6d3a302 llama : migrate to llama_batch_ext (#32356)
* llama : migrate to llama_batch_ext

* mtmd : migrate zonos3-mini code

Added in https://github.com/ggml-org/llama.cpp/pull/32020
2026-11-16 15:16:12 +01:00
Jeff Bolz ef3848f9d7 vulkan: Support split_k for conv3d. (#29929)
This tries to use the 256x128 shape (coopmat2-only) with split_k when
it appears profitable. It reuses the mul_mat split_k reduce shader.
2026-11-16 15:25:12 +02:00
uvos 47fe133318 HIP/CUDA: enable GDN mma kernel on CDNA (#31327) 2026-11-16 15:24:19 +02:00
Ruben Ortlam 301db63370 vulkan: fix Nvidia fp32 FA shmem overflow, disable shmem staging if it takes too much memory (#32244) 2026-11-16 15:22:57 +02:00
BlivionIaG a001af915e hip: add gfx1013 to rdna1 (#32236) 2026-11-16 15:22:35 +02:00
Pascal 164ab018f9 webgpu: support per-head flash attention masks (#32394)
The flash attention shaders indexed the mask without its head
dimension, so masks with ne[2] > 1 gave wrong results. The mask offset
and the vec block states now include the head, broadcast as
head % mask->ne[2] like the other backends. The params buffer grows to
256 bytes to fit the two new mask fields.

The shared memory estimate dropped kv_shmem when only one of K and V
is read directly, while the shaders keep it unless both are, which made
the K f16 / V q8_0 cases exceed the workgroup storage limit. The vec kv
tile now takes both flags: the memory estimate drops kv_shmem only when
both are direct, and the tile still divides the KV padding when either
is. get_alloc_size and supports_op use the same rule as the pipelines,
and supports_op accounts for the register tile fallback.
2026-11-16 14:20:18 +01:00
Georgi Gerganov 5cb9516798 readme : fix line endings 2026-11-16 14:03:30 +02:00
8bfd64b5ac readme : update (#32382)
* readme : update

* authors : update

* readme : minor

* cont : minor

* cont : typos

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Niklas Wenzel <dev@nikwen.de>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Niklas Wenzel <dev@nikwen.de>
2026-11-16 13:28:31 +02:00
Xuan-Son Nguyen e1925b3695 jinja: support json sort_keys=true (#32366) 2026-11-16 12:28:19 +01:00
Georgi Gerganov 95db855bce ci : use vX.Y.Z tags as docker tags (#32388)
* ci : use vX.Y.Z tags as docker tags

When the Docker workflow is manually dispatched from a vX.Y.Z tag, use that tag verbatim for the image tags instead of appending the build number and commit hash.

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)

* ci : require a tag for manual docker runs

Manual dispatches of the Docker workflow must now use a tag ref, so the image tags are derived from the selected tag. The check can be skipped with the new skip_ref_check input. Scheduled nightly builds are unchanged.

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
2026-11-16 13:27:10 +02:00
miyan 1d2e880bad vulkan : fix 32-bit compilation (#32275) 2026-11-16 11:50:31 +01:00
PascalandRuben Ortlam adf554d239 vulkan: make the radix top-k tie selection deterministic (#31931)
* vulkan: make the radix top-k tie selection deterministic

The radix select gathers the elements above the threshold and the ties
with an atomic counter, so with more ties than remaining slots the kept
columns depend on the thread scheduling.

Each subgroup now owns a contiguous chunk of the row with the same
column mapping in every pass, counts its kept elements, and writes them
at ranks from a scan of the subgroup totals; overflowing ties are ranked
in column order with a ballot, so the lowest tied columns win, as on
CUDA and Metal. The radix pipelines require subgroup arithmetic, ballot
and full subgroups. Enable the deterministic top-k check on Vulkan.

* optimize shader for AMD, Intel and Nvidia

---------

Co-authored-by: Ruben Ortlam <rortlam@redhat.com>
2026-11-16 11:41:44 +01:00
Ruixiang Wang 6b01a4767f CI: trigger spec dec testing for relevant PRs (#32281)
* ci: trigger spec dec testing after refactoring

* add options to run spec dec ci nightly manually
2026-11-16 11:14:07 +01:00
liminfei-amd f477c1b90a cuda : fix MUL_MAT_ID abort with odd src0 columns on AMD (#32385)
On AMD devices MUL_MAT_ID with an F32, F16 or BF16 src0 and at most 8 tokens always used MMVF.
MMVF reads src0 in pairs of values and asserts an even ne[0], so a rank 1 LoRA on MoE expert weights (lora_b has ne[0] == 1) aborted.
Use MMVF there only for an even ne[0]; other shapes take the existing fallback.

Fixes #30576

Assisted-by: GitHub Copilot

Signed-off-by: AMD ROCm Contributor <91481003+liminfei-amd@users.noreply.github.com>
2026-11-16 17:53:06 +08:00
Titaniumtown 21fa9c043f kv-cells: avoid an O(n_cells) scan per sequence removal (#32380) 2026-11-16 10:52:09 +01:00
Pascal a946eb541a server: answer reasoning_effort levels the template rejects with 400 (#32332)
The reasoning effort probe already renders every candidate level, but
it dropped the ones the template raises on, so a request with such a
level (e.g. max on Qwen3.8) reached the template and came back as a
500 on /v1/chat/completions, /v1/responses and /v1/messages.

Keep the raising levels in reasoning_effort_rejected and reject them
in oaicompat_chat_params_parse with an invalid_argument, which the
server answers with a 400 listing the supported levels. Aliases and
levels a template ignores are not rejected, so high on Qwen3.8
still renders as xhigh.

Also trim the output_config.effort test to a single level.
2026-11-16 06:55:56 +01:00
thelittlefireman 05db09a4d3 ggml-cuda: fix MoE regression on HIP with #29948 (#32344) 2026-11-16 06:37:42 +01:00
BeinseziiandSigbjørn Skjæret 9fba51d2c9 convert: fix ripple dflash2 NoPE metadata (#32363)
* convert: fix ripple dflash2 NoPE metadata

was wrong

https://huggingface.co/ricursiveAI/Ripple-65B-Preview-DFlash2/blob/743c4669f7daa11d3ca350b9a057c059481b1afd/config.json#L45-L47

* Update conversion/base.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-11-16 00:47:27 +01:00
Xuan-Son NguyenandAldehir Rojas b4c0e00a4f jinja: add 4 missing features (#32352)
* jinja: add 4 missing features

* apply suggestion

Co-authored-by: Aldehir Rojas <hello@alde.dev>

* fix error msg

---------

Co-authored-by: Aldehir Rojas <hello@alde.dev>
2026-11-16 00:40:44 +01:00
Evan Huus 1719dbb0ee server : skip MCP discovery for router children (#32359) 2026-11-15 23:55:33 +01:00
Xuan-Son Nguyen fc504f91d8 mtmd: support higgs v3 STT and vibevoice-ASR (#32329)
* mtmd: support higgs v3 and vibevoice-ASR

* use proper mapping

* address review comment

* nits

* rm hacky get_audio_length, add mtmd_helper_get_media_meta
2026-11-15 21:25:34 +01:00
Seungjun Bae 18b74fa41d model : read qwen3vl output head size from the tensor (#32349)
Allows Qwen3-ForcedAligner, whose output head (5000 timestamp classes) is smaller than the vocab.
2026-11-15 19:45:17 +01:00
Xuan-Son Nguyen 7ae3c6b231 mtmd: add zonos3-mini TTS model (#32020)
* zonos3 init

* add audio gen pipeline

* fix string_format

* add default voice as gguf metadata

* better ref

* support voice clone

* add delivery params

* optimize speed

* nits

* nits 2

* add text normalization

* nits
2026-11-15 19:44:44 +01:00
Xuan Son Nguyen f4d0673917 Revert "jinja: add 4 missing features"
This reverts commit a681a53b60.
2026-11-15 19:35:31 +01:00
Xuan Son Nguyen a681a53b60 jinja: add 4 missing features 2026-11-15 19:33:06 +01:00
Xuan-Son Nguyen 75fd7f9ec9 docs: move per-model TTS to /docs, refer it from other pages (#32304) 2026-11-15 18:35:27 +01:00
Xuan-Son Nguyen a68799c609 model: support MiniMax-M3.1 (#32301) 2026-11-15 17:39:59 +01:00
Roshan Kumar 56fd8a375f server : ignore empty stop strings (#32093) 2026-11-15 17:23:05 +01:00
Saejin Kim-HeinertandSigbjørn Skjæret b54752df40 ggml-cpu: add x86 VNNI Q2_0 dot product (#26348)
* ggml-cpu: use VNNI for Q2_0 dot products

Keep the upstream AVX2 unpacking, accumulation, and fallback. Use VNNI for the integer dot product on supported x86 CPUs.

Assisted-by: Codex

* ggml-cpu: use VBMI for Q2_0 unpacking

Use multishift to unpack Q2_0 weights on CPUs with AVX-512 VBMI and VL. Retain the existing shift-and-mask unpacking on other CPUs.

Assisted-by: Codex

* Update ggml/src/ggml-cpu/arch/x86/quants.c

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-11-15 16:19:33 +01:00
uvosandSigbjørn Skjæret 900f01ba09 CI/HIP: update fattn symbol names in hip-quality-check (#32331)
* CI/HIP: update fattn symbol names in hip-quality-check

* ignore import-not-found

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-11-15 15:53:22 +01:00
Anton SokolchenkoandJames Pickering de32b268c4 Support DFLASH2 for Recursive Ripple-65B-Preview (#31904)
* fixed support for DFlash2 for Ripple 65B

* spec: Add NoPE handling to DFlash

(cherry picked from commit 89ea8e3e77e3359d17902a2db136022d5d419d88)

* spec: Fix NoPE not being picked up by DFlash

(cherry picked from commit 43393b95fda2ce7e7d243e2650bf7b1cadf1c732)

---------

Co-authored-by: James Pickering <james_pic@hotmail.com>
2026-11-15 19:36:08 +08:00
eddiechen 4e5116f476 server : fix output_config.effort being dropped in /v1/messages (#32321) 2026-11-15 11:09:46 +01:00
uvos 278a89cb78 CUDA/HIP: Refactor the use of _sync warp primitives (#31617)
Uses the rocm-native primitives, increases the rocm version requirement to 6.3 and makes top-k selectable warp.
2026-11-15 10:58:36 +01:00
Xuan-Son Nguyen dd099a6bf3 chat: fix inkling 2 begin/end of image tokens placement (#32294)
* chat: fix inkling 2 begin/end of image tokens placement

* add test
2026-11-15 09:41:44 +01:00
Xuan-Son Nguyen 73f53317eb vendor: sync stb_image (#32305) 2026-11-15 09:37:30 +01:00
Xuan-Son Nguyen 429460dc0e server: refactor, add server_slot_sched, prompt running hash (#32154)
* server: refactor, add server_slot_sched, prompt running hash

* use sha256 hash
2026-11-15 09:37:17 +01:00
Aayan Karmali 58a4a6b49c server : clear slot cache when the global lora config changes (#30367) 2026-11-15 09:05:10 +01:00
Ed Addario 5a4afd8077 tests: Fail test-backend-ops and exit with error status 1 when missing and/or unsupported types and/or unsupported backends are provided in the command line (#30279)
* Fail tests on missing or unsupported type

* Fix output

* Remove whitespace

* Fail if no tests ran regardless of backend support

* Revert unnecessary cosmetic change
2026-11-15 08:26:23 +01:00
shaofeiqi 6d3568e31d opencl: add A7 Q6_K non-MoE binary kernel (#30949) 2026-11-14 21:49:26 -08:00
Tarek Dakhran e574a17b06 model : metal : workaround fusion for LFM3 (#32296)
* lfm3 : put the normed tensor first in residual adds to enable norm fusion

Assisted-by: Claude Opus 5.5

* tests : update Metal fusion baseline for lfm3

Assisted-by: Claude Opus 5.5
2026-11-15 01:05:20 +01:00
Piotr Wilkin (ilintar)andXuan Son Nguyen 5c008e0008 feat: add Inkling 2 support (text + vision) (#32217)
* convert : add Inkling2 text GGUF conversion

Assisted-by: Codex

* llama : implement Inkling2 text inference

Assisted-by: Codex

* tests : cover Inkling2 text support

Assisted-by: Codex

* conversion : dequantize packed Inkling2 experts in row chunks

Decode per-layer NVFP4 and MXFP8 schemes, validate companions, and use stored scale shapes with a warning for the declared MXFP8 [32,32] grid. Reorder packed w13 rows and their scale rows together. Keep normal GGUF outtype selection and 3D expert tensors.

Stream 128 rows per expert through LazyChunkedTensor. Real w13 (384x4608x4608) needs a 2.25 MiB F32 chunk instead of a 30.375 GiB expanded layer. Real-shaped sparse probes peak below 5.8 MiB of Python/NumPy chunk allocations, plus 5.2 MiB of callback metadata: budget about 16 MiB per projection including mapped pages, excluding runtime, other tensors and OS cache. Full-model chunk metadata is roughly 370 MiB.

Packed synthetic GGUF weights match the independent pinned vLLM decode bit-for-bit. CPU prefill/decode logits remain within 3.1e-6 absolute error, with 100% argmax agreement. Float fixtures, original-name conversion and CPU architecture checks pass. Real checkpoint conversion remains untested.

Assisted-by: Codex

* convert : add Inkling2 vision mmproj support

Assisted-by: Codex

* mtmd : implement Inkling2 still-image encoder

Assisted-by: Codex

* inkling2 : scope image loading

Assisted-by: Codex

* inkling2 : enable flash attention with folded bias and sinks

Assisted-by: Codex

* cuda : route per-head flash attention masks through vec

Assisted-by: Codex

* mtmd : support Inkling2 temporal frame means

Assisted-by: Codex

* style : restore shared image comment

Assisted-by: Codex

* tests : enlarge Inkling2 context fixture

Assisted-by: Codex

* inkling2 : load per-layer expert feed-forward widths

Assisted-by: Codex

* tests : add Inkling2 Metal fusion baselines

Assisted-by: Codex

* inkling2 : group and align registry entries

Assisted-by: Fable

* convert : regenerate the Inkling2 tokenizer hash with the update script

Assisted-by: Fable

* conversion : revert the print_registered_models change

Assisted-by: Fable

* conversion : sort the Inkling2 model map entries

Assisted-by: Fable

* convert : map Inkling2 vision checkpoint names in the tensor table

Assisted-by: Fable

* convert : map Inkling2 text checkpoint names in the tensor table

Assisted-by: Fable

* tests : document the Inkling2 packed expert conversion test

Assisted-by: Fable

* tests : cover Inkling2 checkpoint and transformers tensor names

Assisted-by: Fable

* llama : place Inkling2 fields and list entries by file convention

Assisted-by: Fable

* gguf-py : align Inkling2 tensor mapping comments

Assisted-by: Fable

* mtmd : place Inkling2 entries by file convention

Assisted-by: Fable

* tests : place Inkling2 cases after the existing ones

Assisted-by: Fable

* cuda : index per-head flash attention masks in the tile and mma kernels

Assisted-by: Claude
Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q

* inkling2 : project the relative bias before the gather, drop arch branches

Assisted-by: Claude
Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q

* convert : require Inkling2 hparams, add inkling2 pre-tokenizer

Assisted-by: Claude
Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q

* mtmd : cap Inkling2 image size, drop video frame averaging

Assisted-by: Claude
Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q

* inkling2 : fix CI build, Meta device assert and Windows conversion test

Assisted-by: Claude

* tests : use the default sliding window for the Inkling2 test model

Assisted-by: Claude

* simplify mtmd part

* give _v arch for vision

* remove alpha channel handling, it should be a follow up

* explicit rule about audio + vision models

* add audio support

* rm audio

* rm gguf mtmd rope base

* address review comment

* nits

* do not normalize embd from mtmd

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-11-15 00:51:31 +01:00
IsNot 9f26fe11c7 common : reject out-of-range grammar_triggers type instead of aborting (#30835) (#31478)
* common : reject out-of-range grammar_triggers type instead of aborting (#30835)

* tests : remove the invalid grammar trigger type regression test (#30835)
2026-11-15 00:32:37 +01:00
Srijan Keshri cf9a9957da server : Fix connection reuse after streamed responses (#30492)
Assisted-by: Codex
2026-11-15 00:29:28 +01:00
Pranav DhinakarandMax Krasnyansky dca216ad63 hexagon: add TIMESTEP_EMBEDDING HVX kernel (#32228)
* hexagon: add TIMESTEP_EMBEDDING HVX kernel

* hex-timestep: add dma, improve kparams, and work-splits, and tracing

* timestep: add testcases that cover WAN2.2

* hex-timestep: relax the gates to cont_rows

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-11-14 15:03:31 -08:00
Niklas Wenzel f8cc5774e7 memory : fix non-consecutive position warning for M-RoPE (#32287) 2026-11-14 23:57:23 +01:00
Aaron Teo acf5d54562 llamafile: introduce more Q0 quant support for s390x (#32203)
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
2026-11-14 23:38:06 +01:00
Yuri Khrustalev 3e41ae69db llama : allocate the tensor staging buffer from the CPU buffer type (#31438)
* llama : allocate the tensor staging buffer from the CPU buffer type

* llama : drop the staging buffer test, reword the comment
2026-11-14 23:37:46 +01:00
Richard Palethorpe 3bc612fe23 vulkan: Explicitly include SPIRV-Headers (#31416)
* vulkan: Explicitly include SPIRV-Headers

Distros that don't put the headers in a system include path need
to specify it to the compiler. So we use the appropriate CMake
target to include it.

Assisted-by: Codex

* vulkan: Add explicit headers location to workaround OSX CI runner conf
2026-11-14 23:37:14 +01:00
Ruben Ortlam d8f30a5b6d vulkan: optimize prompt processing Flash Attention for RDNA3 and RDNA4 (#31323)
* vulkan: optimize Flash Attention cm1 for RDNA3 and RDNA4

* reduce shmem use through buffer reuse

* clean up and fixes

* limit q_in_regs to prefill

* vulkan: add flipped FA path, each subgroup owns 16 Q rows

* cleanup

* move flip shader into own file

* replace kvsep with direct load mode

* fix nvidia hang, tune AMD

* only store pv in sfsh if necessary

* fix skip condition

* nvidia tuning

* reduce occupancy on RDNA4 for large attention head sizes

* fix sparse FA fp16 underflow bug
2026-11-14 23:36:58 +01:00
Kartik Sirohi e964175653 cuda : keep columns in registers for cooperative softmax (#28982)
Template soft_max_f32_parallelize_cols_single_row to keep column
elements in registers when they fit, reducing memory traffic from
5 global memory passes to 2 passes.

Assisted-by: Gemini 3.8 Flash
2026-11-14 23:36:34 +01:00
Jeff BolzandRuben Ortlam d20426bbbe vulkan: optimize matrix multiplication for narrow N (#31990)
* vulkan: optimize matrix multiplication for narrow N

Add new "narrow" tile configs for coopmat1/coopmat2 that are tuned for N<=32.

* Attempt to handle wave64

* add n=16 tile for cm1

* apply RDNA split_k heuristic to narrow_n generally

---------

Co-authored-by: Ruben Ortlam <rortlam@redhat.com>
2026-11-14 23:36:05 +01:00
Ruben Ortlam 19f31e61b0 vulkan: use GPU copies on UMA with uncached memory on Intel (#32024) 2026-11-14 23:35:54 +01:00
Nicolas Mowen 07037ba133 sycl : optimize mtmd image processing (#32109) 2026-11-14 23:35:16 +01:00
Tarek Dakhran 0cc522e45d chat : allow undeclared tool calls in LFM2 tool-call grammar (#32164)
* chat : honor OpenAI tool strict flag in LFM2 tool-call grammar


* chat : drop strict, always allow undeclared LFM2 tool calls
2026-11-14 23:34:18 +01:00
Takamichi Ohara f08bca8a46 docs : add OpenVINO backend to ops matrix (#32259) 2026-11-14 23:33:55 +01:00
3c311796f2 chat : allow tool calls inside reasoning blocks for Ministral-3 (#32271)
* chat: allow tool calls inside reasoning blocks for Ministral-3 and Qwen3

Fixes #29248
Fixes #20837

* chat: address review feedback for Ministral-3 reasoning end

- Remove <function= from Qwen3 (intentionally not used outside outdated coder models)
- Ministral-3: use choice between [/THINK] and peek([TOOL_CALLS]) instead of optional

* chat: add [TOOL_CALLS] to Ministral-3 thinking_end_tags

* chat: simplify Ministral-3 reasoning end choice

* chat: restore peek lookahead for Ministral-3 tool call start

* chat: keep [TOOL_CALLS] only in parser, not in thinking_end_tags

* cont : revert tool call tag removal and update tests

---------

Co-authored-by: Alen Požun <kukirin-ui@users.noreply.github.com>
Co-authored-by: Alde Rojas <hello@alde.dev>
2026-11-14 23:33:35 +01:00
Alex 8b4ac2496c common : fix infinite recursion in debug build for peg parser dump (#32282)
Change dump()->dump_impl() calls to avoid infinite recursion.
2026-11-14 23:33:21 +01:00
lhez 1ae047c926 opencl: fixes for q4_k mv oob access, f16 swiglu_oai, cpy with strided dst, batched group_norm and bf16 sigmoid (#32251)
* opencl: fix q4_K mv oob read

Assisted-by: Codex

* opencl: fix kernel dispatch and oob writes for  f16 swiglu_oai

Assisted-by: Codex

* opencl: fix the row broadcast for add/mul

Assisted-by: Codex

* opencl: respect destination tensor strides for cpy

Assisted-by: Codex

* opencl: handle batches in ne3 for group_norm

Assisted-by: Codex

* opencl: avoid double rounding for bf16 sigmoid

Assisted-by: Codex
2026-11-14 23:32:48 +01:00
Cristian Adam 1c2adc75f6 chat : allow Gemma4 tool call during thought (#32232)
The thought channel is normally closed with <channel|>, but the model
may open it and go straight to a tool call without closing it. Stop the
reasoning at <channel|> OR <|tool_call>, so an unclosed thought does
not swallow the following tool call.

Fixes: #32231

Assisted-by: Qwen 3.8 27b
2026-11-14 23:32:19 +01:00
miyan a956a852c9 Fix ctz64 on 32bit (#32273) 2026-11-14 19:06:52 +01:00
Jhen-Jie Hong c6c0ec6d5c hexagon: fix fa no-mask tail (#32243) 2026-11-14 10:06:01 -08:00
Aman Gupta d0764c5ee2 qwen2vl : apply FFN weight scales (#32241)
The qwen2vl graph passed NULL for the ffn_up/gate/down scales, so the
per-tensor NVFP4 scales loaded by the generic pass were never applied
and the model produced garbage (PPL ~1e6 on CUDA, NaN on CPU).

Pass the scales to build_ffn like qwen2 does.

Fixes #32101
2026-11-15 02:02:15 +08:00
Matt Rutkowski 3549c274dc convert : register Granite4VisionForConditionalGeneration for text (#32237)
Signed-off-by: Matt Rutkowski <mrutkows@us.ibm.com>
2026-11-14 15:19:01 +01:00
Aman Gupta c56be864f4 ggml-backend : start a new split at an incompatible weight in a split without inputs (#31775) 2026-11-14 22:14:48 +08:00
Jonathan BuckandJonathan Buck a9a5d216e6 convert : pplx-decider must match qwen3.5 model_type (#32156)
* convert : require matcher to check backbone definition (model_type) in config.json

Fixes #32152

* Revert the modified llama_test_cmd_entry.

---------

Co-authored-by: Jonathan Buck <jonabuck@amazon.com>
2026-11-14 15:05:30 +01:00
Xuan-Son Nguyen 9d925e7c02 mtmd: fix video pixel aspect ratio and video rotation (#32267) 2026-11-14 14:57:21 +01:00
Jhen-Jie HongandXuan Son Nguyen e3483e40b0 mtmd : allow custom video decoder (#32188)
* mtmd : allow custom video decoder

* also adapt to wllama requirements

* clean up

* add test

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-11-14 14:12:07 +01:00
HaoJun ZHANGandXuan Son Nguyen 4f2187e73d jinja : reduce template rendering overhead (#32023)
* jinja : reduce value allocation and lookup overhead

Assisted-by: GPT-6 Astra

* jinja : address rendering review feedback

Assisted-by: GPT-6 Astra

* jinja : test multipart strings and simplify lookup

Assisted-by: GPT-6 Astra

* fix "undefined hint"

* fix divisibleby 0

* clean up

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-11-14 13:34:20 +01:00
Hera 0cce7580b8 mtmd : avoid padding-only Whisper audio chunks (#30403)
Assisted-by: Claude Opus 5
2026-11-14 13:33:59 +01:00
Ruixiang Wang 32bdc3a89e spec + model: refactor DFlash based speculative decoding methods (#32016)
* dflash : move the DSpark markov head and backbone graphs to dspark.cpp

Move-only: build_dspark_markov_head, graph_dsv4, graph_dsv41 and
graph_mistral_large4 move verbatim. build_inp_target_features and
build_dspark_markov_head become static members of llama_model_dflash so
both files can use them.

* dflash : detect the head and the backbone once, split the loaders per file

load_arch_hparams sets llama_model_dflash::variant (DFlash, DFlash2,
DSpark, LiliCorr) and ::backbone (base, DSV4, DSV4.1, Mistral Large 4).
The head, backbone and base layer loaders become member helpers in
dflash.cpp, dflash2.cpp, lilicorr.cpp and dspark.cpp. build_arch_graph
dispatches on the backbone.

The DFlash2 and LiliCorr loaders create the DFlash2 conv tensors through
a helper in dflash2.cpp, right after their head tensors instead of in
the base layer loop; the other tensors keep their creation order. The
DFlash2 loader drops its copy of the conv group size check, which master
never reached.

* dflash : share the injection input, token embeddings and lm head lookups

build_inject_inp (target features + fc + encoder norm), get_tok_embd and
get_lm_head replace the copies in the base, DSV4, DSV4.1 and Mistral
Large 4 graphs. build_inp_target_features is file-static again.

* dflash : split the DFlash decoder into a graph per head

graph_base creates the shared inputs and provides build_inject,
build_inp_tokens, build_layers and build_logits. graph_dflash
(dflash.cpp), graph_dflash2 (dflash2.cpp), graph_dspark (dspark.cpp) and
graph_lilicorr (lilicorr.cpp) assemble them and add their head; the
selector, scorer and markov head builders move to the file of their
graph, and the DFlash2 local conv run by build_layers to dflash2.cpp.
build_arch_graph dispatches on the backbone, then on the head.

The op order and the tensor names are unchanged.

* speculative : split the DFlash driver into a class per head

common_speculative_impl_draft_dflash stays the DFlash implementation and
becomes the base of draft_dspark, draft_dflash2 and draft_lilicorr. Its
protected constructor only reads the draft metadata; init() then sets up
the contexts with the options of the draft (draft length, backend
sampling, device features, masked h_nextn). Each derived class overrides
draft() with the path of its head from the former draft(). DFlash2 also
overrides begin() to reseed its generator per request, and LiliCorr
overrides process() to store the context features of each injected
chunk, which its noise block carries as its state.

The factory keeps the current routing: draft-dspark picks DSpark,
draft-dflash picks DFlash2 or LiliCorr from the draft metadata and
DFlash otherwise. draft-dspark with a DFlash2 or LiliCorr draft now runs
the DSpark reader instead of a mix of both. sample_from_anchor is logged
by DSpark only.

* speculative : add the draft-dflash2 and draft-lilicorr types

draft-dflash2 and draft-lilicorr name the DFlash2 and LiliCorr drafts,
which draft-dflash keeps running too: it picks them from the draft
metadata. draft-dflash2 and draft-lilicorr must match the draft model,
draft-dspark rejects a DFlash2 or LiliCorr draft, a model that is not a
DFlash draft is rejected, and a run uses at most one DFlash family type;
these errors leave speculative decoding uninitialized.

The type detected from the draft GGUF is now the concrete one, and
draft-dflash2 and draft-lilicorr download the dflash sidecar.

* docs : describe the DFlash2 and LiliCorr drafts

Add the DFlash2 and LiliCorr sections and the new types, and explain
that draft-dflash also runs DFlash2 and LiliCorr drafts.

* speculative : drop the unused DFlash is_mrope

Nothing reads it since the migration to common_batch (#29385), which
picks the number of M-RoPE position rows from the rope type of the draft.

* doc: update spec dec

* address review comments
2026-11-14 09:49:27 +01:00
Ehsan BateniandMax Krasnyansky 61dbf49228 Hexagon: add support for IQ2_XXS and NVFP4 (#31954)
* ggml-hexagon: IQ2_XXS and NVFP4 tiled repack layouts

* ggml-hexagon: IQ2_XXS HVX matmul kernel

* ggml-hexagon: NVFP4 HVX matmul kernel

* ggml-hexagon: NVFP4 per-tensor scale

* ggml-hexagon: HMX paths for IQ2_XXS and NVFP4

* ggml-hexagon: supports_op coverage for NVFP4 and IQ2_XXS

* fix HMX IQ2_XXS dequant scale underflow in fp16

* hexagon: IQ2_XXS 2-gather tile decode with compact grid, small-weight tests

* IQ2_XXS vpopcount sign parity and gather pipelining

* pipeline IQ2_XXS HMX dequant gathers and add large_w tests

* hex-iq2-xxs: rebase and cleanup

Remove unused helper functions.
Add lut_size to the overall vtcm required size.
Trim down (way way down) the new tests.

* hex-iq2-xxs: actually trim down the tests

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-11-13 20:57:05 -08:00
Aldehir Rojas 9d25e0014c chat : remove mistral workaround (#32230) 2026-11-13 21:51:44 -06:00
Aman Chadha(IVIXMMI) 9183a552e5 metal : fix mul_mv destination offset overflow (#32221) 2026-11-14 07:49:32 +08:00
Todor BoinovskiandMax Krasnyansky ded522f830 hexagon: add support for SSM_SCAN op (#31921)
* hexagon: add SSM_SCAN Op support

* hex-ssm-scan: fix multi-device spliting

* hex-ssm-scan: remove redundant checks and make kernel use host config

* hex-ssm-scan: remove direct ddr access

* hex-ssm-scan: remove DMA ordering

* hex-ops: update supported ops

* hex-ssm-scan: rewrite the dma pipeline to match other ops (dbl.buf, etc)

* hex-ssm-scan: remove all fallbacks, only dma/vtcm paths

* hex-ssm-scan: add kernel_type to kparams and remove redundant recomputes

* hex-ssm-scan: avoid stalls in the overall pipeline (dma flush between chunks)

* hex-ssm-scan: vectorize softplus

* hex-ssm-scan: optimize hvx loops

* hex-ssm-scan: add HMX support

* hex-ssm-scan: add support for GGML_HEXAGON_SSM_SELECT

* hex-ssm-scan: cleanup kernel names

* hex-ssm-scan: simplify mdev checks

* hex-mdev: improve mdev row-split solver

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-11-13 12:27:51 -08:00
AMIR 5193ef1d4f ui: configure allow-remote=root in .npmrc for npm 12+ (#32158) (#32196) 2026-11-13 20:54:49 +01:00
Max Krasnyansky 4df9b47976 ci: reduce qdc timeouts to avoid stalling pipelines and runners (#32148)
* ci: reduce qdc timeouts to avoid stalling pipelines and runners

* ci: fix flake8 warnings
2026-11-13 11:49:33 -08:00
thelittlefireman b79696f1f5 ggml-cuda: hip: tune GCN MMQ configs (#30021) 2026-11-13 18:12:51 +01:00
Pascal 4f9dfa0e36 requirements, Fix CI: support Python 3.15 (#32199)
* requirements : support Python 3.15

torch 2.11.0, numpy 2.4, matplotlib 3.10, pandas 2.3, pillow 11.3 and
cffi 1.16 have no Python 3.15 wheels, so the CI runner on 3.15 cannot
install the requirements.

Python 3.15 gets the first releases that ship 3.15 wheels through
version markers, and every version below 3.15 stays as is.
pyproject.toml now allows 3.15 and picks torch 2.15.0 there.

* requirements : apply review suggestions

numpy 2.5.4 from Python 3.14, matplotlib 3.11.2 and pandas 3.0.6 from
3.11, cffi 2.1.1 and pillow 12.3.0 for every version. cffi 1.16.0 has
no 3.14 wheel and fails to build there, so this also fixes the
tokenizer test requirements on 3.14.
2026-11-13 17:50:03 +01:00
Max Krasnyansky 9daa0aa22b hexagon: improve buffer allocations and mappings (mmap stacking) (#32144)
* hex-bufs: combine queue and fence buffers

* hex-bufs: use n_threads for queue sizing

* hex-mmap: optimize buffer stacking at the host to avoid iova fragmentation

* hex-mmap: add simply retry for spurios mmap failures

* hex-mmap: use eager mode for extended mmap retries
2026-11-13 07:56:20 -08:00
ynankani fe14cad51b CUDA: add SM103 SASS to non-native builds (#30099)
Signed-off-by: ynankani <ynankani@nvidia.com>
2026-11-13 20:32:24 +05:30
anujj 31953bcf74 ggml : add BF16 support to RMS_NORM (CPU, CUDA) (#32135)
* ggml : add BF16 support to RMS_NORM (CPU, CUDA)

* cuda : use one helper for the F32/BF16 RMS_NORM launches
2026-11-13 15:41:17 +02:00
Yonghye Kwon ae93877c8f vulkan: add TOP_K_MASK (#31544)
* vulkan: add TOP_K_MASK

Assisted-by: Claude Opus 5.5

* vulkan: fix the TOP_K_MASK barrier and use the common offset helper

Assisted-by: Claude Opus 5.5

* vulkan: declare ggml_vk_top_k_mask in the common header

Assisted-by: Claude Opus 5.5
2026-11-13 14:02:36 +01:00
Pranesh GonegandlaandPranesh Gonegandla 7874988b45 spec : Warn when --spec-draft-p-min is used with probabilistic draft sampling (#31294)
* spec : skip p_min for sampled MTP and DFlash2 drafts

Under rejection sampling the acceptance at a position is 1 - TV(p, q),
which the draft's top probability does not tell, so the p_min cutoff
dropped draft tokens the target would have accepted. Skip it when
MTP or DFlash2 samples the draft. draft-simple keeps it: each extra
draft token costs a full draft model pass.

* spec : replace the sampled-draft p_min skip with a warning

p_min behaves as on master again for every drafter and sampling mode.
Instead, log a warning when --spec-draft-p-min > 0 is used with
--spec-draft-sampling probabilistic, suggesting p_min 0: under
rejection sampling the draft's top probability does not tell how much
of the draft the target accepts. The cut is not skipped because with
MTP at long drafts it can still pay off.

* spec : warn about p_min only for drafters that use it

The n-gram drafters ignore p_min, and DSpark uses it as the cutoff of its confidence head, so the warning is skipped when only these are configured.

---------

Co-authored-by: Pranesh Gonegandla <pgonegandla@nvidia.com>
2026-11-13 13:25:32 +01:00
8c2c4c39c5 ggml-vulkan: add optional device keepalive (#29267)
* ggml-vulkan: add optional device keepalive

* simplify implementation, move ownership to backend, change env var to seconds

---------

Co-authored-by: Acmmi <acmmi@Acmmis-MacBook-Air.local>
Co-authored-by: Ruben Ortlam <rortlam@redhat.com>
2026-11-13 13:53:55 +02:00
o7siandPiotr Wilkin 108f73adfe grammar : optimize matching with left factoring (#29557)
* grammar : limit the number of alternatives

* grammar : introduce an AST for grammar parsing

* grammar : revert AST refactor and alternatives limit

This reverts commits:
- 84f3b57c4d6c482e7187a30735d16b02de3784fa
- 393a707ebb7e6f798abdb8af34cb1bc102f4aaa6

* grammar: optimize big grammar matching by using left-factorization

* grammar : preserve character class boundaries during left factoring

* grammar : prevent infinite expansion during left factoring

* grammar : use ASCII characters in comments

* grammar : test empty suffixes after left factoring

* grammar : deduplicate left factoring logic

* grammar : add comments

* grammar : use number parameters and clarify RSS output

* grammar : clarify character class test names

* grammar : remove performance benchmark

* grammar : share immutable rules

---------

Co-authored-by: Piotr Wilkin <piotr.wilkin@syndatis.com>
2026-11-13 13:52:21 +02:00
Aman Gupta 951d465f5c llama: move moe-cache code out of llama-graph (#32157) 2026-11-13 18:48:29 +08:00
Pascal 2d8a325029 CUDA: split the PAD_REFLECT_1D grid to stay within the HIP grid limit (#32179)
The kernel launches one block per row and 256-wide tile, so the
{2, 16777217} test case asks for 16777217 blocks of 256 threads. HIP
refuses a grid dimension above 2^32 threads and the ROCm jobs fail
with "invalid configuration argument".

The host now launches the grid in chunks of at most UINT32_MAX / 256
blocks and passes the chunk offset to the kernel. Real sizes fit in a
single chunk and keep their timings.
2026-11-13 11:15:25 +02:00
Daniel Bevenius b1b770d90d examples : remove explict version from test-cmake (#32191)
This commit removes the llama version test-cmake and instead uses the
version that is supplied by CMAKE_PREFIX_PATH.

The motivation for this is to avoid issues when bumping major versions.
2026-11-13 11:09:55 +02:00
Georgi Gerganov 4cda6227a0 llama.cpp : bump version to 1.0.0 (#32190)
* llama.cpp : bump version to 1.0.0

* readme : bump major version in link

[no ci]

* examples : bump test-cmake version

[no ci]
2026-11-13 11:06:06 +02:00
Georgi Gerganov 6859813352 sync : ggml
[no ci]
2026-11-13 10:37:44 +02:00
Georgi Gerganov 3bfde28f91 ggml : bump version to 1.0.0 (ggml/1690)
* ggml : bump version to 1.0.0

* examples : bump test-cmake (ggml/0)
2026-11-13 10:37:44 +02:00
Georgi Gerganov 28d37fe2e2 examples : clean-up old/obsolete stuff (#32184)
* examples : clean-up old/obsolete stuff

* examples : update references to removed files

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)

[no ci]
2026-11-13 10:33:17 +02:00
Pascal bd2767d06b ci: fix HIP quality check by ignoring the FP8 vec FA kernel spill (#32183)
The D=256 vec flash attention kernel with F8_E4M3 K and V uses 257
VGPRs (255 + 2 spilled) on gfx908, one over the limit of the check.
The spill is a perf issue only, like the Q4_0 D=256 vec kernel
already in the ignore list.

Both variants of the kernel join the ignore list.

[no ci]
2026-11-13 10:25:08 +02:00
Sigbjørn Skjæret 82503b42fa ci : set windows-cuda ccache size to 1G (#32185) 2026-11-13 08:53:37 +01:00
Awss 5676dfbe01 chat : accept </ifm|think> as K2 Horizon closer at medium/low effort (#30335) 2026-11-13 00:11:58 +01:00
Pascal 0a66ac7189 server, fix CI: read a small temp file in the read_file tool test [no release] (#32168)
* server: read a small temp file in the read_file tool test

test_tools_builtin_read_file read its own file, which grows with every
added test. On Windows the CRLF checkout puts it at 16664 bytes, over
the 16384 byte read limit, and the tool refuses it.

The test now writes a small file of its own and checks the exact
content read back.

* server: use a temp cwd in the x-tool-cwd tool test

test_tools_builtin_cwd_header read test_tools_builtin.py through the
x-tool-cwd header, which hits the same 16384 byte read limit on the
Windows CRLF checkout, and wrote its shell marker into the source tree.

The test now uses a temp directory as cwd, reads a small file from it
and checks the shell marker lands there.
2026-11-12 23:47:14 +01:00
Pascal 892b40b910 server: allow private media URLs in the vision tests [no ci] (#32167)
The vision tests download their images from huggingface.co, which
resolves to a private address on the runners inside the HF network.
The media URL check refuses it with "host is not a public address"
and the server-cuda job fails.

The vision fixture enables media_allow_private_urls. The private
address check keeps its own coverage in
test_vision_image_url_private_address.
2026-11-12 23:15:28 +01:00
Pascal e970101470 server: skip LAN address tests when the host cannot reach it [no ci] (#32166)
The same-origin tests over the LAN address block forever on the Metal
runners: the request to the host's own LAN address never returns and
has no timeout, so the job runs until the 6h limit.

get_lan_ip() now probes /health on that address with a short timeout
and returns None when the host cannot reach it, which skips the test.
The test requests themselves use the default request timeout so a
hang fails the test instead of stalling the job.
2026-11-12 23:13:37 +01:00
SaidSultaniandMax Krasnyansky 92d6c8a5ca fix(hexagon): make host build work with clang-cl on Windows ARM64 (#32011)
* fix(hexagon): make host build work with clang-cl on Windows ARM64

Two CMake fixes for building ggml-hexagon natively on Windows-on-ARM64
with x64-hosted clang-cl (validated on Snapdragon X Elite, discussion in
qualcomm/GenieX#1390):

- Some clang drivers (e.g. x64-hosted toolchains used on ARM64) don't
  predefine the arch macro, which makes winnt.h fail with "No Target
  Architecture". Define _M_ARM64/_ARM64_ explicitly for the host TUs.
- The Hexagon SDK's incs/stddef include (AEEStdDef.h and friends) poisons
  the MSVC header chain when it leaks onto dependent TUs' include paths
  via htp_iface's PUBLIC includes. Keep it PRIVATE to the interface stub
  and system-level for the backend's own TUs (which include <AEEStdErr.h>
  directly).

Originally validated 2026-08-26 as reported in the issue; applies
cleanly to 15759e0.

* hex-cmake: trim down excessive comments

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-11-12 13:52:56 -08:00
Georgi Gerganov 8ca5a8008b llama : prune (#30565)
* llama : remove deprecated `llama_set_warmup`

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)

* llama : remove remaining deprecated API

Drop the APIs marked DEPRECATED in llama.h, the deprecated
llama_context_params::defrag_thold field and the --defrag-thold option.

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)

* fit-params : remove stale `llama-ext.h` include

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)

* llama : document `llama_model_dflash_selector_top_k`

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)

* llama : promote the NextN / speculative API from `llama-ext.h`

Move the NextN, layer input, embd row and state row APIs to `llama.h`
and rename `llama_set_embd_state_src` to `llama_set_ctx_state_src`, with
the internal `llama_context::set_state_src` following the same name.

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)

* llama : polish the NextN API promotion

Move the NextN draft model queries next to the Adapters section, rename
the `src` parameters of `llama_set_ctx_state_src` and
`llama_context::set_ctx_state_src` to `ctx_src`, and drop the now-redundant
`llama-ext.h` includes from `common/speculative.cpp` and `tools/imatrix`.

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
2026-11-12 20:59:09 +02:00
Georgi Gerganovandlosewayy 29dbbad2b1 ggml : prune (#29801)
* ggml : prune deprecated API

* cont : bump RPC version

* cont : minor

* ggml : remove outdated documentation from ggml.h

The 172-line comment block at the top of the header has not been updated in a
substantive way since the backend v2 split (Nov 2023) and now documents APIs
that no longer exist:

- `ggml_set_f32()`, `ggml_get_f32_1d()`, `ggml_set_f32_1d()` were removed
- `ggml_conv_1d_1s()`, `ggml_conv_1d_2s()` ops no longer exist
- `ggml_set_param()` no longer takes a context argument
- `ggml_graph_compute_with_ctx()` moved to the CPU backend header
- the optimization API now lives in `ggml-opt.h`

The block also ended with 5 empty `TODO` sections.

Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)

* ggml : remove DIAG_MASK_INF and DIAG_MASK_ZERO (#31946)

* rpc : bump RPC protocol major version

The tensor op enum is sent over the wire, so removing GGML_OP_ADD1,
GGML_OP_DIAG_MASK_INF and GGML_OP_DIAG_MASK_ZERO shifts the value of every
op that follows them and peers built before and after this change would
misinterpret each other's ops.

The handshake compares the major version for exact equality and ignores the
patch version, so bumping the major version is what rejects incompatible
peers instead of silently misbehaving.

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)

---------

Co-authored-by: losewayy <1134285612@qq.com>
2026-11-12 20:58:24 +02:00
Aparna M P 2a4daf0e08 hexagon: add Q5_0 and Q5_1 support (#32094)
* hexagon: add q5_0 and q5_1 support

* hex-mm: dedicated q5_0 kernels
2026-11-12 08:33:16 -08:00
Hamish M. Blair d8c8cc4d9b server : cancel MCP tool calls on client disconnect or timeout (#32025)
* server : cancel MCP tool calls on client disconnect or timeout

* server : shorten MCP cancellation tests and fix a type-check error
2026-11-12 17:19:26 +01:00
Oliver Simons 3ba1ce0e28 qwen3.5: merge tensors on convert also (#32102)
* qwen3.5: merge tensors on convert also

* Remove conversion test for qwen3.5
2026-11-12 15:54:58 +01:00
anujj 0fb8f15770 Title: CUDA: few-row MMQ for BS 2-24 (#31533)
* CUDA: few-row MMQ for 2-24 src1 columns

New int8 MMA kernel for quantized mat-muls with 2-24 src1 columns, e.g. speculative decoding verify passes.
Supports Q8_0, Q3_K, Q4_K, Q5_K, Q6_K, IQ3_S, IQ4_NL and IQ4_XS on NVIDIA GPUs from Ampere on.

* nit fixes

* CUDA-only guard for few-row MMQ, weights flag in test_mul_mat

* few-row quantize: DS4 sum from the quantized values, as in #31247

* few-row MMQ: clearer HIP/MUSA abort, test_mul_mat two-context build

* few-row MMQ: dispatch order MMVQ, few-row, MMQ
2026-11-12 21:12:45 +08:00
Sigbjørn Skjæret 827466b9ef ci : build ui on non-slim runner (#32138) 2026-11-12 12:35:31 +01:00
Xuan-Son Nguyen 8954f19483 server: fix SSRF for CORS proxy and media URLs (#32115) 2026-11-12 12:33:48 +01:00
Xuan-Son Nguyen a1a68eb797 server: explicit opt-in tools via non-loopback addr, check cors host port (#32113)
* server: explicit opt-in tools via non-loopback addr, check host port

* fix ci
2026-11-12 12:16:50 +01:00
Xuan-Son Nguyen e182fdad4a server: disallow direct access child instance from browser (#32120) 2026-11-12 12:16:29 +01:00
PascalandVenkata Vineel 09452cdedd server: keep the server awake while a request uses its state (#32133)
With --sleep-idle-seconds, a handler passes wait_until_no_sleep()
while the server is awake, then keeps using the server state, for
example to parse the body and tokenize. If the idle timer expires in
that window, the main loop enters sleep and the callbacks destroy the
model: the handler reads freed vocab memory, and a task posted after
that stays in the queue with nobody awake to process it.

Count the requests that passed wait_until_no_sleep() and are not done
yet, and do not enter sleep while the count is positive. The response
generator releases its count in its destructor, so every return path
of a handler gives it back.

Co-authored-by: Venkata Vineel <vyalamar@users.noreply.github.com>
2026-11-12 12:16:16 +01:00
Pascal cfb9d4b754 mtmd: require an F32 position embedding for HunyuanVL (#32131)
HunyuanVL resamples its position embedding on the host into a float
buffer sized for F32, then reads ggml_nbytes() of the tensor into it.
The graph never uses the tensor, so any type loads, and an F64 tensor
overflows the buffer at the first image request.

Require the position embedding to be present and F32 when loading the
HunyuanVL tensors.
2026-11-12 12:15:28 +01:00
Pascal 72f4603c7c mtmd: read the MiniCPM-V projection width from ne[1] (#32132)
The MiniCPM-V resampler ends with build_mm(mm_model_proj), whose output
width is ne[1], while n_mmproj_embd() returns ne[0]. Both are equal in
real models, the projection is square. A non square projection passes
the text model n_embd check and only aborts at the encode, on the
output buffer size check.

Return ne[1], so the width matches the graph output.
2026-11-12 12:14:50 +01:00
Pascal 8c388364e1 mtmd: require the MLP projector and patch embedding tensors (#32129)
* mtmd: require the MLP projector and patch embedding tensors

The loader marks the patch embedding and the LLaVA projector tensors
as optional, while the graphs and n_mmproj_embd() dereference them
unconditionally. A GGUF without them crashes on a null tensor during
the warmup.

Load the patch embedding as required for the vision encoders that
start with it, which is all of them except GEMMA3NV, DEEPSEEKOCR and
DEEPSEEKOCR2. Load mm.0 and mm.2 as required for MLP, and mm.0, mm.1,
mm.3 and mm.4 for the Yi-type MLP_NORM variant, selected by mm.3.

The graph no longer tests mm.2 and the yasa2 assert on the patch
embedding is covered by the loader.

* mtmd: keep the Yi-type llava TODO

The Yi-type variant is still detected by the presence of mm.3, the
TODO on that detection stays.
2026-11-12 12:14:36 +01:00
Pascal 08f49598c1 mtmd: require a 2x2 spatial merge for the Qwen-VL projectors (#32130)
The Qwen2-VL, Qwen2.5-VL, Qwen3-VL and Ling3-VL graphs build their
input with a temporal merge that assumes a 2x2 patch grid, and the
Qwen2-VL merger reshapes by 4. Any other spatial_merge_size from the
GGUF reaches a GGML_ASSERT at the first image request and aborts the
server.

Reject a spatial_merge_size other than 2 when loading the hparams.
2026-11-12 12:14:10 +01:00
Kong Jiale a8575a3ef4 ggml : route GATED_DELTA_NET to the SpacemiT RVV kernel (#31897)
The SpacemiT compute case was commented out, so the op stayed on the scalar CPU path. Type dispatch stays with the other ops, and use_ref keeps the reference path on the generic CPU kernel.

Assisted-by: Codex

Signed-off-by: gitGurugu <2049447146@qq.com>
Signed-off-by: Wang Yang <yangwang@iscas.ac.cn>
2026-11-12 12:20:25 +02:00
sliu39 a963f65957 vulkan : Intel FA kernel for prefill. (#29357)
* vulkan : Intel customized FA shader for prefill usage

* vulkan : adding check of max_bias/logit_softcap for Intel FA kernel path

* vulkan : add m16n16k16/SIMD32 support for Xe FA prefill shader

* vulkan : Intel FA shader and host refinement, with accuracy issues fixes

* vulkan : Limit the SLM usage to 48 KiB for Intel prefill FA shader

* vulkan : Update Intel FA shaders to handle cases with large head dimension K, unaligned Q head stride and multi head mask cases

* vulkan : Intel FA shader and host refinement to align with existing design, extreme cases handling, shape check relaxing.

* vulkan : Intel FA shader and host update to support q as tensor view, unaligned batch stride q. Add env variable for enforcing XE FA path for other vendor GPUs with 16x16x16 CM1 and SIMD32 support

* vulkan : Inte FA shader and host update to fix the potential SLM over 48 KiB issue, enhance the buffer access robust and add asymmetric qk/pv ration support.

* vulkan : Inte FA shader and host update to add SLM 48 KiB size check, remove unnecessary extensions and finetune the unroll/dont_unroll loops

* vulkan : Intel FA shader and host update to add SLM size check function, use real device max SLM size in shader specialization constants.

* vulkan : Intel FA shader and host update to consolidate host helper function, remove redundant buffer sync, bypass unused pipeline and buffer, fix subgroup shmem conflict and add negative case handling.

* vulkan : Intel FA host update to correct get_xe_fa_tuning_params() function log message, and add workgroupMemoryExplicitLayout/workgroupMemoryExplicitLayout16BitAccess prerequisite‌ feature check.
2026-11-12 11:19:17 +01:00
Marco Colombo ab4a77b9ad ggml-cpu: use the 4x4 simd_gemm tile on ARM NEON (#32128)
Assisted-by: Claude Fable 5.2
2026-11-12 12:13:59 +02:00
Konrad Moren 0d69255c52 CUDA: Optimize prefil via fuse of w_s scale in epilogue MMQ for nvfp4 checkpoints (#26048)
* ggml-cuda: Apply NVFP4 weight scale in MMQ write-back

* ggml-cuda: Fuse mul_mat(_id) + scale into MMQ

* ggml-cuda: Tests for the fused scale

* ggml-cuda: reuse existing graph to test fusion in mmq epiloque

* ggml-cuda: rename test_mul_mat_vec_fusion to test_mul_mat_fusion for consistency

* ggml-cuda: refine comments for clarity on MMQ dispatch and fusion checks in mul_mat functions

* ggml-cuda: refactor fusion checks for mul_mat operations and remove unused function
2026-11-12 10:28:36 +01:00
Aman Gupta 3bce8bd50c ggml-backend : prefetch only inputs computed by earlier splits (#32100)
* ggml-backend : prefetch only inputs computed by earlier splits

The copy callback of a split runs after the prefetch and can write tensors that a later split reads, e.g. the MoE cache uploads missed experts into its bank. A prefetch of such a tensor reads it before the write.

* llama : skip the MoE cache when its device cannot run the cached mul_mat_id

Otherwise the scheduler runs the matmul on another backend and copies the whole bank every ubatch.
2026-11-12 17:24:40 +08:00
Kong Jiale 81f21f4b27 ggml : dispatch SpacemiT Q4_1 32x256 through the HP kernel (#31929)
The 32x256 pack already keeps a separate uint8 zero point. Complete the M1 path and use the same HP kernel as Q4_0.

Assisted-by: Codex

Signed-off-by: gitGurugu <2049447146@qq.com>
Signed-off-by: Wang Yang <yangwang@iscas.ac.cn>
2026-11-12 10:59:09 +02:00
Xuan-Son NguyenandAleksander Grygier b19f1d9e63 ui: add search_chats tool (#32051)
* ui: add search_chats tool

* add from_timestamp / to_timestamp

* ui: search_chats review fixes

- keep search_chats disabled on import, copy tool policy on fork
- stream messages without attachment payloads, one transaction for time ranges
- keep the output within budget, cut whole messages from the older end
- reject impossible date-times, quote titles and attachment names
- typed readOnly tables, move localIsoString to formatters as formatLocalIso

Assisted-by: Claude (Opus 5.5 Medium)

---------

Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>
2026-11-12 09:44:35 +01:00
Aleksander Grygier d90cfd4a9b ui: install llama-brand from a GitHub tarball instead of git (#32112)
npm 11.20 refuses git dependencies by default (EALLOWGIT), so npm ci
failed on fresh installs. Pin llama-brand to the same commit through
an HTTPS tarball, which also adds an integrity hash to the lockfile.

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-12 09:36:28 +01:00
Xuan-Son Nguyen 60d46a5abe server: when --tools-runtime is specified, ignore x-tool-runtime (#32119) 2026-11-12 09:34:16 +01:00
Aldehir Rojas 93cf736a8d chat : refactor gemma4 to build loose schema rules and clean up (#31915)
* chat : build a schema-shaped PEG parser for Gemma 4 tool arguments

* chat : enforce only the key structure of Gemma 4 tool arguments

* chat : remove old gemma4 workarounds

* cont : add node.as<>() to simplify static casts

* cont : add depth_guard and use numerical rule names

* cont : clean up

* cont : extract gemma 4 parsers

* cont : clean up

* cont : clean up

* cont : remove dynamic cast
2026-11-12 09:44:42 +02:00
Joel Fernandes 4fffab623e llama : avoid touching unused graph metadata pages (#32043)
Each graph result reserves metadata for max_nodes, but many graphs use
only a fraction of that capacity.  The vector resize initializes the
whole reserve and makes otherwise unused pages resident.  Allocate the
reserve with new uint8_t[] instead.
2026-11-12 09:42:45 +02:00
Titaniumtown 09338b85e8 sycl: make oneDNN FA cache thread-safe and memoize runtime failures (#31138) 2026-11-12 08:58:26 +02:00
Neo Zhang dcbff21661 sycl : fix AOT for more than 4GB (#31873)
* fix aot for >4g garbage output issue; add --aot in build script

* correct the aot flag

* support no-xmx build, update the code for no-xmx building, rename nossl to no-ssl
2026-11-12 08:54:58 +02:00
zhaoyshineandzhaoyshine f3d7725a77 sycl : use cached device-id in set_main_device (#32013)
ggml_sycl_set_main_device() runs at the start of every graph compute. Its
early-return check called dpct::get_current_device_id(), which locks the
device manager mutex and looks up the thread-to-device map. Use the
thread-local cached id that the rest of the backend already uses.

Assisted-by: Claude Code

Co-authored-by: zhaoyshine <zhaoyshine@users.noreply.github.com>
2026-11-12 08:50:55 +02:00
Joseph Patricio 7021f7e099 CUDA: add cuda debug buffer pool (#31919)
* CUDA: add cuda debug buffer pool

* CUDA: add code comment for debug pool with compute-sanitizer
2026-11-12 08:50:32 +02:00
shaofeiqiandLi He e0949d09b1 opencl: add bin kernel kernel_gemm_noshuffle_q4_0_f32_32b_trans_ila_a7_bin (#30947)
* opencl: add A7 Q4_0 non-MoE binary kernel

* opencl: refactor

---------

Co-authored-by: Li He <lih@qti.qualcomm.com>
2026-11-11 21:53:51 -08:00
Feiteng 5d4c8eeb24 CUDA: skip empty MoE tiles in MMQ stream-k fixup (#32067)
* CUDA: skip empty MoE tiles in MMQ stream-k fixup

mul_mat_q_stream_k_fixup loads ids_dst[col_low + j] for all j < J before
checking j_sup. mul_mat_q returns early on col_low == col_high, the fixup
kernel did not. mm_ids_helper emits one extra tile per expert that may be
empty; for trailing experts col_low == ncols_dst, so the load reads one
element past the J_best-1 padding added in #29953. The kernel returns at
j >= j_sup before storing anything, so results are unaffected.

Return early when j_sup <= 0, like the main kernel. The return is
block-uniform (tsj is loaded once per block) and precedes __syncthreads().

Add test_mul_mat_id_unused_tail to test-backend-ops: only the first
n_mats/2 experts are selected, so the trailing experts have zero rows and
their tiles are empty; m=33, k=512 keeps tile efficiency low enough that
stream-k splits along K and launches the fixup kernel.

* CUDA: drop the extra MUL_MAT_ID test case and the explanatory comment (review)
2026-11-12 11:45:56 +08:00
Aleksander Grygier dd6b945be2 server: use the moved test model in the router CORS test (#32117)
test_router_restricted_cors_same_origin_reaches_child still named
ggml-org/tinygemma3-GGUF, which the router no longer finds since the test
models moved to ggml-org-dev, so the Server test job fails on master.

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-12 02:01:19 +01:00
e1f7593c32 opencl: add decomposed FA for Adreno (#32046)
* opencl: add decomposed FA

Assisted-by: Claude Opus 5.5
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>

* opencl: build FA MQ kernel on first FA call

* so allows warmup to early build neccessary kernels

Assisted-by: Claude Opus 5.5

* opencl: handle inf for fully masked row

* opencl: refine condition for using fa decomp

* opencl: support dp4a for PV mm

Assisted-by: Codex

* opencl: add direct PV for FA decomp to avoid V transpose for small batch

Assisted-by: Codex

* opencl: enable direct pv by default

* opencl: expand X2E decomposed FA to smaller heads and batches

- Lower the default head dimension threshold from 256 to 128 on X2E.
- Allow DK64 with P8, no attention sinks, GQA >= 2 and n_kv >= max(1024, 2*n_q).
- Allow P8 with V transpose for 2-31 queries when direct PV is unavailable, DK >= 128 and n_kv >= 1024.

Assisted-by: Codex

* opencl: adjust format

---------

Co-authored-by: Li He <lih@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
2026-11-11 16:18:24 -08:00
Xuan-Son Nguyen 7bcbc04fbf ui: move js sandbox setting out of "developer" (#32054)
* ui: move js sandbox setting out of "developer"

* make js sandbox enabled by default
2026-11-11 23:42:34 +01:00
Iñigo Imaz dd00306bcf docs: update image link in README.md (#32038)
* Update image link in README.md

I guess the image in the main README is now this one https://github.com/ggml-org/llama-brand/blob/master/logo/svg/cover/llama-cpp/cover-llama-cpp-dark.svg

* Fix image link in README.md

* Update image alt text in README.md
2026-11-11 23:39:32 +01:00
Aleksander Grygier 1d81992784 ui: keep downloads in Discover during search and fix their progress with a draft (#31467)
* ui : keep downloads in Discover sidebar during search, sort base-org repos first

- Show the sticky Downloading section regardless of an active search query
- Order search results so repos published by their base model's org lead

Assisted-by: Claude (Opus 5.5 Medium)

* server : throttle download progress per file

Files of a download run in parallel threads, but progress updates were
throttled with one shared timer. A small draft sidecar reports far more
often than the main weights, so it took nearly every slot and the main
file's updates, including the first one carrying its total size, were
dropped until the draft finished. The UI then showed the draft's progress
first and restarted from 0% for the model.

Throttle each file on its own and always pass a file's first update.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: list a downloading repo once and match base orgs case-insensitively

A search hit that is downloading already shows under Downloading, so the
results leave it out. Base model orgs compare without case, as the hub
treats them.

Assisted-by: Claude (Opus 5.5 Medium)

* server : tell clients a download stopped on request

When a client paused or cancelled a download, every other client following
/models/sse got a bare download_failed and could not tell it from an error,
so the web UI showed "Download failed" for a pause made in the Llama app.
A stop asked for through unload or delete now fails with {"stopped": true}.

Deleting a paused download, which has no instance left, now sends
model_remove as deleting a running one does, so other clients drop its row.

Assisted-by: Claude (Opus 5.5 Medium)

* ui : follow downloads other clients start, pause or cancel

A download another client started (e.g. the Llama app) already showed up
from the feed; it now also picks up the sidecars the list reports for it.
A stop another client asked for reads as a pause instead of a failure, and
a removal clears the download row even when the list does not carry it yet.

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-11 23:26:39 +01:00
Aleksander Grygier bd717e08f3 ui: take the logo and favicons from the llama-brand package (#32039)
* ui : take the logo and favicons from llama-brand

The web UI's logo mark and favicons now come from llama-brand, added as a
dev dependency from GitHub and pinned to a commit, so the UI draws the
same icon as the README cover instead of its own copy of an older one.
Logo.svelte inlines the brand's icon and fills it with currentColor, and
the PWA asset generator starts from the brand's web favicons, which carry
their own margin and optical centring, so the UI's colouring and padding
script, its test and its logo.svg go.

Assisted-by: Claude (Opus 5.5 Medium)

* ui : follow llama-brand's master instead of a pinned commit

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-11 23:25:35 +01:00
Xuan-Son NguyenandPascal 2cbe0ec429 server: fix --cors-origins handling + protect against DNS rebinding (#32098)
* server: fix --cors-origins localhost handling

* server: validate Host on loopback connections against DNS rebinding

A page whose own name resolves to 127.0.0.1 reaches the server with
that name in Host and a matching Origin, so the same-origin shortcut
let it through. On a loopback connection, Host must now name loopback
or the host of an allowed --cors-origins entry. The LAN tests reach
the server through a real interface instead of a forged Host.

* fix python type check

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-11 21:17:46 +01:00
Xuan-Son Nguyen 664f031f49 ci: move test models to ggml-org-dev (#32096)
* ci: move test models to ggml-org-dev

* fix test spec
2026-11-11 21:17:16 +01:00
CC 4180722eeb metal : prevent out-of-bounds reads in SOLVE_TRI kernel (#32103) 2026-11-11 22:02:06 +02:00
Ruben Ortlam 055c3c26ef vulkan: support batched tensor copy API (#31418)
* vulkan: support batched tensor copy API

* allocate staging buffer only for copies that actually need it
2026-11-11 21:56:39 +02:00
Alessandro de Oliveira Faria (A.K.A.CABELO)andAlessandro de Oliveira Faria 2656b71a19 vendor : update LibreSSL to 4.4.1 [no ci] (#32106)
Co-authored-by: Alessandro de Oliveira Faria (A.K.A.CABELO) <cabelo@multicortex.ai>
2026-11-11 20:42:10 +01:00
Georgi Gerganov cb44fdb189 llama : reuse graph result instances across scheduler reserves (#32089)
Allocate gf_res_reserve and both gf_res_prev entries once in the llama_context constructor. Add llm_graph_result::set_max_nodes() and reset the existing instances in sched_reserve() instead of recreating them. get_gf_res_prev() now only selects the arena for the current batch.

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
2026-11-11 19:58:39 +02:00
Aldehir Rojas cd2db800db jinja : support chained comparisons and argument unpacking (#32072) 2026-11-11 11:53:04 -06:00
Xuan-Son Nguyen f7b886fe6f chat: allow TTS model to provide custom text normalizer via jinja (#32053)
* chat: allow TTS model to provide custom text normalizer via jinja

* rm tts_norm_is_reserved_name
2026-11-11 17:34:55 +01:00
Sigbjørn Skjæret 956591f457 ci : remove dead snakes (#32084) 2026-11-11 15:13:14 +01:00
Georgi Gerganov ccd360f060 llama : remove non-fused GDN and ubatch-dependent graph nodes (#32078)
- remove build_delta_net_chunking and build_delta_net_autoregressive
- always use the fused gated delta net
- make graph_max_nodes independent of the ubatch size
- drop the fused GDN probes and fused node tags

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
2026-11-11 15:16:16 +02:00
Asahi-PrvandAsahi-Prv b326dd6752 devops : add missing MKL runtime DLLs to the Windows SYCL release package (#32005)
ggml-sycl.dll built with oneAPI 2026.2 imports mkl_intel_thread.3.dll and libimalloc.dll, which are not part of the release package. LoadLibrary fails and the SYCL backend is silently skipped, llama.cpp falls back to CPU.

Adding both DLLs fixes the load: measured on Arc B570 with LFM2.5-8B-A1B Q4_K_M, pp4096 goes from 404 t/s (CPU fallback) to 2232 t/s (SYCL) with the extracted b12244 package.

Assisted-by: GLM (z-ai/glm-5.3-flash)

Co-authored-by: Asahi-Prv <Asahi-Prv@users.noreply.github.com>
2026-11-11 14:14:43 +01:00
Alessandro de Oliveira Faria (A.K.A.CABELO) b8abe41a50 vendor: update cpp-httplib to 0.66.0 (#32049) 2026-11-11 20:07:30 +08:00
Tarek DakhranandClaude Opus 5.5 2e325dcd57 backend : re-instantiate CUDA graphs when the cluster dims of a kernel change (#31284)
* CUDA: re-instantiate graphs when the cluster dims of a kernel change

* CUDA: keep the cluster dims with the graph instance, check them only on Hopper+

Assisted-by: Claude Opus 5.5

* CUDA: use CUDA_CHECK and a helper struct for the graph cluster dims

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* CUDA: add a TODO for the graph cluster dims test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 14:00:45 +02:00
Oliver Simons 1648d64117 test-fp8.cpp: use only the public ggml API (#32083) 2026-11-11 13:33:12 +02:00
fox5611 432433637b ROCm/HIP: fixed BLAS workspace instead of rocBLAS allocating one per call (#31838)
* ROCm/HIP: fixed BLAS workspace instead of rocBLAS allocating one per call

Assisted-by: Claude Opus 5

* ROCm/HIP: corrected by only setting the workspace on ROCm 7.0 or newer
2026-11-11 12:32:39 +01:00
JamieBordoni 71a6a114bc quantize: fix imatrix GGUF filepath metadata leak (#32000) 2026-11-11 19:13:28 +08:00
Pascal 7f24aacde3 cpu: accumulate flash attention V in F32 (#29810)
* cpu: accumulate flash attention V in F32 in the one-chunk kernel

The one-chunk kernel accumulated an F16 V cache in an F16 running sum
that is only normalized at the end, which overflows past 65504 on long
contexts: inf, then NaN on every following token.

Every V type now accumulates in F32, like the tiled kernel. F16 rows go
through ggml_vec_mad_f16_f32, which converts and multiplies in the same
register (F16C/AVX512, NEON, RVV) with no round trip through a temporary
F32 row, so decode keeps master speed on x86 and loses about 4% at 4k
context on ARM, where master used native F16 FMA.

* cpu: widen the F16 V rows like ggml_vec_dot_f16_f32

ggml_vec_mad_f16_f32 required F16C and FMA, so AVX builds without FMA
fell back to a scalar loop and ran 3x slower than master. It now
follows the guards of ggml_vec_dot_f16_f32: AVX512, then AVX with F16C
through GGML_F32x8_FMA, then NEON and RVV, then the scalar loop.
2026-11-11 11:55:19 +01:00
Leon af6b5243d6 webgpu: add PAD_REFLECT_1D support (#31877)
* webgpu : add PAD_REFLECT_1D support

* webgpu: remove redundant PAD_REFLECT_1D contiguity check

* tests: simplify PAD_REFLECT_1D setup
2026-11-11 12:11:06 +02:00
Jeff Bolz d2b3eb9b60 vulkan: allow dequantized KV optimization with CM2 (#31526)
* additional FA test cases

Assisted-by: Codex

* vulkan: allow dequantized KV optimization with CM2

Enable the KV dequantization optimization with coopmat2, guarded by a heuristic
based on query tokens*GQA ratio.

While doing this we saw very large speedups for certain head sizes, and it was
because those head sizes were incorrectly treated as unaligned. So also fix
those to be treated as aligned, which should help for when the dequant
optimization doesn't apply.

* gqa uses nbq2 for the stride, so we need to check that value for alignment

* tests : use VARS_TO_STR18 for flash attention

Assisted-by: Codex
2026-11-11 12:08:06 +02:00
figkcoci-debugandfigkcoci-debug eeb2104794 vulkan: allocate the device fault vendor binary buffer (#32008)
* vulkan : allocate the device fault vendor binary buffer

* vulkan : drop the redundant vector include

---------

Co-authored-by: figkcoci-debug <307500910+figkcoci-debug@users.noreply.github.com>
2026-11-11 11:02:48 +01:00
Oliver Simons 03d6c824e1 tests : use only the public ggml API (#32077) 2026-11-11 12:01:24 +02:00
Pascal adde94c11b ui: open the virtual keyboard in the chat editor on Android (#32060)
CodeMirror takes text input through EditContext on Chrome for Android
only, and there the chat editor never brings up the virtual keyboard.
The editor now always takes its input through contenteditable, as on
desktop and iOS.

A press in the input box outside the text also focuses the editor, in
the collapsed form too, so the whole box is a tap target again as the
textarea was.
2026-11-11 10:42:02 +01:00
Aman Gupta 459ab1cdd3 cuda : use MMVF for BF16 matrices of few rows at small batch sizes (#31959)
MMF tiles 32 rows per block, so a BF16 matrix with few rows (e.g. the
qwen4exp hyper-connection projections, 320 rows with K = 10240) runs only
a handful of blocks and leaves most SMs idle at batch sizes 2..8. MMVF
runs a block per row and wins there when K is long enough.

MMVF's cost grows with rows x batch while MMF's stays roughly flat in that
range, so use MMVF only while rows * ne11 * 256 <= nsm * min(K, max_k) and
K >= 1024. The budget scales with the SM count, which keeps GPUs with few
SMs (e.g. the A16) on MMF. Blackwell allows up to 8 columns and
max_k = 6144; older GPUs stop at 5 columns and max_k = 4096.

Add eval cases at 128x2560, 512x2560 and 320x10240 for n = 2, 3, 4, 5 and 8.
2026-11-11 10:46:13 +02:00
Jiří SvítilandJirka Svítil 77e2db8cab cuda : add sm70 FlashAttention config case for DKQ=256, DV=256, ncols=64 (#27997)
* CUDA : add sm70 FlashAttention config case for the 256x256 head

* CUDA : tune sm70 FlashAttention for 256x256 ncols=32

Assisted-by: Codex

---------

Co-authored-by: Jirka Svítil <mistrjirka@users.noreply.github.com>
2026-11-11 10:45:34 +02:00
Sihan Yu 9d7b2d7397 cuda : fattn mma head sizes 40 72 (#30285)
* cuda : add MMA FlashAttention support for head sizes 40 and 72

Head sizes 40 and 72 (used e.g. in vision towers of Gemma 3/4 and Qwen3-VL) were excluded from every tensor-core dispatch branch and fell back to the tile kernel.

The fp16 MMA fragment covers 16 halfs in the head direction, so the in-kernel geometry is padded to the next multiple of 16 (40 -> 48, 72 -> 80). Register arrays, shared memory tile strides and MMA loop bounds use the padded size, while global memory loads and stores the real one.

Enabled on NVIDIA Turing and newer while Volta and AMD keep the tile kernel. The RDNA/CDNA config rows and the mma_tile_sizes specialisations are needed because the explicit instantiations are compiled for every target of a translation unit.

* tests : add the head size = 40 encoder perf shape

* cuda : address review comments on the head size 40 and 72 kernels
2026-11-11 10:45:01 +02:00
Oliver SimonsandAman Gupta 9e9619d030 CUDA: Add OCP FP8 support (#28413)
* Initial CUDA WIP version

* Use fp8 intrinsics

* Do W8A16 for FP8 in GEMV case

(base) osimons@ub-osimons:~/lcpp_fp8.git$ ./build-x64-linux-gcc-reldbg/bin/llama-bench -m /mnt/share/gguf/nvidia/Qwen3.6-27B-NVFP4/Qwen3.6-27B-NVFP4_fp8.gguf -p 8196 -ub 512,2048 -d 0,32000 -n 256 --load-mode dio
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 32165 MiB):
  Device 0: NVIDIA RTX PRO 4500 Blackwell, compute capability 12.0, VMM: yes, VRAM: 32165 MiB
| model                          |       size |     params | backend    | ngl | n_ubatch |         lm |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -------: | ---------: | --------------: | -------------------: |
| qwen35 27B NVFP4               |  19.56 GiB |    27.32 B | CUDA       |  -1 |      512 |        dio |          pp8196 |       2781.53 ± 8.48 |
| qwen35 27B NVFP4               |  19.56 GiB |    27.32 B | CUDA       |  -1 |      512 |        dio |           tg256 |         37.56 ± 0.02 |
| qwen35 27B NVFP4               |  19.56 GiB |    27.32 B | CUDA       |  -1 |      512 |        dio | pp8196 @ d32000 |       1616.31 ± 2.61 |
| qwen35 27B NVFP4               |  19.56 GiB |    27.32 B | CUDA       |  -1 |      512 |        dio |  tg256 @ d32000 |         34.11 ± 0.03 |
| qwen35 27B NVFP4               |  19.56 GiB |    27.32 B | CUDA       |  -1 |     2048 |        dio |          pp8196 |       2542.00 ± 0.51 |
| qwen35 27B NVFP4               |  19.56 GiB |    27.32 B | CUDA       |  -1 |     2048 |        dio |           tg256 |         37.36 ± 0.02 |
| qwen35 27B NVFP4               |  19.56 GiB |    27.32 B | CUDA       |  -1 |     2048 |        dio | pp8196 @ d32000 |       1762.86 ± 1.33 |
| qwen35 27B NVFP4               |  19.56 GiB |    27.32 B | CUDA       |  -1 |     2048 |        dio |  tg256 @ d32000 |         34.09 ± 0.02 |

* Optimize GEMV kernel for FP8

* Guard __nv_cvt_fp8x2_to_bf162raw on CTK 13.2

Fixes https://github.com/ggml-org/llama.cpp/pull/28413#discussion_r4044500099

* Remove trailing whitespace

* Add supported for strided copies to/from FP8 in CUDA

* Prepare Q_reg for FP8 FlashAttention in fattn-vec.cuh

* Fix odd-rows for convert/getrows, possible with FP8

* Fix FP8 fallback in cuBlas

* Gate mmvf_f8x2_e4m3_to_bf162 on FP8 availability, i.e. CTK >= 11.8

* Restrict (fused) MMVF to 2*ts alignment, and cublas FP8 fo 16-byte

* Add todo about accumulator/precision honoring

This is beyond the scope of the stacked PR

* Avoid races in fp8_fallback

* Allow Hopper and other BW SKUs for FP8

* Use intrinsic over explicit manual conversions

* Resolve TODO

As per spec, we don't have to preserve signs

* Add commend on cublaslt workspaces

* Use convert_unary_cont_cuda over block dequant in convert.cu

* Use get_rows_cuda_float for fp8 kv cache

* Make FP8 copies work on all IHV's

* More cleanup

* Simplify by falling back to shared cuBLAS helper

* Fuse weight_scale multiplication for FP8 also

* More code-cleanup

* Guard __nv_cvt_fp8x2_to_bf162raw on CTK 13.4

Only cvt.rn.bf16x2.e4m3x2 is exposed from 13.2 onwards, so
93e98b9203dfacb85f9ac665f5f0482a8c6b4192 was incorrect fix

* Use F32 activations in fallback mmvf path

* Explicitly link against cublaslt

* Move FP8 conversion intrinsics into ggml_cuda_cast

Cleaner this way

* WIP src1/prec for fp8 in MMVF

* Remove comment

* Keep only f8 x f8 KV cache for now

* Just do FP8 x FP32 x FP32 in MMVF as its fast enough

* Update ggml/src/ggml-cuda/ggml-cuda.cu

Co-authored-by: Aman Gupta <amangupta052@gmail.com>

* Add batched FP8 GEMMS in cublas as suggested by code review

https://github.com/ggml-org/llama.cpp/pull/28413#issuecomment-6676015644

* Extract fp8x2 -> float2 load into helper

* Searapte copy checks to/from fp8

* Compiler massaging to avoid repeated address offsets

---------

Co-authored-by: Aman Gupta <amangupta052@gmail.com>
2026-11-11 10:44:30 +02:00
Michael Taylor f3b407ca1b sycl : fixed sycl CLAMP to actually support strided view (#31784)
Modify `ggml_sycl_op_clamp` to use the unary helper function
`ggml_sycl_detail::ggml_sycl_op_unary` to handle non-contigous or
strided views. The fix uses a pattern that matches other stride-safe
unary ops, and is a simpler execution path.

Deleted dead template `unary_op_clamp_kernel`, the flat `clamp` kernel
launcher from element_wise.cpp, and deleted (now) unused
`SYCL_CLAMP_BLOCK_SIZE` from presets.hpp.
2026-11-11 13:48:20 +08:00
Ramzi aa75ff3239 chat : fix Muse Spark ignoring response_format json_schema (#32017) 2026-11-10 23:30:14 +01:00
Aleksander Grygier b5cb8f6f47 ui: keep a hidden sidebar from intercepting clicks again (#32035)
The hidden desktop strip relied on inherited `visibility: hidden` to keep
its content from catching clicks. Since the sidebar's sections fold
(#31936), an open section's CollapsibleRegion sets `visibility: visible`
again, and its rows' context-menu triggers carry an inline
`pointer-events: auto`, so they caught clicks meant for the page.

Make the hidden strip's content inert, which no descendant can undo, and
add a client test that hit-tests the hidden sidebar's area.

Fixes #31971

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-10 22:44:23 +01:00
Aleksander Grygier 85822a7ab9 ui: update the README banner to the new llama.cpp cover and remove the legacy brand assets (#31789)
* media : remove the legacy llama0 and llama1 brand assets

The brand assets live in ggml-org/llama.brand; none of these is referenced
in the repository.

Assisted by: Claude (Opus 5.5 Medium)

* readme : use the new llama.cpp cover from llama.brand

Assisted by: Claude (Opus 5.5 Medium)
2026-11-10 22:43:27 +01:00
Aleksander Grygier 2b66bfbf71 ui: show audio in Discover for models whose mmproj has an audio encoder (#31837)
* ui: show audio in Discover for models whose mmproj has an audio encoder

Discover took audio only from the pipeline tag, so omni models tagged
any-to-any, such as Gemma 4 and LFM 3 Omni, showed vision but never audio,
though their mmproj carries an audio encoder.

Read the encoders from the mmproj's GGUF metadata instead: a ranged read of
its first 64 KB, once per repo, for clip.has_vision_encoder and
clip.has_audio_encoder. The Discover list and details now show audio when
either the pipeline tag or the mmproj says so.

Assisted by: Claude (Opus 5.5 Medium)

* ui: take Discover modalities from the mmproj's encoders in one place

Review fixes for the mmproj header read:

- drop vision for audio-only mmprojs: the converter writes
  clip.has_vision_encoder only when true, so a header read in full without
  it has no vision encoder; a header cut short leaves vision as it was
- accept only a 206 to the ranged read, so a server that ignores Range
  never streams the whole mmproj
- encode the mmproj path in the resolve URL
- derive the list and details modalities with one HuggingFaceService
  helper, so both views agree
- point the key comment at tools/mtmd/mtmd-impl.h, where the keys live now

Assisted-by: Claude (Opus 5.5 Medium)

* ui: keep a gated repo's mmproj from being read again on every mount

A gated, private or missing mmproj answers 401, 403 or 404 to every read,
so caching nothing made each mount of its Discover row request it again and
log an error. Cache it as unknown instead: no audio, and vision from the
mmproj being there, as before the header was read.

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-10 22:34:46 +01:00
Sigbjørn Skjæret 582eab7186 ci : bump ty to 0.0.100 (#32030)
* bump ty to 0.0.100

* rename rules

* type fixes

* type fix

* proper type fix
2026-11-10 21:54:14 +01:00
Xuan-Son Nguyen 7c2653acc7 mtmd: fix string format, rename to mtmd_format (#32040) 2026-11-10 20:23:25 +01:00
Aleksander Grygier 7592b61d3b ui: move Models, MCP and Settings to a sidebar footer (#32022)
* ui : move Models, MCP and Settings to a sidebar footer

The sidebar's actions split in two: New chat, Search and Projects stay at
its top, and Models, MCP and Settings move to a footer at its bottom, under
a thin rule, in the expanded sidebar and its collapsed strip alike. On a
phone the footer keeps clear of the home indicator, and Models stays
desktop-only as before. The search dialog still lists every action.

Assisted-by: Claude (Opus 5.5 Medium)

* ui : group the sidebar footer in an inset surface

The design system gains inset-group-surface, for rows grouped inside a glass
panel: white on the light glass and a faint tint of the text colour on the
dark one, with the sidebar's own ambient edge, no inner highlight, a faint
wide shade under it and the card radius. It is matte-glass pointed at its
own component tokens, like the sidebar and chat form surfaces, and reads as
part of the panel: no tight contact shadow or solid border like a button,
no inset shade like a field.

The sidebar footer uses it in place of the rule above Models, MCP and
Settings. DESIGN.md and the Surfaces page document it.

Assisted-by: Claude (Opus 5.5 Medium)

* ui : give the sidebar's logo button the add button's key look

Whenever the sidebar shows, open or peeking, its logo button takes the same
key as the chat form's add button (matte-key, white in light mode, a light
tint in dark), in place of its own muted fill. While the sidebar is hidden
the logo has no fill and no shadow, and hover and focus never change the
fill. It keeps its round shape and its own size.

Assisted-by: Claude (Opus 5.5 Medium)

* ui : put search in a round key at the top of the phone sidebar

On a phone the sidebar's Search row moves to a round key at the right end
of its top row, across from the logo, with the same key look as the logo
and the chat form's add button. It opens search as the row did. Desktop
keeps the Search row.

Assisted-by: Claude (Opus 5.5 Medium)

* ui : size the phone sidebar for a thumb and list its footer with the actions

On a phone the sidebar's actions and rows are 48px tall at 17px, flush one under
the other with 16px corners, its section headings 15px and grey, and its groups
28px apart. MCP and Settings join the top actions there: the footer is a desktop
pattern.

Assisted-by: Claude (Opus 5.5 Medium)

* ui : mark the selected sidebar row with a tinted inset surface

Assisted-by: Claude (Opus 5.5 Medium)

* ui: keep the phone sidebar's round keys white on the grey page

Assisted-by: Claude (Opus 5.5 Medium)

* ui : open Settings and MCP as pages on a phone, dialogs elsewhere

On a phone the sidebar opens them at #/settings and #/mcp-servers, with the menu key and a
title like any phone page. Pinned under the status bar, Settings' section tabs fill the
screen above them in the page's colour.

Assisted-by: Claude (Opus 5.5 Medium)

* ui : open a phone sidebar's pages behind it, settled before they slide back

A tap in the phone sidebar now opens its page while the current one is still aside and slides
back once it has rendered, instead of swapping pages mid-slide. The slid page's fixed bars are
pinned again as the page under them changes, the lock ends with the slide itself, and a
phone's new chat shows its form and greeting at once, without the move and entry they have
on a desktop.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: style the sidebar footer group and the selected row with master's classes until the design system lands

Assisted-by: Claude (Opus 5.5 Medium)

* ui: add GlassKey with master's styles for the phone sidebar's round keys

Assisted-by: Claude (Opus 5.5 Medium)

* ui: center the top edge of a new chat's form on the screen

The form rested 42px below the middle up to 2xl and 6px above it past
that; one drop, from the resting card's height, now puts its top edge on
the middle at every width from md up.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: round the sidebar's action rows concentric with the footer group

The sidebar stacks four surfaces: the page, the sidebar, the footer group
inset in it and a row in the group. app.css now names the group's radius
and padding, and derives the row radius from them (16px less 4px), so the
footer's rows run parallel to its corners; the actions at the top of the
sidebar share that radius. The footer moves to its own component, the
README documents the pattern, and a client test holds the rule on the
rendered footer.

A selected action row was bg-accent, the same grey as a hovered row in
light and darker than the footer's fill in dark, so it read as hovered or
not at all; it is now raised and edged, while a hover only tints.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: import demo chats through the Settings page on a phone

A phone opens Settings as a page now, not a dialog, so the demo harness
waited for a dialog that never came and every phone demo with traces
failed; it now imports through whichever of the two opens.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: count the sidebar's rows from their items in the surface test

The test expected three footer rows and two top actions, which the
stacked PRs change as they add Projects, Library and Servers; it now
expects as many rows as the sidebar has items.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: add a line on Meta+Enter off macOS too

The chat form takes Meta+Enter as Ctrl+Enter and leaves the newline to
the editor, but the editor bound only Mod-Enter, which is Ctrl off
macOS, so on Linux Meta+Enter neither sent nor added a line. This failed
the client test on CI's Linux runners, on master too.

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-10 18:58:57 +01:00
Daniel Bevenius cd4ed902e1 cuda : remove unused ids and type_src0 parameters (#32018)
This commit removes the unused ids and type_src0 parameters from
quantize_row_q8_1_cuda.

The motivation for this is that these parameters were required in the
past because this function was used in combination with the function
pointer quantize_cuda_t, which has been now been removed, and this
function no longer needs to adher to the function pointer signature.

Refs: https://github.com/ggml-org/llama.cpp/pull/31884.
2026-11-10 18:26:53 +01:00
Oliver Simons 202c1a229f llama : Add FP8 KV-cache support (#28392)
* Add FP8 KV-cache support

* Add tests for scaled, FP8 KV-cache to test-backend-ops

* Fix get_*_scale mapping and dequnatize FP8 also before rope shift

* Normalizes cache-scale names independently of projection layout.

While fused KQV is currently impractival for non-shared scales, this is
an improvement for future use regardless

* Enable Gemma 4 Assistant with FP8 by storing and fetching scale ptrs

* Update comment wording

* Move E2E FP8 KV cache tests to test-llama-archs

They belong there as they are not just a test for ggml's backend ops

* Increase support surface, remove seed

* make scaled kv generic

* Fix CI failings for test-kv-cache

https://github.com/ggml-org/llama.cpp/actions/runs/38718763447/job/116367842578?pr=28392

* CI: execute kv-cache-scale tests

* exclude fp8-kv tests from OpenVINO

* Fix linting errors of test_kv_cache_scales

* Disable kv-ffload=false + multi-GPU tests for FP8

See https://github.com/ggml-org/llama.cpp/actions/runs/38952722084/job/117148353699?pr=28413

* Adjust test label for test-kv-cache.cpp

* Fix conflict-resolution mistakes in test-kv-cache.cpp

https://github.com/ggml-org/llama.cpp/actions/runs/40554674463/job/122367426461?pr=28392

* Move KV tests to test-fp8.cpp

https://github.com/ggml-org/llama.cpp/pull/27512#discussion_r4410518158

* Remove tmp file for test-kv-cache-scales.py on Windows as well

* Fix fp8 kv cache test by population kv_source_impl

* Exclude models that consume KV cache directly

* Remove obsolete comment

* Allow BF16 scales as well for KV cache

* Rename *_cache_scale to *_cache_s

https://github.com/ggml-org/llama.cpp/pull/28392#discussion_r4517194999

* Move `is_type_scaled()` to `llama-kv-cache.cpp`
2026-11-10 18:08:32 +02:00
Georgi Gerganov e24d0b4c34 Merge pull request #32015 from ggml-org/gg/fix-sampling-output-aliasing 2026-11-10 17:12:27 +02:00
Georgi Gerganov 3e86ce2c2d llama : reuse backend scheduler when sched_params are compatible (#32009) 2026-11-10 17:10:11 +02:00
Georgi Gerganov 494d0f47bd ggml : add MUL_MAT src1 precision tests and gating (#31896) 2026-11-10 17:03:27 +02:00
Aleksander GrygierandPascal 281e7dad7c ui: gather chats into named groups in the sidebar (#32012)
* ui: let a menu action open a submenu

Group and project pickers open as flyouts from a chat's menus. Master's menu
list has no submenus, so bring them back next to its tooltips, with a close
callback for picks made inside them.

Assisted-by: Claude (Opus 5.5 Medium)

* ui : store groups and let conversations join them

Groups live in their own IndexedDB table (schema version 2) with a name.
groupsStore creates, renames and deletes them; deleting a group keeps its
chats, out of any group. A conversation records the group it is sorted
into, and a group picked before the first message is kept pending and
applied when the conversation is created.

Assisted-by: Claude (Opus 5.5 Medium)

* ui : show each group as a sidebar section

Every group is a sidebar section next to Pinned and Recent, folding,
showing its first six chats and dragged into place like them. A chat
dropped on a group joins it and leaves Pinned; dropped on Recent, it
leaves its group. Recent is called Ungrouped once a group exists, and its
menus create a group in a small naming dialog; so does a right-click
anywhere in the list that no row or heading takes, down to the empty space
under the last section. A group's heading starts a new chat in it, and its
menus rename or delete it; deleting keeps its chats, back in Ungrouped.

Assisted-by: Claude (Opus 5.5 Medium)

* ui : move chats into groups from their menus

A chat's menus, in the sidebar and on right-click, move it into a group
or take it out of its group. Desktop menus show a searchable group flyout
whose New group entry names a group in the group dialog and moves the chat
into it; a phone gets a picker dialog instead, which creates a group from
what is typed.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: open the group picker from a phone's hold menu

The hold menu is anchored to the row, so it showed the desktop flyout,
which ran off the left edge of the screen and whose New group closed
without naming one. On a phone, Move to group now opens the picker
dialog, as intended.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: open a new chat in a tab of its own

With conversation tabs on, New chat from the sidebar, a group's heading
or the shortcut replaced the open chat's tab. It now adds a tab, as the
tab bar's "+" does, and the open chat keeps its own.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: list a phone's groups under the picker's search

On a phone the picker fills the screen, and its grid spread the rows
over the height, so the list sat in the middle. It is now a column: the
list starts under the search and scrolls in the rest of the screen.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: fix gaps in chat groups found in review

- A phone shows a group heading's New chat and menu, and the Ungrouped
  heading's menu, which were reachable only by hover or right-click.
- A pinned chat moved into a group from its menu leaves Pinned, as a
  drag onto the group already did, so the move shows.
- Only the sidebar's New chat, a group's New chat and the new-chat
  shortcut open a new tab; stepping past the last chat, search and the
  models drawer show the new-chat screen in the current tab again.
- An export carries each chat's group, and an import sorts the chat
  into the group of that id or name, or creates it; a group the file
  does not carry is dropped.
- The group picked with a group's New chat survives leaving the
  new-chat tab and coming back.
- A phone's menus skip any submenu in the shared menu list, not only
  the group one.
- Deleting a group forgets its section's place and fold.
- Groups load newest first by creation, so a rename no longer moves one
  after a reload.
- groupsStore looks groups up in a map and answers which live group a
  chat is in, for the three places that checked it by hand.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: keep a forked chat in its source chat's group

A fork copied the working directory but not the group, so it landed in
Ungrouped.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: move a chat with its whole fork family between groups

Moving a chat into a group, or out of one, takes its parents and its
children along, so a fork tree never splits across sidebar sections. A
pinned member that joins a group leaves Pinned with the rest. Group
sections render the tree with its depth, as Ungrouped does.

* ui: show the group name prompt as a small centered card on phones

The new and rename group prompt opened full screen on a phone, like a
large dialog. It is now built on the alert dialog, as the chat rename
prompt is, so a phone shows it as a card in the middle of the screen.
Its Cancel is a plain button, so Enter in the name field saves.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: space a phone's group sections like Ungrouped

On a phone a group or Pinned section put its first chat right under the
heading, where Ungrouped leaves a row gap. They now leave the same gap,
and the sections sit a little further apart than the rows inside them.

Assisted-by: Claude (Opus 5.5 Medium)

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-10 15:44:46 +01:00
Hamish M. Blair 90e3841d9f ui: show settings save button even while focused (#32010) 2026-11-10 14:14:11 +01:00
Daniel Bevenius c75fba30cf cuda : remove unused quantize_cuda_t typedef (#31884)
This commit removes the unused quantize_cuda_t from quantize.cuh.

The motivation for this is that the usage of this typedef as a function
parameter was removed in Commit 74976e1aef
("CUDA: remove -sm row, refactor cuBLAS (#24216)") and there is no longer
any usages of it.
2026-11-10 14:08:15 +01:00
Georgi Gerganov 40f4f4ac9d graph: mark every sampling chain as an output to prevent aliasing
The sampling chains are read on the host after the graph compute, and the
graph addresses are reused across builds with different selected rows: the
token-generation reserve graph has one output row per sequence, while the
actual block draft has n_max rows per sequence. The reserve allocates the
unselected chains without keeping them, so their buffers alias, and the
actual graph then reuses those addresses by node index.

set_outputs only marked the tensors stored for the selected rows, so the
unselected per-(sequence, row) chains were freed and reused. Collect every
chain tensor in t_sampling_all and mark all of them as outputs there.

Reproduces with -np 2 (non-unified KV) and 2 concurrent requests for
draft-dspark, draft-dflash and draft-eagle3; with different prompts one
sequence's acceptance collapses (eagle3 0.15 vs 0.40). After the fix:
dspark 0.71/0.78, dflash 0.56/0.44, eagle3 0.35/0.42.

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
2026-11-10 14:44:14 +02:00
Aleksander Grygier 54cde7106a ui: expand the chat form into a large editor for longer messages (#31973)
* ui: expand the chat form into a large editor for longer messages

The chat form expands into a large editor once a message has markdown
beyond plain paragraphs or reaches four lines, and a Maximize button in its
top-right corner toggles it by hand. The expanded form starts around half the
screen height and grows with the text up to the space below the tabs bar,
animating its height by mirroring the editor's content height.

On a new chat the expanded form keeps the collapsed form's top edge and the
greeting only fades out, waiting for a collapse to settle before it fades
back in. In a conversation the messages follow the form as it grows, so the
latest one stays in view above it.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: make the expanded chat form a full-screen sheet on phones

In the expanded editor Enter adds a line and Cmd/Ctrl+Enter sends.

On a phone the expanded form is a full-screen sheet with no corners or
border. The page around it and Safari's bars take its colour, and the
sidebar's logo button and the working-directory chip step aside while it is
open. It moves by relative offsets and a clip on the main thread, so its
edges, text, actions and chip stay in step in every engine, and its content
keeps the collapsed card's places and widths, so nothing reflows when it
settles. Expanding does not focus the editor on a phone.

On a new chat on desktop the form is lifted by its sticky bottom offset
instead of a translate, and the expanded size is built on the collapsed
card's real height, so its top edge holds still through any sequence of
quick toggles.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: hide the phone menu toggle and the sheet's focus edge while expanded

On a phone the menu toggle rides on the page, above the chat form: it now
fades out while the expanded form covers the screen, and comes back when
it collapses. The expanded sheet also drops the glass rim and inner ring,
so focusing the editor no longer outlines the whole screen.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: keep the expanded phone sheet clear of the status bar

The expanded chat form fills the screen on a phone, under the status bar
an installed app draws over the page. Its text now starts below the bar
(pt-safe) and its collapse button sits below it too (top-safe-2.5).

Assisted-by: Claude (Opus 5.5 Medium)

* ui: hide the phone's context gauge while the chat form is expanded

The gauge pinned to the phone's top right sat over the expanded sheet's
collapse key. Like the menu toggle across from it, it now fades out while
the expanded form covers the screen and comes back when it collapses.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: lift the expanded phone sheet's actions above the iOS keyboard

Assisted-by: Claude (Opus 5.5 Medium)

* ui: move the chat form with a short, quick drift, without sliding after it lands

Its position transition came back on before the browser had laid out the new position,
so it slid the form there after the fade; the drift is now 10px each way and quicker.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: style the expanded chat form with master's surfaces until the design system lands

Assisted-by: Claude (Opus 5.5 Medium)

* ui: tighten the expanding chat form after review

- Expand on markdown only in finished lines: a lone `-`, `#` or `1.` on the
  line being typed no longer opens the large editor.
- Show "Cmd/Ctrl+Enter to send" by the send button while expanded.
- Move the expansion state, the phone sheet and the held collapsed height
  into useChatFormExpansion, and the sheet's geometry into pure functions.
- Drop the working-directory chip's phone animation, which never ran since
  the phone chip moved to the + menu.
- Clearing or sending mid-move cancels the sheet's animation.
- A click only continues at the end of the text when it lands below it.
- Reuse ActionIcon for the toggle and INPUT_CLASSES for the card, derive the
  centered form's offset from its drop, name the shared easing and fade.
- The greeting owns its fade behind the expanded form.
- MarkdownEditor's view is reactive, so following its content starts once it
  exists.

Assisted-by: Claude (Fable 5.2)

* ui: keep the expanded chat form to the message being written

- The form expands on its own only for what is written or pasted, and
  another conversation starts collapsed: a restored draft no longer opens
  the large editor, or the phone sheet, on arriving at a chat.
- In the expanded editor the pickers open by the line being written, above
  it or, short of room, below, instead of above the form's top edge.
- The sidebar's swipe is off while the phone sheet covers the screen.
- The send hint shows only beside the send button, where the row has room.
- The inner area keeps master's focus shadow, except on the phone sheet.
- The card and the form's position honour reduced motion.
- The phone greeting's place accounts for the wrapper it is counted from.
- The sheet's animation tolerates the form going away mid-move.

Assisted-by: Claude (Fable 5.2)

* ui: turn the typed pickers off on phones, and scroll long messages

- On a phone, typing `/` or `@` opens no picker for now: each mobile
  browser moves the page its own way as the keyboard opens, so a list by
  the form can not stay clear of the text being written. A note marks it
  for later.
- The editor scrolls inside the form again: MarkdownEditor's own box sat
  between the input and the editor without taking its height, so a message
  longer than the form ran under the actions instead of scrolling.

Assisted-by: Claude (Fable 5.2)

* ui: settle the expanding chat form's review findings

- A phone expands the form only by hand: the sheet switching in mid-typing
  moves the text under the keyboard.
- The typed pickers are off wherever a finger is the main pointer, not
  below a viewport width, so a narrow desktop window keeps them and an
  iPad loses them. The prompt picker opened from the + menu floats by the
  sheet's actions on a phone.
- A message put back after a failed send expands the form again.
- Only the left button moves the caret from a click below the text.
- The sheet's fill is one token, used by the sheet, the page and Safari's bars.
- The collapsed-height hold is its own module and no longer finds the
  editor's elements by class name.
- Shorter comments.

Assisted-by: Claude (Sonnet 5.5)

* ui: keep a one-line message on the chat form's actions row

On a desktop the main chat form is a single pill while its message is one
line: the add button, the text, then the model selector and send. A second
line, or a line that wraps beside the actions, brings back the two rows,
and they stay until the text is shorter than where it wrapped, so the
wider row can not flip it straight back. The editor stays mounted: the
actions join the text's row by CSS alone.

The single row's corners are `--chat-form-single-row-radius` in app.css, a
pill by default; the "Scrapbook / ChatForm single row" story shows the real
form with other radii.

Assisted-by: Claude (Opus 5.5)

* ui: give the one-line chat form 1.25rem corners

Assisted-by: Claude (Opus 5.5)

* ui: pick the working directory from the + menu on desktop too

The chip under the chat form is gone: the + menu offers the working
directory everywhere but a phone's chat, whose header shows it. The menu
item names the directory once one is set, and the picker gains a row to
clear it, which the chip's X used to do.

Assisted-by: Claude (Opus 5.5)

* ui: put the single-row chat form behind a Display setting

"Single-row chat form for short messages", on by default.

Assisted-by: Claude (Opus 5.5)

* ui: send the chat form on Enter, and add a line with Shift or Ctrl/Cmd

Enter sends wherever the caret is, the expanded editor and code blocks
included; Shift+Enter and Ctrl/Cmd+Enter add a line, continuing lists and
quotes. With "Send message on Enter" off the two swap as before. The
expanded editor's send hint and the code-block rule are gone with it.

Assisted-by: Claude (Opus 5.5)

* ui: start the chat form on the row layout it will settle on

Assisted-by: Claude (Opus 5.5)

* ui: show a set working directory under the chat form again

On desktop the chip under the form is back once a directory is set, and
the + menu offers the directory only until then.

Assisted-by: Claude (Opus 5.5)

* ui: add a line on Enter with an on-screen keyboard

A phone's Return has no Shift, so on a touch screen Enter adds a line and
the send button sends, as "Send message on Enter" turned off does; on a
desktop Enter still sends.

Assisted-by: Claude (Opus 5.5)
2026-11-10 13:33:50 +01:00
Ramzi 7d6fd845ec convert : mark Sarvam byte fallback tokens as BYTE (#32006) 2026-11-10 13:20:56 +01:00
LinesandSigbjørn Skjæret 4ed88da2ea convert : workaround conversion for p-e-w/gpt-oss-20b-heretic-ara-v4 (#31860)
* fix(conversion-base): changed tokenizer init only for gpt-oss checkpoints, no content modification

* fix(conversion-gpt_oss): normalized 4 tensor suffixes, no content modification

* fix: whitespaces

* revert fix(conversion-base) per #31860

* updated conversion/gpt_oss.py alt->erro reframe

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-11-10 13:00:03 +01:00
Michael Paulauski ac931b5bcb cuda : halve VMM pool reservation size and retry on cuMemAddressReserve failure (#30542)
* cuda : Lowering VMM pool reservation size to 8 GB on GPUs with combined memory due to CUDA out of memory errors

Assisted-by: Gemini Argon

* cuda : Halve VMM pool reservation size and retry on failure

Assisted-by: Gemini Argon

* cuda : only shrink VMM pool on CUDA_ERROR_OUT_OF_MEMORY

Assisted-by: Gemini Argon

* cuda : log warning when VMM pool reserves less than max size

Assisted-by: Gemini Argon
2026-11-10 12:36:29 +01:00
Aman Gupta 15759e0ca9 CUDA: skip graph compat check when graph is being reused (#32004) 2026-11-10 18:53:47 +08:00
Xuan-Son Nguyen a591cbe17c mtmd: share compute buffer among subcontext (#31945)
* shared compute buffer

* add load-time alloc for audio gen
2026-11-10 11:45:09 +01:00
Xuan-Son Nguyen 203b5f404c server: fix output lang for realtime api (#31982) 2026-11-10 11:35:30 +01:00
Xuan-Son Nguyen 351025ec53 mtmd: add kimi K3.1 vision (#31980)
* mtmd: support kimi k3.1 vision

* fix preproc (both k2.5 and k3.1) and projector mlp

* small renaming
2026-11-10 11:28:47 +01:00
Adrien Gallouët 0d9eacadf2 ggml : fix FreeBSD Vulkan build by setting _XOPEN_SOURCE=700 (#31701)
Fixes #31541

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-11-10 11:57:40 +02:00
Aman Gupta 17701e48d7 CUDA: move graph and concurrent stream state out of common.cuh (#32001)
ggml_cuda_graph, ggml_cuda_concurrent_event and ggml_cuda_stream_context
are only used by ggml-cuda.cu, but keeping them in common.cuh meant any
change to them rebuilt every CUDA translation unit. Move them to graph.cuh
and leave forward declarations in common.cuh. The context constructor,
cuda_graph() and any_cuda_graph_*() are now defined in ggml-cuda.cu, and
the stream context is held through a unique_ptr.

Also define ggml_cuda_kernel_can_use_pdl once in ggml-cuda.cu instead of
as a static function in the header, so there is one PDL cache instead of
one per translation unit, and fold the GGML_CUDA_PDL env check into it.
2026-11-10 17:03:49 +08:00
Aleksander Grygier ac007054e7 ui: use a live-preview markdown editor for the chat form input (#31955)
* ui: use a live-preview markdown editor for the chat form input

Replace the contenteditable rich input with a CodeMirror editor that shows
the markdown source of the block being edited and renders every other block
through the MarkdownContent pipeline and stylesheet, so they match chat
messages. Plain text types exactly as before, the value stays the exact
markdown source, and mention links show as atomic badges.

The editor lives in a shared MarkdownEditor component, and the highlight.js
theme loader is shared between MarkdownContent and SyntaxHighlightedCode.
The old rich input, its tokenizer, source history and their tests go away.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: give the production build more memory, note what mdsvex is for

The single-bundle build now includes CodeMirror and runs past Node's default
heap limit on CI runners. Raise it in the build script, so local and CI builds
behave the same on every platform.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: keep the live preview's blocks while only the cursor moves

Moving the cursor re-walked the syntax tree and rebuilt every block's
decoration. Keep the blocks with the decorations and rebuild only when the
text changes or the cursor enters another block.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: point the chat form story at the editor and drop the engraved placeholder until the design system lands

Assisted-by: Claude (Opus 5.5 Medium)

* ui: name the markdown editor for screen readers

CodeMirror's content is a textbox with no name of its own, so the chat form
input was announced as an unnamed text box, and could not be found by its name
as the old input could. It takes an aria-label, its placeholder by default,
which makes the chat form's "Type a message..." again.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: format the chat form input's classes

Assisted-by: Claude (Opus 5.5 Medium)

* ui: share the markdown styles and diagram drawing between messages and the editor

The editor and MarkdownContent each @imported markdown-content.css into their own
<style>, so the bundle carried two scoped copies of it. Both now render into a
MarkdownSurface, the one element that carries the stylesheet; the few rules for
MarkdownContent's own template elements are global under .markdown-content.

Drawing mermaid and svg blocks and the actions of their header buttons move to
markdown-diagrams.ts, so the editor's rendered blocks can draw diagrams the same
way messages do.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: harden the live preview editor after review

- onChange skips only the editor's own syncs from `value`, marked by an
  annotation, instead of reporting only edits that carry a user event
- Enter and the mention badges ask the syntax tree whether they are in code, so
  ~~~ fences and indented code count, and a closing fence ends a block
- a picked mention is one undoable edit, through replaceText
- rendered blocks draw mermaid and svg diagrams with their header buttons, a $$
  math block with blank lines renders as one block, and a link reference
  definition stays raw instead of rendering to nothing
- rendered blocks render again when the mention path settings change
- a drag in a rendered block selects its text and copy takes it, across blocks;
  a click still opens the block
- the block margins are the ones a chat message renders (each heading is a first
  child there), checked by a test; the blank lines after a raw block are its gap
- the tree comes from syntaxTree, without forcing a full parse on every edit
- placeholder, label, extensions, mention widgets and render key follow prop
  changes after mount, through compartments
- the editor uses the markdown language alone, which drops the html, css and
  javascript parsers markdown() pulls in for html tag completion (-70 KB gzip)

Assisted-by: Claude (Opus 5.5 Medium)

* ui: pin the CodeMirror packages and set the build heap in .npmrc

The new dependencies are pinned like the rest of package.json. The build heap
moves from a node call on vite's bin path to npm's node-options, so the build
script is plain `vite build` again and the setting reaches it on every platform.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: update the live preview block by block and estimate block heights

An edit used to build every block's decorations and widgets again, which the
editor then compared one by one. The previous decorations, and the mention badges
of raw blocks, now move through the edit, and only the blocks whose range, text,
raw or rendered state, or gap changed get new ones; the texts of the blocks an
edit left alone are reused instead of sliced again. A test drives random edits,
cursor moves and focus changes and checks each result against a full build.

Rendered blocks report the height they were last measured at, or a guess from
their text, so CodeMirror's height map for blocks off screen in a long document
stays close and scrolling stays steady.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: skip the editor's redundant syncs, reconfigures and theme reloads

- syncing `value` compares it with the text the editor last reported or synced, so
  a keystroke coming back through onChange no longer serializes the document again
- the prop effects reconfigure the editor only when a prop changed, not once more
  right after it is created (which rebuilt the whole preview)
- loadHighlightTheme leaves the theme's style element alone while the mode stays,
  so a message or an editor mounting no longer restyles the whole page

Tests check that typing in one block leaves the other rendered blocks without a
single DOM mutation, and that mounting reconfigures nothing.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: simplify the live preview to a plain rebuild in three small modules

Rebuilding the decorations of every block on an edit measures the same as
patching only the changed ones (0.09 ms against 0.11 ms per keystroke at
200 blocks, 0.9 ms against 1.1 ms at 2000), and equal widgets keep their
DOM either way, so the incremental update and its test go.

- markdown-blocks.ts collects the document's blocks, markdown-block-widget.ts
  renders one, markdown-live-preview.ts decides which ones show raw text
- one bounded cache holds a block's HTML and its measured height
- the render key is a plain option: a new one reconfigures the preview,
  which keeps the editor's focus
- MarkdownEditor follows its props with one helper instead of five effects

Assisted-by: Claude (Opus 5.5 Medium)

* ui: keep setext headings raw in the live preview editor

`test` over `---` is a setext heading, so the preview rendered it as an h2
and the underline vanished once the cursor left the block. The editor now
shows a setext heading's text as typed at all times, like a link
reference definition.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: add a setting to turn the live markdown preview off

"Live markdown preview in the message input" (General) is on by default.
Off, the input shows the plain markdown text, with file mention badges
still in place; switching it takes effect without a reload.

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-10 09:51:01 +01:00
Alexey Kopytko 940a30ee3b model : stop renaming the recurrent cache tensors in plamo2 (#31984) 2026-11-10 17:42:30 +09:00
chinmaykamjith001 45c8ea8374 common/json-schema : fix allOf with plain types (e.g. integer) generating an empty object grammar (#31834)
* common/json-schema : fix allOf with plain types (e.g. integer) generating an empty object grammar

* Fix empty range error and incorporated bot review feedback
2026-11-10 09:57:35 +02:00
Vladimir Sotnikov c90aa996ba common : escape control characters in tool argument strings (#31937)
normalize_quotes_to_json copies raw control characters verbatim while
converting pythonic tool arguments to JSON, so calls like
python(code='a<LF>b') produce invalid JSON in the arguments returned to
clients; replaying that history then fails hard. Escape C0 control
characters inside quoted strings via escape_json_string_inner.
2026-11-10 09:56:53 +02:00
Raul Torres a2c942dcfe cann: fail early when checking if buffer type is supported (#31270) 2026-11-10 09:42:58 +02:00
18c5a95e2b sycl: Add device-side MoE scheduler (#30156)
* sycl: Add device-side MoE scheduler

Assisted-by: Codex

* Add width-heuristic override bit and Q2_K and Q3_K types

Assisted-by: Codex

* tests: cover skipped MUL_MAT_ID slots on the device MoE schedule

Assisted-by: Claude Opus 5.5

* sycl: keep graphs on for lazily reordered MUL_MAT and reject BN=64 combos on the device MoE schedule

The graph check disabled SYCL graphs until every MUL_MAT weight with a
reorder extra was reordered, which never happens for some (dense MXFP4,
the fused GLU plain path), so graphs stayed off and the check reran on
every compute. Upstream already reorders those during recording with
the async memory extension, so only MUL_MAT_ID weights are reordered
ahead of recording now.

ggml_sycl_grouped_dequant_gemm_device_supported() accepted any XMX
combo, but the device schedule only runs with BN == GGML_SYCL_FG_BN, so
a 32x64 combo (GGML_SYCL_XMX_GATHER_SHAPES=4 or 8) made the graph check
pass and the compute fall back to the host sync path while recording.

Assisted-by: Claude Opus 5.5

* sycl: keep the ggml_sycl_mul_mat_id_needs_sync name for the graph check

Assisted-by: Claude Opus 5.5

* fix: stop erroneously reordering host-offloaded tensors

With -ncmoe, prompt processing offloads the CPU-resident expert matmuls
to the GPU: the scheduler uploads the experts into a copy in the compute
buffer before every run. init_tensor gave that copy a reorder record, so
the MUL_MAT_ID gather path reordered every offloaded expert tensor on
every ubatch, and the next upload overwrote the result.

Qwen3-30B-A3B Q4_K_M, 1x B60, llama-perplexity -c 4096 -ub 2048
-ncmoe 48, 8 chunks, on master ac8e1694:
  master                        116 s, PPL 7.739
  master XMX_GATHER_TYPES=0      67 s, PPL 7.738
  with this change               67 s, PPL 7.729

Refs ggml-org/llama.cpp#31262

Assisted-by: Claude Opus 5.5

* Remove SYCL-specific test comments

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* sycl : name the MUL_MAT_ID path predicates after what they decide

ggml_sycl_mul_mat_id_mmvq_ok -> ggml_sycl_mul_mat_id_should_use_mmvq
ggml_sycl_mul_mat_id_mmvq_shape_ok -> ggml_sycl_mul_mat_id_mmvq_eligible
ggml_sycl_mul_mat_id_grouped_shape_ok -> ggml_sycl_mul_mat_id_device_sched_eligible
node_is_device_local -> graph_node_is_in_local_device
ggml_sycl_gg_rows -> ggml_sycl_grouped_gemm_rows
ggml_sycl_gg_tile -> ggml_sycl_grouped_gemm_tile (from #29245, renamed to match)

"eligible" is a necessary condition; the type and width checks follow
elsewhere. "should_use" means the path runs. No functional change.

Assisted-by: Claude Opus 5.5

* sycl : reorder Q3_K MoE experts and load them as vectors in the grouped GEMM

Assisted-by: Claude Opus 5.5

---------

Co-authored-by: cwriter <cwriter@localhost>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-11-10 09:42:25 +02:00
ae4893a092 vulkan: add tiled-transpose concat shader for Qwen models (#30591)
* vulkan: tiled transpose for dim-0 concat with a transposed source

The delta-net conv-state path does ggml_transpose() straight into a dim-0
ggml_concat(), so the generic concat kernel reads src1 with the transposed
stride, one element per thread, and neighbouring lanes land on different cache
lines. Stage a 32x32 tile through shared memory instead, which keeps both the
load and the store coalesced. 33 columns pads away the bank conflicts.

Gated narrowly: dim 0, 2D, unquantized 4-byte elements, contiguous src0/dst
rows, and src1 actually transposed. Everything else falls through to the
existing kernel.

Qwen3.6-35B-A3B on gfx1151/RADV, 30 gated-delta-net layers: the CONCAT op goes
11877 us -> 957 us, and pp2048 at ub2048 goes 1224 -> 1628 t/s. Generated text
is unchanged.

The approach follows the tiled transposed concat Gaetan Puleo wrote for the HIP
backend in his Strix Halo fork.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* enable by default, add smaller shader sizes

* clean up comments

* improve shader

* Add non-contiguous concat tests

Co-authored-by: ravel7524 <58877666+ravel7524@users.noreply.github.com>

* handle x or y axis overflow

---------

Co-authored-by: Nathan Wilson <nath.flagman151@passmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: ravel7524 <58877666+ravel7524@users.noreply.github.com>
2026-11-10 09:41:45 +02:00
figkcoci-debugandfigkcoci-debug 6122b566b0 ggml-rpc : fail gracefully on unreachable endpoints and failed handshakes (#30415)
* ggml-rpc: fail gracefully on unreachable endpoints and failed handshakes

rpc_dispatcher::start() aborted the whole process (GGML_ABORT -> SIGABRT) when
the endpoint could not be parsed, the transport failed to initialise, the socket
could not connect, or the HELLO handshake failed. A typo in --rpc or a
temporarily down node killed the host application, and the reason went to a log
stream the host may never see.

Make the connect-time failure reportable instead of fatal: start() returns bool
(LOG_ERROR + false; the dispatcher thread only starts after a successful
handshake), negotiate_hello() returns false on a half-open socket,
get_dispatcher() returns nullptr and does not cache failed attempts, and every
consumer treats nullptr as "endpoint did not answer" with the conservative
behaviour of its domain (allocation -> nullptr, alloc-size -> lower bound,
device memory -> 0/0, events -> no-op, device count -> 0, and add_server()
already maps 0 devices to nullptr).

Mid-stream failures keep the existing RPC_STATUS_ASSERT by design (fail fast
against silent corruption) and are intentionally out of scope.

Field motivation: on Android (llama.rn + phone RPC cluster) an unreachable
worker crashed the app; the crash buffer showed
abort <- ggml_abort <- rpc_dispatcher::start <- ggml_backend_rpc_add_server.

* tests : add a regression test for unreachable rpc endpoints

Calls ggml_backend_rpc_add_server with a socket that nothing listens on and with a
malformed endpoint, and expects nullptr from both. Without the fix in this branch
the first call aborts the process, so the test cannot even fail cleanly.

* ggml-rpc remove low-value comments

* tests : drop the internal device-count check after the rpc registry refactor

---------

Co-authored-by: figkcoci-debug <figkcoci-debug@users.noreply.github.com>
2026-11-10 09:40:35 +02:00
Masato Nakasaka 3837bf0bc0 vulkan: Divide GROUP_NORM workgroup to 2D to avoid maxComputeWorkGroupCount limit (#31930)
* Divide workgroup to 2D to workaround maxComputeWorkGroupCount limit

* Replaced with existing macro
2026-11-10 08:22:28 +01:00
Ed Addario c1a4febe9d vulkan: Fix out of bounds reads/writes on MoltenVK (#31890)
* Fix test-backend-ops sentinel mismatch errors on MoltenVK

* Set disable_robustness for norm_f32, l2_norm_f32, and sum_rows_f32's pipelines
2026-11-10 08:19:11 +01:00
Hongqiang Wang 78ffbaa185 vulkan: cap scalar flash attention at 8 rows for large heads on Qualcomm (#30829)
With head sizes above 128 and KV >= 1024, scalar flash attention uses
16 rows per workgroup. On the Qualcomm Adreno driver this variant
spills registers to scratch memory (264 bytes per invocation on the
X1-85, none at 8 rows) and runs about 40x slower. The largest
test-backend-ops case then becomes a single dispatch of several
seconds, past the Windows GPU timeout, and the test never completes on
the X1 Elite.

Use at most 8 rows per workgroup on Qualcomm when hsk or hsv > 128.
Smaller heads keep 16 rows, which is faster for them on the X2 Elite.

Adreno X1-85, FLASH_ATTN_EXT hsk=256 kv=4096: nb=64 711 ms -> 17 ms,
nb=512 no result -> 125 ms; test-backend-ops completes. Adreno X2-90:
Gemma 4 E4B pp512 at depth 4096 with flash attention 81-90 -> 117 t/s.
2026-11-10 08:00:55 +01:00
Oscar M. Herrera 58e38bdf65 llama : apply the iGPU lazy-mode fallback only when mmap is off (#31957)
On a device without mmap support, --lazy-mode auto resolved to off even
with --load-mode mmap, so llama_prefetch_rows never ran and the rows of
the large tensors faulted in one page at a time. Resolve it to off only
when mmap is off; the default iGPU load path is unchanged.

Fixes #31943
2026-11-10 14:55:46 +08:00
Asahi-PrvandNeo Zhang d5bbcd1f31 ci: update oneAPI 2026.2, oneDNN 2026.1 and Level Zero SDK 1.34.0 (#31540)
Co-authored-by: Neo Zhang <zhang.jianyu@outlook.com>
2026-11-10 01:18:44 -05:00
Gotya-OFFCIALandTitaniumtown 8766c0273c sycl: un-gate oneDNN fused SDPA for Alchemist at head_dim=64 (#31574)
Co-authored-by: Titaniumtown <titaniumtown@proton.me>
2026-11-10 01:08:06 -05:00
Ramzi 3b4d6ab3f7 chat : don't use lazy grammar when response_format is set with tools (#31956) 2026-11-10 01:44:07 +01:00
Dmitrii Panchev cf6463ace7 hexagon: fix 64-bit FastRPC handle truncation on close (#30952) 2026-11-09 16:10:13 -08:00
Matt Thompson 711f9f4c11 server: responses api: support conversion of text.format.json_schema to completions (#26145) 2026-11-10 00:43:38 +01:00
Aldehir Rojas 26d52b0f95 server : use common_chat_input in realtime api (#31978) 2026-11-09 23:59:16 +01:00
Xuan-Son NguyenandPascal 08f945ffb5 server: add oai-compat /v1/audio/speech endpoint (#31763)
* server: add oai-compat /v1/audio/speech endpoint

* add e2e pytest

* server: share the s16 rounding, type speech form fields

The WAV writer of the audio helper truncates while pcm_f32_to_s16
rounds, so wav and pcm differ by one LSB. Both now round.

pcm_f32_to_s16 takes a pointer and a count and converts the TTS
bytes without an intermediate copy.

/tts and /v1/audio/speech parse their body with parse_tts_body, so
multipart fields such as speed get their type before the speech
conversion checks them.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-09 23:58:52 +01:00
Xuan-Son NguyenandPascal 20c2e96655 mtmd: support gemma 4 input audio stream (#31965)
* mtmd: support gemma 4 input audio stream

* use 480ms step

* server: enable realtime embeddings only for audio generation

The realtime slot forced embeddings on, which marks every row of a
batch as an output. Only a model that generates audio reserves
n_batch outputs, so a chat model with an audio stream (Gemma 4)
aborted on its first batch of audio rows. The hidden state is only
read by the audio generation pipeline.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-09 23:36:31 +01:00
Xuan-Son Nguyen 3d94ffe452 mtmd: add mtmd_helper_resampler (#31964)
* mtmd: add mtmd_helper_resampler

* update docs

* add test

* MA_MAX_FILTER_ORDER
2026-11-09 23:04:39 +01:00
Aleksander Grygier 0a94a3f289 ui : retry opening the sidebar when a demo imports its traces (#31952)
A cold dev server reloads the page once it has optimized its
dependencies, and that reload can land right after the harness opened
the phone sidebar, closing it again before Settings is clicked. Repeat
the pass over the menu keys up to three times while Settings is not
reachable.

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-09 21:08:01 +01:00
Aleksander Grygier 610f7ab40a ui: make the app feel native on phones (#31949)
* ui: reveal the mobile sidebar by sliding the page aside

On a phone the sidebar now stays put under the page, and the page slides
right to uncover it, as a rounded card with the card border. The menu
button rides on the page; a swipe from the left edge opens the sidebar,
and a swipe back or a tap on the page closes it.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: polish the mobile sidebar reveal and add a hold menu to rows

Holding a conversation row on a phone opens its action menu anchored to
the row, in place of the hidden three-dot trigger. The page no longer
widens the document when it slides aside, so iOS stops panning it on the
first open.

The page and sidebar backgrounds start in the page color and blend into
the sidebar tint as the sidebar opens, following a drag frame by frame,
and a theme-color tag follows them for the browser bars. The sidebar's
contents fade in on a log curve and scale up slightly, and sit at the
same inset as the menu toggle, which now uses a two-line icon in a
filled circle.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: pin the context gauge on phones and lock the slid-aside page

On a phone the context gauge leaves the chat form for the top right of
the chat screen, in a round button across from the menu toggle. Both
take the scroll-down button's frosted look.

While the sidebar is out, and until the page is fully back in place,
the page neither scrolls nor takes taps; a tap or drag on it still
closes the sidebar.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: open the mobile sidebar with a swipe from anywhere

A rightward swipe from anywhere on the page now opens the phone sidebar
too, beside the edge swipe that competes with the browser's own back and
forward gesture. It starts after a short sideways move, skips text
fields and content that scrolls sideways, and keeps the page from
scrolling under it. Letting go opens or closes the sidebar past 30% of
the way or on a flick.

The top buttons and the chat form now hold their place on screen while
the page is slid aside, instead of jumping to the far ends of a
scrolled conversation.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: hide the model org name on mobile unless a setting keeps it

A phone's model selector is narrow, so it now drops the organization
from model names by default. A new Display setting, "Show organization
name on mobile", brings it back; it applies only while the organization
name is shown at all.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: close the mobile sidebar before following a tap in it

On a phone, picking a conversation, New chat, Search, MCP or Settings
in the open sidebar now slides it shut first and acts 180ms later, so
the page or dialog appears once the sidebar is out of the way. With the
sidebar already closed, or on a desktop, the action runs at once.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: make the app feel native on phones and run it under the status bar

Applies the mobile-native checklist: no tap highlight or landscape text
inflation, instant taps on buttons and links, no label selection or
callout on a held control, hover styles only where a pointer can hover,
and 16px textareas on phones so focusing them does not zoom the page.

The page now runs under the notch (viewport-fit=cover), so the installed
iOS app tints its status bar from the page and the sidebar. The menu
toggle, the context gauge, the sidebar and the top of the chat and
models pages pad themselves clear of it, and the locked, slid page
reaches down by the status bar's height, which iOS leaves out of it.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: add a viewport height readout for debugging on phones

With VITE_DEBUG set, a dev build shows the window, viewport unit and
safe-area heights, and the elements that reach past the window's bottom
and make the page scroll. It is loaded on demand, so production builds
leave it out.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: fill the screen from launch in the installed iOS app

iOS 26 gives an installed app with viewport-fit=cover and a translucent
status bar a viewport short of that bar (WebKit bug 301108), clipping
the bottom of the page and lifting the chat form, but only while the
page is shorter than the screen. In the installed app the page now
reaches the screen plus the status bar, capped at the screen, so it
gets the full screen from the start without ever scrolling.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: raise the phone's corner controls in the installed app

The installed app's status bar already sets the menu toggle, the context
gauge and the open sidebar's logo apart from the screen's edge, so they
sit just below it rather than a full 1rem lower. They share one value,
which stays at 1rem below the inset in a browser tab.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: add safe-area utilities for what is pinned to a screen edge

The page runs under the status bar and over the home indicator, which an
installed iPhone app always shows. New pt-safe, pb-safe, top-safe and
bottom-safe utilities, each with a spacing step (pt-safe-12 is 3rem plus
the top inset), keep pinned content clear of them instead of hand-written
calc()s. A --safe-area-bottom joins --safe-area-top, and the chat screen and
Models page use pt-safe-12.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: open search in a full-height drawer on phones

A phone used to leave for the /search page. It now opens the same search
as the desktop dialog, in a bottom drawer that fills the screen below the
status bar. Keyboard shortcuts, the Enter hint and the keyboard highlight
stay on desktop, and so does the models manager, as in the phone sidebar.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: show every dialog full screen on phones, clear of the status bar

A phone now shows every dialog full screen, whatever size it asks for on a
desktop, instead of each full-screen dialog repeating its own classes. Its
top and bottom borders take the status bar's and the home indicator's
height, transparent, so an installed app's dialogs keep their content clear
of both with no change to their padding. The attachment preview, which has
its own full-screen content, pads itself and lowers its close button the
same way.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: fade the page out under the status bar of the installed app on phones

Assisted-by: Claude (Opus 5.5 Medium)

* ui: show search in the full-screen dialog on phones instead of a drawer

Assisted-by: Claude (Opus 5.5 Medium)

* ui: start a phone page's title a row below the corner controls

Assisted-by: Claude (Opus 5.5 Medium)

* ui: pin the slid page's bottom bars against the screen in the installed iOS app

Assisted-by: Claude (Opus 5.5 Medium)

* ui: cast a soft shadow from the slid page onto the phone sidebar

Assisted-by: Claude (Opus 5.5 Medium)

* ui: hide the scroll-down key at the real bottom on iOS and show it only well above

Assisted-by: Claude (Opus 5.5 Medium)

* ui: fade the status-bar fade out as the page slides aside for the sidebar

Assisted-by: Claude (Opus 5.5 Medium)

* ui: draw the slid page's shadow above its rounded frame, so it follows the corners

Assisted-by: Claude (Opus 5.5 Medium)

* ui: show a phone chat's title and working directory at the top, and pick it in a drawer

Assisted-by: Claude (Opus 5.5 Medium)

* ui: give iOS Safari a solid top and fade the bottom of the screen on phones

Assisted-by: Claude (Opus 5.5 Medium)

* ui: let iOS Safari find the top strip, which it skips when it takes no taps

Assisted-by: Claude (Opus 5.5 Medium)

* ui: lengthen the top fade and lift the chat form a little in phone browsers

Assisted-by: Claude (Opus 5.5 Medium)

* ui: give iOS Safari's bottom toolbar the page's colour, not the chat form's

Assisted-by: Claude (Opus 5.5 Medium)

* ui: draw the phone's context gauge key like the menu key across from it

Assisted-by: Claude (Opus 5.5 Medium)

* ui: confirm opening the phone sidebar, its hold menu and its rows with a haptic tap

Assisted-by: Claude (Opus 5.5 Medium)

* ui: keep the screen-edge fades on the page as it slides aside for the sidebar

Fading them out with the slide made the page flicker; the rounded frame over the
slid page already clips them at its corners.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: keep the working directory chip optional for hosts with their own trigger

The phone layout opens the picker from the chat's header, so ChatForm hides the
chip there. Master's version of the chip has no such switch; bring it back.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: style layer 5's additions with master's tokens until the design system lands

Assisted-by: Claude (Opus 5.5 Medium)

* ui: drop the design system's stylesheets and fonts from app.css until it lands

Assisted-by: Claude (Opus 5.5 Medium)

* ui: keep master's font stacks until the design system brings its fonts

Assisted-by: Claude (Opus 5.5 Medium)

* ui: restore master's colour palette in app.css until the design system lands

Resolving the safe-area commit took the design system's app.css, which drops
the palette for its token files and aliases the shadcn names to semantic
tokens. With the tokens gone those aliases point at themselves, so every
colour went invalid: transparent surfaces and currentColor borders.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: center alert dialogs on phones instead of sliding them up as drawers

An alert is a short question, not a drawer: on a phone it is now a centered
card, 90% of the screen wide and as tall as its text, scrolling only past the
screen's height. It zooms in as on desktop, and its footer drops the margin
the sheet needed.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: space the working directory drawer for a phone

The drawer reused the popover's compact picker: Browse sat against the search
field, the divider ran past the rows and each row had its own inset. In the
drawer the picker now takes the header's inset, gives Browse room, lines its
rows up with the search results and sizes its text for a phone.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: tap the haptic engine when a message sends, and once per hold

A phone now confirms a sent message by feel, at the moment the send goes
through (not on every press of the button), and before any await, so it still
runs inside the tap or keypress that iOS needs to play its tick. The hold that
opens a row's menu now taps the engine from the long-press hook itself, once:
the hook vibrated and the row tapped as well, which buzzed twice on Android.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: name the gauge dial sizes, the chat form's bottom offset and the sidebar fade

Review follow-ups: the context gauge dial reads its grid, radius, stroke and
track from one table per size instead of ternaries; the chat form's distance
from the screen's bottom per platform and the sidebar content's fade duration
are named constants. The viewport debug readout moves out of src, next to the
tests, as it is a debug tool the app only loads in a dev build.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: say why the microphone can't record instead of failing silently

A llama-server opened over plain HTTP on the local network or over Tailscale
is not a secure context, where browsers hide the microphone, so a tap on the
mic did nothing but log to the console. The button now shows a crossed-out mic
in the warning tone with the reason in its tooltip, and a tap says it in a
toast, as a phone shows no tooltip. A start that fails names the browser's
reason (access denied, no microphone, in use by another app) the same way,
and the next tap tries again. The recorder passes the browser's error on
instead of replacing it with a generic one.

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-09 21:06:28 +01:00
Piotr Wilkin (ilintar)andXuan Son Nguyen 5f490d7bfe feat: external PLE file loading (#31185)
* model : allow loading external PLE files

Assisted-by: Codex

* convert : add separate PLE exports

Assisted-by: Codex

* tests : fix PLE export test lint

Assisted-by: Codex

* model : make --ple an alias of --shard, allow multiple shard locations

Reuse the multi-shard loading path for external PLE files instead of a
separate option.

* --ple is an alias of --shard: both take a comma separated list and may
  be given more than once, and the shards may live anywhere
* metadata comes from the main file; numbered shards are still
  auto-discovered and strictly validated
* explicit files outside the discovered set are appended as extra shards,
  exempt from split.no and split.count checks
* -isnm/--ignore-shard-num-mismatch loads only the given files, in order,
  with no auto-discovery and no split metadata validation
* ple_path, path_ple and llama_model_loader::add_ple_file are removed;
  common_model_load() uses llama_model_load_from_splits for two or more
  paths
* common/fit.{h,cpp} take the shard path vector, so --fit and the router
  memory measurement load every shard

Assisted-by: GPT-6-Astra

* rm -isnm

* make sure to dedup paths

* rm test_convert_ple

* address comments

* allow ple shard to override tensor

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-11-09 19:41:01 +01:00
Julius TischbeinandOliver Simons 9d340ec341 Windows unbuffered model load (#26014)
* Windows unbuffered io

* Fix read len calculation

* Add test and model load timing

* chunked fallback read + removing filepointer semantic divergence between linux and windows

* tests : remove unused mmap test arguments

Assisted-by: Codex

* remove timing, set file pointer after chunk read, test refinement

* Update tests/test-llama-mmap.cpp

Co-authored-by: Oliver Simons <osimons@nvidia.com>

* Update src/llama-mmap.cpp

Co-authored-by: Oliver Simons <osimons@nvidia.com>

* test refinement

Assisted-by: Codex

* remove duplicate constexpr

Assisted-by: Codex

* adding label behavior to tests

Assisted-by: Codex

---------

Co-authored-by: Oliver Simons <osimons@nvidia.com>
2026-11-09 19:22:08 +01:00
nv-vankitandPascal 359dc5858e server : functionality test for new spec dec (#31027)
* server : functionality test to make sure every time when we add new spec dec method the existing ones are not broken.

* server: use compact models for speculative decoding CI

* ci: avoid GHA cache for speculative models

* ci: run speculative tests on DGX Spark

* server: address speculative test review

* server: use consolidated speculative models

* server: refine speculative test calibration

* docs: record speculative Spark validation

* ci : run the server-speculative steps with bash

* server: recalibrate speculative test floors on GB10

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-09 19:08:20 +01:00
Aaron Teo a4d4f2fd4e ggml-zdnn: accept non-zdnn tensors, invalidate compute tensors after every op (#31824)
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
2026-11-09 15:48:32 +02:00
Ethan Linandlinsen 1945793e00 server : fix response_format json_schema when schema is not nested in the json_schema wrapper (#28697)
* server : accept top-level schema in response_format with type json_schema

The OpenAI-compatible chat completions endpoint only applied the schema
when it was nested inside response_format.json_schema.schema. When the
schema was provided directly as response_format.schema (as documented in
the server README, and used by some clients), it was silently ignored
and generation ran unconstrained.

Also mention json_schema in the error message for invalid response_format
types, document the OpenAI wrapper form in the README, and add a test
case for the top-level schema form.

Fixes #10732

* server : let the json_schema wrapper take precedence by presence

Pick the schema source based on whether response_format has a
json_schema key, instead of falling back when the parsed wrapper schema
is empty. An empty schema {} is valid (any JSON value), so the old check
let a top-level schema override an explicit {} in the wrapper. Use
json_value() for the top-level schema too, like the rest of the
function, and add tests for the precedence.

---------

Co-authored-by: linsen <251731047+linsen458-spec@users.noreply.github.com>
2026-11-09 14:10:11 +01:00
losewayy 0eebb6eddc ggml : implement the backward pass for TRI (#30712)
* ggml: implement the backward pass for TRI

GGML_OP_TRI had no case in the backward switch, so a trainable graph containing it aborted with "unsupported ggml op for backward pass: TRI". The op is linear, so its gradient is masked the same way. The incoming gradient is not necessarily contiguous (the backward of a transpose hands over a view) and ggml_tri requires contiguity, so it is made contiguous first.

test_tri gains a post_transpose variant (out = transpose(tri(a))) that aborts as above without this change.

* ggml : simplify the TRI backward pass

Pass the gradient directly to ggml_tri and drop the transpose test. The existing TRI cases cover the backward pass for contiguous F32 gradients.
2026-11-09 15:03:00 +02:00
Georgi Gerganov d628a98f09 tests : keep dummy ffn.act scales away from zero (#31940)
The dummy-model generator fills every tensor from a plain normal pool.
`blk.N.ffn.act.scales` is a divisor in the mpt graph, so with seed
1224489258 it draws |v| ~ 1e-6, and `ffn_act = gelu(...) / scales` reaches
2.7e5. Metal stages both operands of `kernel_mul_mm_f32_f32` as half, so
those activations overflow to inf, the norm downstream turns the inf into
nan and the logits come out nan, while the CPU reference stays in f32 and
stays finite. Both NMSE values then become nan and the check fails.

Give `ffn.act` the same treatment as `rope_factors` and keep its magnitude
away from zero.

Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (RTX 5090 32GB, DDR5 128GB)
2026-11-09 14:54:27 +02:00
Aman Gupta a8fad21e99 mtp: read the target hidden states from a device buffer (#31927) 2026-11-09 20:52:32 +08:00
Aleksander Grygier b39680c41a ui: fold, preview and rearrange the sidebar's sections (#31936)
* ui : fold the sidebar's conversation sections

Pinned and Recent conversations fold under their heading. The whole heading
row is the toggle; its chevron follows the title, shows on hover and stays
while the section is folded. Folded sections are remembered across reloads.

Assisted-by: Claude (Opus 5.5 Medium)

* ui : show a sidebar section's first six chats

Pinned and Recent list their first six chats and end with a Show more
row that opens the whole section; Show less, sticking to the bottom of
the sidebar while the section scrolls under it, closes it back. The
expanded sections last for the session.

Assisted-by: Claude (Opus 5.5 Medium)

* ui : rearrange the sidebar's sections and chats by dragging

Pinned and Recent can be dragged by their title. The real section follows
the pointer, tinted like the sidebar's resize handle and edged with a
faint border, the other sliding aside as it passes; let go, it glides from
there into its new slot. The order is remembered across reloads, and a
section never moved keeps its default place. All sections now scroll
together.

A chat dragged onto Pinned is pinned, onto Recent unpinned. A lifted copy
of the row follows the pointer while the row slides out of its section.
The section under the pointer takes the same tint and border; a section
ignores its own chats. Let go, the row slides in where its section sorts
it, the rows below making room, and the copy glides into it; anywhere
else, it slides back into its own section.

While dragging, the rows and headings passed over take no hover. Dragging
needs a mouse or a pen, so a finger still scrolls the sidebar; the sidebar
scrolls near its edges, and Escape cancels.

Assisted-by: Claude (Opus 5.5 Medium)

* ui : set the sidebar's sections apart from their rows

A section heading now reads above its chats: larger, semibold and in the
full foreground colour. Chat titles drop to a regular weight in a slightly
softened colour, full again when the chat is open or hovered, and "Show
more" drops to a regular weight in the muted colour. Chats inside a
section sit closer together, sections further apart, and the sidebar's
actions get more room above and below.

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-09 13:42:44 +01:00
ArshM17-NV 806b3c2bed ggml-backend: don't upload inputs that are not read (#31419)
* stop uploading inputs nothing reads

* Address nitpicks
2026-11-09 14:40:17 +02:00
jjperry69andGabe Goodhart 0c6dca2fae models : Attach Granite NVFP4 weight scales (#31449)
* models : Attach Granite NVFP4 weight scales

* feat: Pass nvfp4 scales through build_ffn for all granite architectures

Branch: granite-nvfp4-moe-scales
AI-usage: full (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

---------

Co-authored-by: Gabe Goodhart <ghart@us.ibm.com>
2026-11-09 13:04:06 +01:00
Iswarya AlexandDavid Hekimian fc0be86182 ci : Run gfx1151 ROCm job on Strix Halo Devlab runners (#31567)
* fix(rocm,ci): Update ci workflow to use devlab ci runners for gfx1151

* ci : move gpu-rocm job to Strix Halo devlab runners

Assisted-by: Claude

* ci : ROCm 10 wheels with per-GPU device packages for gpu-rocm; add gfx1201

The runners' system ROCm 7.2.1 cannot compile the current HIP code for
gfx1151 ("Invalid dpp_ctrl value" in mmf-instance-ncols_*.cu). Go back
to the TheRock ROCm 10 wheels with the device-<gfx> extra, so rocBLAS
has the target's kernels, and keep ci/run.sh's results and models under
$RUNNER_TEMP instead of the runners' RAM-backed home. Add a gfx1201
(Radeon AI PRO R9700) row with HIP graphs disabled.

Assisted-by: Claude

---------

Co-authored-by: David Hekimian <dhekimian@users.noreply.github.com>
2026-11-09 13:01:52 +01:00
CCandZhuomin Chen e01c9f1875 metal : prevent out-of-bounds writes to the Metal FA output buffer (#31662)
Co-authored-by: Zhuomin Chen <moanachen@Mac.attlocal.net>
2026-11-09 13:32:27 +02:00
losewayy 4b46d96538 metal : make ggml_tri select instead of scaling by the comparison (#30711)
* metal: make ggml_tri select instead of scaling by the comparison

The kernel multiplied by the 0/1 comparison, which turns a +-inf outside the triangle into NaN, while the CPU kernel writes a plain zero there. Select, like the CPU does.

Adds test_tri_special, which puts +-inf into the cells that have to be discarded, for all four triangle types. It fails against the previous kernel and passes with this change.

* Just trim down the repetitive comments.
2026-11-09 13:31:16 +02:00
Sigbjørn Skjæret f3178366b9 opencl : include climits (#31935) 2026-11-09 11:49:03 +01:00
Marco Colombo 2d99adb108 mtmd: Pocket TTS - add generic support for temperature set via API (#31655)
* mtmd, server : add temperature to /tts, default Pocket TTS to 0.3

Assisted-by: Claude Opus 5.5

* feat: add temp parameter in llama-tts

* temp: fix default hardcoded values

* fix: remove too narrow test

* fix: addressing PR comments
2026-11-09 11:20:45 +01:00
Aman GuptaandGeorgi Gerganov cf92c1323f RPC: Improve loading times (#29413)
* RPC: improve loading

* cont : bump RPC protocol version

* address review comments

* cont: more review comments

* cont : clean-up

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-11-09 12:01:10 +02:00
Takamichi Ohara 9f0d80cf4a ggml : support YaRN truncation config on openvino, sycl, vulkan, webgpu, rpc (#30802)
* openvino : support YaRN truncation config

* sycl : support YaRN truncation config

* vulkan : support YaRN truncation config

* webgpu : support YaRN truncation config

* rpc : support YaRN truncation config
2026-11-09 11:59:07 +02:00
Titaniumtown a0ad695cb2 sycl: cache device-id (#31126) 2026-11-09 10:51:18 +02:00
Pascal 6590d25a32 tests: bound the test_rope tolerance by the f32 angle resolution (#31874)
The rotation angle of a position p is rounded in f32 to about
p*FLT_EPSILON radians, and the CPU reference and the GPU kernels reach
it through different rounding sequences (cumulative product versus
pow). With positions up to 8192 this puts the NMSE floor between
backends near 8e-8, so the default 1e-7 threshold fails at random on
Metal. Derive the tolerance from n_ctx instead: it stays at 1e-7 for
n_ctx=512 and becomes 9.5e-7 for the YaRN cases at n_ctx=8192.
2026-11-09 09:31:13 +01:00
lhez 53eb669432 opencl: add f32 MUL_MAT_ID (#31783)
* opencl: add f32 mul_mat_id

* opencl: handle -1 index

* opencl: make loading moe_sort_by_expert general
2026-11-09 10:10:59 +02:00
Aaron Teo d78f4255d4 ggml-zdnn: fix release binary not detecting backend lib (#31905)
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
2026-11-09 10:10:26 +02:00
Xuan-Son Nguyen 7d7f028753 mtmd: refactoring: remove clip.h, consolidate code into mtmd (#31865) 2026-11-08 15:49:06 +01:00
Kong Jiale b65bf69213 cpu: enable SpacemiT IME2 MXFP4 dense repack (#31776)
Assisted-by: Codex

Signed-off-by: gitGurugu <2049447146@qq.com>
2026-11-08 16:45:08 +02:00
Aaron Teo a3c7cf08f9 llamafile: impl bf16 and q8_0 vxe (#31866)
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
2026-11-08 16:44:25 +02:00
57d04820df ggml : update CUDA/Metal radix Top-K to make tied selections deterministic (#31084)
* ggml : fix for the radix TOP_K thread timing bug

* make the key an ordered 64-bit pair

* a rewrite without shift-skip logic

* fix casts

* warp-contiguous layout

* reduce the scope removing vulkan+metal changes

* turn it off for Metal/ROCm, note about cub::DeviceTopK

* fix the hip build

* skip Vulkan as well

* popc lambda from fattn

* do block_cols math in int64_t

* make it allowlist bc SYCL looks non deterministic too

* ggml : make Metal radix Top-K tied selections deterministic

The radix top-k kernel gathered the selected indices with atomic
counters, so the order of the tied candidates at the threshold (and,
with more ties than remaining slots, the subset itself) depended on
the thread scheduling.

Replace the two atomic compaction loops with a deterministic scheme:
count the elements above and tied with the threshold in contiguous
per-thread chunks, rank them with a simdgroup scan and a serial scan
over the (now unused) radix histogram, and write each kept element
exactly once in column order. The output is now byte-identical
across runs.

This is also faster than the atomic version: the strided atomic
loops become contiguous passes, reducing top-k times by ~20-45% on
M5 Ultra, with tie-heavy shapes within a few percent.

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)

* tests : enable deterministic top-k check on Metal, add radix-path tie cases

Metal now produces deterministic tied selections for the radix top-k
path, so the determinism self-check runs for the MTL backend as well.

Add eval cases covering the radix path (ncols > 2048) with more ties
at the threshold than remaining slots, and tie-heavy perf cases to
monitor the cost of the deterministic tie handling.

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)

* ggml : use coalesced accesses in the Metal radix Top-K compaction

The deterministic compaction walked contiguous per-thread chunks with
scalar sequential loads, which the memory system of the M2 Ultra can
not overlap well: the radix Top-K times regressed by ~1.5-2.4x on the
large-ncols shapes.

Give each warp a contiguous chunk of the row instead and let the lanes
interleave within it with stride 32, so every access is fully
coalesced. The lanes write their kept elements at lane-major ranks,
which is deterministic, and only a tie overflow falls back to a
per-step cross-lane ranking to keep the lowest tied indices in column
order.

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)

* tests : remove the extra top-k tie and perf cases

The radix-path eval and tie-heavy perf cases were added to validate and
benchmark the Metal deterministic tie handling. Keep only the enabling
of the determinism self-check for the MTL backend.

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)

* metal: scan the radix bins and warp totals with a simdgroup

The radix top-k selected the threshold bucket with a serial scan of
the 256 bins by thread 0 in each of the 4 passes, and ranked the warps
of the compaction with two serial scans by thread 0. The first
simdgroup now does both: each lane sums 8 bins, an exclusive prefix
sum and a ballot find the lane that reaches the k-th value, and the
warp totals get their exclusive bases from simd_prefix_exclusive_sum.

* tests: enable the deterministic top-k check on ROCm

HIP has no cub::DeviceTopK, so every row goes through the radix or
bitonic path, which break ties in the same way in each run. The check
covers all tie cases on ROCm, including the 1 and 2 row ones that CUDA
skips.

* cuda: remove the redundant radix top-k counter reset

Address review from ORippler. top_k_radix_init already zeroes
greater_count for each chunk of rows and top_k_radix_select keeps it,
so the reset kernel before the gather is one launch with no effect.

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-08 16:42:57 +02:00
sargentmajordandSargentmajord 9982004913 ggml-cuda : add d_state=64 fast path to SSM_SCAN kernel (#31818)
Zamba2 and similar architectures use a state size of 64 in their
recurrent-scan step, which fell outside the GPU kernel's existing
fast paths (96, 128, 256). Every scan step was falling back to CPU,
bottlenecking generation to 6-7 tok/s against the 50-70 tok/s band
other architectures achieve on the same hardware.

Add a 4th template instantiation for d_state=64, following the
existing pattern; no change to the underlying math. Verified against
the existing CPU-reference correctness harness with 3 new test cases
built from Zamba2's real hyperparameters (28/28 passing).

Measured ~35% throughput improvement (11 -> 15.1 tok/s) on Zamba2-7B-
Instruct. Full parity with other architectures not yet reached - a
separate, unrelated fraction of scan calls (~28%) still falls back to
CPU for a cause not yet diagnosed.

Assisted-by: Claude Sonnet 5

Co-authored-by: Sargentmajord <331229907+sargentmajord@users.noreply.github.com>
2026-11-08 15:48:20 +02:00
Alexey Kopytko cf34d31133 moe-cache : ensure CPU part does not wait for the cached experts (#31888) 2026-11-08 22:29:11 +09:00
LeonandMasashi Yoshimura a691b23458 webgpu : add COUNT_EQUAL support (#31578)
* webgpu : add COUNT_EQUAL support

Assisted-by: Codex

* webgpu : reduce small COUNT_EQUAL inputs in one dispatch

Assisted-by: Codex

* webgpu : simplify COUNT_EQUAL and reuse existing tests

* webgpu : move COUNT_EQUAL pipeline cache declaration

* Update ggml/src/ggml-webgpu/ggml-webgpu.cpp

Co-authored-by: Masashi Yoshimura <yoshimura.masashi.frbs@gmail.com>

* tests : simplify COUNT_EQUAL coverage

---------

Co-authored-by: Masashi Yoshimura <yoshimura.masashi.frbs@gmail.com>
2026-11-08 11:50:09 +02:00
Aaron Teo 2277e75297 ci: move runners to Ubuntu 26.04 (#31529)
* ci: update to ubuntu26.04

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* ci: bring back toolchain workaround

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* ci: remove workaround completely

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* ci: more switching to ubuntu 26.04

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* ci: pin android cmake version

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* ci: restore ubuntu 24.04 for s390x

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* ci: fix missing s390x ubuntu 24.04 release link

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* ci: switch release-publish to ubuntu-latest

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* devops: revert --break-system-packages

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* ci: bring back ubuntu 24.04 for arm release

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* ci: revert ubuntu 26.04 from release.yml

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* ci: fix broken links

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* ci: fix openvino job naming

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* devops: revert vulkan dockerfile to use system python3

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

---------

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
2026-11-08 11:47:29 +02:00
Georgi GerganovandPascal 12ab597986 ggml : switch to F32 src1 in the CPU backend (#31854)
* ggml : switch to F32 src1 in the CPU backend

Change the default computation for GGML_TYPE_F16 to use F32 for src1
data in matrix multiplications: vec_dot_type(F16) is now F32 and the dot
product goes through the new ggml_vec_dot_f16_f32 (x = F16 src0, y = F32
src1), so src1 is no longer quantized to F16 and worst-case inputs that
only fit in the F32 FP range are handled correctly.

- conv_2d/conv_3d: store im2col patches (the inner mul_mat src1) as F32
  and pass separate src0/src1 types to the inner mul_mat; the previous
  F16 patches made the inner mul_mat widen src1 into the conv work
  buffer, corrupting the patches and crashing on large inputs
- conv_transpose_1d/2d (F16 kernel): keep src1 in F32 in the work buffer
  and use ggml_vec_dot_f16_f32; update work buffer planning accordingly
- supports_op: restore MUL_MAT support for F16 src1, which is now
  widened to F32
- ggml_vec_dot_f16_f32: add GGML_SIMD support by converting chunks of x
  to F32 and reusing the F32 SIMD dot
- mul_mat/mul_mat_id: add TODOs for a more efficient F16 x F16 path that
  can be used if ggml_prec_get_src(dst, 1) allows converting src1 to F16

MUL_MAT f16 x f32 perf on M5 Ultra (GFLOPS):
- m=2048, n=512, k=16 : 120.9 -> 101.7
- m=2048, n=512, k=64 : 150.5 -> 344.2
- m=4096, k=14336 GEMV:    63 -> 343-415
- m=4096, n=512, k=14336: 65.6 -> 770.2

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)

* cont : add some F16 x F32 vectorized paths

* cont : enable the fused F16 x F32 dot on AVX without FMA

The 256-bit fused path of ggml_vec_dot_f16_f32 required AVX2 and FMA,
so CPUs with AVX and F16C only fell back to the chunked conversion and
F16 decode was 4x slower than before. Enable it on __AVX__ && __F16C__
and accumulate with GGML_F32x8_FMA, which does mul + add without FMA.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-08 11:44:00 +02:00
PascalandGeorgi Gerganov d6be065572 llama, mtmd: follow the reference precision of ggml_prec_set_acc calls (#31851)
* llama, mtmd: follow the reference precision of ggml_prec_set_acc calls

Requests that the reference implementation does not keep in F32 move
to GGML_PREC_BF16: the GLM4 family down and output projections, the
MiMoVL vision matmuls and the Qwen3-TTS speaker encoder conv1d. The
DeepSeek V4/V4.1 routers and the MiniMax-M3 indexer stay in F32 with
a link to the reference line that computes them in F32.

* ggml: handle GGML_PREC_BF16 the same way as GGML_PREC_F32 in backends

Backends that reject GGML_PREC_F32 on the src1 of MUL_MAT_ID now also
reject GGML_PREC_BF16 (CUDA, Hexagon, OpenCL, WebGPU, Vulkan), and
backends that accept GGML_PREC_F32 for flash attention now also accept
GGML_PREC_BF16 with the same F32 accumulation (CPU, SpaceMIT, ET).
Vulkan keeps the F32 transfer between devices for both. The CUDA
cuBLAS path already handles GGML_PREC_BF16 and is unchanged.

* tests: cover GGML_PREC_BF16 in the flash attention tests

The flash attention cases run with GGML_PREC_BF16 next to F32 and
DEFAULT, and var_to_str prints bf16 so these cases keep their own name.

* llama: request GGML_PREC_BF16 for the KQ multiplication

* cont : add TODO in clip.cpp

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-11-08 08:37:46 +01:00
bosh a2b301831d speculative : do not apply the MoE cache to the draft model (#31497) 2026-11-08 15:11:07 +09:00
c19043f67e opencl: improve q8_0 and MoE small-n verification for spec (#31831)
* opencl: add kernel_gemv_noshuffle_q8_0_f32_mc

Assisted-by: Claude Opus 5.5
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>

* opencl: small-n MoE GEMV for q5_1, q4_k

Assisted-by: Claude Opus 5.5
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>

* opencl: adreno kernel k floor 256 for x2

Assisted-by: Claude Opus 5.5
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>

* opencl: small-n MoE GEMV for more quants

Assisted-by: Claude Opus 5.5
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>

* opencl: rename function

* opencl: disable bin FA for small verify batch

---------

Co-authored-by: Li He <lih@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
2026-11-07 22:04:10 -08:00
WuXin 372dc5ca15 common : sanitize invalid UTF-8 in PEG tool call arguments (#31861) 2026-11-07 22:06:15 -06:00
Georgi Gerganov f477f795cb server : downgrade per-request proxy logs to trace (#31859)
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
2026-11-08 01:29:52 +01:00
Aparna M PandMax Krasnyansky 76a67730ff hexagon: support for automatic loading of libhtp without legacy env vars (#31781)
* hexagon: use absolute skel path

* hexagon: support non-ascii dir and long path in skel-abs path

* hex-libdir: add support for FASTRPC_MANAGE_DSP_LIBRARY_PATH and GGML_HEXAGON_LIBHTP_DIR on all platforms

* hex-libdir: keep extra fwd slash in abs path on Android and Linux

* hex-libpath: always start with absolute path then fallback to session ctl then env

* hex-libdir: add CDSP1,2,3 to the fallback, needed for the IQ10

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-11-07 16:25:20 -08:00
Jesse Symons 12e9084dc1 sycl: exact rounding in CPY to Q4_1, Q5_1 and Q8_0 (#31465) 2026-11-07 18:47:26 -05:00
KyleHagy 818ac6b078 sycl: refuse device buffers that would exceed VRAM (#31285) 2026-11-07 18:13:04 -05:00
Max Krasnyansky e9a2a44fe9 hexagon: n_hvx and n_threads cleanup (#31857)
* hex-nhvx: apply nhvx updates to the latest master

* hex-nhvx: set N_HVX and N_THREADS based on the arch to avoid overallocation

* hex-build: add support for enabling HTP_DEBUG via build.py

* hex-trace: move tracing state/helpers into new htp-trace.h header

* hex-dcache: cleanup dcache #defines

* hex-hnvx: minimize the diff by reflowing the macros

* hex-ci: fix python type/linter warnings

* hex-nhvx: consitent logging
2026-11-07 15:03:31 -08:00
Xuan-Son Nguyen a2654ec431 mtmd: move log-mel computation to cgraph (#31808)
* mtmd: move log-mel computation to cgraph

* rm stray file

* also include source in mtmd_audio_track

* rm stray file

* rm mtmd_audio_stft_op

* add test
2026-11-07 23:53:11 +01:00
Xuan-Son NguyenandPascal 9f0e5098a5 mtmd: support MageVL-Cowork model (#31819)
* mtmd: support MageVL-Cowork model

* mtmd: keep magevl rope_freqs at model level and check its shape

The rope factors are shared by every layer, so they live in clip_model
instead of a pointer copied into each clip_layer. The loader now checks
that v.rope_freqs is F32 with d_head elements and that d_head splits
into the 4:6:6 pairs the graph expects.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-07 19:36:44 +01:00
e95426f169 llama_moe_cache: run missed experts on CPU, add tuning script (#30966)
* moe-cache : compute part of the misses on the CPU

--moe-cache-cpu-share F computes a share F of the MoE cache misses on the CPU
from the host experts instead of uploading them. The graph builds the expert
FFN twice: on the device from the cache slots, where the misses left to the CPU
read zero slots, and on the CPU from the host experts, where the experts
computed on the device have id -1. The two results are added before the down
bias.

- ggml-cpu: MUL_MAT_ID skips negative ids and writes zero rows
- sched: the copy callback is called for all the inputs of a split, and before
  the host sync when there are no events
- the slot lookup callback reads the inputs of the CPU part together with the
  selected experts, so the CPU part does not wait for the device, and its
  result is copied back asynchronously
- each token uses distinct zero slots, mul_mat_id does not support an expert
  selected twice by a token
- LLAMA_MOE_CACHE_TRACE writes the selected experts of each lookup, for a
  simulator of the cache

* scripts : add moe-cache-tune.py to pick the MoE cache arguments

The script wraps a llama-server command: it starts the server at two cache
sizes and three CPU shares, fits the upload and CPU speeds of the machine, and
replays the routing of the runs through a copy of the cache logic to predict
other sizes and shares. It prints the recommended --moe-cache-mib and
--moe-cache-cpu-share.

docs/moe-cache.md describes the MoE cache arguments and the script.

* add am17an as codeowner for MoE cache

* moe-cache : upload all the misses until the cache is full

With a high --moe-cache-cpu-share, few misses are uploaded, so an empty cache
fills slowly and more misses than needed go to the CPU. The CPU share now
starts once all the slots of a cache group hold an expert. moe-cache-tune.py
replays the same rule.

* add backend sync for async transfer queue

* moe-cache : let the scheduler copy the results of the CPU part

* Fix MSVC compilation

* remove sync change

* llama-graph : extract the CPU share condition into can_cpu_share

* llama-graph : make llm_graph_params::moe_cache const

The graph construction no longer updates the MoE cache directly: the CPU
parts of the layers are accumulated in the graph result and applied to
the cache in llama_context::process_ubatch, after the graph inputs are
set and before the graph is computed.

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)

* scripts : replay the ClassLRU policy in moe-cache-tune.py

* moe-cache : pick CPU misses by CRF instead of random (#31853)

* moe-cache : remove the backend logic of the CPU part from the graph

The CPU part reads the host experts, so the scheduler places it on the
CPU without set_tensor_backend. The cache gives no CPU map for batches
that a device would offload. Graphs with a CPU part look up their own
slot map, so the graph does not register the part with the cache.

ggml-backend : the inputs of the next split that are already computed
are copied to host memory before the current split runs, so a CPU split
does not wait for the device split before it. This replaces the stash
of the cache, and the copy callback is again called for host weights
only.

* llama-graph : return the MoE cache slots and CPU ids in a struct

The scheduler prefetch no longer checks that the next split runs on
another backend: the inputs of a split are never on its own backend,
so the input loop already skips that case.

---------

Co-authored-by: Gaurav Garg <gaugarg@nvidia.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Co-authored-by: Alexey Kopytko <alexey@kopytko.com>
2026-11-08 00:20:58 +08:00
Michael Taylor f7591a6a01 sycl: add f16 support for SWIGLU_OAI (#30857) 2026-11-07 09:02:58 -05:00
Georgi Gerganov 80d2c0c2c5 metal : fix GROUP_NORM for ne03 > 1 and non-contiguous inputs (#31852)
The kernel_group_norm_f32 kernel and its host dispatch ignored the
batch dimension (ne03) of the source tensor: only ngrp threadgroups
were dispatched and each normalized the flat range
[tgpig*gs, (tgpig+1)*gs) with gs = ne00*ne01*ceil(ne02/ngrp). For
tensors with ne03 > 1, all but the first ne00*ne01*ne02 elements were
left unwritten (exposed by the new GROUP_NORM test with
ne=[3,1,1,65537], num_groups=1).

Dispatch one threadgroup per (group, i03) pair (ngrp in x, ne03 in z,
same work distribution as the Vulkan backend) and compute per-batch
statistics, matching the CPU reference. Also clamp the group element
count for the last group when ne02 is not divisible by ngrp and use
int64_t indexing.

The kernel now only requires ggml_is_contiguous_rows(op->src[0])
instead of full contiguity: it addresses src0 and dst with their own
strides (nb03 and the dst strides are added to the kernel args - dst
can be a strided view of src0 with ggml_group_norm_inplace) and falls
back from the flat fast path to a strided path that decodes the linear
element index when rows or channels are padded. The flat/strided
selection is done on the host in
ggml_metal_library_get_pipeline_group_norm and compiled into one of
two specialized pipeline variants via a FC_group_norm_flat function
constant, so the dead path is eliminated at pipeline creation.
GROUP_NORM support is also gated on F32 src and dst in supports_op -
the kernel only implements f32 and an f16 input would silently be read
as f32.

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
2026-11-07 13:04:13 +02:00
Harkanwal BediandPascal 0956684529 server: visit parent slots before child slots (#30779)
* server: visit parent slots before child slots
A parent slot wakes its waiting child slots once it finishes the shared
prompt prefix, while slots are visited in id order. When a child task
lands on a lower slot id than its parent, the child is visited before
the parent wakes it and waits an extra batch to process its suffix.
Non-child slots are now visited before active child slots, so every
child woken by its parent joins the same batch.

* server: check systemone test events through test tags

The systemone tests matched free-form debug messages, including the
formatted f_keep value of the prompt cache. The server now emits
__TEST_TAG_WAIT_PARENT_SLOT__ when a child slot waits for its parent,
__TEST_TAG_PROMPT_CACHE_RESTORE__ on every prompt cache restore and
__TEST_TAG_PROMPT_CACHE_FULL_RESTORE__ when the whole cached prompt is
a prefix of the new one, compared on token counts.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-07 13:03:38 +02:00
Nikhil Jain cf659f240b webgpu : Implement group norm kernel (#31490)
* Only run webgpu CI on my fork

* Add webgpu only workflow

* Add WebGPU group_norm kernel

* clean up

* resolve merge conflicts

* parameterize data type; clean up cmake changes

* add nullptr check

* clean up webkit experiments

* more cleanup

* more cleanup

* clang format

* Remove unused feature check for WebGPU

* clean up supports_op condition and update documentation

* add perf tests for group_norm

* clean up doc with script and undo change in test-backend-ops

* Do 2d dispatch of workgroups

* Do 2d dispatch of workgroups

* Unstage unrelated changes

* undo docs churn

* add new test case

* undo small change

* undo small change
2026-11-07 10:01:27 +02:00
Jeff Bolz 1938152648 vulkan: Add 'readonly' to decodeBuf declarations. (#31833)
In a future driver version I hope to be able to get some perf gain out of this.
2026-11-07 10:00:40 +02:00
Ali F. Al-Ramadan 5d7d4aec16 CUDA: fix shared memory bank conflicts in the lightning indexer (#31803) 2026-11-07 09:59:26 +02:00
Adrien Gallouët c0ab6e5c0b llama : add missing link to pthread (#31785)
* llama : add missing link to pthread

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Link Threads::Threads privately

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

---------

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-11-07 08:43:50 +02:00
shaofeiqiandLi He 31424b7cf1 opencl: add arange (#31829)
Assisted-by: Claude Opus 5.5

Co-authored-by: Li He <lih@qti.qualcomm.com>
2026-11-06 21:44:05 -08:00
Yonghye Kwon fd27b20349 meta : raise the compute view limit for buffers with few tensors (#31447)
* ggml-backend-meta : minimum number of compute views per buffer

Assisted-by: Claude Opus 5.5

* tests : revert the deepseek41 test model to 6 layers

Assisted-by: Claude Opus 5.5
2026-11-07 11:05:24 +08:00
Ehsan BateniandMax Krasnyansky ca89968a23 hexagon: fuse post-norm residual add into RMS_NORM_MUL (#31822)
* hexagon: fuse post-norm residual add into RMS_NORM_MUL

* hex-fusion: refactor RMS_NORM fusion and add dedicated RMS_NORM_MUL_ADD opcode

Improves performance by keeping things in HVX registers.
Also simplifies overall fusion logic and logging.

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-11-06 16:06:59 -08:00
usrlocalben bcb0152876 model: add Kimi-K3.1-Pro text model (#31779)
* model: add Kimi-K3.1-Pro text model

K3 with the full-attention MLA layers replaced by DSA: on this checkpoint 24
of the 93 layers carry a lightning indexer (32 heads x 128, top_k 2048, no
rope). The KDA (linear-attention) layers and the rest of the K3 skeleton -
cross-layer residuals, latent MoE, situ, MLA output gate, full-rank KDA gate -
are unchanged, so this reuses the K3 graph and converter and adds the indexer.

Indexer:
  - the selection is an additive mask over the whole cache, not a gather of
    the top-k rows, so it reuses the existing sparse flash attention path;
    off that path the mask stays correct but still costs a kq over the cache
  - the indexer key LayerNorm takes its eps from the model's rms_norm_eps,
    because dsa_config declares no eps of its own

Graph and cache:
  - new arch with four indexer tensors per DSA layer, and a hybrid memory
    that keeps a K-only MLA cache for those layers, recurrent state for the
    KDA layers, and the indexer key cache
  - the graph node budget shares the K3 entry

Conversion:
  - KimiK31Model(KimiK3Model), picked by KimiK31ForConditionalGeneration
  - dsa_config becomes attention.indexer.{head_count,key_length,top_k,types}
    plus the indexer LayerNorm eps; the layer list is 1-indexed like
    linear_attn_config's, and the converter asserts that it covers exactly
    the full-attention layers
  - the routed experts stay MXFP4 and are repacked as in K3, not dequantized
  - the vision tower and mm_projector are skipped: text only for now
  - the arch key follows the checkpoint's own model_type ("kimi_k31") and the
    loader registers KimiK31ForConditionalGeneration; the release name carries
    a -Pro suffix that nothing in the config or the tensor names uses

Quantisation: the indexer's q_b, k and proj are kept out of the quant
overrides, since the top-k selection is sensitive to their precision.

Verified on an 8-layer slice against the HF checkpoint: 38 tensors checked
exactly, the indexer metadata against the config, and the selection against
full attention - identical PPL at top_k beyond the context length, divergent
once top_k is forced below it. Arch test fixture and the full arch suite pass.

* converter: avoid clutter in the k3 base

* narrow comment to match model

* update metal fusion expectations

* restore

* sync with PR 31733 refactoring
2026-11-06 23:42:46 +01:00
669204ee2b hexagon: conv_2d optimizations and support for conv_3d (#31814)
* hexagon: Generalization of CONV_2D implementation for CONV_3D

- New op HTP_OP_CONV_3D is wired through htp-ops.h, main.c (shares op_conv_2d) and the op remap in htp-kparams.cpp.
- conv_3d runs on the existing HMX tiled (implicit GEMM) conv2d path. KD and KH are folded into one virtual kernel row (KD*KH), and output rows are counted as N*OD*OH.
- htp_kparams_solve_conv3d checks the conv_3d layout and calls the tiled solver htp_kparams_conv2d_tiled, which takes an is3d flag.
- The tiled kernel takes the sizes and strides that differ between conv_2d and conv_3d from a struct htp_conv_geom. Only the input stage unfolds depth.
- 256-entry DMA queue can no longer overflow

Assisted-by: Claude Code

* hexagon: modify checking to eliminate divide-by-zero hazards

* hexagon: conv2d tiled: compute the conv_3d unfold divisors on the host

The KH and OH fastdiv values used to unfold conv_3d kh into (kd, kh) and
oh into (od, oh) were built per op on the device, each with a 64-bit
divide. Pass them in the kernel params instead; they fit in the space the
window variant already reserves, so the struct size does not change.

Assisted-by: Claude Code

* hexagon : preserve fold scheduling in conv2d for small size

hexagon : support arbitrary positive conv2d strides
Support positive horizontal strides for WHCN and CWHN inputs. Preserve
load/shuffle paths for strides 1/2 and use gather for larger strides.

hexagon : improve conv2d tile size selection

Prefer the largest feasible output-channel block and derive spatial
chunk capacity from VTCM usage. Compare capacity and thread-aligned
candidates using per-thread work and chunk overhead.

Co-authored-by: Yiwei Shao <yiwei@aizip.ai>

* hex-conv: fix compiler warning

---------

Co-authored-by: Yiwei Shao <yiwei@aizip.ai>
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-11-06 11:51:57 -08:00
Xuan-Son Nguyen 954de248ac docker: add explicit /usr/local/nvidia search path (#31806) 2026-11-06 18:53:04 +01:00
Sofian Elmotiem 6c01da6278 cli : fix crash on invalid UTF-8 in prompt or files (#31563)
Assisted-by: Claude
2026-11-06 13:54:48 +01:00
Georgi Gerganov 8820b088de tests : qualify isnan and signbit with std:: (#31791)
* tests : qualify isnan and signbit with std::

The unqualified calls rely on the C math.h names also being present in the
global namespace, which is not the case for the NetBSD 11.99.9/x86_64 gcc
14.4.0 build reported in the issue.  Prefix the calls with std:: to select
the standard C++ overloads declared by <cmath>.

ref: https://github.com/ggml-org/ggml/issues/1681

Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)

* tests : qualify C math calls with std::

Follow-up to the isnan / signbit fix: make the remaining math calls in
test-quantize-fns.cpp consistent with the std:: isfinite / max / numeric_limits
already used in the same file.

The float variants are overloads in C++, not distinct names, so the mapping is
fabsf -> std::fabs, copysignf -> std::copysign, nextafterf -> std::nextafter -
not std::fabsf & friends, which do not exist in libstdc++ and would break the
gcc build.  NAN and INFINITY are macros and are left unqualified.

Only behavioural difference: array_rmse() fed a double accumulator to sqrtf(),
now std::sqrt() keeps the computation in double.  The result is more accurate
and differs from the previous value by at most 1.4e-7 relative, three orders of
magnitude below the tightest tolerance in this test.

Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
2026-11-06 14:53:04 +02:00
Georgi Gerganov 9a1d03c7ca tests : override hardware_concurrency in CI (#31797)
The self-hosted CPU runners are containers with 8 CPUs, but
std::thread::hardware_concurrency reports the cores of the host. The
table mode of test-llama-context sizes its pool from that count, so it
runs 4 models x 8 threads on an 8-CPU job and hangs.

Add TEST_ARG_HARDWARE_CONCURRENCY and TEST_ARG_PARALLEL_WORKERS as the
defaults for the auto-selected counts of test-llama-context and of
test-backend-ops, and set them to 8 and 4 for cpu-x64-high-perf.

Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (RTX 5090 32GB, DDR5 128GB)
2026-11-06 14:52:26 +02:00
Adrien Gallouët 18ab62a8ec cmake : generate RC file directly without win-app.rc.in (#31718)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-11-06 11:02:47 +01:00
lhez 52c9bad946 opencl: support repeat for f16, i32, i16, bf16 (#31782)
* opencl: add f16 repeat

Assisted-by: Codex

* opencl: support repeat for i32, i16, bf16

Assisted-by: Codex

* opencl: update ops

Assisted-by: Codex
2026-11-06 11:36:27 +02:00
Masashi Yoshimura 5d3ed2ae7e webgpu: add batched tensor API (get_tensor_n and set_tensor_n) (#31473)
* add set_tensor_n and get_tensor_n

* fix for editor-config checker

* remove the outdated comment
2026-11-06 11:35:47 +02:00
Ruben Ortlam 80088fa60b vulkan: detect ReBAR support, avoid BAR allocations if unavailable (#31658) 2026-11-06 11:35:03 +02:00
Alexey Kopytko c4fb579695 moe-cache : LRFU/MQ-like hybrid cache, ClassLRU (#31233)
* moe-cache : LRFU/MQ-inspired hybrid cache

* make the clock advance by N tokens: use the real token count (ne[1] is always 1, ids have n_expert_used entries per token)

* sel is a RESHAPE node, the orig input is in src[0], so sel->src[0]->ne[1] is the count

* c for the class index, last_use for last use time

* explain the roles of these

* compute the key once

* use the key directly instead of re-reading

* show how the score is computed from the age

* remove the branch and clarify exp2 and log2 relationship

* reword the header comment and elaborate on unused slots and sorting

* not a copy, not just skip

* std::array maybe

* Revert "std::array maybe"

This reverts commit 7649b1f172db4941a854ad115bec0c7e062b65b9.

* half life it is

* add key and iteration helpers

* extract decayed_crf

* bump classes to 8, document the formula, display the cache size in token worth count

* assert the computed number of classes matches the half life
2026-11-06 11:32:54 +02:00
Samriddha Sinha 3ed7284f2d ci : bump CUDA 13 release builds to 13.5 (#31788)
* ci : bump CUDA 13 release builds to 13.5

* docker : bump cuda13 images to 13.5.0
2026-11-06 09:52:47 +01:00
Alessandro de Oliveira Faria (A.K.A.CABELO) d9cf1870be vendor: update BoringSSL to 0.20261105.0 (#31757)
[no ci]
2026-11-06 09:52:16 +01:00
Johannes Gäßler 240d68fcfe CUDA: fix Maxwell build (#31787) 2026-11-06 09:06:27 +01:00
Adrien Gallouët bbc2e4b657 common : restrict HF token to huggingface.co hosts (#31730)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-11-06 08:43:33 +01:00
Shawn GuandLi He 4b77dee052 opencl: add q6_k multi column tiled GEMV for speculative verification and refactoring (#31758)
* opencl: use the multi-column q6_K vocab head GEMV for 2..4 tokens on X2E

Co-authored-by: Shangqing Gu <shawngu@qti.qualcomm.com>
Assisted-by: Claude Opus 5.5

* opencl: rename GGML_OPENCL_Q6K_MC3 to GGML_OPENCL_Q6_K_MC3

Co-authored-by: Shangqing Gu <shawngu@qti.qualcomm.com>
Assisted-by: Claude Opus 5.5

* opencl: rename GGML_OPENCL_Q40/Q41/Q4K_MC3 to GGML_OPENCL_Q4_0/Q4_1/Q4_K_MC3

GGML_OPENCL_Q4K_MC3_DQ and GGML_OPENCL_Q4K_MC3_LDS become GGML_OPENCL_Q4_K_MC3_DQ
and GGML_OPENCL_Q4_K_MC3_LDS.

Co-authored-by: Shangqing Gu <shawngu@qti.qualcomm.com>
Assisted-by: Claude Opus 5.5

* opencl: remove unnecessary debug probe

Co-authored-by: Shangqing Gu <shawngu@qti.qualcomm.com>
Assisted-by: Claude Opus 5.5

* opencl: clean up

---------

Co-authored-by: Li He <lih@qti.qualcomm.com>
2026-11-05 21:11:07 -08:00
Max Krasnyansky 67107ac34f hexagon: Implement set_tensor_n and improve repacking/tiling robustness (#31744)
* hex-repack: move repacking/tiling into graph-optimize

* hex-tiling: simplify tiling rules and remove obsolete helpers

* hex-ops: use a hash for de-duplicating tensor descriptors

* hex-bufs: improve tensor hash

* hex-ops: accept empty (zero-elements) ops

* hex-bufs: enable hostbuf by default and reject tensors that do not fit into max_vmem

* hex-bufs: improve error message and update readme

* hex-bufs: make sure to sync flush opbatch in all sync set_tensor paths
2026-11-05 18:15:58 -08:00
Xuan-Son Nguyen e14a779719 mtmd: enable cmake unity build (#31760) 2026-11-06 00:16:13 +01:00
Sigbjørn Skjæret d38bea3422 tests : drop u8path() (#31755)
[no ci]
2026-11-05 23:06:51 +01:00
Eve 186eab1cb8 ci: fix build-cmake-pkg ccache (#31705) 2026-11-05 21:16:56 +00:00
Xuan-Son Nguyen 6023692b1e llama: fix kv shifting, add tests, add per-model build_rope (#31683)
* llama: fix kv shifting and add tests

* llama: add graph build_rope (#31693)

* llama: add graph build_rope

* improve

* migrate new models

* update review skill

* fix ci

* add --kv-shift to CI
2026-11-05 21:10:36 +01:00
Xuan-Son Nguyen b1c902f026 llama, mtmd: (chatterbox) add use_classifier_free_guidance as gguf metadata (#31698)
* llama, mtmd: add use_cfg as gguf metadata

* use_classifier_free_guidance
2026-11-05 20:09:33 +01:00
Sigbjørn Skjæret e8df20d238 models : recursive fix (#31749) 2026-11-05 20:06:55 +01:00
fecdc81eec model : add Ripple (ricursiveai) (#31692)
* feat : added ricursiveai ripple support

* fix : mandatiry SWA, scale through helper, comments back

* fix : metal

* model : make ricursive RoPE/NoPE explicit and validate the layer pattern

Write attention.rope_pattern and read it as optional, falling back to the
sliding window pattern so existing GGUFs keep loading. Refuse a
sliding_window_pattern that does not cover every layer, a model with no
sliding layer, and a gate or value residual present in only some layers.
The converter derives layer_types when the config omits it, ignores a stale
hybrid_layer_pattern and drops tensors of features the config disables.

Assisted-by: Claude Opus 5.5

* model : make ricursive swiglu_clamp per-layer

Store swiglu_clamp as a per-layer array, read it with get_key_or_arr so the
scalar in existing GGUFs still applies to every layer, and refuse a negative
or non-finite value. Add a test-llama-archs fixture with mixed per-layer
clamps and no rope_pattern, so the saver roundtrip checks both.

Assisted-by: Claude Opus 5.5

* model : use the ricursive per-tensor scales and RoPE parameters

Pass the attn_gate and output weight scales to build_lora_mm, take
ext_factor and attn_factor from the graph instead of hardcoding them, and
copy rope_freq_scale_train to the SWA layers instead of pinning it to 1.
Cite the reference lines for the value residual and the NoPE layers.

Assisted-by: Claude Opus 5.5

* convert : drop the ricursive chat_template.jinja override

The precedence between tokenizer_config.json and chat_template.jinja is to
be changed in SpecialVocab for all models instead. Until then a new
conversion stores the short tokenizer_config.json template; GGUFs that
already carry the tool-aware template are not affected.

Assisted-by: Claude Opus 5.5

* fix : remove NVFP4 checkpoint quantization block

* match #31733 refactor

---------

Co-authored-by: Piotr Wilkin <ilintar@gmail.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-11-05 19:55:15 +01:00
45b6604b20 model: Add support for the xing4_0 (#31589)
* model: add support for Xing4.0

* Update src/models/xing4_0.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update src/models/xing4_0.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update src/models/xing4_0.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update src/models/xing4_0.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update src/models/xing4_0.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update src/models/xing4_0.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* fix: rename model file to xing4-0.cpp to match CI expectations

* model: rename xing4_0.cpp to xing4-0.cpp

* lint: fix E128 continuation indent in conversion/xing.py

* tests: add xing4_0 fusion baseline for Metal

* reduce comments

---------

Co-authored-by: shixq7 <shixq7@chinatelecom.cn>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-11-05 17:47:27 +01:00
Nick Towle e30ad80fd7 ui: prevent a hidden sidebar from intercepting clicks (#31712)
* ui: prevent a hidden sidebar from intercepting clicks

* ui: also fix desktop sidebar
2026-11-05 15:44:34 +01:00
Xuan-Son Nguyen e0f96a1a12 llama: refactoring: move model to class member of llm_graph_context (#31733) 2026-11-05 15:41:03 +01:00
Adrien Gallouët 468fde90e8 server : simplify windows MCP command resolution (#31728)
* server : simplify windows MCP command resolution

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* common : remove utf8_to_wstring()

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

---------

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-11-05 15:34:54 +01:00
Adrien Gallouët 1416f76aa1 vendor : update cpp-httplib to 0.65.0 (#31731)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-11-05 15:34:18 +01:00
Zheyuan Chen 10d4846a57 ggml-cuda: sanitize post-bias NaNs in MoE selection (#31005)
Fixes #28370
2026-11-05 13:50:16 +01:00
Jeff Bolz 61d4472344 vulkan: Implement test-fusion interface (#30769) 2026-11-05 13:31:44 +01:00
ManuelPDMandDukeNukem-dev 016c1d914c CUDA: use highest compiled arch in get_mmvq_mmid_max_batch (#31679)
Co-authored-by: DukeNukem-dev <54203477+DukeNukem-dev@users.noreply.github.com>
2026-11-05 20:18:38 +08:00
Alex-JP-93 da3416d807 vulkan: RDNA3/4 Large-Tile Split-K Optimization (#30280)
* RDNA3/4 Vulkan Large-Tile Split-K Optimization

* Add single-operator performance test cases

* Update the policy to use driver-reported LDS usage

* Add comments

* Use the simpler split_k rule from review

Assisted-by: GitHub Copilot

* tests : drop the split_k test cases

* vulkan : simplify the RDNA3/4 split_k condition
2026-11-05 13:11:07 +01:00
Sigbjørn Skjæret cc78da97c0 gguf-py : external chat templates have precedence over tokenizer_config (#31717) 2026-11-05 13:04:48 +01:00
Georgi Gerganov ea40a3a1e1 metal : fuse DeepSeek-V4/V4.1 and GLM MoE routing (#31615)
* metal : support scale-bias routing norm in the TOPK_MOE fusion

The DeepSeek-V4.1 graph normalizes the routing weights with a scale+bias
(sum + 1e-20) instead of the lower clamp used by the other models. Add
norm_bias/norm_scale_bias fusion chains that accept a SCALE node with a
unit scale factor at the clamp position, pass an additive bias in the
kernel args, and apply it as max(wt_sum, clamp) + bias so both variants
share one kernel. Cover the new chains in test-backend-ops.

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)

* metal : fuse sqrt(softplus) MoE routing with a selection bias

Add TOPK_MOE chains anchored at the softplus node (UNARY+SQRT+ADD+
ARGSORT+GET_ROWS plus optional norm/scale) so the DeepSeek-V4/V4.1
routing fuses on Metal. The kernel scores with sqrt(softplus) and ranks
selection-biased keys while the gathered weights and the norm stay
unbiased, matching the reference graph. Dispatch hooks at the softplus
node, cover the norm-bias combination in test-backend-ops and record the
new deepseek4/deepseek41 fusion counters.

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)

* metal : fuse sigmoid MoE routing with a selection bias

The sigmoid-gated MoE archs (GLM, DeepSeek-V2/V3.2, ...) rank
sigmoid(logits) + exp_probs_b with an optional weights-norm, but only
the softmax and sqrt(softplus) routing chains were fusable.

- add 6 sigmoid-anchored TOPK_MOE chains: sigmoid + ADD + ARGSORT +
  GET_ROWS with optional SCALE and SUM_ROWS + CLAMP/SCALE + DIV norm
  tail; the selection-bias ADD is required, same as the sqrt(softplus)
  chains
- generalize the gating function constant from a sqrt(softplus) bool
  to an int (0 = softmax, 1 = sqrt(softplus), 2 = sigmoid); sigmoid
  scores are 1 / (1 + exp(-logits)), matching the CPU, Metal unary and
  CUDA kernels
- anchor the unary dispatch hook at sigmoid nodes too
- test-backend-ops: 4 sigmoid routing cases with and without a
  weights scale
- re-record tests/fusion/MTL.csv: the sigmoid archs (glm5-next,
  deepseek2/32, kimi, minimax, ...) now fuse their routing, and
  muse-spark prefill drops one MOE_REDUCE fire because the reordered
  graph allocates the routing weights inside the reduce output range,
  which the dst/input overlap check correctly declines

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)

* cont : remove low-signal comments

* ocd : valign ggml_metal_fusions
2026-11-05 13:52:56 +02:00
15e50990bb CUDA/HIP: matrix-core (MFMA) lightning indexer for CDNA2 (gfx90a) (#29050)
* CUDA/HIP: matrix-core (MFMA) lightning indexer for CDNA2 (gfx90a)

* Addressing reviewers comments, mma.cuh and physical warp size

* CUDA/HIP: single mma.cuh matrix-core lightning indexer ::CDNA::

* Support RDNA3

* CUDA/HIP: stream heads through the mma lightning indexer kernel

Process 16 heads at a time with double-buffered Q staging instead of keeping
all heads resident in shared memory, and drop the GGML_INDEXER_NO_MFMA env.

* enable for CUDA

* CUDA: Remove wmma lightning-indexer, also run head 32 through mma kernel

* CUDA: Nv mma dosent transpose accumulator

---------

Co-authored-by: Carl Philipp Klemm <carl@uvos.xyz>
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
2026-11-05 11:44:15 +01:00
Johannes Gäßler 69ee2ecc01 CUDA: linearize MoE data layout for MMQ/MMF (#31047)
* CUDA: linearize MoE data layout for MMQ/MMF

* Apply MMQ tile size optimization more generally

* Fix F16 performance regression
2026-11-05 12:33:25 +02:00
bri-prism 2bc46baa92 vulkan: FWHT kernels for block widths above 512 (#29772)
* vulkan: add F16 input to the FWHT

The Vulkan FWHT accepts F32 input only. This adds an F16 source variant of the
shader, for both the subgroup and the shared-memory paths, so the kernel reads
an F16 source directly instead of requiring a converted copy.

The F16 shader needs shader-float16, so the pipelines are only created when the
device reports it. A null pipeline makes ggml_vk_can_use_fwht fall back, which
is the same path an unsupported width already takes.

supports_op and the dispatch both already route through ggml_vk_can_use_fwht,
so there is a single place asking whether an op qualifies; nothing to
consolidate here the way the CUDA/Metal ports needed.

test-backend-ops on an M5 Pro (MoltenVK, fp16: 1): MUL_MAT 1128/1128,
including all 24 Hadamard cases (18 existing F32, 6 new F16).

Assisted-by: Codex

* vulkan: FWHT kernels for block widths above 512

The Vulkan FWHT covers block widths 64 to 512; wider blocks fall back to the
dense matmul. Extend the width list to 8192 and pick the shader per width: the
subgroup shader keeps n/subgroup_size values per invocation, so widths above
512 (or a subgroup narrow enough to exceed 64 values per invocation) use the
shared-memory shader with one row per workgroup and a 256-wide block. The rows
per workgroup become a specialization constant so the dispatch matches.

Pipelines for a width are only created when the device's shared-memory and
workgroup-invocation limits allow it; otherwise that width keeps the fallback.

This is the Vulkan counterpart to #29100 (CUDA). Stacked on #29101; the last
commit is the one to review.

Assisted-by: Codex
2026-11-05 12:03:44 +02:00
Nisbenz 3b83d12b08 vulkan : check combined K+V f16 scratch size in FA dequant path (#30487)
Fixes #30048

Assisted-by: Claude
2026-11-05 12:02:39 +02:00
cwriterandcwriter e152332a15 llama-server: Use reference-counted checkpoints instead of deep-copies (#30137)
* checkpoints: reduce overhead

Assisted-by: Codex

* rebase

Adapt the slot save/restore checkpoint appendix from #26004 to the
shared_ptr checkpoint list and the common_state_data buffers.

Assisted-by: Claude Opus 5.5

---------

Co-authored-by: cwriter <cwriter@localhost>
2026-11-05 11:02:36 +01:00
Takamichi Ohara 1fa7f1c488 ggml : support truncate=false for YaRN (#30206)
* ggml : support `truncate=false` for YaRN

YaRN calculates correction intervals to switch frequency interpolation
methods, but the setting to not round the boundaries of these intervals
to integers (`truncate=false`) was not supported.

Models that use `truncate=false` include gpt-oss and some PLaMo-3
models; to support them correctly, the GGML operations and CPU backend
are modified to handle calculations without rounding. The default
behavior is `truncate=true`, and existing calls not explicitly
configured will maintain their previous behavior.

* support CUDA backend

* add tests

* move RoPE helpers to internal header

* format
2026-11-05 12:02:05 +02:00
Daniel Bevenius 9c9074e14e conversion : resolve quantized_layers from source tensor names (#31611)
* conversion : resolve quantized_layers from source tensor names

This commit changes how the per layer NVFP4 quant_algo are resolved from
hf_quant_config.json quantized_layers.

The keys in quantized_layers are module paths in the source checkpoint,
and these can be either be modules like 'lm_head' or parent modules
like 'layers.0.ffn.experts'. Currently these keys are passed through
filter_tensors, but they are not tensor names. So a filter_tensors that
keeps an explicit list of tensor names, like Nemotron's with --mtp
would drop them. That means there was no `_nvfp4_layer_algo entry`, and
`_tag_prec_a4` fell back to `_nvfp4_global_algo`.

This commit records the mapping from source tensor name to filtered
tensor name in index_tensors. Then in prepare_tensors each source
tensor is resolved to its closest quantized_layers parent, and
the algo is stored under the filtered tensor name, which is what
_tag_prec_a4 later looks up.

Converting Nemotron 3.5 Lightning NVFP4 with --mtp now produces:
```console
    general.tensor_extra.name = ['output.weight']
    general.tensor_extra.prec_a4 = [False]
```
whereas on master no general.tensor_extra fields are written. For the
non-MTP conversion, the general.tensor_extra fields are identical to
master.

Also manually verified (using pdb) that this works for
DeepSeek-V4.1-Flash-1017-NVFP4.

Refs: https://github.com/ggml-org/llama.cpp/pull/31602#issuecomment-6603337403

* conversion : shorten comment

* conversion : also remove '.bias' suffixes
2026-11-05 10:05:06 +01:00
Hoang Vien DuyandGeorgi Gerganov f7b41cca4f metal : use mul_mm for 32 <= K < 64 (#30982)
* metal : use mul_mm for 32 <= K < 64

* metal : add bf16 x bf16 mul mat kernel

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-11-05 10:58:41 +02:00
Adrien Gallouët c55b094880 server : reject non-object session updates (#31714)
Prevent malformed realtime session updates from throwing an uncaught
exception and terminating the server

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-11-05 09:38:08 +01:00
Aman Gupta 2c192444c9 fit : account for skipped NextN layers when setting n_gpu_layers (#31569)
* fit : account for skipped NextN layers when setting n_gpu_layers

Without MTP, fit counts n_layer + 1 layers while the model loader counts
n_layer_all + 1 for n_gpu_layers, including the NextN layers which hold no
weights. A fit result that offloads all layers then leaves layer 0 on the CPU
("offloaded 65/66 layers" for Qwen3.8-27B on two GPUs).

Add the skipped NextN layers to n_gpu_layers and to the tensor split of the
device that holds the output layer so the loader assigns the layers that
fit measured.

Fixes #31531

* fit : pass hp_ngl_skip by reference like the other hparams

* fit : add hp_ngl_skip to common_get_device_memory_data
2026-11-05 10:31:48 +02:00
Leon 553d2c2e7e webgpu : add MEAN support (#31591)
Assisted-by: Codex
2026-11-05 10:29:14 +02:00
Pascal 040ce199a1 graph: store the MTP conv snapshots of every recurrent model in one set_rows (#31660)
* graph: store the conv state of every rollback arch in one set_rows

The rollback slots of the causal conv history were written with one
copy per slot in qwen4exp, glm5-next, kimi-k3, bailingmoe3, lfm and
mamba2, while delta_net already wrote them with a single set_rows.
build_conv_state_store now does this for all of them: one copy without
rollback, otherwise one set_rows of the overlapping windows into the
min(n, n_rs_seq + 1) slots a rollback can read.

The slot order {n_rs_seq, ..., 0} is computed once per graph and each
conv layer takes a view of its tail. Its size does not depend on the
ubatch, so the reserved graph covers every ubatch shape, and the graph
input that held it is gone.

A model that is not causal never rolls back, so the context clamps
n_rs_seq to 0 for it, like for an arch without rollback support.

* nits: compute the conv slot order in the graph and keep the n_head name of bailingmoe3

* graph: build the conv slot order in each conv layer

The slot order {K-1, ..., 0} of the conv state store was one tensor
shared by every conv layer of the graph, built on the device of the
first one, so each layer on another device copied the view of its
tail. Each conv layer now builds its own order next to its set_rows,
and llm_graph_input_rs no longer holds it.
2026-11-05 09:28:03 +01:00
Georgi Gerganov d4c345d6c4 ci : remove unused model generation (#31412)
* ci : re-enable test-llama-archs on Windows

* cont : remove unused test
2026-11-05 10:25:04 +02:00
Adrien Gallouët 2ada83079a common : drop u8path() and use generic_string() (#31681)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-11-05 08:37:59 +01:00
Adrien Gallouët 5a081c0a70 server : drop u8path() and use generic_string() (#31682)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-11-05 08:37:13 +01:00
Adrien Gallouët 7e42b95f23 rpc : align code with common.cpp (#31684)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-11-05 08:36:51 +01:00
Alex-JP-93 3259ab6e5d vulkan: Re-enable large coopmat matmul configuration on AMD proprietary driver 26.10.1+ (#31587) 2026-11-05 07:40:39 +01:00
Pascal a01dff908b tests: fix CI flakes of TOPK_MOE caused by near ties (#31426)
* tests: fix CI flakes of TOPK_MOE caused by near ties

With uniform random logits, two experts sometimes land within one f32
ulp of each other at a routing boundary, mostly with sqrt_softplus
gating whose output range is narrow. A backend whose gating differs by
a few ulps from the CPU then picks the other expert and the test fails.

Draw each row of logits again until the group selection and the final
top-k both have a relative margin of 1e-4, computed on the host in
double precision.

* tests: build the TOPK_MOE logits by inverting the gating function

Draw the selection scores (probs plus bias) of each row as a random
permutation of a grid, move the experts of the best groups up by half a
step, and set the logits to the inverse of the gating function at these
scores. Every expert and group decision is then one grid step apart on
all backends, and the host no longer recomputes the routing.

The softmax probs are built to sum to one so that the softmax keeps the
scores, and the views of the logits are not initialized again after the
logits are set.
2026-11-05 07:31:48 +01:00
Pascal 44f91a02b2 metal: keep fusable node groups inside one command buffer (#31688)
The first command buffer takes the first 10% of the graph nodes and the
rest is split evenly between the extra command buffers. A fusion cannot
span two command buffers, so a boundary that falls inside a fusable
group runs that group unfused, and the output then depends on the node
count of the graph.

Move each boundary to the end of the fusable group it falls into, so
the encoders fuse the same nodes as with a single command buffer.
2026-11-05 08:21:13 +02:00
Daniel Bevenius 330f0189c9 qwen2 : add nvfp4 scales to build_ffn (#31657)
This commit adds nvfp4 scales to qwen2's build_ffn call which are
currently passed as NULL.

The motivation for this is that a model converted to NVFP4 will
currently crash because the scales are loaded but never applied in
the FFN:
```console
llama-completion: llama.cpp/src/llama-sampler.cpp:1218:
void llama_sampler_dist_apply(llama_sampler*, llama_token_data_array*):
Assertion `found' failed.
```
The attention already applies its scales but ffn is missing them.
2026-11-05 06:29:50 +01:00
shaofeiqiandLi He 7d516e25ea opencl: enable f32 and q5_k multi column GEMV by default for Adreno X2 (#31694)
* opencl: use the small-N f32 GEMV by default on X2E

Assisted-by: Claude Opus 5.5
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>

* opencl: use the q5_K multi-column verify GEMV by default on X2E

Assisted-by: Claude Opus 5.5
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>

---------

Co-authored-by: Li He <lih@qti.qualcomm.com>
2026-11-04 21:22:41 -08:00
shaofeiqiandLi He 8bed7cfb99 opencl: support non-contig sigmoid (#31687)
Co-authored-by: Li He <lih@qti.qualcomm.com>
2026-11-04 19:18:24 -08:00
shaofeiqiandLi He 8415bd7bb3 opencl: add DSV4_HC_PRE and DSV4_HC_POST (#31623)
* opencl: add DSV4_HC_PRE and DSV4_HC_POST

Co-authored-by: Shafei Qi <shaoqi@qti.qualcomm.com>

* opencl: improve dsv4_hc_post

Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>

---------

Co-authored-by: Li He <lih@qti.qualcomm.com>
2026-11-04 19:14:11 -08:00
anujj 5ce06fc71f Title: CUDA: fuse NVFP4 gate/up matmuls and GLU in MMQ (#31296)
* cuda: fuse NVFP4 gate/up and GLU in MMQ

* cuda: honor the requested src1 precision in fused gate/up MMQ

* cuda: share NVFP4 scale checks, skip unused fused FP4 kernels

* cuda: test NVFP4 fused MMQ at W4A8 without scales and with batches, clean up naming

* cuda: check gate_scale before loading the fused NVFP4 gate scale
2026-11-05 10:47:33 +08:00
Hongqiang Wang e56f006ed6 llama : fix on-device state copy sizes for quantized KV caches (#30541)
* llama : fix on-device state copy sizes for quantized KV caches

The on-device sequence state copies (LLAMA_STATE_SEQ_FLAGS_ON_DEVICE) computed
their element count as size / ggml_element_size(). For a quantized type such as
q8_0 or q4_0 that is the number of blocks, not elements, so only part of each
range was copied: the restored state was wrong, or the copy crashed.

Compute the count as size / ggml_type_size() * ggml_blck_size() for the save,
the restore and the copy between differently chunked ranges, and assert that
each range covers whole blocks.

* llama : drop the one-line comments

* llama : drop the redundant cast in the state copy size
2026-11-05 10:31:01 +08:00
Eve 3de52ac388 ci: run self hosted cpu jobs on hf or github (#31439) 2026-11-04 22:34:01 +00:00
Tom Vaucourt 6f1e654370 server : realtime: send an error event instead of crashing on exceptions (#31632)
* server : realtime: send an error event instead of crashing on exceptions

* server : route websocket handlers through ex_wrapper, remove the realtime test

* server : realtime: keep the previous tools when the chat template rejects new ones

* server : realtime: reject tool schemas the grammar cannot convert, drop ws_ex_wrapper

* server : realtime: clarify the tool schema check comment
2026-11-04 21:34:58 +01:00
shaofeiqiandLi He 9e2a04a439 opencl: improve decode perf with FA (#31633)
* opencl: use more workgroups for q4_K gate up + GLU

Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>

* opencl: handle FA for GQA 2 and 6 with head size 256

Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>

* opencl: handle FA for GQA 16 with head size of 128, 512

Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>

* opencl: handle FA for GQA 2 with head size 512

Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>

---------

Co-authored-by: Li He <lih@qti.qualcomm.com>
2026-11-04 10:45:30 -08:00
Aleksander Grygier c6a0ff11a0 ui: open actions on right-click for chats, tabs, messages and models (#31545)
* ui : share directory resolution and the cwd row

Move the lookup of a browser-picked folder to ToolsService.resolveDirectoryPath, let the working directory picker run without its chip, and split the 'Set working directory to' row into its own component, so other views can pick and show a folder the same way.

Assisted by: Claude (Opus 5.5 Medium)

* ui : open an item's actions on right-click

Sidebar conversations and models manager rows open the same actions as
their "more actions" menu on right-click. Both menus render one shared
list of actions, so they cannot drift apart. Phones keep the browser's
long-press; the dropdown's drawer already holds the actions there.
An action can render a flyout submenu in place of its row, through the
kit of the menu it is in, and close the menu after a pick.

Adds the shadcn-svelte context-menu component, mirroring the
dropdown-menu one.

Assisted by: Claude (Opus 5.5 Medium)

* ui : close conversation tabs from a right-click menu

A tab's right-click menu closes it, the other tabs or the tabs to its
right, and stops a response the tab is generating. Closing tabs that
include the active one keeps the tab the menu was opened on.

Assisted by: Claude (Opus 5.5 Medium)

* ui : open a message's actions on right-click

User and assistant messages open the same actions as their action icons
on right-click. Selected text, links, media, code and fields keep the
browser's own menu, so copying them works as before. The fork dialog
opens from the actions context, like the delete one, so both the icon
and the menu can open it.

Assisted by: Claude (Opus 5.5 Medium)

* ui : scope right-click menu shortcuts and selection to their item

Show the conversation shortcuts only on the active row, as they act on
the active conversation, and keep the browser menu only for a selection
inside the message. Name the close-tab shortcut, and drop the unused
chip toggle, menu submenus and shared cwd row.

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-04 18:40:55 +01:00
Xuan-Son NguyenandPascal 5851a8e505 model, mtmd: drama-3 tts support (#31613)
* model, mtmd: drama-3 tts support

* clean up

* mtmd: drama3 helper stops at the codebook tower limit

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-04 15:15:09 +01:00
Aleksander Grygier efbf6e80a5 ui: show text and page thumbnails for PDF attachments, stop endless conversion retries (#31468)
* fix(ui): show extracted text in PDF attachment preview

Assisted-by: Claude (Opus 5.5 Medium)

* feat(ui): show page thumbnails for PDF attachments sent as images

Assisted-by: Claude (Opus 5.5 Medium)

* ui: stop retrying PDF preview conversion after it fails

A failed PDF-to-image conversion or text extraction reset its loading
flag, which re-ran the effect and retried forever. Remember the failed
item for each view and only retry when the item changes, so the error
state stays on screen.

Add a PdfConversionError story that feeds a broken PDF.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: keep the text of PDFs sent as images and tidy the PDF preview

Store the extracted text of a PDF sent as images in its content, so the
Text view reads it instead of parsing the PDF again; only messages stored
with the old placeholder extract it. The model still receives the images
only. Guard retries on the error state, as the item remounts per id, skip
an empty base64 payload, and share the status markup between views.

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-04 15:46:51 +02:00
Masashi Yoshimura b393d73bb0 webgpu: return maxBufferSize from get_max_size and avoid whole-buffer bindings in memset/clear (#31348)
* Fix get_max_size and improve the memset shader

* Fix buffer_memset to  handle the buffer size more than max binding size.
2026-11-04 15:46:06 +02:00
Neo ZhangandSigbjørn Skjæret 00df7c22d9 sycl : fix docker issue (#31605)
* add oneDNN in build image

* Update .devops/intel.Dockerfile

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-11-04 15:45:29 +02:00
Aaron Thomas 034e2535a9 gguf-py : validate every tokenizer merge pair, not only the first (#31631)
* gguf-py : validate every tokenizer merge pair, not only the first

The transformers 4.45 pair format was detected by checking merges[0],
then every element was converted unchecked: a non-pair element raised
a bare TypeError, and pairs with the wrong arity or non-string parts
were silently mangled into invalid merges.

Validate each element (2-string list) up front and raise the existing
"Unknown tokenizer merges format" error for anything else.

Assisted-by: Devin

* gguf-py : remove test_vocab.py per review
2026-11-04 14:42:39 +01:00
Adrien Gallouët 52e53a76df common : delete common_get_path_from_env() (#31444)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-11-04 14:21:38 +01:00
Georgi Gerganov cd959bae99 llama.cpp : bump version to 0.9.0 (#31646)
* llama.cpp : bump version to 0.9.0

* authors : update
2026-11-04 14:06:26 +02:00
Georgi Gerganov 13ec8f614c sync : ggml 2026-11-04 12:00:37 +02:00
Georgi Gerganov cca14f64fe ggml : bump version to 0.29.0 (ggml/1680)
* ggml : bump version to 0.29.0

* authors : update
2026-11-04 12:00:37 +02:00
Aman Gupta e853ec9a8c llama : follow ctx_other devices and flash attention in Gemma4 assistant (#31635)
* llama : follow ctx_other devices and flash attention in Gemma4 assistant

* llama : include <algorithm> for std::any_of
2026-11-04 11:23:25 +02:00
Kirill A. Korinsky feaf800060 ggml : Gate RVV code on the target extension (#31607)
The RISC-V vector intrinsic specification requires __riscv_v_intrinsic
to be defined when the compiler supports the API, even when the vector
extension is not selected. Clang therefore defines it for rv64gc,
causing ggml to enter RVV branches where vector types are rejected
because the required ISA extensions are absent.

Use __riscv_v for optional RVV implementations; this macro represents
the selected target ISA. Retain __riscv_v_intrinsic checks in SpaceMIT
sources, where they validate mandatory compiler API support rather than
select optional vector code.

Closes: https://github.com/ggml-org/ggml/issues/1535
2026-11-04 11:11:52 +02:00
Georgi Gerganov 4c4ca5b188 common : do not allow _ in sidecar (#31639) 2026-11-04 10:32:45 +02:00
EugeoSynthesisThirtyTwoandYour Name 162e2bb2e4 ggml : fix bf16 (#31609)
Co-authored-by: Your Name <you@example.com>
2026-11-04 10:10:22 +02:00
lhez 6900fb5ac8 opencl: improve q2_0 GEMV for Adreno (#31577)
* opencl: improve q2_0 decoding and small f32 matmuls

Assisted-by: Codex

* opencl: optimize small f32 matmuls on Adreno 8xx

Assisted-by: Codex
2026-11-04 10:07:14 +02:00
Pascal bbb02c93e9 ggml: bound views by the bytes their strides reach (#31584)
* ggml: bound views by the bytes their strides reach

A view was checked against its source with the size of a contiguous
tensor of the same shape, computed before its strides were set. Views
whose rows overlap failed that check although they stay inside their
source, and views with wider strides passed it while reaching past
the end. The view functions now pass their strides to the tensor
constructor, which checks the bytes they actually reach, like
ggml_nbytes. The conv windows of delta_net are created with all their
rows at once instead of being widened after the check.

* ggml: assert that a strided view starts from contiguous elements

The view functions set nb[0] to the type size, so the source must have
contiguous elements along dim 0 for the view to read them.
2026-11-04 09:47:11 +02:00
Georgi Gerganov 960b1d51b7 ggml : write host-buffer tensor data directly in gguf writer (#31144)
The gguf writer staged each tensor into a temporary vector before
writing it. For host-buffer tensors the data pointer is directly
writable, so skip the staging copy and write it straight to the file.
This avoids the per-tensor allocation, zero-init and memcpy.

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
2026-11-04 09:29:26 +02:00
Georgi GerganovandPascal 4b29d02629 common : fix -hf tag matching inside longer quant names (#31620)
* common : fix -hf tag matching inside longer quant names

":F16" picked a BF16 file when the repo had both, since "BF16"
contains "F16". The same applied to ":Q4_XS" vs "IQ4_XS". The tag
now has to start at a token boundary in the path.

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)

* common : accept underscore as -hf tag boundary

Some repos separate the quant with an underscore, e.g.
stepfun-ai/Step-3.5-Flash-GGUF-Q4_K_S ships
step3p5_flash_Q4_K_S-00001-of-00012.gguf, which :Q4_K_S no longer
matched. Add _ to the token boundary and pin the tag and default
lookup cases.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-04 09:22:54 +02:00
Todor BoinovskiandMax Krasnyansky bee23322f1 hexagon: add LIGHTNING_INDEXER support (#31289)
* hexagon: initial support for LIGHTNING_INDEXER op

* hex-lght-idxr: hmx for prefill

* hex-lght-idxr: q8_0 decode gather transpose, two-block decode compute

* hex-lght-idxr: pipelined hmx path for decode and prefill

* hex-lid: add support for K (f32)

* hex-lid: add support for K (f32)

* hex-lid: add support for q4_0

* hex-lid: remove scalar VTCM reads in the HMX q4_0 K prep and remove vector spills

* hex-lid: host/kernel kparams alighnment and remove sw devices in kernel

* hex-lit: update dma pipeline to use canonical dbl buffering in all paths

* hex-lit: add GGML_HEXAGON_LI_SELECT to force hmx/hvx paths

* hex-lit: add support for dims collapse

* hex-lit: make kparams constistent with other ops

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-11-03 21:17:30 -08:00
Xuan-Son Nguyen 149f5641b9 mtmd: accurate support for qwen3-vl video prompt format (#31619) 2026-11-03 23:57:26 +01:00
b97627967c model, mtmd: add chatterbox TTS support (#30993)
* add chatterbox TTS support

Co-authored-by: Pascal <admin@serveurperso.com>

* add built-in voice support

* rm redundant code

* clean up

* clean up 2

* Apply batched suggestions from code review

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* improve

---------

Co-authored-by: Pascal <admin@serveurperso.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-11-03 23:47:11 +01:00
Todor Boinovski 7071ba378a hexagon: add TOP_K_MASK op (#31552)
* hexagon: add TOP_K_MASK op

* Simplify Hexagon TOP_K_MASK kernel and do not duplicate host configuration
2026-11-03 12:04:03 -08:00
6cb3e672e8 gemma4: graph optimizations (massive gains on pp for E2B and E4B) (#31356)
* Optimizing Gemma4 graph for prompt processing

TODO:
	- Currently complete microbatch needs to contain tokens which does not require the logits; need to optimize this further
	- There are a few more operations that are performed after the kv cache is calculated, need to safely optimize them.

* Not filtering the rows when they are unmasked

* Cropping out the attention mask too.

* gemma-4: improved pruning, mask cropping and PLE scaling

Prune before the first KV-shared layer with the single-stream, causal, non-embedding safety gates.
Use cropped masks only during shared-layer attention.
Preserve requested layer-input embeddings in the KV prefix.
Remove the PLE contiguous materialization and fold equivalent PLE scaling into RMSNorm parameters/weights.

* spec-mtp: Enable masked h_nextn extraction only when the draft context shares the target KV cache

This lets Gemma 4 prune unused prompt rows before its shared layers so that we keep the full perf P.P gains with MTP.

Avoid extracting target hidden states for output-free prompt chunks in that shared-MTP path.
When a batch has a selected final token, extract only that row unless the whole verification batch is selected.

Keep ordinary MTP and classic speculative decoding behavior unchanged: they still use unmasked h_nextn because the draft
context must catch up on all target rows.

speculative-simple needs an update to select the final prompt row only for shared-KV MTP,
so the optimization works there without changing other speculative modes.
And it looks like the server already selects its final prompt row for first-token sampling.

* spec-mtp: make sure to not drop first draft round

* gemma4: enable pruning based on static cparams.kv_unified/n_seq_max=1

* gemma4-mtp: simplify prunning logic and avoid graph branches/re-allocs

* gemma4: update pruning gates to use only n_seq_max==1, kv_unified can now be used for multi-seq

* spec-simple: revert spec-simple changes for now

* gemma4: remove causal_attn from pruning gate (last source of reallocs)

* cont : adapt to llm_attn_mask changes

* cont : add TODO

---------

Co-authored-by: Meet Amin <meetamin378@gmail.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-11-03 20:30:37 +02:00
Zack LiandMax Krasnyansky 6b9c0cfcba hexagon: faster TOP_K for the backend sampler (#31279)
* hexagon: faster TOP_K for the backend sampler (skip tiles, then block scan)

Skip the bitonic sort for tiles whose max cannot reach the running k-th
best value, then replace the tile sort with top_k_scan(): reject 256-float
blocks with one HVX max-reduce and insert only surviving elements into a
sorted k-entry list. Used for k in 2..64 and rows wider than 1024, in both
the single-row and multi-row paths.

Ties keep the first-seen index. NaN is treated as -INF.

* hex-top-k: eliminate scalar reads from vtcm

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-11-03 10:06:28 -08:00
Aleksander Grygier d1c4e5bec1 ui: run local browser tests in installed Chrome with its sandbox on (#31596)
Playwright's bundled Chromium lags behind Chrome security releases
(1.56.1 ships 141, and even 1.64.1 ships 156.0.8078.4, before the
156.0.8078.73 fixes), and Playwright launches it with --no-sandbox by
default. Outside CI, use the installed, auto-updated Chrome with the
sandbox enabled. CI keeps the bundled Chromium, so the self-hosted
runner's preinstalled 1.56.1 browsers still work.

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-03 19:02:54 +01:00
Todor BoinovskiandMax Krasnyansky 20a51f24a8 hexagon: add DSV4_HC_PRE and DSV4_HC_POST ops (#30953)
* hexagon: add support for DSV4_HC_PRE

* hexagon: hsv4-hc-pre plain/gated split kernel and l2 prefetch optimization

* hex-dsv4-hc-pre: double-buffered DMA for prefill

* hex-dsv4-hc-pre: refactor DMA block setup

* hex-dsv4-hc-pre: precompute kernel params

* hex-dsv4-hc-pre: support for n_hc != 4

* hexagon: add support for DSV4_HC_POST

* hex-dsv4-hc-pre: refactor gated PRE prefill DMA to dst writeback after each block

* hex-dsv4-hc-pre: remove in-loop vector spills and redundant device-side checks

* hex-run: add support for test-llama-archs tool

* hex-dsv4: refactoring and optimizations

Remove non-dma paths.
Clean up DMA pipelines and dma_addr_t casts.
Add dims collapsing support.

Tested with test-llama-archs -a qwen4ext and test-backend-ops

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-11-03 08:26:18 -08:00
randomizedandzouyongxing 0f35d6d7b0 spec : save and restore MTP hidden state (#30141)
* speculative : save and restore MTP hidden state

Assisted-by: Codex

* tests : remove MTP state test

Assisted-by: Codex

* speculative : simplify MTP checkpoint state

Save only the boundary hidden row; verification rows are rebuilt on the supported restore paths.

Assisted-by: GPT-6

---------

Co-authored-by: zouyongxing <zouyongxing@bytedance.com>
2026-11-03 16:53:57 +01:00
Aaron Teo a464960ed3 ggml-zdnn: fix fattn test failure (#31583) 2026-11-03 23:51:07 +08:00
Giulianno 9309bf1bcf model : Add support for Metis-1.6 models (#31134)
Assisted-by: Claude
2026-11-03 16:24:45 +01:00
Tarek Dakhran d1044bc827 model : enable NVFP4 LFM3 ggufs (#31608) 2026-11-03 15:36:44 +01:00
Prabhsimran Singh 28b812d6cb mtmd : add Nemotron 3.5 Transcribe support (#31471)
* mtmd : add Nemotron 3.5 Transcribe support

Assisted-by: Claude

* convert : fix vocab of draft models for Nemotron-H targets

Assisted-by: Claude

* convert : reject non-BF16 Transcribe checkpoints

Assisted-by: Claude

* mtmd : make speaker FIFO predictions local

Assisted-by: Claude

* mtmd : normalize the mel in encode_windows

Assisted-by: Claude

* convert : map the generated speaker kernel

Assisted-by: Claude

* mtmd : validate Nemotron Transcribe mmproj shapes

Assisted-by: Claude
2026-11-03 14:27:05 +01:00
Pascal 749db7f034 qwen4exp: write the conv state tail without an intermediate cont (#31443)
build_conv_state_at makes a contiguous copy of the conv input tail
before writing it into the recurrent state, so every GDN layer issues
two device copies per generated token where one is enough. On
Qwen3.8-Flash-Next (36 GDN layers) that is 72 copies per token, now 36.
Speculative decoding adds one more per rollback slot. ggml_cpy reads
the strided view directly, as the shared build_conv_state already does,
so the extra copy is dropped.
2026-11-03 13:47:23 +02:00
Aleksander Grygier 0943ca5472 ui: add Playwright harness for recording demo videos (#31318)
* ui: add Playwright harness for recording demo videos

Scripted walkthroughs under tools/ui/demo are recorded with Playwright and
exported as H.264 mp4 for pull requests (demo/pr) and docs (demo/docs).
Fixture MCP servers and seeded settings keep recordings deterministic, a
drawn cursor and captions make the flow readable, and DEMO_UI_DIR points
the recording at another checkout such as a PR worktree.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: record demo scripts from PR descriptions and a configurable storage

Demo scripts no longer live in the repo. `npm run demo:pr -- <number>`
records the `ts demo` block from a PR description against the PR head,
uploads the mp4, a GIF preview and the script to DEMO_STORAGE, and links
the preview in a `## Demo` section of the PR description.
`npm run demo:docs` records the docs scripts kept in DEMO_STORAGE.
DEMO_STORAGE (hf:// buckets or file://) and the public URL are read
from tools/ui/.env.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: import generated conversations into demo recordings

Demos can now start with existing chats: `test.use({ traces })` imports
llama.app JSONL traces through Settings > Import/Export before the video
starts. Built-in traces are generated deterministically, other names are
downloaded from DEMO_TRACES_DATASET, and `demo:traces` pushes the built-in
ones there. Adds `demo.scroll` for wheel scrolling.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: keep demo captions clear of the chat form and isolate concurrent runs

Captions move to the top of the frame, since at the bottom they covered
the chat form a demo is often about. Each recording run gets its own
Playwright output folder, so a run starting elsewhere no longer clears
another run's video before it is saved.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: record demos at 2x and 60 fps with human-like timing

Recording goes through Chromium's screencast instead of Playwright's
video, which encoded VP8 at about 1 Mbit/s at 1x. Frames are captured at
the device pixel ratio (DEMO_SCALE, default 2) and encoded into a 60 fps
H.264 mp4, starting once the page has settled.

The cursor now glides faster along a slight arc with Fitts-like
durations, settles briefly before each click, and typing and pauses vary
a little. The randomness is seeded per demo, so re-recording gives the
same motion.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: document the demo harness prerequisites

The demo README now lists the tools the harness needs and what each is
for, and walks through creating the Hugging Face bucket and the optional
traces dataset, with the matching .env settings.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: version linked demo URLs so re-recorded previews refresh

GitHub's image proxy caches a preview by URL, so a demo re-recorded at
the same PR head kept showing the old GIF. The Demo section now links
the files with a content hash in the query string.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: label PRs that feature a video demo

Adds a `video demo` label and a workflow that applies it when a PR
description has a demo written by the UI demo pipeline (the managed
demo:start/demo:end markers, or a preview linking to a demo video),
and removes it when that section is gone.

Assisted-by: Claude (Opus 5.5 Medium)

* ci: label video demos with gh instead of github-script

The label workflow now runs the gh CLI in a shell step, like
pr-draft-label.yml, so it needs no action outside the repository's
allowed list. The video link check only counts videos uploaded for the
PR itself, so a description linking other PRs' demos isn't labeled.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: zoom demos in on details with a two-finger pinch

demo.zoom pinches in on a control or a point, holds, and pinches back out
to 1x, so a detail too small to read in the frame can be shown up close.
It drives Chromium's own pinch gesture, so the page's fixed bars stay put,
and works at desktop and phone sizes alike.

The pinch is evened out where Chromium misses: one anchored near the top
does not zoom at all, so it starts lower, and one near the bottom
overshoots its ratio, so a small pinch from the middle corrects it. The
way out reads the real scale until the page is back at 1x, which the next
gesture needs.

The overlay also draws a fingertip per touch, so a pinch on a touch
device shows as two fingers.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: wait out a cold dev server before a demo starts

A fresh checkout's dev server optimizes its dependencies on the first
load and reloads the page, which can take well past an action's timeout,
so a demo failed before it started. The demo now waits up to 90 seconds
for the app; the recording only starts once it is ready.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: record a video per test in a PR demo, shown in a table

A large PR can split its demo into several shorter videos: each test()
in the ts demo block is recorded as its own video (<sha>-1.mp4, ...),
and the Demo section lays their previews out in a table, each column
headed by its test's title.

GIF previews now keep their longer side at DEMO_PREVIEW_WIDTH and shrink
until they are under 5 MB, since GitHub's image proxy refuses anything
larger and shows a broken image instead.

Importing traces works on a phone viewport too, where Settings sits in
the sidebar under the page until the menu key slides it aside.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: fix demo trace imports on desktop and lay landscape demos out two to a row

Importing traces failed on a desktop whose collapsed sidebar sits under
the page: it now expands the sidebar from the page's menu key on a phone
or from the sidebar's logo on a desktop, whichever makes Settings
reachable. Afterwards it forgets the expanded sidebar, so a demo starts
with it collapsed, as on a first visit.

A PR demo with several videos shows up to four portrait (phone) videos
to a row, or two landscape ones, so each preview stays readable.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: only record other people's PR demos at a reviewed head commit

demo:pr runs the PR's code and demo script on this machine, with the
gh and hf logins, so a PR by someone else now needs --trust <sha>
naming the head commit that was read, and is refused again after new
pushes. Recordings also launch Chromium with its sandbox on, which
Playwright turns off by default.

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-03 12:30:49 +01:00
Tarek Dakhran f9448cddc4 model : record layer inputs in LFM3 for spec dec (#31598) 2026-11-03 13:29:10 +02:00
Jasmine-tim 0978a6e902 ggml: avoid signed overflow in ggml_roll bounds check (#30375)
ggml_roll validated each shift with abs(shift), but abs(INT_MIN) is signed
integer overflow UB evaluated before the bounds check runs. Compare with a
signed-safe test instead.
2026-11-03 13:26:29 +02:00
Daniel Bevenius 0ef27309b7 conversion : use original model tensor names in _flush_nvfp4_experts (#31427)
This commit updates the _flush_nvfp4_experts to take a new parameter
which is a dictionary mapping the key to the original model tensor name.

The motivation for this is that I noticed that the tensor names being
looked up were not the original model tensor names, for example:
```console
  model.layers.1.mlp.experts.up_proj.weight
```
But _nvfp4_layer_algo uses the original per-expert names:
```console
(Pdb) p self._nvfp4_layer_algo
{
'backbone.layers.1.mixer.experts.0.up_proj': 'W4A16_NVFP4',
...
```
This means that the lookup will never match for experts and fall back to
_nvfp4_global_algo. If the global algo is W4A16_NVFP4 then this would
go unnoticed, but for a model like
nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 which is
MIXED_PRECISION, the expert tensors will not get prec_a4=false so they
would use the native W4A4 path on Blackwell even though the original
model tensors declare them as W4A16_NVFP4.
2026-11-03 11:40:51 +01:00
mmalpe-nvandGaurav Garg 8fc0376721 cuda: potential fix for Q8_1 correction sum (#31247)
* cuda: potential fix for Q8_1 correction sum

* Update ggml/src/ggml-cuda/quantize.cu

Co-authored-by: Gaurav Garg <gaugarg@nvidia.com>

* addressed review comment to change the quantize_mmq_q8_1 flow as well

---------

Co-authored-by: Gaurav Garg <gaugarg@nvidia.com>
2026-11-03 15:54:42 +05:30
Bhargav Krish 740251a920 cuda : gather batched tensor reads into mapped host memory (PR to master) (#31391)
* cuda: gather batched tensor reads into mapped host memory

Assisted-by: Codex

* add comments

Assisted-by: Codex

* simplify the implementation based on checks that exist earlier in the flow

* simplify batched tensor gather indexing

* add conditional dispatch

* address comments - make host registration raii, better test mindful of ci time, assert instead of return false

* cuda : gather small reads and use pinned DMA for large reads

* cuda : use std::vector for host gather descriptors

* cuda : add 8-byte gather copies, generalize tensor test and log registration failures

* common buffer for descriptors

* simplify stream sync + increase to 128k descriptor pinned buf
2026-11-03 15:31:50 +05:30
zyang-dev 71efc559a0 Add nccl support for multi-node tensor parallelism (#28967)
* Add a backend interface for collectives across ranks running in separate processes, implement it in the CUDA backend using NCCL, and use it in the RPC server when available.

* Drop ggml_backend_comm_allreduce_rank_t and reuse ggml_backend_comm_allreduce_tensor for per-rank communicators

* reuse ggml_backend_cuda_comm_allreduce_nccl for per-rank communicators. Remove the helper function backend_comm_proc_address

* Change ggml_backend_comm_get_unique_id_t and ggml_backend_comm_init_rank_t to more generic multi-process backend communicator interfaces

* set maxCTAs in ncclConfig_t to 8 for dgx spark in ggml_backend_cuda_comm_init_rank
2026-11-03 14:56:19 +05:30
Nidhi 19e8a86fe8 tools : reject mismatched shards in GGUF split merge (#31579) 2026-11-03 11:11:05 +02:00
Aman GuptaandPratik Narola 259d91709d meta: support GLM-5.3 next sm tensor (#30499)
* meta: support GLM-5.3 next sm tensor

* glm5-next : cont q_absorbed for -sm tensor with non-unified KV

* meta : keep GGML_OP_ARANGE in ggml.h order

* glm5-next : split MLA heads in pairs for -sm tensor

The MLA cache has a single KV head, so the heads on a device are its GQA
ratio. The HIP tile FA kernel for D=512 needs an even GQA ratio; with 3
devices the 16 heads of the test model were split 5/5/6 and it aborted.

* meta : keep the split for views inside one slab of the split axis

With a non-unified KV cache, build_attn_mha splits q into streams with a
view. For a permuted q, e.g. the absorbed query of glm5-next, that view
aborted with "view of permuted tensor not implemented". The split is now
kept if every other dim of the source and the view stays inside one slab
of the split axis. The per-device strides are only rescaled above it.

From tetherto/qvac-fabric-llm.cpp#353.

Co-authored-by: Pratik Narola <pratik.narola@tether.io>

* glm5-next : drop the q_absorbed cont

The meta backend can now split the per-stream view of the permuted q, so
the cont is not needed for -sm tensor with a non-unified KV cache. With
the cont, the q of flash attention was not contiguous, and when FA ran on
the CPU (WebGPU) the meta get_tensor aborted.

---------

Co-authored-by: Pratik Narola <pratik.narola@tether.io>
2026-11-03 10:51:10 +02:00
Georgi Gerganov 36ee3caf7a metal : fix cpy kernel for dst non-contiguous in dim 0 (#31581)
kernel_cpy_t_t indexed the dst innermost dim with an implicit element-size
stride instead of args.nb0, so cpy into a permuted view (nb0 != type size)
wrote the wrong addresses. caught by the new test-backend-ops cpy case with
permute_dst = {1, 0, 2, 3} from #31446. addressing is unchanged when dst is
contiguous in dim 0.

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
2026-11-03 10:41:24 +02:00
FamousMonsterrandFamousMonsterr fac3a894fc vulkan: skip maxStorageBufferRange pre-check for non-compute tensors (#31114)
* vulkan: skip maxStorageBufferRange pre-check for non-compute tensors

Pre-allocated leaf tensors and views (GGML_OP_NONE / GGML_OP_VIEW) are never
executed by the backend, but were still rejected by the descriptor-size
pre-check in ggml_backend_vk_device_supports_op when larger than
maxStorageBufferRange on devices without shader_64b_indexing. The scheduler
then aborts with 'pre-allocated tensor (...) in a buffer (Vulkan0) that cannot
run the operation (NONE)' - e.g. the MoE expert cache banks (llama-moe-cache)
of a 256-expert model with --moe-cache-mib on GCN GPUs, where a whole bank
(2.4 GB) exceeds the limit while the consuming ops only access small
per-step views.

Skip the size pre-check for non-compute ops (NONE and VIEW); the buffer type
allocation limit still bounds the actual allocation, and compute ops keep
validating their sources.

Tested on AMD Radeon Pro VII (gfx906, no shader_64b_indexing):
- before: GGML_ABORT 'pre-allocated tensor (moe_cache.0.blk.0.ffn_down_exps.weight) ... cannot run the operation (NONE)' with llama-cli -ngl 99 --cpu-moe --moe-cache-mib 6144 (Cyber-Tiel-Coder-35B-A3B, qwen35moe, 256 experts per layer)
- after: loads and generates at 8.1 t/s (vs 0.6 t/s without the cache)
- regression: -ncmoe 16 path unchanged (19.2 t/s), qwen3moe/qwen35moe without cache unchanged

Fixes #30697

* vulkan: use ggml_op_is_empty for the non-compute size-check skip

---------

Co-authored-by: FamousMonsterr <FamousMonsterr@users.noreply.github.com>
2026-11-03 08:53:04 +01:00
Aman GuptaandPascal b74298c109 delta_net_base: refactor conv state (#30878)
* delta_net: access conv state via views

* fix sm tensor

* delta_net: fill the conv slot index on the host

The rollback slot index of the conv windows is a constant per graph
shape. It is now an I32 input of the recurrent memory filled on the
host, instead of an arange and an argsort computed on the device, and
the meta backend no longer needs a mirrored split for ARANGE.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-03 08:28:35 +01:00
Dongsheng LuoandDongsheng Luo 37b0f9c36a vulkan: fix out-of-bounds NVFP4 UE4M3 scale lookup (#31079)
* vulkan : fix out-of-bounds NVFP4 UE4M3 scale lookup

On the non-OCP path, ue4m3_fp32_lut has 128 entries and is indexed with the raw scale byte. ggml_validate_row_data accepts every byte and ggml_ue4m3_to_fp32 ignores bit 7, so a byte >= 0x80 reads past the shared array. Vulkan robustness does not cover shared memory.

Size the table, both init loops, and the two host shared-memory budgets to 256 entries. ue4m3_to_fp32_build() already masks the exponent and treats only 0 and 0x7F as special, so entries 128-255 match the CPU. A 256-entry table keeps 0xFF distinct from 0x7F: the CPU decodes 0xFF as a non-zero value and 0x7F as zero.

Each NVFP4 pipeline uses 512 more bytes of shared memory. ggml_fp32_to_ue4m3 never sets bit 7, so models quantized by llama.cpp are unaffected.

Assisted-by: Claude
Assisted-by: Codex

* vulkan : fix out-of-bounds NVFP4 UE4M3 scale lookup

Assisted-by: Codex

---------

Co-authored-by: Dongsheng Luo <7137760+flyingdoog@users.noreply.github.com>
2026-11-03 07:58:56 +01:00
Georgi Gerganov ca8daad199 llama : drop the redundant cnv from llm_attn_mask (#31558)
The dspark block mask is now derived at input-fill time: llm_attn_mask::block_uniform
fills every token of a sequence with the mask row of its first token, so the
graph-level reshape/repeat in dflash is gone and cnv always aliases raw.

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
2026-11-03 08:11:06 +02:00
Aman Gupta 0f663d2471 dsv4.1: fix llama-context tests with partial offload (#31292) 2026-11-03 08:08:42 +02:00
hesiru 00f113eceb cuda : prevent premature eviction of graphs about to be used (#30662)
* cuda : prevent premature eviction of graphs about to be used

* add last_graph_use_time for entire cuda graph map and testcase for cuda graph

Assisted-by: Codex

* delete the test
2026-11-03 08:06:03 +02:00
shaofeiqiandLi He e6192de0a6 opencl: add initial iq4_xs support (#31339)
* opencl: add iq4_xs AOS mv

Assisted-by: Codex

* opencl: add iq4_xs SOA mv and tiled mm

Assisted-by: Codex

* opencl: add iq4_xs noshuffle gemm/gemv for adreno

Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Assisted-by: Codex

* opencl: add iq4_xs dp4a for adreno

Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Assisted-by: Codex

* opencl: optimize iq4_xs adreno decode and narrow batches

Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>

Assisted-by: Codex

---------

Co-authored-by: Li He <lih@qti.qualcomm.com>
2026-11-02 21:58:19 -08:00
frogscooper 4402e65401 chat : allow constraint type after gpt-oss final channel (#31556) 2026-11-03 07:55:11 +02:00
Aman Gupta 74101b3cda ci : update the HIP VGPR spill ignore list for the kv_rows FA kernel argument (#31550)
The kv_rows argument added to the FA kernels in #29510 changes their mangled names, so the known spills were no longer ignored.
2026-11-03 07:54:47 +02:00
Leon 713319e284 webgpu: add ARANGE support (#31455) 2026-11-03 07:54:00 +02:00
Aman Gupta ee17f293b6 kv-cache : never return 0 kv_rows for sequences without cells (#31570)
A cache that shares the cells of another one (f.ex. the gemma4 assistant) can build a graph before its sequences have any cells, f.ex. at the server warmup. get_n_kv_rows() then returned 0, which means that kv_rows are not used, so the graph fell back to the dense KQ mask and the scheduler re-planned for it. Later graphs with larger masks then aborted with GGML_SCHED_DEBUG_REALLOC=1.
2026-11-03 07:51:53 +02:00
51fe42c5d1 hexagon: tiled hvx permute (#31446)
* hexagon: add tiled HVX transpose path for permuted CONT/CPY

Permuted copies where dst dim 0 is strided in src, but some other src dim
is contiguous, used to fall back to HTP_COPY_KERNEL_RESHAPE. That issues
one DMA descriptor per element. The channel-first <-> channel-last
permutes around per-channel norms (e.g. [W,H,T,C] <-> [C,W,H,T] in video
VAEs) ht this path, and these copies ran at about 45 MB/s.

Add HTP_COPY_KERNEL_TRANSPOSE for same-type, same-shape f32/f16 copies
into a contiguous dst.

Host:
  - drop extent-1 dims and merge dims that stay adjacent in src into a
    canonical view, stored in the kernel params. There are no minimum
    size limits.
  - the copy is treated as a 2D transpose between dst dim 0 and the
    contiguous src dim, batched over the remaining dims.
  - tiles are sized to 64 KB per buffer and halved until they fit in
    session VTCM. blkk is capped so the halfword vgather offsets fit in
    16 bits.

Kernel, per tile:
  - 2D DMA of blk0 strided src rows (blkk contiguous elements each)
    into VTCM
  - vgather transpose in 32x32 (f32) or 64x64 (f16) blocks
  - 2D DMA of blkk contiguous rows out to dst

Staged and transposed buffers are double-buffered, so the next tile's
input DMA overlaps the current transpose and output DMA. Tiles are split
across threads and mdev sessions.

Small dims:
  - packed mode: when dim k follows dim 0 and dim 0 divides the HVX
    lanes (f32: 2/4/8/16, f16: 2/4/.../32), one vgather writes several
    whole dst rows, and each tile goes out as one contiguous DMA.
  - when the contiguous src dim fits in one vector, it is staged exactly
    and the tile is made long along dim 0.

All other copies select the same kernels as before.

Tests: add test_cont cases for perms {1,2,3,0} and {3,0,1,2}, f32 and
f16, at various sizes.

* hexagon: cpy transpose cleanups

- fix vgather halfword offset comments: offsets are unsigned 16-bit, caps are kept lower for speed
- set packed directly in the packed case
- add dim_k == 3 CONT tests (full reversal)

Assisted-by: Claude Opus 5.5

* hexagon: use cpy transpose path for ne[0] == 1 permutes

- drop the nb[0] check in the transpose solver: size-1 dims are skipped, and the canonical tp.nb[0] check covers packed dim 0
- try the transpose path before SAMESHAPE_SAMETYPE, which did one DMA per element for these
- add ne[0] == 1 permute CPY tests

Assisted-by: Claude Opus 5.5

* hex-cpy: further improve cpy

Bring in improvements from aizip branch/PR.

Cleanup kparams and replace divs with fastdiv.

Co-authored-by: Yiwei Shao <yiwei@aizip.ai>

* hex-cpy: avoid vtcm bank conflicts

* hex-cpy: trim down the comments

* cpy-test: trim down new tests that cover permute scenarios

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
Co-authored-by: Yiwei Shao <yiwei@aizip.ai>
2026-11-02 21:13:38 -08:00
Aman Gupta 2cbca389f5 dflash: copy hidden features via device copies (#31506) 2026-11-03 09:02:32 +08:00
Aaron Teo 29c720eaf7 docs: update s390x build docs (#31546)
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
2026-11-02 20:52:39 +02:00
d02b4c0d66 mtmd: Pocket tts - support for voices without voice cloning (#31429)
* feat: support PocketTTS voices with no voice cloning

The kyutai/pocket-tts-without-voice-cloning weights ship predefined voices as the
backbone KV state after the voice prompt. The converter turns each state back into
the prompt embeddings and stores them in the mmproj (voice.<name>), the zeroed voice
encoder is left out, and the "voice" field of /tts and --tts-voice select one of them.
A speaker reference is rejected when the model has no voice encoder.

Assisted-by: Claude Fable 5.1

* convert : pocket-tts voices never fail the conversion, log the drift of the state

The voice states shipped by Kyutai are not always the ones the pack's
weights give for the prompt: italian, german and spanish match within the
bf16 rounding, english_2026-04 is 2% off and french 20% (its layer 0 state
comes from other weights), while the reference loads them as is and they
still sound like the same voice. The converter now re-prompts the backbone
with the recovered embeddings, compares the whole state and logs the
distance per voice, warning above 5%. A state that does not belong to the
weights at all is skipped, so the packs of kyutai/pocket-tts convert too.

Assisted-by: Claude Fable 5.1

* mtmd : list the predefined voices, expose them in GET /props

mtmd_gen_audio_info gains n_voices and mtmd_gen_audio_voice_name() the
names, with the voices worded as a feature of any audio generation model.
The server lists them under "voices" in /props when the model ships some.

Assisted-by: Claude Fable 5.1

* server : an unknown voice is a bad request

A model with predefined voices answers 400 naming the voice instead of 500
"failed to process TTS prompt". A model without them ignores "voice" with a
warning, as the other audio generation models do.

Assisted-by: Claude Fable 5.1

* refactor api

* fix conversion

* small renaming

* fix lint

* tts : a model with predefined voices speaks with the first one by default

Without a voice prompt such a model only produces near silence. When neither
a voice nor a speaker reference is given, llama-tts and the server now take
the first predefined voice and say so in the log, as the LFM3-Audio template
does with its default voice.

Assisted-by: Claude Fable 5.1

* mtmd : one comment per voice accessor, converter states the rule only

The clip.h comment described clip_gen_get_voice but sat above all three
functions; the converter comment quoted the drift measured per pack, which
goes stale with the next upload of the weights.

Assisted-by: Claude Fable 5.1

* clean up redundant code and comments

* mtmd: match the LFM3-Audio voice name case-insensitively

The tts chat template lowers the voice name, so Male speaks with
the male voice; the voice lookup lowers it the same way and names the
bitmap after the canonical voice.

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-02 18:39:51 +01:00
Maphist0andSigbjørn Skjæret e4b6c8ca6a model: add Clover support (#31306)
* model : add AR-only Clover MoE support

Add GGUF conversion, model loading, and causal hybrid attention for Clover, reusing shared attention, recurrent-state, MoE, and LoRA helpers.

* Update tests/test-llama-archs.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* convert : filter Clover tensors using filter_tensors

* clover : use the standard ssm_a tensor name

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-11-02 18:37:27 +01:00
Aman GuptaandGeorgi Gerganov 1b6935b227 llama : flash_attn_ext_rows - more efficient unified kv for multi-seq (#29510)
* llama : use llm_attn_mask and kv_rows for unified attention

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* cont : optimize `get_seq_cells`

* llama : build the kv_rows lists from per-sequence cell bitmaps

Keep a bitmap over the cells for each sequence (the transpose of `seq`), updated in seq_pos_inc/dec, and build the cell lists by walking its set bits. The lists stay in cell order, cells shared by several sequences go to every list, and nothing has to be rebuilt when cells are removed.

Also leave out of the lists the cells that the SWA hides from every token of the slice (standard and chunked SWA).

* cont : make get_kv_rows_p_min private

* cont : minor metal

* llama : use kv_rows for every ubatch of a unified cache

A ubatch made of equal-length runs with a single sequence each (f.ex. the drafts of several slots) now splits into per-sequence slices.

Any other ubatch (interleaved, uneven or coupled sequences) becomes a single slice over the cells of all its sequences instead of falling back to the dense KQ mask, so the graph keeps the topology of the reserved graph.

* llama : merge get_kv_rows_seq_ids and get_kv_rows_p_min into get_kv_rows_slices

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-11-02 18:13:33 +02:00
Alexey Kopytko 3f8af15a76 imatrix : fix --nextn crash (#31522) 2026-11-02 14:20:02 +01:00
Niklas Wenzel cd8427d018 context : simplify setters (#29730) 2026-11-02 14:13:31 +02:00
Maternion 6b00d5627b readme : fix links in README.md for maintainer PRs (#31487) 2026-11-02 14:12:18 +02:00
li-lizhe 458fcfc340 docs: CANN backend supports BF16 on 910B, document GGML_CANN_NO_PINNED (#31249)
The note under 'Run the inference' still said that the CANN backend only
supports FP16/Q4_0/Q8_0 models, while the DataType table in the same file,
and ggml-cann.cpp (BF16 is enabled for MUL_MAT, GET_ROWS, CPY and CONT for
every non-310P device), say otherwise. Spell out the per-device support and
mark BF16 as unsupported on 310P instead of leaving the cell empty.

GGML_CANN_NO_PINNED is read by ggml-cann.cpp when allocating host buffers
and when advertising the host_buffer device capability, but it was not
documented, so the escape hatch for a failing pinned host allocation was
not discoverable.

Signed-off-by: li-lizhe <147392333@qq.com>
2026-11-02 14:11:30 +02:00
Aaron Teo 6ed812057c ggml-zdnn: share buffers with host (#31481)
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
2026-11-02 14:11:07 +02:00
Yuri Khrustalev ea24b952cd ggml : use vm_allocate instead of posix_memalign on all Apple platforms (#31437)
* ggml : use `vm_allocate` instead of `posix_memalign` on iOS

* ggml : use vm_allocate on all Apple platforms, run test-host-buffer-release by default
2026-11-02 14:08:21 +02:00
lhez 84d9ce0fad opencl: add topk_mask (#31441) 2026-11-02 13:59:05 +02:00
dcbb091677 vulkan: add tensor-parallel allreduce function (#25051)
* vulkan: add tensor-parallel all-reduce communication

* vulkan: require shared semaphores for tensor communication

Assisted-by: Codex

* vulkan: reduce directly from shared host buffers

Assisted-by: Codex

* vulkan: fix ring all-reduce races with 3+ devices

Two hazards in the tensor-parallel ring all-reduce made -sm tensor output
non-deterministic with three GPUs:

- With n = 3, step t's host-slot region starts at t * cels * 2 bytes, which
  is not a multiple of 32, so consecutive steps' transfer copies share a
  32-byte sector of the imported host memory. On NVIDIA the later copy's
  leading bytes (up to 10 F16 values at the start of every chunk) end up
  stale in host memory, even when the copies are ordered by a semaphore.
  Give every step its own 256-byte-aligned region. n = 2 and n = 4 already
  had aligned regions for the usual sizes, so their layout is unchanged.

- A single-shot all-reduce leaves every device's host slot being read by all
  peers, but the next ring round only ordered its first transfer after nextd,
  and only from round 2 on. With n >= 3 a device could overwrite its slot
  while prevd was still summing it. Wait for all peers on the first ring
  round after a single-shot all-reduce.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* vulkan: fold the tensor-parallel all-reduce into the compute submissions

With -sm tensor the meta backend drives every device from one thread and calls
graph_compute once per subgraph (two per layer), so decode was bound by the CPU
cost of submitting, not by the GPUs: ~1500 vkQueueSubmit per token for a 64-layer
model on two GPUs, with the GPUs idle ~40% of the time.

- Don't split subgraphs by flops while a communicator is active. The heuristic
  was tuned for whole graphs and turned each small subgraph into ~5 submits.
- Fold the small (single-shot) all-reduce into the compute submissions: the
  last command buffer of a subgraph stays open, the all-reduce appends the
  gather copy to it and submits it, then opens the next command buffer with the
  peer waits and the reduce, which the next subgraph is recorded into. Each
  subgraph now costs one submit per device. The reserved progress value is
  attached by whichever path submits the open context.
- Alternate between two host slots, placed outside the ring's regions, so the
  gather no longer needs a WAR wait on the peers' previous reduce.
- On synchronize, submit the deferred work of every device in the group first,
  so the tail of the graph runs on all devices in parallel.
- Wait for the transfer queue as well as the compute queue before freeing or
  reallocating communication buffers, and don't leave a stale cap behind when a
  reallocation fails.

qwen3.5 27B Q8_0 on RTX 4090s, tg128: 2 GPUs 32.2 -> 49.7 t/s, 3 GPUs 20.9 ->
40.0, 4 GPUs 15.0 -> 27.6; pp512 on 2 GPUs 2138 -> 2320. Greedy output is
unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* - Two-phase all-reduce over shared host memory: write own slot + signal, wait peers + add from their slots
- One exported timeline per device, no backend hooks; comm work recorded in the backend compute ctx
- F16 wire above 1 MiB, F32 below; small reductions ride with the next graph submit
- Double-buffered, grow-only host slots; zero-size splits zeroed
- Remove ring, transfer-queue leg, device staging, shrink, Windows shims
- comm_init returns nullptr when unsupported; add GGML_VK_COMM_DISABLE
- No flops-based submit splitting for graphs below 1 GFLOP

[rebased onto the ring all-reduce fix and the folded all-reduce, which this design replaces; their backend hooks are removed here]

* add minimal submit split size

* replace opaque_fd with sync_fd for spec compliance

* add precision check

* add semaphore pool

* rename env var, add logging if requirements not met

* reduce comments

* code improvements

* vulkan: use posix_memalign for the tensor-parallel host buffers

std::aligned_alloc is unavailable on Android bionic below API 28, which
upstream switched away from in #30828; do the same for the communicator's
host slots.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* vulkan: synchronize the tensor-parallel all-reduce through the host

Exchanging SYNC_FD semaphores costs a kernel wake-up on every all-reduce:
on NVIDIA the waiting GPU starts ~35 us after its peer has finished, which
is most of the idle time in a decode step. OPAQUE_FD semaphores avoid that
but must not be imported into another physical device.

Use only device-local timeline semaphores instead. Each device signals
done[i] once its partial sum is in host memory (after a barrier that makes
it available to the host), and a bridge thread signals go[i] from the host
once all peers of i are done. The waiting side makes the host memory
visible with a host-write barrier, since its submit predates the data.
The thread spins for up to 200 us on each wait (GGML_VK_ALLREDUCE_SPIN_US)
and then blocks, so it is idle when there is no work. This also takes the
per-round fd export, dup and import calls off the main thread and drops the
VK_KHR_external_semaphore_fd requirement and the binary semaphore pool.

Graphs below 1 GFLOP, such as tensor-parallel decode slices, are no longer
split into several submits.

qwen3.5 27B Q8_0 on RTX 4090s, tg128: 2 GPUs 37.2 -> 45.4 t/s, 3 GPUs
24.1 -> 35.5, 4 GPUs 16.5 -> 27.1; pp512 unchanged. Greedy output is
unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* use sync_fd on mesa drivers

* refactor for readability

* move sync_fd semaphore handling to thread, trigger mesa submit threads

* move allreduce functions to new cpp file

* clean comments

* smaller fixes, code improvements

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Ruben Ortlam <rortlam@redhat.com>
2026-11-02 13:57:56 +02:00
ravel7524 07ec7c8af0 vulkan: improve small-n split-K selection on RDNA3 and RDNA4 (#30146)
* vulkan: improve small-n split-K selection on RDNA3 and RDNA4

* Clean up test cases by removing commented code

Removed tests as requested
2026-11-02 13:54:09 +02:00
Amadeus dyw cb02250b5b vulkan: avoid redundant scans when zeroing skipped rows (#31266)
* vulkan: avoid redundant scans when zeroing skipped rows

* vulkan: remove unreachable zero-expert fallback
2026-11-02 13:53:27 +02:00
92es 2c93f0b6e1 sycl : fast ggml_sycl_comm_exchange() on Intel Arc Pro B70s (#30956)
* Update ggml-sycl.cpp

Increases -sm tensor prompt eval t/s by +19% with zero PPL change.

* Update ggml-sycl.cpp

Replace first pair of wait() calls with ext_oneapi_submit_barrier().  The wait() calls was discovered to cause MTP degradation.

* Update ggml-sycl.cpp

Adding comm_exchange function pointer to switch between 2 and 4 buffer ggml_sycl_comm_exchange versions @ initialization time (versus hot path) controlled by SYCL_ENABLE_DEFAULT_CONTEXTS=1 setting.

* Update ggml-sycl.cpp

adding back 2nd half of 4-buffer free.

* Update ggml-sycl.cpp

Removing extra whitespace.

* Update ggml-sycl.cpp

Excluding Windows until it is validated on Windows.
2026-11-02 13:52:53 +02:00
Titaniumtown 5e18eb021b sycl: dmmv code cleanup (#31325)
* sycl: fold q8_0 reorder dmmv into the generic template

* sycl: template the dmmv non-K launchers

* sycl: template the dmmv K-quant launchers
2026-11-02 13:51:20 +02:00
Aleksander Grygier 5c2ea1ab11 ui: disable model deletion the server would refuse (#31305)
The router only removes models it took from the Hugging Face cache, but
the models manager offered "Delete from disk" for every model, so a model
from --models-preset or --models-dir failed with a "not removable" toast.
Read the source and can_remove fields of /models and show the delete
action disabled for those models, with a tooltip (a hint line in the
mobile drawer) saying how to remove them instead.

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-02 13:36:00 +02:00
Aleksander Grygier 23426d797e ui: align alert dialog styling with other overlay surfaces (#31312)
Use the popover background, softer border, rounded-xl corners and
shadow-md like dropdowns, hover cards and the settings dialog, and
match the dialog title line height.

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-02 13:35:25 +02:00
Aleksander Grygier 90080ce094 ui: keep queued steering messages in their own conversation (#31304)
A message queued while a conversation was busy was re-sent through
sendMessage, which targets the active conversation, so switching chats
before the turn ended delivered it into the wrong one. Re-send only when
the owning conversation is active; otherwise keep it queued there and
flush it when that conversation is opened again.

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-02 13:34:17 +02:00
Aleksander Grygier 64be95b163 ui: show interrupted state for tool calls that never returned (#31314)
* ui: show interrupted state for tool calls that never returned

The chat-level streaming flag was applied to every assistant message, so a
tool call left without a result in an earlier message kept spinning while a
newer reply streamed. Scope streaming to the last assistant message and mark
finished calls without a result as interrupted.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: mark a shell command stopped before any output as interrupted

exec_shell_command creates its result message before running, so stopping
it mid-run leaves an empty result instead of none.

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-02 13:34:00 +02:00
Aleksander Grygier b12b405c17 ui: don't reload a loading model on select, prefer it in pre-selection (#31326)
* ui: don't reload a loading model on select, prefer it in pre-selection

Assisted-by: Claude (Opus 5.5 Medium)

* ui: skip hidden models in automatic model selection

Assisted-by: Claude (Opus 5.5 Medium)

* ui: don't count a model being unloaded as loading

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-02 13:33:28 +02:00
Liu Jia aac6da8012 docs : fix Vulkan example command in build.md (#31530) 2026-11-02 12:01:33 +01:00
Niklas Wenzel 00ef136649 tests : merge identical if branches (#31523) 2026-11-02 11:59:00 +01:00
MuoDoo 59695669f1 jinja : implement map('filter') (#24033)
* jinja: implement map('filter')

* jinja : validate filter keyword arguments

Assisted-by: Codex
2026-11-02 11:45:40 +01:00
Uber a7a490a490 ggml : fix clang-cl build with AVX512 BF16 (#31276)
* Correctly gate immintrin.h inclusion when building with clang-cl and avx512bf16

* ggml : explicit vector casts for clang-cl
2026-11-02 18:39:53 +08:00
Li Chuyang f72dc30928 docs: fix Huawei ascend brand (#31463) 2026-11-02 18:36:45 +08:00
Sigbjørn Skjæret fcfe4b59df requirements : minor adjustments for python 3.14 wheels (#31534) 2026-11-02 11:34:45 +01:00
o7si 4e910a86e8 cmake : add explicit thread linkage (#31414) 2026-11-02 09:34:17 +01:00
Pascal 1c5ed702c9 webgpu : fix CI by keeping scratch alloc sizes monotonic in batch length (#31422)
* webgpu: keep scratch alloc sizes monotonic in the batch length

The flash attention vec path and the MMVQ path append a scratch area
after dst, but get_alloc_size only accounted for it when the current
batch takes that path. The size was therefore not monotonic in the
batch length: a graph reserved at n_ubatch (tiled path, no scratch)
or at one token did not cover batches of 2 to 19 query rows, and
GGML_SCHED_NO_REALLOC builds aborted on an unexpected reallocation.

Pass the row count to both path predicates and size the scratch with
the row count clamped to the longest batch the short path serves.

* nit: align the mmvq pipeline key with clang-format 22

* webgpu: address review @yomaytk, name the mat-vec row threshold
2026-11-02 09:36:04 +02:00
Sigbjørn Skjæret 785322d17f requirements : fix numpy dependencies (#31508) 2026-11-01 18:23:40 +01:00
Michael Taylor dde879bc4f sycl: Try PkgConfig for LevelZero includes / libs (#31076) 2026-11-01 01:46:43 -05:00
shaofeiqiandLi He 060b2fea4e opencl: add initial iq3 support (#31338)
* opencl: add iq3_s AOS mv

Assisted-by: Codex

* opencl: add iq3_s SOA mv and tiled mm

Assisted-by: Codex

* opencl: add iq3_s noshuffle gemm/gemv for adreno

Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Assisted-by: Codex

* opencl: add iq3_s dp4a for adreno

Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Assisted-by: Codex

* opencl: optimize iq3_s adreno decode and narrow batches

Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Assisted-by: Codex

* opencl: add iq3_xxs AOS mv

Assisted-by: Codex

* opencl: add iq3_xxs SOA mv and tiled mm

Assisted-by: Codex

* opencl: add iq3_xxs noshuffle gemm/gemv for adreno

Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Assisted-by: Codex

* opencl: add iq3_xxs dp4a for adreno

Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Assisted-by: Codex

* opencl: optimize iq3_xxs adreno decode and narrow batches

Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Assisted-by: Codex

---------

Co-authored-by: Li He <lih@qti.qualcomm.com>
2026-10-31 22:26:47 -07:00
Mark TodorovichandMax Krasnyansky a91eab67d8 hexagon : fix DMA transfers with strides > 24 bits (#31442)
* hexagon : fix DMA transfers with strides > 24 bits

On v75+ the 2D DMA descriptor stores row_size and strides in 24 bits.
dma_queue_push sent oversized strides to fallback_2d, which only splits
nrows, so strides of 16 MB or more were truncated - wrong addresses.
Fix will fall back to one transfer per row in that case.

Also added two CONT tests with source strides over 16 MB.

* hex-dma: refactor the fallback paths to remove duplication

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-10-31 16:49:23 -07:00
bosh d5ef408e20 mtmd : fix InternVL CLS token order (#30886) 2026-11-01 00:30:01 +01:00
Niklas Wenzel 79bce0c881 tests : enable Gemma 4 in test-llama-archs (#31401)
* tests : enable Gemma 4 in `test-llama-archs`

* tests : use GQA for Gemma 4 in `test-llama-archs`
2026-10-31 23:55:02 +01:00
Tarek Dakhran bc428e407a llama : enable tensor split for LFM3 (#31431)
Assisted-by: Claude Opus 5.5
2026-10-31 23:40:50 +01:00
Alessandro de Oliveira Faria (A.K.A.CABELO) f6312ad75e vendor : update miniaudio to 0.11.26 (#31459)
[no ci]
2026-10-31 23:37:20 +01:00
Niklas Wenzel 0138937e3f llama : fix rotation of attention tensors in layers with shared KV (#31406)
* llama : fix rotation of layers that reuse another layer's KV cache

* llama : use KV source rotation only for attn_* tensors
2026-10-31 23:30:49 +01:00
Anthony Maltsev ddf48f85c9 reshaping so mm on 2d instead of 3d tensor for lfm sc (#30831) 2026-10-31 20:34:57 +01:00
aiqubits f3586e66bc server: cap shared decision prefixes at the first output (#30710)
Limit the shared prefix to the first required output position of the parent and each child task.
This keeps decision output embeddings available in the final batch when questions share option tokens.

Fixes #30482

Signed-off-by: aiqubits <aiqubits@hotmail.com>
2026-10-31 19:09:48 +01:00
David Garcia 2871678a1d server : fix /v1/responses endpoint to accept extra_params (#30959) 2026-10-31 19:08:08 +01:00
boshandXuan-Son Nguyen 7bd1808308 mtmd : resize Gemma 4 images without letterbox padding (#30922)
* mtmd : resize Gemma 4 images without letterbox padding

* Apply suggestion from @ngxson

---------

Co-authored-by: Xuan-Son Nguyen <thichthat@gmail.com>
2026-10-31 18:55:33 +01:00
shaofeiqiandLi He 0ec4027a2a opencl: add fwht (#31434)
Co-authored-by: Li He <lih@qti.qualcomm.com>
2026-10-30 22:13:48 -07:00
Tarek Dakhran 1052f7e1d6 chat : strip the think block in the qwen3-coder with reasoning_format none (#31380)
* chat : accept the think block in the qwen3-coder response format with reasoning_format none

Assisted-by: Claude Opus 5.5

* chat : define both qwen3-coder think block variants up front
2026-10-30 22:38:04 -05:00
Jesse Symons 8956c769e8 sycl: fix f32 to q2_0/q4_0/q5_0 rounding in CPY (#31120) 2026-10-30 21:54:41 -04:00
Adrien Gallouët 5047748f11 rpc : delete fs_path_to_utf8() (#31424)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-10-30 20:36:30 +01:00
shaofeiqiandLi He 792fcf4e75 opencl: add initial iq1 support (#31341)
* opencl: add iq1_s AOS mv

Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Assisted-by: Codex

* opencl: add iq1_s SOA mv and tiled mm

Assisted-by: Codex

* opencl: add iq1_s noshuffle gemm/gemv for adreno

Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Assisted-by: Codex

* opencl: add iq1_s dp4a for adreno

Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Assisted-by: Codex

* opencl: optimize iq1_s adreno decode and narrow batches

Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Assisted-by: Codex

* opencl: add iq1_m AOS mv

Assisted-by: Codex

* opencl: add iq1_m SOA mv and tiled mm

Assisted-by: Codex

* opencl: add iq1_m noshuffle gemm/gemv for adreno

Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Assisted-by: Codex

* opencl: add iq1_m dp4a for adreno

Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>

* opencl: optimize iq1_m adreno decode and narrow batches

Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>

---------

Co-authored-by: Li He <lih@qti.qualcomm.com>
2026-10-30 10:43:30 -07:00
Adrien Gallouët a34de8618b common : delete fs_path_to_utf8() (#31423)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-10-30 16:20:04 +01:00
Adrien Gallouët aa588a3463 common : drop the Windows argv reconstruction (#31417)
The manifest from #31329 already makes argv UTF-8.

Fixes #26416

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-10-30 14:02:00 +01:00
Tarek Dakhran 889f2ec96d quant : allow quantization for LFM3-Audio codec decoder (#31386)
* quantize : quantize the LFM3-Audio codec decoder, it is Q4_0 QAT

* quantize : drop the QAT note from the LFM3-Audio comment
2026-10-30 11:52:59 +01:00
Alessandro de Oliveira Faria (A.K.A.CABELO) 1ab23c33b7 vendor : update cpp-httplib to 0.64.0 (#31400) 2026-10-30 11:49:58 +01:00
Sigbjørn Skjæret 27b8464e82 ci : add missing ccache-clear to windows behavior check (#31413) 2026-10-30 11:34:25 +01:00
Adrien Gallouët d63fbfe794 cmake : embed an app manifest in all Windows exes (#31329)
* cmake : embed an app manifest in all Windows exes

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Remove LLAMA_WIN_APP_CMAKE

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

---------

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-10-30 10:56:53 +01:00
Daniel Bevenius e216f4ec87 ci : extract most apple jobs into daily workflow (#31408)
This commit extracts all apple jobs except for macos-latest-arm64 into a
new daily workflow.

The motivation for this is to reducd the load on the macos runners.
2026-10-30 10:01:28 +01:00
Lorenzo Bona 344846b4fc gemma3n, gemma4 : create token_embd before the tied output (#31394)
The duplicated output looks up token_embd in its buffer context, so
token_embd must exist first. With the old order a load where input and
output share a buffer type (CPU-only, -ngl 0) held the table twice:
gemma-4-E2B-it Q4_K_M, no mmap, CPU model buffer 3163.73 -> 2947.73 MiB.
2026-10-30 09:36:27 +01:00
Anton IandPascal fb5997274e kv-cache : keep rotation constants on the GPU (#31073)
* kv-cache : keep rotation constants on the GPU

Assisted-by: Codex

* tests : verify resident KV rotation buffers

Assisted-by: Codex

* kv-cache : document remaining multi-device rotation work

Assisted-by: Codex

* kv-cache : support per-layer rotation buffers

Assisted-by: Codex

* kv-cache : fix hybrid SWA rotation inputs and mixed placement check

Assisted-by: Codex

* models : use per-layer rotation in mistral-large4 and dflash

* tests: drop the rotation buffer tests

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-10-30 10:15:06 +02:00
Pascal ab3768cdc2 memory: refuse rollbacks past the snapshots of the last ubatch (#31368)
* memory: refuse rollbacks past the snapshots of the last ubatch

The recurrent memory and the DSV4 cache accepted any rollback up to
n_rs_seq, whether or not the last ubatch wrote that snapshot. Each now
records the deepest rollback its last ubatch guarantees and seq_rm
refuses anything past it:

- recurrent: n - 1 for a ubatch of n <= n_rs_seq + 1 tokens, 0 for a
  longer one, which only guarantees the final state
- DSV4: min(n, n_rs_seq), the snapshots are rebuilt from the state
  before the ubatch

The dead rollback block of the rollback test becomes the assertion it
asked for, and rollback_bound checks the exact bound of each memory
after ubatches of 1, 2, n_rs_seq + 1 and n_rs_seq + 2 tokens.

* tests: address review from ggerganov

Detect architectures with llama_arch_from_string in comp_padding and
arch_reserves_single_seq, as in rollback_bound.
2026-10-30 10:11:58 +02:00
shaofeiqiandLi He e0f142d5dd opencl: add initial q2_0 support (#31340)
* opencl: add q2_0 AOS mv

Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Assisted-by: Codex

* opencl: add q2_0 SOA mv and tiled mm

Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Assisted-by: Codex

* opencl: add q2_0 noshuffle gemm/gemv for adreno

Assisted-by: Codex

* opencl: add q2_0 dp4a for adreno

* opencl: optimize q2_0 adreno decode and narrow batches

---------

Co-authored-by: Li He <lih@qti.qualcomm.com>
2026-10-29 21:30:05 -07:00
Tarek Dakhran 3c809f4438 common : end reasoning budget forcing on an end tag from the prompt (#31374) 2026-10-29 19:27:50 -05:00
Michael Taylor 36a0e2ffc0 sycl: FLASH_ATTN_EXT failures (#30761) 2026-10-29 17:37:28 -04:00
Georgi Gerganov 418f905a4e ggml : add batched tensor copy API (#30881)
* ggml : add generic batched tensor copy API

Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)

* llama : use batched tensor copy API in context io

Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)

* ggml : add batched tensor copy API

Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)

* llama : batch device io tensor copies

Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)

* ggml : add CUDA support for batched tensor APIs

Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)

* ggml : remove async tensor_n APIs and add null buffer hooks

Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)

* ggml : fix batched tensor 2D detection

Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)

* ggml : group batched tensor operations by buffer type

Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)

* cont : minor indentantion

* cont : const correctness

* tests : add mock buffer type tests for batched tensor API

Restructure test-backend-tensor to run a unified test set on each backend buffer type and on an ad-hoc mock buffer type. The mock set exercises the dispatch logic in ggml_backend_tensor_{set,get,copy}_n: hook invocation, per-tensor fallback, cpy_tensor_n all-or-nothing, and buft grouping.

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)

* cont : add TODO
2026-10-29 20:52:17 +02:00
Slobodan JosicandMario Limonciello eb4e244be7 ROCm/HIP: add RDNA4_M family for gfx1170/gfx1171 (#31373)
gfx117X is more like RDNA4, with 128b WMMA. But they're not identical. Adjust the defines accordingly to have a new RDNA4_M and use it where it makes sense.


Assisted-by: Codex
Assisted-by: GitHub Copilot

Co-authored-by: Mario Limonciello <mario.limonciello@amd.com>
2026-10-29 18:23:29 +01:00
Oliver SimonsandSigbjørn Skjæret 68d20b884c Quant: OCP FP8 E4M3 support (#27512)
* WIP OCP FP8 E4M3 support

* Fix python type-checker

* Materialize input_scales also for FP8 when converting

* Fix conversion of https://huggingface.co/RedHatAI/Qwen3.8-27B-NVFP4

1. Per row FP8 wrongly fell to NVFP4 conversion, as NVFP4 checked on
   dims alone. Also check on dtype for FP8
2. Need to reshape FP8 QKV projections scales in the same way that
   weights are reshaped

* Allow remaining FP8 tensors to be dequantized during conversion

* Allow tensor-strategy for FP8-only compressed-tensor ckpts

* Fix and test gguf-py behavior for FP8_E4M3

* Disallow QKV-fusion for FP8 temporarily

* Fix conversion of fused qkvz & FP8 for Qwen3-Next

* Avoid fusing FP8 for --fuse-gate-up-exps as we may have different ws

* Emit warnings when not fusing FP8 during conversion

* More FP8 conversion fixes

* Add supported for strided copies to/from FP8 in CPU

* remove unneeded scratch buffer from FP8 cpu cpy/dup path

* add explanatory comments and TODOs

* Add dedicated encoder and decoder tests for FP8

* Whitelist ggml_op_get_rows in metal instead of blacklisting

required to fix failing CI, and white-listing should be more
future-proof than adding FP8 to the black-list

https://github.com/ggml-org/llama.cpp/actions/runs/38433916498/job/115422880633?pr=27512

* Fix more conversion bugs

* Defer weight materialization during conversion

* Blacklist fp8 in metal backend instead of white-list

* Adjust indents

* Fix lint + execute tests

* Add fast CPU kernel for F8_E4M3

* Conversion fixes

* Fix remaining gaps

* Enable tied weights with shared scales

* Make mandatory-F32 rules apply to FP8 during conversion

* Fix python type checks

https://github.com/ggml-org/llama.cpp/actions/runs/39765670052/job/119780379595?pr=27512

* Fix kaleidai unit-test failures

https://github.com/ggml-org/llama.cpp/actions/runs/39765670125/job/119780383150

* Relax numeric precision constraints for pre-check

* Leave fast-FP8 CPU on RISC-V for follow-up

We already exclude arm from the fast-path

* Allow output scale tensors to be located in any gguf shard

* Move test_conversion_fp8.py test execution to behavior-check.yaml

* Apply batched suggestions from code review

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Move mixed compressed-tensor logic into dequant_fp8 + add unit-tests for it

* Install all requirements also on Windows for behavior-check

Fixes https://github.com/ggml-org/llama.cpp/actions/runs/40535852590/job/122300758223?pr=27512

* Unmap lazy tensors before clean-up in Windows

Should fix https://github.com/ggml-org/llama.cpp/actions/runs/40539797824/job/122314782459?pr=27512

* LF line endings

https://github.com/ggml-org/llama.cpp/actions/runs/40539798168/job/122314791118?pr=27512

* Extract FP8 tests from llama-arch to its own test-fixture

See https://github.com/ggml-org/llama.cpp/pull/27512#discussion_r4400628221

* Update gguf-py/gguf/scripts/gguf_convert_endian.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-10-29 18:14:37 +01:00
Xuan-Son NguyenandGeorgi Gerganov 0893ec98f5 skills: specify rule about ggml_build_forward_expand (#31375)
* skills: specify rule about ggml_build_forward_expand

* Update skills/code-review/SKILL.md

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-10-29 15:44:12 +01:00
Xuan-Son Nguyen 78feec062d model: mistral le chonk (#31376)
* llama: extend DSA and sdpark to new model

* fix ci
2026-10-29 15:43:33 +01:00
Aman Gupta 0ac1a7fd35 llama : reuse the tensor staging buffer and skip the direct-io bounce buffer (#31321) 2026-10-29 19:22:43 +08:00
7konigsbergandZeshawn Zahid 7df8b876b9 server : restore the spec checkpoint for drafts longer than n_rs_seq (#31350)
Assisted-by: Claude

Co-authored-by: Zeshawn Zahid <zzshawnn@gmail.com>
2026-10-29 11:28:03 +02:00
Aleksander Grygier 8ab29faec4 ui: keep chats at the bottom until the reader scrolls up (#31311)
* ui: land at the bottom of a conversation every time it opens

Opening a long conversation often stopped short of its end:

- the press grace counted from page load (pressedAt = 0), so a chat that
  landed within 400 ms of a refresh did not follow the rows realizing
  after the jump
- the click that opened a chat (sidebar row, tab) started the same grace,
  so navigating by click had the same problem
- any scroll event away from the bottom stopped following, including the
  ones the browser fires itself while the chat settles
- the chat screen outlives its routes, so going back to the last landed
  conversation through the new-chat screen skipped the landing entirely

scrollToBottom() now clears the grace, only an upward scroll away from the
bottom stops following, and every navigation lands anew. Navigations into
a chat also disable SvelteKit's scroll reset, which moved the previous
conversation to its top for a few frames.

Where overflow-anchor is supported, a 1px pin after the messages is the
only scroll anchor while following, so the browser keeps the bottom in
place itself; WebKit keeps the resize observer.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: let nested agentic content scrollers chain to the chat

Reasoning, shell output and code blocks used overscroll-behavior:
contain, so the wheel stopped dead at their edges instead of
scrolling the conversation.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: stop following the stream as soon as the reader scrolls up

A trackpad or smooth wheel scrolls in steps smaller than the at-bottom
threshold, so each step up was re-stuck and snapped back on the next
growth while a response streamed. Any upward move that is not content
shrinking now releases at once.

The attach effect also read the hook's own state, so every flip
re-attached and re-measured, which undid such a release.

Assisted-by: Claude (Opus 5.5 Medium)
2026-10-29 10:18:13 +02:00
Aleksander Grygier f5d4d75419 ui: show submit button when the chat input has content (#31324)
ChatFormActions gated the mic and stop buttons on a `canSubmit` prop that
ChatForm never passed, so it was always false and typing text did not swap
them for the submit button. Use `canSend` instead and also hide the
skip-reasoning button while there is content to send.

Assisted-by: Claude (Opus 5.5 Medium)
2026-10-29 10:15:54 +02:00
Aleksander Grygier 3ed915e309 ui: reveal the collapsed sidebar from its logo and toggle it there (#31307)
* ui: reveal the collapsed sidebar from its logo and toggle it there

The collapsed sidebar now shows only the logo; hovering it reveals the
icon strip until the pointer leaves. The expanded sidebar drops its
top-right collapse button and the logo collapses it on hover instead.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: fade the peeked sidebar in at full width and keep the toggle slide

Hovering the logo of the collapsed sidebar now fades in the fully open
sidebar, floating over the page at its full width, instead of resizing
it. Opening or closing with the toggle still resizes and slides: the
sidebar docks to strip width while it slides, then floats out of sight,
and it stays hidden under the pointer after a toggle close. The unused
icon-only strip is removed from the sidebar actions.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: stack the sidebar above the chat tabs on desktop

The peeked sidebar floats over the page, but the chat tab bar shared its
z-index and came later in the page, so the tabs were drawn on top of it.

Assisted-by: Claude (Opus 5.5 Medium)
2026-10-29 10:15:31 +02:00
Aleksander Grygier f8fcef2546 ui: add MCP group checkboxes in tools settings and fix indeterminate checkbox colors (#31303)
MCP server groups in the settings Tools tab now get a group checkbox like the
built-in, server and browser groups, toggling the server-scoped tools key.
Indeterminate checkboxes now use the checked state colors.

Assisted-by: Claude (Opus 5.5 Medium)
2026-10-29 10:15:09 +02:00
Aleksander Grygier b7cde589e4 ui: request persistent storage on startup (#31299)
Ask the browser to keep IndexedDB and localStorage data under storage
pressure, so conversations and settings are less likely to be evicted.

Assisted-by: Claude (Opus 5.5 Medium)
2026-10-29 10:14:53 +02:00
lhez 3dd0f60bb4 doc: update ops for opencl (#31347)
Assisted-by: Codex
2026-10-29 10:14:29 +02:00
uvos 78febbad9d HIP/CUDA: enable sparse attn on CDNA (#31259)
* HIP/CUDA: enable sparse attn on CDNA

* Use __ballot for __ballot_sync on old ROCm versions

* Ignore new large head fattn mma variant
2026-10-29 10:14:11 +02:00
Giuseppe Scrivano abb76ab380 model : Support Limite 1B - Violetto (#29433)
* gguf-py : add Limite arch

Assisted-by: Claude Opus 5.5
Signed-off-by: Giuseppe Scrivano <giuseppe@scrivano.org>

* convert : add Limite support

Assisted-by: Claude Opus 5.5
Signed-off-by: Giuseppe Scrivano <giuseppe@scrivano.org>

* llama : add Limite support

Assisted-by: Claude Opus 5.5
Signed-off-by: Giuseppe Scrivano <giuseppe@scrivano.org>

---------

Signed-off-by: Giuseppe Scrivano <giuseppe@scrivano.org>
2026-10-29 10:13:41 +02:00
Keyu Zhu 41be0bdc98 common: select A100 cores for SpaceMiT default threads (#30284)
* common: select A100 cores for SpaceMiT default threads

# Conflicts:
#	tests/CMakeLists.txt

* ggml: export SpaceMiT build marker

* common: scope SpacemiT thread detection macro to llama-common

* common: detect A100 cores in all Linux RISC-V builds

* tests: link barrier test against llama-common

* tests: drop SpacemiT default thread test

* common: follow platform check order
2026-10-29 10:12:42 +02:00
lhezandHongqiang Wang 22183e51f9 opencl: initial support for iq2 (#31267)
* opencl: add iq2_s AOS mv

* opencl: add iq2_s SOA mv and tiled mm

* opencl: add iq2_s noshuffle gemm/gemv for adreno

* opencl: add iq2_s dp4a for adreno

* opencl: optimize iq2_s adreno decode and narrow batches

Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>

* opencl: add iq2_xs AOS mv

* opencl: add iq2_xs SOA mv and tiled mm

* opencl: add iq2_xs noshuffle gemm/gemv for adreno

* opencl: add iq2_xs dp4a for adreno

* opencl: optimize iq2_xs adreno decode and narrow batches

Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>

* opencl: add iq2_xxs AOS mv

* opencl: add iq2_xxs SOA mv and tiled mm

* opencl: add iq2_xxs noshuffle gemm/gemv for adreno

* opencl: add iq2_xxs dp4a for adreno

* opencl: optimize iq2_xxs adreno decode and narrow batches

Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>

---------

Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
2026-10-29 10:12:11 +02:00
Aman Gupta 6d194531e1 metal : faster MXFP4 mat-vec, full threadgroups for unary/binary/GLU (#31075)
* metal : faster MXFP4 mat-vec kernel

Decode the fp4 values with the fp16 bit trick instead of a threadgroup lookup table, and load the packed nibbles as aligned 32-bit words. The kernel no longer uses threadgroup memory.

* metal : launch contiguous unary and binary ops with full threadgroups

The contiguous unary path and the row-broadcast binary path used one thread per threadgroup. Use up to 256 threads per threadgroup and bound-check the tail.

* metal : split GLU rows across threadgroups

A GLU row ran on one threadgroup, so single-row decode used a single threadgroup for the whole op. Split each row over threadgroups of up to 256 threads.
2026-10-29 10:11:29 +02:00
Jesse Symons fade717ed8 tests : remove duplicate FA test cases added in #31125 (#31357)
Requested in the #31125 review. The cases duplicate coverage that is
already in the file and add CI time. Test-only, nothing changes in the
kernel, the docs or the CMake option.

Assisted-by: Claude
2026-10-29 10:10:25 +02:00
Georgi Gerganov c9daee149d ggml-metal : fix unary kernels compile on older Metal toolchains (#31360)
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
2026-10-29 10:09:51 +02:00
Max Krasnyansky 8a199ad31a hexagon: support for UPSCALE op (#31331)
* hex-upscale: basic upscale ops (not fully vectorized)

* hex-upscale: optimize nearest 2x, 4x and bilinear 4x

* hex-upscale: add support for non-contig rows and dims collapsing

* hex-upscale: use vgather instead of scalar staging

* hex-upscale: move all kernel select logic to the host, review comments and cleanups

* tests: add permuted UPSCALE test to cover qwen3-vl image->text scenarios
2026-10-29 00:22:34 -07:00
Trivikram ReddyandMax Krasnyansky da6c30ade8 hexagon: improve DDR bw utilization in transposed CONCAT (#31277)
* hex-concat: double row block size for transposed CONCAT

* hex-concat: double buffer scratchpads

* hex-concat: chunked src1 rows in transposed concat

* hex-concat: pad src1 VTCM stride to spread gather lanes

* hex-vgather: apply similar vgather opts to SSM_CONV and GDN

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-10-28 23:57:18 -07:00
Jesse Symons f7c01c5226 [SYCL] add DPAS flash attention kernel for quantized KV cache (#31125)
* sycl : add DPAS flash attention kernel for quantized KV cache

Decode and small batches (1 to 31 query tokens) now run flash attention
straight on q4_0 and q8_0 KV blocks, instead of scalar dequant or
converting the whole cache to f16 per call. Q.K uses s8 x s4 and s8 x s8
DPAS on XMX, V.P stays f16. dpas.s4.s8 is inline vISA since joint_matrix
has no s4.

Covers head sizes 64 to 512, GQA up to 32 and attention sinks. Prompts
stay on the existing kernels. GGML_SYCL_FA_DPAS=0 turns it off.

B580, Llama 3.1 8B Q4_K_M, q4_0 KV, tg128: 31.3 to 54.2 t/s at 16K
context. FLASH_ATTN_EXT 4582/4584, same 2 failures as master.

Assisted-by: Claude

* sycl : 192 head dropped, Wildcat Lake added to docs

Assisted-by: Claude

* sycl : stricter checks from review

Assisted-by: Claude

* [SYCL] : rename XMX macro, add GGML_SYCL_SUPPORT_XMX option, document DPAS

Assisted-by: Claude
2026-10-29 14:01:21 +08:00
Anant Shrivastava bf48103ad1 sycl : vectorize reordered-layout dequantization stores (Q4_0, K-quants, IQ3, MXFP4) (#30976)
* sycl : widen reordered q4_0 dequantization stores

The reordered-layout Q4_0 conversion kernel processed one qs byte per
work-item with two scalar stores, strided 16 elements apart. Give it
the same treatment the Q8_0 kernel got in #30775: a flat
sycl::range<1> launch with four qs bytes (8 outputs) per work-item
inside one 32-element block and two vec<dst_t,4> stores.

Same float math and rounding as the scalar kernel; output is
bit-identical. The existing k % QK4_0 == 0 assert guarantees whole
32-element blocks, so the k/8 launch covers every row exactly. The
now-unreferenced scalar reorder kernel is removed.

Arc Pro B70, primed prefill (prefill after decode on a
reorder-installed tensor): +6.4% to +7.3% pp512/pp2048 on Q4_0
models (tested up to 27B dense); fresh-layout prefill and decode
unchanged; PPL within run noise.

* sycl : vectorize reordered q4_K dequantization stores

The reordered-layout Q4_K conversion kernel already loads its
quantized inputs through sycl::vec but writes the dequantized
results with scalar stores. Build the results in a vec<dst_t,n>
register and emit two vector stores instead. Same per-element
math; output is bit-identical.

Arc Pro B70, primed prefill (prefill after decode on a
reorder-installed tensor): +14.7% to +15.3% pp512/pp2048 on
Q4_K-dominant models (tested up to 27B dense); fresh-layout
prefill, decode and PPL unchanged.

* sycl : remap reordered q5_K dequantization to vec4 loads/stores

The reordered-layout Q5_K conversion kernel gave each thread only
four outputs fed by scalar ql/qh byte loads plus a local-memory
scale broadcast, so the stores were never its bottleneck: widening
the stores alone measured within noise on mixed-quant models and
+0.5% on a q5_K-dominant model. Re-decompose to 32 threads per
block, each consuming four consecutive ql and qh bytes and emitting
two aligned vec<dst_t,4> stores; a 4-element run shares one scale
pair, so the per-element math and the output are identical.

Arc Pro B70, primed prefill on a q5_K-dominant 27B model: +3.6%
pp512/pp2048 over the scalar kernel; fresh-layout prefill, decode
and PPL unchanged. The GET_ROWS reorder path moves to the same
32-thread decomposition.

* sycl : remap reordered q6_K dequantization to vec4 stores

The reordered-layout Q6_K conversion kernel wrote four scalars per
thread, strided 32 elements apart, each pair of neighbors sharing
neither instruction stream nor store transaction. Re-decompose the
same 64-thread work-group so each thread emits one aligned
vec<dst_t,4>: a 4-element run shares one 16-element scale group,
one ql nibble source and one qh bit pair, so the per-element math
and the output are identical to the scalar kernel.

Arc Pro B70, primed prefill: +28.7% pp512/pp2048 on a Q6_K 27B
model; fresh-layout prefill, decode and PPL unchanged.

* sycl : vectorize reordered q3_K dequantization stores

Each thread of the reordered-layout Q3_K conversion kernel already
computes four consecutive outputs; build them in a vec<dst_t,4>
register and emit one vector store instead of four scalars. Same
per-element math; output is bit-identical.

Arc Pro B70, primed prefill: +16.8% pp512/pp2048 on a Q3_K-dominant
27B model; fresh-layout prefill, decode and PPL unchanged.

* sycl : vectorize reordered iq3_xxs/iq3_s dequantization stores

Both reordered-layout IQ3 conversion kernels write eight contiguous
outputs per thread as scalar stores (four from each grid lookup).
Build two vec<dst_t,4> registers and emit two vector stores. Same
per-element math; output is bit-identical.

Arc Pro B70, primed prefill: +20.3% to +21.1% pp512/pp2048 on
IQ3_XXS- and IQ3_S-dominant 27B models; fresh-layout prefill,
decode and PPL unchanged.

* sycl : remap reordered q2_K dequantization to vec4 stores

Re-decompose the reordered-layout Q2_K conversion kernel the same
way as Q6_K: one aligned vec<dst_t,4> per thread instead of four
scalars strided 32 apart. A 4-element run shares one scale byte and
one 2-bit lane, so the per-element math and the output are
identical to the scalar kernel.

Arc Pro B70, primed prefill: +13.1% pp512/pp2048 on a Q2_K-dominant
27B model; fresh-layout prefill, decode and PPL unchanged.

* sycl : widen reordered mxfp4 dequantization stores

Give the reordered-layout MXFP4 conversion kernel the same flat
sycl::range<1> launch as Q4_0: four qs bytes (8 outputs) per
work-item inside one 32-element block and two vec<dst_t,4> stores.
Same LUT lookups and per-block scale; output is bit-identical. The
k % QK_MXFP4 == 0 assert guarantees whole blocks, so the k/8 launch
covers every row exactly. The old scalar kernel is removed.

The reordered layout for MXFP4 is installed on the MoE expert path
(reorder_qw_mxfp4_moe); dense MXFP4 tensors keep the normal layout.

Arc Pro B70, MXFP4 MoE 20B model: +57% pp2048 / +61% pp512 prefill
(primed and fresh alike), decode unchanged, PPL within the model's
run-to-run noise.
2026-10-29 13:57:53 +08:00
Yonghye Kwon 6d5ac3c8d5 tests : fix comp_padding CUDA graph reservation (#31344)
Assisted-by: Codex
2026-10-29 07:04:28 +02:00
Ravi Panchumarthy 7ac5727d60 ggml-openvino: slice view inputs of SCALE (#31334)
SCALE now accepts a src0 view with arbitrary row strides. translate_scale
used the raw input, and on the dynamic path a VIEW translates to its whole
source tensor, so a strided view input scaled the entire source. The model
output then had the source's size, and copying it back overran the result
buffer, which crashes test-backend-ops on the SCALE view cases.

Slice the input with process_view_input_new, as the other translators do.
Reject an in-place SCALE on a non-contiguous view: its result is copied
back densely and would land in the wrong elements.
2026-10-28 21:11:13 -07:00
Marco ColomboandMax Krasnyansky 2e0d86bebc Hexagon: IM2COL improvement - block based HVX parallelization (#31050)
* hexagon: IM2COL over multiple threads with output-col blocks

Assisted-by: Claude Opus 5.5

* hex-im2col: fix padding handling

* hex-im2col: loop unroll common cases

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-10-28 19:04:18 -07:00
Pascal e09a78a25d tests: fix CI flakes in server cancel and sleep tests (#31330)
test_cancel_request enables context shift so the generation only ends
when the request is cancelled, then polls /slots with a deadline.
Without context shift the generation fills the context and frees the
slot on its own on fast machines, so the test passes even with
cancellation disabled.

test_server_sleep uses the existing wait_for_sleep helper instead of a
fixed sleep.
2026-10-29 01:35:01 +01:00
Nikhil Sharma 369dcf309c ggml-cpu: add SVE implementation of ggml_gemv_q8_0_4x8_q8_0 (#30585) 2026-10-29 08:21:05 +08:00
Abhijith Jajam d0ff6317cc docs : fix DSpark confidence threshold flag name (--spec-draft-p-min) (#31178) 2026-10-29 08:17:38 +08:00
Pascal cf24317bea common, mtmd: bound control vector layer indices and M-RoPE position loops (#31336)
* common: bound control vector layer indices by the model layer count

A direction tensor layer index is parsed from the tensor name with no
upper bound, and the buffer size and write offset are computed in int.
A large index overflows both, so the accumulation writes before the
start of the buffer.

Pass the model layer count to the loader, reject indices above it like
the existing zero index check, and compute the sizes in size_t.

* mtmd: fill M-RoPE positions by whole merge tiles

The position loops of the Qwen-VL, GLM, PaddleOCR and Qwen2.5-VL
encoders hardcode a 2x2 merge inside a stride of n_merge, so a patch
grid that is not a multiple of the merge writes past the positions
buffer. Iterate over whole merge tiles with n_merge inner steps, which
bounds the writes by the patch count and gives the same positions on
aligned grids.

PaddleOCR also requires a merge aligned image at graph build, as the
Qwen-VL family already does.
2026-10-28 23:15:01 +01:00
Hongqiang WangandLi He 74a4fb0037 opencl: add tiled q4_k GEMM for adreno (#30412)
* opencl: run the tiled q4_K lm_head through a tiled GEMM at batch > 1

* opencl: split tiled dp4a q4_k gemm into its own file and test for dp4a

Assisted-by: Codex

---------

Co-authored-by: Li He <lih@qti.qualcomm.com>
2026-10-28 14:55:28 -07:00
5220213255 hexagon: add conv2d support (#30995)
* hexagon: add conv2d op

* hex-conv2d: add double-buffered staging

* hex-conv2d: full 64-wide activation block transpose straight from staged buffer

* hex-conv2d: implicit gemm

* hex-conv2d: remove scalar path and kernel

* hex-conv2d: unify conv2d kernel params for all paths

* hex-conv2d: scale HMX result before f16 storage and then restore during unpacking to prevent overflow

* hex-conv2d: fix f16 overflow in HMX paths

* hex-conv2d: refactor to rebase to new op fusion machinery

* hex-conv2d: restructure DMA pipelines and trace scopes

* hex-conv2d: removed the HVX_WIDTH and HVX_TILED paths

* hex-conv2d: remove in-loop vector spills

* hex-conv2d: refactor kparams, vtcm-layout and dma-pipeline to match the other ops

* hex-conv2d: add basic support for CONV_2D_DW

Co-Authored-By: Pranav Dhinakar <pdhinaka@qti.qualcomm.com>

* hex-conv_2d_dw: use DMA in all paths

* hex-conv_2d_dw: support for chunking larger inputs

* hex-conv_2d_dw: remove scalar/hvx access for w tensor (replaced with sync dma copy)

* hex-conv-2d: fix warnings

* hex-conv2d-dw: vectorize all paths

* hex-conv-2d-dw: tree-based reductions and unrolling to optimize

* hex-conv-2d-dw: uninline compute loops from dma loops

* hex-conv-2d-dw: move all kernel selection into the host solver and simplify/specialize kernel dispatch

* hex-softmax: update obsolete gates to enable SOFT_MAX for mmproj

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
Co-authored-by: Pranav Dhinakar <pdhinaka@qti.qualcomm.com>
2026-10-28 13:43:40 -07:00
Bartowski c3df4b7e22 Speed up IQ4 quant creation with pre-computed halfway points (#31201) 2026-10-28 14:35:56 -04:00
anubhavfujitsu ac8e1694d8 ggml-cpu: SVE 256 bit implementation of gemm_q6_K_8x8_q8_K kernel (#27472)
* q6_k GEMM working

* cleanup

* review comments

* review comment addressed - removed pg16_8

* incorporated the review comments - replaced switch with if and other points

* marking q8_k_blocklen UNUSEDto disable warning for non-i8mm cases
2026-10-29 00:01:37 +08:00
abhijain1204fujitsuandAnubhav Srivastava c02b5c641e ggml-cpu: SVE 256-bit Implementation of mxfp4 gemv kernel (#28141)
* mxfp4 sve 256 impl completed

* working with pretty good perf improvement

* review comments addressed

* review comments for SVE 256 check

* trailing white spaces removed

---------

Co-authored-by: Anubhav Srivastava <anubhav.srivastava@fujitsu.com>
2026-10-28 23:58:20 +08:00
Aaron Teo ff0d396e3d ci(s390x): switch to Ubuntu 26.04 runners (#31317) 2026-10-28 23:32:38 +08:00
Yonghye Kwon 838093f192 llama : remove simple batch splitting for DSV4 (#31300)
* llama : fix DSV4 compressor padding for simple batches

Count actual tokens per sequence when padding compressor writes. Add a graph reuse regression test for odd-sized DeepSeek V4 and V4.1 batches.

Assisted-by: Codex

* tests : cover multiple sequences in DSV4 padding regression

Exercise one and two sequences at different positions while keeping a fixed ubatch shape.

Assisted-by: Codex

* llama : remove simple batch splitting for DSV4

Use the existing sequence splitting to keep n_seq_tokens consistent with
the compressor padding calculation. Extend the graph reuse regression to
6-token batches.

Assisted-by: Codex
2026-10-28 23:25:47 +08:00
Georgi Gerganov 604cc43023 codeowners : refresh teams and maintainer PRs link (#31316)
* codeowners : refresh teams and maintainer PRs link

Update the CODEOWNERS team legend with the current `ggml-org` team
memberships, and extend the maintainer PRs link in the README to cover
all current members of the `maintainers` team.

Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)

* codeowners : list remaining backend sub-teams

Add the `ggml-*` sub-teams that were missing from the team legend at the
top of the file: base, cpu, et, hip, openvino and riscv. The legend now
covers every sub-team of the `ggml` and `llama` maintainer groups.

No ownership rules change - only the comment block is extended.

Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)

* codeowners : use ggml-hip team for HIP backend

Route `/ggml/src/ggml-hip/` to `ggml-org/ggml-hip` instead of a single
collaborator, so review requests reach the whole team.

Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)

* codeowners : add llama-examples team

List the newly created `llama-examples` team in the team legend at the
top of the file.

Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)

* codeowners : use llama-examples team for examples

Collapse the per-directory rules that only listed a single collaborator
into a single `/examples/` rule owned by `ggml-org/llama-examples`.
Directories owned by other collaborators keep dedicated rules, since the
last matching pattern takes precedence in CODEOWNERS.

Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)

* cont : update zdnn

* codeowners : use llama-base team for src

Route `/src/` to `ggml-org/llama-base` and list the new team in the
legend at the top of the file. Rules for individual `src` files are kept,
since the last matching pattern takes precedence; `/src/models/` drops
its individual owner, which is already covered by `llama-models`.

Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
2026-10-28 17:00:54 +02:00
Frosty40 39febd2adb sycl: work-group argmax for short TOP_K rows (#31110)
* sycl: work-group argmax for short TOP_K rows

Used angelX to optimize SYCL/drama 3 coordination, results are as follows

Rows with ncols <= max work-group size: one work-item per column, k rounds of work-group argmax. Ties take the lower column, NaN is never picked. Longer rows keep the existing paths.

Arc Pro B70, oneAPI 2026.1, test-backend-ops perf (existing short-row cases), base 9b56ccf00 vs patch, median ratio of 6 rotating A/B/A rounds:

| ncols | nrows | k | base us | patch us | speedup |
|---|---|---|---|---|---|
| 16 | 1024 | 1 | 19.97 | 2.88 | 6.95x |
| 16 | 1024 | 2 | 23.83 | 2.90 | 8.25x |
| 16 | 1024 | 8 | 66.12 | 4.66 | 14.21x |
| 16 | 8192 | 1 | 110.42 | 5.96 | 18.51x |
| 16 | 8192 | 2 | 130.21 | 8.82 | 14.77x |
| 16 | 8192 | 8 | 420.62 | 27.27 | 15.42x |
| 128 | 1024 | 1 | 20.49 | 2.94 | 6.97x |
| 128 | 1024 | 2 | 25.89 | 3.45 | 7.52x |
| 128 | 1024 | 8 | 101.67 | 8.95 | 11.36x |
| 128 | 8192 | 1 | 112.19 | 13.29 | 8.44x |
| 128 | 8192 | 2 | 141.18 | 20.60 | 6.85x |
| 128 | 8192 | 8 | 643.21 | 67.60 | 9.51x |
| 1024 | 1024 | 1 | 17.32 | 14.98 | 1.16x |
| 1024 | 1024 | 2 | 28.77 | 24.02 | 1.20x |
| 1024 | 1024 | 8 | 151.87 | 77.88 | 1.95x |
| 1024 | 8192 | 1 | 156.49 | 138.78 | 1.13x |
| 1024 | 8192 | 2 | 201.47 | 209.18 | 0.96x |
| 1024 | 8192 | 8 | 1006.25 | 632.60 | 1.59x |

The 1024 x 8192, k=2 case is 3.7% slower.

Correctness: test-backend-ops -o TOP_K 533/533 and -o ARGSORT 98/98 on the patch, again on the rebase onto 3f03b1772.

Full test-backend-ops aborts on base and patch alike at the non-F32 SWIGLU_OAI assert (element_wise.cpp:1187, see #30857). With only those cases excluded: 20145/20173 on base and patch. One CPY f16->q2_0 [256,2,3,4] failure was seen once on the patch, and 3/3 re-runs pass on base and patch. All other patch-run failures also fail on base.

Assisted-by: Claude Opus 5.5

* made changes as suggested

Assisted-by: Claude Opus 5.5
2026-10-28 22:18:07 +08:00
anujj f5222f25c1 cuda: fuse MoE (MUL_MAT_ID) gate/up and GLU in MMQ (#31188) 2026-10-28 18:34:28 +05:30
Xuan-Son NguyenandSigbjørn Skjæret 28cb47c7c8 gguf-py: fix np_roundf dropping the sign (#31310)
* gguf-py: fix np_roundf dropping the sign

* Update requirements/requirements-convert_legacy_llama.txt

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* requirements.txt

* Update tools/server/tests/requirements.txt

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-10-28 14:03:59 +01:00
Georgi Gerganov 8642541b25 llama.cpp : bump version to 0.8.0 (#31309)
* llama.cpp : bump version to 0.8.0

* authors : update
2026-10-28 13:30:38 +02:00
Tarek DakhranandXuan Son Nguyen d4f1b18b3a mtmd: enable video input support for LFM3 model family (#31283)
* mtmd : add LFM3-VL and LFM3-Omni video input

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Assisted-by: Claude Opus 5.5

* mtmd : reserve the soundtrack command to avoid a GCC 11 warning

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-10-28 12:23:06 +01:00
Tarek DakhranandXuan Son Nguyen d1eb0debad server : support tool calls in /v1/realtime (#31287)
* server : add tool calls to /v1/realtime

Assisted-by: Claude Opus 5.5

* refactor

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-10-28 12:17:13 +01:00
Georgi Gerganov 5d7793d6fc sync : ggml 2026-10-28 12:36:51 +02:00
Georgi Gerganov ec941ee852 ggml : bump version to v.28.0 (ggml/1676)
* ggml : bump version to 0.28.0

* authors : update
2026-10-28 12:36:51 +02:00
Pascal e486a4b559 ui: build the reasoning menu from the template effort levels (#31273)
* ui: build the reasoning menu from the template effort levels

When the chat template reports effort levels, the menu offers Default,
Off and those levels, and a pick is sent as reasoning_effort. A
template that reports none keeps the token budget menu.

Picking a model other than the one that produced the last answer resets
the conversation effort to Default, and an effort the current menu
lacks resolves to Default.

* ui: wait for a cold router model before building the request

A router serves the props of a loaded model only, so a request sent to
a cold model was built without its template effort levels and fell
back to a token budget. The request now waits for the model to load,
joining a load already in flight, and a stop during the wait sends
nothing.

Status waiters hold a list per model so a load and the requests that
wait on it each get theirs, and a props request in flight is shared
with a second caller instead of answering null.
2026-10-28 11:35:57 +02:00
Georgi Gerganov f1a7dea99a tests : adjust constants to reduce irrelevant warnings (#31254) 2026-10-28 11:26:38 +02:00
Yonghye Kwon a7af149a2a cuda : add top-k MoE fusion for 384 experts (#31288)
Assisted-by: Codex
2026-10-28 10:13:01 +02:00
Kris 2864937a77 vulkan: clamp A prefetch to end_k in coopmat1 int8 matmul (#30283)
* fix: clamp the A prefetch to end_k in the Vulkan coopmat1 int8 matmul, like the B prefetch

* fix: zero the A scale for k steps past end_k in the Vulkan coopmat1 int8 matmul instead of relying on the clamped block
2026-10-28 10:12:22 +02:00
a71c00f0bb chat : stop qwen3-coder reasoning and content before template separators (#30557)
* chat : strip trailing whitespace from parsed reasoning and content

Assisted-by: Claude Fable 5.1

* chat : stop qwen3-coder reasoning and content before template separators

Assisted-by: Claude

* tests : remove the Qwen parser case from the chat session tests

Assisted-by: Claude

* common : handle partial prefix until match at end of generation

---------

Co-authored-by: Zeshawn Zahid <zzshawnn@gmail.com>
Co-authored-by: Alde Rojas <hello@alde.dev>
2026-10-28 10:11:47 +02:00
Aleksander Grygier f4d8af608a ui : show save and reset settings buttons only when relevant (#31248)
* ui : show save and reset settings buttons only when relevant

Assisted-by: Claude (Opus 5.5 Medium)

* ui : move reset settings button next to reload app, pad settings caption

Assisted-by: Claude (Opus 5.5 Medium)
2026-10-28 08:54:34 +02:00
Aleksander Grygier 02d502f314 ui: Support OGG audio attachments by converting them to WAV (#31237)
* ui: support OGG audio attachments by converting them to WAV

Fixes #24164

Assisted-by: Claude (Opus 5.5 Medium)

* ui: convert all non WAV/MP3 audio uploads to WAV

Assisted-by: Claude (Opus 5.5 Medium)
2026-10-28 08:54:15 +02:00
Aleksander Grygier b1756b23f3 ui: Fix draft for new message disappearing after using "Add System Message" action (#31214)
* ui: keep new chat draft when adding a system prompt

Fixes #24338

Assisted-by: Claude (Opus 5.5 Medium)

* ui: remove unused initialMessage from chat form

Assisted-by: Claude (Opus 5.5 Medium)
2026-10-28 08:53:56 +02:00
Aleksander Grygier 6886628cb1 ui : Search dialog (#31173)
* ui : replace the sidebar search mode with a search dialog

Search opens a spotlight dialog (Cmd+K or the Search entry under New chat in the sidebar) with keyboard navigation over recent chats and sidebar actions. Phones keep the search page.

Sidebar actions show their shortcuts, with new ones for Settings (Cmd+,) and Models (Shift+Cmd+M).

Assisted-by: Claude (Opus 5.5 Medium)

* ui : ignore search dialog keys while an IME is composing

Enter confirms an IME candidate and the arrow keys move through the candidates, so the search field leaves them to the IME instead of opening the highlighted result or moving the highlight.

Assisted-by: Claude (Opus 5.5 Medium)
2026-10-28 08:53:41 +02:00
Tarek DakhranandXuan Son Nguyen 8ca4060b7e server : add realtime conversation and transcription on /v1/realtime (#31269)
* server : add realtime conversation and transcription on /v1/realtime

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Assisted-by: Claude Opus 5.5

* examples : default the realtime client to the server port 9931

Assisted-by: Claude Opus 5.5

* server : sample realtime responses right after their own decode

Slots respond in turns on one context: sampling at the start of the next step read the logits of whichever slot decoded last, so concurrent sessions mixed their outputs.

Assisted-by: Claude Opus 5.5

* move more stuff to server-realtime, making it self-contained

* examples : fix the type check of the realtime client

Assisted-by: Claude Opus 5.5

* examples : use the model card prompts in the realtime client

Assisted-by: Claude Opus 5.5

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-10-28 01:26:12 +01:00
Tarek DakhranandXuan Son Nguyen d05706b4f3 model : add LiquidAI/d1-2B and LiquidAI/d1-omni-2B (#31275)
* model : add LiquidAI/d1-2B and LiquidAI/d1-omni-2B

Assisted-by: Claude Opus 5.5

* convert : keep the speech task templates out of the d1 GGUFs

Assisted-by: Claude Opus 5.5

* small refactoring: no_batching

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-10-27 23:36:32 +01:00
Pascal 1b4554eff1 server: expose the reasoning effort levels of the chat template (#31271)
* server: expose the reasoning effort levels of the chat template

The capability probe renders a user turn for each candidate level
(minimal, low, medium, high, xhigh, max) and for an unknown level. A
level is honored when it renders without raising and differently from
the unknown level, which shows the template fallback. Levels sharing a
render are aliases and the highest one is kept.

The caps serialize to JSON, and chat_template_caps in /props reports
the list as reasoning_effort_levels, from lowest to highest.

* jinja: keep the caps free of the json dependency

The caps keep reporting their flags as a map, and the effort levels
stay a plain member. common_chat_templates_get_caps() builds the JSON
the server reports from both, so jinja depends on json only for the
probe inputs.
2026-10-27 22:39:37 +01:00
Pascal a919179942 tests: fix CI riscv timeout in test-llama-context (#31265)
In --models mode the thread count per model is computed and printed,
but never set in the params, so every context runs on the common
default, the number of physical cores. With several models in
parallel this oversubscribes the CPU, and the busy-wait barriers of
the CPU backend make it collapse.

Set the count in cpuparams and cpuparams_batch.
2026-10-27 22:18:31 +01:00
uvos 338fce6511 HIP/CUDA: remove pragma unroll from loops in conv2d where itteration count cant be infered at compile time (#31255) 2026-10-27 19:42:04 +01:00
Tarek DakhranandXuan Son Nguyen 19ff817fb2 model : add LiquidAI/LFM3-Audio-2B and LiquidAI/LFM3-Omni-2B (#31257)
* model : add LiquidAI/LFM3-Audio-2B and LiquidAI/LFM3-Omni-2B

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Assisted-by: Claude Opus 5.5

* convert : read the LFM3-Audio config.json as released

A transformers that knows lfm3_audio rewrites the config into another layout (no lfm/encoder keys).

Assisted-by: Claude Opus 5.5

* mtmd : disable flash attention for the LFM3-Audio depthformer

Assisted-by: Claude Opus 5.5

* tts : add a voice option for LFM3-Audio and LFM3-Omni

The model card offers a female and a male voice; --tts-voice for llama-tts, "voice" for the server /tts.

Assisted-by: Claude Opus 5.5

* clean up template handling

* convert : detect LFM3-Audio and LFM3-Omni by model_type

Assisted-by: Claude Opus 5.5

* move banned tokens --> logits bias --> mtmd_helper

* leave mtmd-cli unchanged

* no more template_asr_system

* left over

* llama : keep the LFM3-Audio encoder convs and codec unquantized

Assisted-by: Claude Opus 5.5

* common, mtmd : shorten comments

Assisted-by: Claude Opus 5.5

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-10-27 18:47:14 +01:00
Tarek DakhranandAldehir Rojas eb7818c0b8 common : support deferred tools and tool_reference content parts (#31256)
* common : support deferred tools and tool_reference content parts

Assisted-by: Claude Opus 5.5

* common : credit the deferred tools co-author

Co-authored-by: Aldehir Rojas <765227+aldehir@users.noreply.github.com>

Assisted-by: Claude Opus 5.5

* tests : fix the type check of the deferred tools test

Assisted-by: Claude Opus 5.5
2026-10-27 18:39:11 +01:00
Mengsheng Wu 6c44591049 snapdragon: switch QDC artifact upload and log download to v2 API (#31240)
Signed-off-by: Mengsheng Wu <mengshengwu@outlook.com>
2026-10-27 18:06:41 +01:00
19cce69199 model : add LiquidAI/LFM3-1.7B and LiquidAI/LFM3-VL-2B (#31252)
* model : add LiquidAI/LFM3-1.7B and LiquidAI/LFM3-VL-2B

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Xuan-Son Nguyen <son@huggingface.co>
Assisted-by: Claude Opus 5.5

* tests : set the SWA rope base of the LFM3 test model

Assisted-by: Claude Opus 5.5

* llama : enable recurrent state rollback for LFM3

Assisted-by: Claude Opus 5.5

* Apply batched suggestions from code review

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Tarek Dakhran <github@dakhran.com>

* gguf-py : align the lfm3vl mapping comments

Assisted-by: Claude Opus 5.5

* llama : fix CRLF line endings from the review suggestions

Assisted-by: Claude Opus 5.5

* tests : add the LFM3 Metal fusion baseline

Assisted-by: Claude Opus 5.5

* llama : keep the LFM3-VL depthwise conv kernels in F32 when quantizing

Assisted-by: Claude Opus 5.5

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Xuan-Son Nguyen <son@huggingface.co>
2026-10-27 17:06:40 +01:00
Mendy Berger e86f5fd1e9 cmake: WASI doesn't support spawning (#30751) 2026-10-27 15:14:03 +01:00
Aman GuptaandGeorgi Gerganov 27b7a8a009 llama : build the k-pool sparse selection independently of the kq_mask shape (#31243)
* llama : build the k-pool sparse selection independently of the kq_mask shape

qwen4exp (QSA) and glm5-next pick cache cells by index and no longer read the kq_mask values, but they still shaped the selection like the dense kq_mask. Build the selection as [n_kv, n_tokens] over the cache cells instead, and split it into the streams of the K view right before the attention.

This lets the dense mask change its layout (e.g. per-sequence row lists) without touching the sparse layers.

* improve comment

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* llama : drop the unused build_fused_top_k_mask args, review nits

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-10-27 21:57:45 +08:00
Katara 7ecb57f7ad rpc : catch std::length_error in recv_msg (#31204) 2026-10-27 15:04:55 +02:00
Georgi Gerganov 8542e94350 tests : speed up tensor data generation and model saving (#31145)
* tests : speed up tensor data generation and model saving

Refactor save_models to create only the model, not a llama_context,
which is not needed for saving. Replace the slow std::normal_distribution
draw with a splitmix64 + box-muller normal generator.

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)

* tests : use a shared pool of normal draws for tensor data

Generate a pool of normal draws once with std::mt19937 and
std::normal_distribution, then have every tensor copy a cyclic slice of
it instead of running a per-tensor box-muller. The name-keyed slice
selection keeps the draw deterministic.

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)

* tests : log the size of saved models

Report the produced file size in bytes and MB after each save.

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)

* tests : simplify the model save log

Put the model name, target and size on one line per saved model.

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
2026-10-27 14:44:41 +02:00
Pascal 922f53ed33 vulkan: fix CI llvmpipe by allowing dim 3 broadcast in MUL_MAT (#31239)
* vulkan: allow dim 3 broadcast in MUL_MAT

supports_op rejected every MUL_MAT whose src0 and src1 differ in ne3,
although mul_mm, mul_mmq and mul_mat_vec already broadcast src0 over
dim 3 through broadcast3. Such a matmul then ran on the CPU with a
copy of the weight, and the split layout changed with the number of
streams in the ubatch.

Keep the strict check for MUL_MAT_ID only, and route the broadcast
cases away from mul_mat_vec_nc, the one path that requires equal ne3.

* tests : harden MUL_MAT with permuted dim 3 broadcast cases

Harden the MUL_MAT coverage with a weight broadcast over dim 3 against
a permuted per-stream activation, the shape of an MLA output
projection with a non unified KV cache, for a single token, several
heads and several tokens per stream.
2026-10-27 14:39:03 +02:00
Sam Malayek 0c8c59ef3d mtmd-cli: Reduced scope of batch pointer (#31155) (#31165)
* mtmd-cli: Reduced scope of batch pointer (#31155)

* mtmd-cli: Remove CLI regression test
2026-10-27 12:56:03 +01:00
Aleksander Grygier 3ab37aedfe ui: fix system message row stretching to full prompt height in WebKit (#31246)
The expand button wrapping the system message card defaulted to
inline-block. WebKit takes an inline-block's baseline from its last
line even when that line is clipped by the card's max-height, so the
row grew to the height of the whole prompt and left a large empty gap
below the card. Make the button a block so the row matches the card.

Assisted-by: Claude (Opus 5.5 Medium)
2026-10-27 13:19:02 +02:00
85cc432e2c server: add router device memory margin parameter for dynamic unloading (#21231)
* server: add --models-memory-max parameter to allow dynamically unloading models when they exceed a memory size threshold

estimate with to-be-loaded model size included

use no_alloc to get memory requirements for model load

only set model memory_mb if not previously calculated

use memory margin instead of total size limit, apply to each device separately

add server memory debug logging

move llama_context_device_memory function to llama-ext.h

fix model count exceeded check

improve memory_per_device map naming

improve variable naming, fix style

also strip models memory margin from child processes

cont : clean-up

replace device memory map with buft memory map. Use llama_get_memory_breakdown

extract duplicated check into helper function

move model memory estimation to subprocess

precompute name->buft map, map GPU host types to CPU buft

cleanup unused variable

remove duplicated init calls

* adapt to upstream changes

* wip

* common_memory_reports internface

* SERVER_ROLE_MEASURE

* server-models-sched

* fix build

* server : fix memory accounting of measure children and claim spin

A measure child runs no_alloc, so the free memory it reports does not
include its estimate: subtracting self there erased the memory held by
other processes, down to zero with --fit on, and the next load could
overshoot the margin. Subtract self only for instance reports.

A queued load only checked --models-max before claiming, so when the
memory margin blocked it behind a busy model, load() threw and the
waiter retried in a tight loop. Refuse the claim while the limits are
exceeded and no idle model can be unloaded, so the waiter sleeps on
the condition variable until one goes idle.

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: Pascal <admin@serveurperso.com>
2026-10-27 12:14:42 +01:00
Julien Chaumond 74a1fd3d03 common : --model or -hf is required (#31242)
Assisted-by: Codex
2026-10-27 11:55:48 +01:00
tc-mbandXuan Son Nguyen 7eed6b7685 model: Add MiniCPM-OCR-Jev (#30980)
* model : add MiniCPM-OCR-Jev support

The model is a Qwen3.5-VL trunk with a value head read at the last token: it
scores an OCR text against an image rendered from it, it does not generate.

Add the openbmb-minicpm-ocr-jev decision type, served by POST /v1/systemone.
The class scores are the pooled output of the sequence, read with last pooling,
so the conversion turns score.weight into cls.output and sets
embedding_length_out. The checkpoint has no LM head, so text generation is
rejected instead of using the output weights duplicated from the token
embeddings.

The task is fixed, so its prompt is entirely in the systemone template, the
same way h2o-lightning keeps its default there. The processor of the checkpoint
allows up to 16384 merged tokens per image, more than the Qwen-VL default of
mtmd, so the mmproj keeps the image budget of the preprocessor config. The
checkpoint declares MiniCPMOCRForSequenceClassification in its config.json, as
other models do.

Signed-off-by: tc-mb <tianchi_cai@icloud.com>

* fix code

Signed-off-by: tc-mb <tianchi_cai@icloud.com>

* use gguf meta classifier_output_labels

* remove redudant code changes

---------

Signed-off-by: tc-mb <tianchi_cai@icloud.com>
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-10-27 11:49:12 +01:00
Georgi Gerganov 276378515c metal : add GGML_OP_TOP_K_MASK (#31230)
* metal : add GGML_OP_TOP_K_MASK

Port the fused top-k mask from the CUDA backend: a grid-stride kernel fills the output with -inf, then one
threadgroup per row copies the mask values of the kept cells over it. Function constants drop the score,
blocks and extra paths that a given graph does not use.

Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)

* metal : clean up top-k mask kernel formatting

Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
2026-10-27 12:13:23 +02:00
Alexey Kopytko d78f40d9d3 glm5-next : QSA without kq_mask (+ DSA) (#31009)
* glm5-next : QSA without kq_mask

* DSA: dsa top-k mask change tested with layers 0,3 of DeepSeek-V3.2
2026-10-27 18:54:36 +09:00
Sreeram fae9ac561f ci : use ccache depend mode for ROCm release builds (#31231) 2026-10-27 09:24:00 +01:00
Sigbjørn Skjæret a8276b70cc ci : force python 3.12 on ubuntu 26.04 (#31222) 2026-10-27 09:09:44 +01:00
Georgi GerganovandPascal de5122d879 ggml : relax scale to non-contiguous views (#31146)
* ggml : relax scale to non-contiguous views

SCALE requires src0 to be 1D-contiguous. Relax this to dim-0
contiguity with arbitrary row strides (ggml_is_contiguous_rows),
same as the unary ops in the Metal backend.

- ggml: relax the assert in ggml_scale_impl
- cpu: address src0 rows through their byte strides (f32, bf16)
- cuda: stride-aware kernel, fastdiv row decomposition
- metal: support bf16 scale (guarded unary instantiations)
- sycl: keep the gate at contiguous, the kernel is flat f32

Add view variants to test_scale. Verified on CPU, Metal and CUDA.

Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (RTX 5090 32GB, DDR5 128GB)

* cuda : skip the softcap fusion on non-contiguous views

The fused softcap kernel reads src and writes dst as dense arrays,
so a view reaching SCALE -> TANH -> SCALE now that SCALE accepts
non-contiguous rows would be read wrongly. Keep such graphs on the
strided kernels until the fusion becomes stride aware.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-10-27 07:55:31 +02:00
Georgi Gerganov 6666a7330c tests : fix multi-seq split replay and partial-state cases for trained models (#31216)
* tests : fix multi-seq split replay against trained models

The rollback snapshot is captured by the graph that decoded the rolled-back
tail, while the reference context decodes the same prefix with different
batch shapes. The two states differ in the last bits and trained models
amplify the difference with every replayed token. Compare only the first
replayed token of each seq, then restore the rollback state into the
reference context before the seq-1-only tail.

Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)

* tests : require bit-exact seq-1 isolation in split replay

The seq state save is not always bit-lossless - SWA-masked cells are not
saved - so reload the restored state back into the rollback context. That
way both contexts enter the seq-1-only tail with byte-identical memory,
including the cell layout, and the tail logits must match bit-exact. Also
assert that the restored state round-trips bit-identical.

Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)

* tests : drop the n_swa partial-state check from ps

The check keys off llama_model_n_swa(), which returns 0 for dsv4 even
though its raw KV cache has a 128-cell sliding window. The partial state
grows with the tokens there by design, so the flatness check is wrong.
Keep the partial <= full invariant.

Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
2026-10-27 07:54:51 +02:00
Georgi Gerganov 7a2e5aba90 llama : remove obsolete n_pad from memory instances (#31200)
Every caller passes n_pad = 1 and the only two users clamp it to 256.
Drop the parameter, the members and the trivially true kv_size assert.

Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
2026-10-27 07:53:44 +02:00
Aleksander Grygier 0d5ca60098 ui: list org-less models first in the model selector (#31215)
The org-less group renders without a header, so when it followed a named
org its models looked like they belonged to that org. Put it ahead of the
named org groups.

Fixes #25227

Assisted-by: Claude (Opus 5.5 Medium)
2026-10-27 07:52:54 +02:00
lhez d930fe3c26 opencl: refine mul mat id -1 index (#31170)
Assisted-by: Codex
2026-10-27 07:52:07 +02:00
lhezandHongqiang Wang a2b3e121ce opencl: initial support for q3_k (#31172)
* opencl: add q3_k AOS mv

* opencl: add q3_k SOA mv and tiled mm

* opencl: add q3_k noshuffle gemm/gemv for adreno

* opencl: add q3_k dp4a gemm for adreno

Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>

* opencl: tune q3_k kernels for adreno

Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>

---------

Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
2026-10-27 07:51:22 +02:00
Aleksander Grygier 823df7d0fb ui: Highlight reasoning toggle on hover and align it with model selector (#31212)
On desktop, the reasoning effort toggle now gets a muted background on
hover and while its menu is open, matches the model selector's 24px
height, uses a 13px icon and sits 5px from the selector.

Assisted-by: Claude (Opus 5.5 Medium)
2026-10-27 07:50:15 +02:00
PascalandAleksander Grygier 1a97e387d3 ui: add conversation compaction (#27401)
* ui: add conversation compaction

Add a compaction action that asks the model to summarize the
conversation so far and stores the summary in a new compaction node
appended to the branch. Later requests slice the branch at the last
compaction node, keeping the system prefix and continuing from the
summary instead of the full history, which frees up context while the
complete history stays visible and navigable in the UI.

The slice is applied once at the request funnel, covering the plain
send, the agentic flow and the KV cache pre-encode, plus the continue
path that bypasses it. The summary request is a one-shot outside the
SSE replay machinery, streamed live into the node with a full rollback
on abort, error or empty result.

The action is exposed as a /compact slash command and a Compact button
in the context gauge popup, with an overridable summarization prompt
in the settings.

* ui: pre-fill KV cache and refresh context gauge after compaction

Once the summary is stored, pre-encode the compacted branch and stamp
the returned prompt timings on the compaction node: re-encoding the
reduced prefix is part of the compaction itself, so the server KV
cache and the context gauge reflect the summary right away.

The summary stream is registered under a prefixed identity
(compact::convId), so the visibility kick and byte-offset resume
protect it through tab switches while the reload probe never
reattaches it as a chat turn. The stamped node becomes the shared
context truth for the gauge and the processing state restore, and the
stale live state that kept the gauge full is cleared.

* chore: format

* ui: allow a dedicated model for conversation compaction

An opt-in setting reveals a model selector whose options come from the
live router model list. The chosen model writes the summary and is
loaded on demand when it is not resident, while the default path keeps
resolving to the conversation's own model.

* ui: align the compaction model field with project conventions

* ui: fix the model option helpers import path

The compaction model selector pulls filterModelOptions and
groupModelOptions from the models component folder, where they no
longer live, so the build fails to resolve the module. Point the
import at the navigation folder, which now owns them.

Missed during the rebase: the helpers moved in a commit that does
not touch any compaction file, so nothing conflicted.

* ui: import the model option helpers from models/utils

* ui: follow the moved models selector in the compaction model field

The models manager moved ModelsSelectorList under ModelsSelector and
the model option helpers to /utils, where groupModelOptions takes
the options alone. Drop the stale copy the rebase kept and read the
helpers from their new home.

* ui: drop the dedicated compaction model

A summary written by another model loses the prefix the conversation
model cached, so the whole history is prefilled again, and the model
then resumes from text it did not write. The conversation model
always writes its own summary.

* ui: keep the last steps of a compacted conversation

A compaction now folds the steps the model reads but the last ones,
a step being a user message or an assistant message with its tool
results, and stores how many it kept on the compaction node. The
model then reads the system prefix, the summary, the kept steps and
what follows.

The summary request is the request the chat sends, with its options,
tools and message shaping, cut before the preserved steps: the server
reuses the prefix it cached and only the instruction is new. The
number of preserved steps is a setting, 10 by default.

* ui: run every compaction summary through one path

Three callers needed the same thing - the automatic pre-encode pass, the
action in the context gauge, and re-running a summary from the block - and
each of them had grown its own copy of the request, the streaming and the
storing.

There is one path now: a summarization owns a node, streams into it, and
reports itself through a ledger the rest of the app reads (`isCompacting`,
which node belongs to which conversation). Re-running a summary is the same
call on an existing node, so the two entry points cannot drift apart again.

* ui: give the compaction row its own summary UI

The summary was a line of text in a tool-call-shaped collapsible, which is
how a compaction reads as a side note rather than as the thing the rest of
the conversation now stands on.

The row is its own block: a divider that names the boundary, and the summary
inside it with the model that wrote it, its statistics, and the same actions
an assistant turn has - copy, edit, fork, regenerate and delete all apply to
the summary, and it collapses to the divider once a summary is in, because
the shrunk context is the headline and the prose is the detail.

The divider is a variant of the shared collapsible block, which also needed
the toggle and the title split into snippets so a caller can lay them out
without copying the markup. The action icons no longer take a role: they
render what a message offers, and a compaction offers the same as any other
stored message.

* ui: report compaction progress on the row

The row owns the run, so the row says how it is going: the prompt progress
in the header while the context is being read, the phase once the summary
starts streaming, and the token count once it is in. Before the first
summary token there is nothing to collapse, so the header stands on its own
instead of offering an empty block.

* ui: keep the summarized turns quiet while a compaction runs

The turns being summarized used to keep their own "Processing..." indicator
and live reading stats while the summary streamed, so the conversation said
the same thing twice in two places. The row reports the run, so the turns it
replaces stay silent until it is done.

* ui: hand the compact action to the context gauge

Compacting is a decision about the context, so the action belongs with the
numbers that show the context filling up rather than inside the form: it sits
at the bottom of both the hover card and the drawer, under the breakdown it
acts on, quiet enough to stay a footnote and disabled - with the row's own
spinner - while a summary is already running.

* ui: follow the bottom, otherwise leave the view alone

The screen scrolled itself with a controller of observers, intervals,
direction heuristics and landing timers, and still managed to move the view
while the reader was reading: any mutation that grew the branch was treated
as a reason to keep following.

The page is the scroller, so it follows when the reader sits at the bottom,
and stays where it is otherwise. A resize observer plus native scroll
anchoring does the rest, and `pointerdown` buys the reader a moment of grace
to be caught mid-drag.

* ui: expand collapsible bodies with a height transition

Expanding a block snapped it open: the body was mounted the moment `open`
flipped, so a tall block moved the conversation around it in one frame.

The body now goes through the same region the models table uses. The height
interpolates and the visibility follows it, so a body is unmounted once the
collapse has finished rather than the moment it starts - a closed block
still costs nothing while tokens stream past it.

* ui: open a compaction summary at the top of the screen

Toggling a summary left the reader wherever they were, so a conversation that
had just compacted showed the divider at the bottom of the screen and read
down from there.

A toggle now scrolls the page once and lands the divider a breath under the
tabs, both ways: the summary is read from its beginning, and closing it returns
to the same boundary instead of wherever the page happens to come to rest. The
scroll waits for the body to reach its height, because before that the page is
too short to scroll the divider up there and the call would clamp.

That wait is the shared collapse duration rather than a transition event: an
engine without `interpolate-size` animates the region's grid row instead of its
height, so a listener on `height` would never fire there and the page would not
move at all.

* ui: spin the compaction icon while the summary runs

The divider reported the phase in its title while the icon still showed a
finished block, so a run in progress and a summary that had landed looked the
same in the one place the eye reads first. The icon spins while the summary is
running and folds back once it is in.

* ui: let the compaction summary speak without the reasoning trimmings

The summary is the thing the conversation now stands on, not a note beside
it, and the conversation model always writes it, so the body drops the
reasoning block's rule and indent (`flushBody` keeps them for the blocks
that borrow them) and the model selector under the summary is gone - there
is nothing left to swap.

* ui: drop the model override from the compaction path

The summary row no longer offers a model selector, so nothing hands
regenerateCompaction a model anymore. Remove the parameter from the
whole path, with the on-demand load of another model it carried: the
summary always comes from the model the request funnel resolves.

* ui: let the compaction summary think as the chat does

The summary request forced thinking off, overriding the reasoning
effort the user chose. The summary is the text the conversation
resumes from, the task where reasoning helps the most, so the request
now carries the very options of the chat.

* ui: pre-encode the compacted branch as the next turn reads it

The post-compaction pre-encode sent the branch without tools, while
the tools render ahead of the messages, so it warmed a prefix the next
turn never reads and that turn prefilled the whole compacted context
again. The pre-encode now sends the tools, message shaping and
reasoning handling of the chat, so the next turn only prefills its new
message.

* ui: pre-encode the branch the next turn actually reads

The per-turn pre-encode sent the messages of the request plus one
assistant message holding the streamed text, without tools. With
tools on, it replaced the slot the next turn extends with a prefix
nobody reads again, and after an agentic turn its history did not
match the stored calls and results. Without agentic, it never ran.

Both pre-encodes now read the stored branch up to its leaf, cut at the
last compaction, and send it with the tools, message shaping and
reasoning handling of the chat. The plain completion path triggers it
too.

* ui: refuse a compaction summary the output limit cut

The stream handler read every chunk but dropped the finish reason, so
a summary cut by max_tokens folded the conversation into half a text.
The handlers now hand the finish reason to onComplete, and a summary
that ends on length rolls the compaction back with an error.

The summary was framed through a replacement string, so $& or 32560 in
it changed the text the model read; it is framed as written now.

* ui: wait for the collapse transition in the tool call tests

The collapsible body now unmounts once its collapse animation ends rather
than as soon as the block closes, so the "folds back" assertions ran before
the body was gone. Wait out COLLAPSIBLE_TRANSITION_MS first.

Assisted-by: Claude (Opus 5.5 Medium)

---------

Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>
2026-10-27 07:49:27 +02:00
Aleksander Grygier bc262f33cc ui: Open model details in a drawer without leaving the page (#31176)
* ui: open model details in a global drawer without leaving the page

The model info actions in the selector used to navigate to the models page
to open its details pane, and a phone opened a separate dialog instead. The
drawer now lives outside the models table, driven by uiStore, so the info
icon opens it over whatever page is showing.

The models dropdown and the phone picker stay open under the drawer's
backdrop, and keep focus while the drawer opens and closes, so the trigger's
tooltip no longer pops over the drawer.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: let the model details drawer reopen after a drag to dismiss

Dragging the drawer down wrote vaul's own open state, which the one-way open prop then kept closed, so the info action could not open it again until a refresh. Bind open both ways through the store instead.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: close the models selector from the details drawer chat actions

Assisted-by: Claude (Opus 5.5 Medium)
2026-10-27 07:46:25 +02:00
Aleksander Grygier f5892ca5ad ui: use shared confirmation dialog for conversation deletion (#31187)
Extract the keyboard-shortcut delete confirmation into DialogConversationDelete
and reuse it in the sidebar and mobile search page instead of the native
window.confirm prompt.

Assisted-by: Claude (Opus 5.5 Medium)
2026-10-27 07:45:53 +02:00
Ehsan Bateni 7aca0d2ea8 hexagon: add GGML_HEXAGON_VTCM_SIZE (#31221)
* hexagon: add GGML_HEXAGON_VTCM_MB and GGML_HEXAGON_NTHREADS caps

Change-Id: Idffe1613c8db6f96aa842d43d1b2efaed45462a8

* hexagon: rename VTCM cap to GGML_HEXAGON_VTCM_SIZE, drop NTHREADS cap

Change-Id: If4c5b9ce9fd6b725c3561421c2295d40599d012d

* hexagon: make GGML_HEXAGON_VTCM_SIZE an override, accept K/M/G suffixes

Change-Id: Ib8eea84c22f24e6d986b37f215e504b046478c2c
2026-10-26 16:56:01 -07:00
begcdn 961c376a2d llama : enable moe amax rescaling for laguna (#31147)
* llama : enable moe amax rescaling for laguna

Assisted-by: Devin

* llama : declare laguna sm tensor support
2026-10-26 23:04:19 +01:00
Sigbjørn Skjæret 4244ad7acd ci : add windows behavior check (#31191) 2026-10-26 23:01:04 +01:00
Hongqiang Wang 63b4b303bd opencl: write KV-cache views in set_tensor so a state restore is not dropped (#30471) 2026-10-26 11:39:22 -07:00
Max Krasnyansky caa8191df1 hexagon: MUL_MAT_ID row skipping optimizations and improvements (#31169)
* hex-mmid: vectorize skipped row zeroing and added zeroing to fused MM_ID_NX

* hex-mmid: remove scalar div
2026-10-26 08:21:14 -07:00
Georgi Gerganov a4699400e3 llama : move no_alloc out of llama_hparams (#31141)
* llama : move no_alloc out of llama_hparams

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (2x DGX Spark (RDMA, TP=2))

* cont : clean-up

* cont : clean-up

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (2x DGX Spark (RDMA, TP=2))

* tests : add a no_alloc case to test-llama-context

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (2x DGX Spark (RDMA, TP=2))
2026-10-26 16:21:25 +02:00
Georgi Gerganov 4b851fbee8 models : replace ggml_tensor** out-params with output structs (#31142)
The hyper-connection and DSA graph builders wrote their extra outputs
(post / comb gates, carried top-k selections, inject gates) through
ggml_tensor** out-params. Return them in a small struct next to the
primary output instead.

Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
2026-10-26 16:19:47 +02:00
Pascal 4d71b42fd8 ggml-cpu : rewrite concat as one row kernel for all types (#31179)
Every dst row comes from src0 or src1, or for dim 0 from the row of
src0 followed by the row of src1, so a single kernel covers every
type: a dense source row is one memcpy, quantized types included,
and a strided source is copied element by element with the element
size known at compile time.

Rows are split across threads instead of i2 alone, which is often 1,
and are copied in tiles of 32: for a transposed source the elements
are walked in the outer loop, so the rows of a tile read neighboring
addresses instead of jumping a full row stride per element.

Add transposed b cases to test_concat, the layout of the delta-net
conv input, and a perf case for it.
2026-10-26 16:18:45 +02:00
virajwad 70c5f26324 vulkan: rms norm optimization using subgroup reductions (#29882)
* RMS Norm Subgroup reductions optimizaton

* Fix build error

* Improved optimization

* Make usage of BLOCK_SIZE consistent

* Add fast path to skip 2nd subgroupAdd

* Add AMD Vendor to subgroup reduction opt

* Add comment clarifying rms_dict and merge_maps behavior

* Add unroll

* Rebase

* Fixed syntax issue

* Add 128/256/512 block sizes to non-subgroup path

* Remove trailing whitespace
2026-10-26 16:14:31 +02:00
Titaniumtown 5cd62ffee9 sycl: template the plain MMVQ launch triples (#31013) 2026-10-26 16:13:16 +02:00
Titaniumtown 38645c2ad9 sycl: template the per-quant concat impls (#31008) 2026-10-26 16:12:56 +02:00
Captain-TrippsandJon Staples 02269b4c00 sycl: add set_tensor_2d for faster tensor-split loading (#30358)
* sycl: add set_tensor_2d for faster tensor-split loading

Pack host rows into chunks of up to 64 MiB when destination rows are
contiguous, avoiding a separate upload call for each row. Keep the
existing per-row copy path for non-contiguous destinations.

On two Arc Pro B70s with gpt-oss-120b UD-Q4_K_XL, total llama-perplexity
runtime drops from 263-268 s to 114-115 s. A full-load readback check
verified 26,784,256 rows with no mismatches.

* sycl: batch 2D tensor uploads through shared staging slots

Implement set_tensor_2d to speed up tensor-split model loading by
packing rows into the existing pinned staging slots and uploading
each batch with a single device copy.

Share staging initialization with set_tensor and wait only when
reusing a slot, allowing CPU packing and device copies to overlap.

Fall back to per-row uploads for noncontiguous device rows,
oversized rows, or unavailable staging.

---------

Co-authored-by: Jon Staples <250234834+Captain-Tripps@users.noreply.github.com>
2026-10-26 16:12:25 +02:00
KyleHagy 5320c87ff0 sycl : bound reorder temp memory to 32 MiB (#31062) 2026-10-26 16:11:57 +02:00
Pascal 74b104c9ba common: pin model resolution on real HF repo layouts (#30548)
* tests: pin model resolution on real repo listings

Extend test-model-resolution with 63 repo listings captured verbatim
from the HF API, chosen to cover every file naming pattern found across
1237 GGUF repos of the major publishers and pruned to whole quant groups.
Each repo pins its resolved plans and its cache listing.

The expectations follow what each file really is per its GGUF header,
not what the current resolver picks, so a mismatch is a resolution bug.
Mismatches are collected and reported together before the test fails.

* tests: leave the listing unchecked on paths the filesystem refuses

Some real split names (DevQuasar's Model.Q2_K.gguf-00001-of-00012.gguf
under a quant folder) exceed MAX_PATH once materialized in the temporary
cache on Windows, so the listing of such a repo is skipped with a note
while its plans stay checked.

* tests: pin real listings to master on GGUF convention names only

Replace the real repo table with 171 HF listings whose file names follow
the GGUF naming convention (lowercase sidecar prefix, - delimited
components, quant last), one per layout pattern found across 1174 GGUF
repos and pruned to whole quant groups. Repos outside the convention are
dropped. Each repo pins the plans and the cache listing master gives
today, so the test is a pure harness: any change in resolution shows up
here, and mismatches are reported together.
2026-10-26 14:18:29 +01:00
Xuan-Son Nguyen c6fb3b5be2 mtmd: add mtmd_helper_get_capabilities (#31183)
* mtmd: add mtmd_helper_get_capabilities

* cap audio_stream

* add back old api for compat

* audio_gen_spk_ref, text_gen
2026-10-26 12:45:35 +01:00
Sigbjørn Skjæret 8b28c23943 ggml-cpu : fix unused variables in arm arch (#31180) 2026-10-26 11:38:55 +01:00
Georgi Gerganov fd8f8451a2 tests : refactor test-llama-archs to not link internal libllama symbols (#30880)
test-llama-archs included src/llama-arch.h and src/llama-model-saver.h,
which prevented building on Windows with shared libraries since the
used symbols are not exported from the DLL.

- declare llama_arch_* wrappers around the internal llm_ arch query
  functions in the llama-ext.h staging header, together with
  llama_arch_kv_name() and llama_arch_supports_save()
- slim llama_model_saver down to the private implementation of
  llama_model_save_to_file(): the fixture-writer half used by the test
  moves into tests/test-llama-archs.cpp as a local kv_writer
- replace the roundtrip saver usage with the public
  llama_model_save_to_file() writing to a temp path (also removes the
  tmpfile() fd leak and its Windows workaround)
- move test-llama-archs, test-generate-models, test-llama-context and
  test-fusion out of the NOT WIN32 OR NOT BUILD_SHARED_LIBS guard

Assisted-by: pi:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (2x DGX Spark (RDMA, TP=2))
2026-10-26 12:16:02 +02:00
Evan Huus 9125a2ebe8 ci : fix race in webgpu memset (#31162) 2026-10-26 18:02:12 +09:00
feni6andfeni6 9b8446b914 metal : stop overflow from happening in large matrix output offsets (#30072)
* Metal : stop overflow from happening in large matrix output offsets

Assisted-by: Codex

* metal : widen batch offsets and MUL_MAT_ID token indices

Assisted-by: Codex

---------

Co-authored-by: feni6 <270037094+feni6@users.noreply.github.com>
2026-10-26 10:13:39 +02:00
Aleksander Grygier dc9e3bed42 ui: UX improvements (#31030)
* ui : land opened conversations at their bottom before the first paint

Pin the page to its bottom with a ResizeObserver until it stops resizing or the reader interacts, and mount the last rows eagerly so the landing frame shows real content.

Assisted by: Claude (Opus 5.5 Medium)

* ui : make conversation tabs browser-style

A plain navigation shows the chat in the current tab instead of adding one, and the new-chat tab turns into the conversation it starts. New tabs come only from explicit actions: the + button, Cmd/Ctrl-click or middle-click on a sidebar chat, and its Open in new tab menu item, which open in the background. Closing a tab moves to its neighbour among the tabs the view shows.

Assisted by: Claude (Opus 5.5 Medium)

* ui : share the tab surface and tab bar fade styles

Move the tab item surface (shape, resting and active looks) to TAB_SURFACE_* class constants and the fade under the tab bar to a global chat-top-fade class, so other elements can match the tabs.

Assisted by: Claude (Opus 5.5 Medium)

* ui : fit the chat column below the tab bar

The tab bar reported 1.25rem of its height while it takes 3.3125rem (pt-3.25 + h-10), so a short conversation still left a few pixels to scroll. Use the bar's full height and drop the extra 1rem from the column's minimum height, so it fills the viewport exactly; the centered new-chat form moves up by the same 1rem to keep its place.

Assisted by: Claude (Opus 5.5 Medium)

* ui : stop the page from bouncing past its ends

Elastic overscroll on the root scroller dragged the sticky and fixed bars (tab bar, chat form) along at the top and bottom of a chat, and they jumped back on release. Set overscroll-behavior: none on html and body.

Assisted by: Claude (Opus 5.5 Medium)

* ui : stop scrolled dialogs from bouncing past their ends

Dialogs scroll on their own (the settings panel, the search results), so the page rule did not reach them and their sticky navigation still jumped on overscroll. Turn off overscroll on dialog panels and on the scroll containers inside them. Only scroll containers: Firefox counts an overflow-hidden box such as truncated text as one, and `none` there kept the wheel from scrolling the list around it.

Assisted by: Claude (Opus 5.5 Medium)

* ui : fit a first exchange opened by the working directory row

The last answer's minimum height leaves room for the last question only, so the synthetic row that a new chat starts with (Set working directory to ...) made a short first exchange overflow by that row. When everything above the last question is synthetic, count those rows as part of it.

Assisted by: Claude (Opus 5.5 Medium)

* ui : keep following a streaming response until the reader scrolls up

Auto-scroll stopped whenever the page moved up, including moves the
reader did not make: scroll anchoring and blocks re-rendering shorter
while a response streams. Only the reader's wheel, touch, scroll keys
or scrollbar drag now count as scrolling up.

The send scroll also looked up the new user bubble by position, which
picked an older message when the answer row had not rendered yet.

Assisted by: Claude (Opus 5.5 Medium)

* ui : fade the chat form between its centered and bottom positions

The form no longer slides across the column between the new-chat center
and the bottom. It fades out drifting toward where it goes and fades
back in where it lands, quicker when opening a chat than when going back
to a new one. Its wrapper keeps the same height in both positions, so
the messages never shift while it moves.

The new-chat greeting fades in as the form settles, also on a refresh or
a first visit. Reduced motion keeps only the fades.

Assisted by: Claude (Opus 5.5 Medium)
2026-10-26 10:13:08 +02:00
Aleksander Grygier 69b3cc406e ui: Sidebar resizing improvements (#31119)
* ui : let the expanded sidebar be resized by dragging its edge

Bring back the desktop sidebar resize handle that was dropped with the
shadcn sidebar primitive. The width is clamped between 18rem and 32rem,
persists across reloads, and resets on a double-click of the handle. The
tab bar follows the same width so a wider sidebar never covers it.

Assisted by: Claude (Opus 5.5 Medium)

* ui : rubber-band the sidebar edge past its bounds and drag it shut

Past the max width the edge resists harder the further it is pulled, and
the same happens below the min width until the pull passes 9rem, where it
snaps to the pointer and letting go hides the sidebar. A release outside
the bounds springs the width back to the nearest one.

Assisted by: Claude (Opus 5.5 Medium)

* ui : show the sidebar edge strip faintly on hover and solid while dragging

The strip is clipped to the sidebar's rounded corners through its own
overlay, so the sidebar itself is never clipped.

Assisted by: Claude (Opus 5.5 Medium)

* ui : remember whether the desktop sidebar is open across reloads

A phone opens the sidebar as an overlay, so it always starts closed there
and its toggling never overwrites the desktop preference.

Assisted by: Claude (Opus 5.5 Medium)

* ui : fade the sidebar labels and conversation list with its width

The fade follows the live width, so it tracks both the toggle's width
transition and a drag below the minimum width.

Assisted by: Claude (Opus 5.5 Medium)

* ui : persist the sidebar width on release and clamp a restored one

A drag now updates only the live width and writes it to localStorage once
when released. A width restored from localStorage is clamped to the
current bounds, so a value saved under older bounds does not stick.

Assisted-by: Claude (Opus 5.5 Medium)
2026-10-26 10:07:25 +02:00
Georgi Gerganov 224d94c02b ci : bump server timeout during startup (#31174) 2026-10-26 09:59:15 +02:00
Georgi Gerganov 48c36110ab ci : drop the macos-latest-x64 job (#31175) 2026-10-26 09:56:19 +02:00
Aman GuptaandGeorgi Gerganov a074215352 deepseekv4.1: follow-up review fixes (#31137)
* kv-cache : use ggml_backend_tensor_copy to refill the DSV4 K mirrors

* llama : move the Engram pad id to llama_hparams and the token map to llama_model

The saver no longer needs the deepseek41 model class.

* deepseek41 : drop the F32 src1 request on the HC mixes matmul

* ggml : add softplus TODOs in the backends that still use log(1 + exp(x))

* convert : make the DSV4/DSV4.1 modify_tensors generators, drop the redundant vision filter

* deepseek41 : clean up comments

* deepseek41 : drop the F32 src1 requests on the router and indexer matmuls

* convert : use format_tensor_name directly in DeepseekV4Model

* s/comp_kv_src/comp_il_src/g

* cont : fix TODOs for log1p

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-10-26 15:52:02 +08:00
Adrien Gallouët d854d98bae cuda : use std::string instead of VLA for archName (#30463)
* cuda : use std::string instead of VLA for archName

Modernize the code (old C style) and fix the following warnings:

```
ggml/src/ggml-cuda/ggml-cuda.cu:180:19: warning: variable length arrays in C++ are a Clang extension [-Wvla-cxx-extension]
  180 |     char archName[archLen + 1];
      |                   ^~~~~~~~~~~
ggml/src/ggml-cuda/ggml-cuda.cu:180:19: note: read of non-const variable 'archLen' is not allowed in a constant expression
ggml/src/ggml-cuda/ggml-cuda.cu:179:9: note: declared here
  179 |     int archLen = strlen(devName);
      |         ^
```

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Add a comment about archName

Signed-off-by: Adrien Gallouët <adrien@gallouet.fr>

---------

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
Signed-off-by: Adrien Gallouët <adrien@gallouet.fr>
2026-10-26 00:08:22 +01:00
48c9f08b7b common, grammar: add recursive depth limit, make it a requirement (#31153)
* common, grammar: add recursive depth limit, make it a requirement

* catch more recursive functions

* common : tidy up peg parser changes

* grammar : reject candidates in a loop, without a depth limit

The depth limit in llama_grammar_reject_candidates_for_stack counted
the code points of a vocab token, so any token of 256 chars or more
was always rejected under a grammar, such as the 512-space token of
GLM-4.7-Flash or the long space tokens of Laguna-S-2.1.

The rejection now walks the code points with its own stack of levels
instead of recursing: there is no limit and no stack risk, whatever
the length of a token from the GGUF, and the masks match master.

* grammar : track rejected candidates as indices into the input

At a given code point a candidate only differs from its input by the
position of its code point pointer, so the levels now carry indices
into the input candidates instead of copies. The first level needs no
special case and the rejects map back to the input directly.

Grammar sampling gets faster than master on GLM-4.7, Laguna, Qwen3.6
and gemma-4 with the json, list and spaces grammars, same masks.

* nit : shorten the comment of llama_grammar_reject_step

* common : add depth_guard for schema walks

* grammar : move GRAMMAR_RECURSIVE_LIMIT into the .cpp

* common : reduce recursion limit for peg parser

* improve

---------

Co-authored-by: Alde Rojas <hello@alde.dev>
Co-authored-by: Pascal <admin@serveurperso.com>
2026-10-25 23:53:08 +01:00
Georgi Gerganov 6050b467c0 tests : adjust default threads for test-llama-context (#31157) 2026-10-25 19:46:44 +02:00
Aman Gupta 2280f0fd16 qwen: merge tensors on convert (#31028)
* qwen: merge tensors on convert

* convert: use generator functions

* qwen4exp: remove conts and pre-scale
2026-10-25 15:23:22 +02:00
Masashi Yoshimura 1a0b56cec7 ci : disable test-llama-context with fa off in the webgpu (#30977) 2026-10-25 13:53:55 +02:00
Trunk Trick 63412d8737 llama: fix exhausting CPU and MEM caused by dry sequence breaker (#31035) 2026-10-25 12:31:09 +01:00
Anthony Wang 5c965fd8ca llama-quant : fix 4D tensor quantization using ggml_nrows() and name-based expert detection (#30820)
* llama-quant : fix quantizing 4D tensors

nrows_total was ne[1]*ne[2], so for a 4D tensor only the first ne[2]
slices were quantized and the rest of the rows were dropped. Use
ggml_nrows(tensor) to count all rows, including ne[3].

Fixes #29916

* llama-quant : detect expert tensors by name and fix imatrix size for 4D tensors

The MXFP4_MOE path used ne[2] > 1 to detect MoE expert tensors. This is
wrong for non-expert tensors with 3 or more dims, e.g. MLA projections
(BailingMoE3) and 4D conv kernels. Use the tensor name instead: all
tensors used by GGML_OP_MUL_MAT_ID are named *_exps or *_chexps. This
also removes the BailingMoE3 special case.

The imatrix holds one vector of ne[0] values per 2D slice, so the
expected size is ne[0]*ne[2]*ne[3], not ne[0]*ne[2]. This matches how
the imatrix is indexed per row for 4D tensors in llama_tensor_quantize_impl.

Assisted-by: Claude Opus 5.5
2026-10-25 13:28:38 +02:00
Hrushikesh Ramilla f1168ac417 grammar: limit recursion depth to prevent stack overflow (#29495) (#29755) 2026-10-25 12:10:10 +01:00
Georgi Gerganov fa3645feb5 llama : respect no_alloc in all memory caches (#31087)
* llama : respect no_alloc in memory caches

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)

* llama : TODO: move no_alloc out of llama_hparams

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)

* llama : TODO: move no_alloc out of llama_hparams

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
2026-10-25 11:56:04 +02:00
ShreyaandShreya Agrawal 7790a9231d tools : reject truncated GGUF tensor data in merge (#31044)
Co-authored-by: Shreya Agrawal <shreyaagrawal@Shreyas-MacBook-Air.local>
2026-10-25 11:25:10 +02:00
anujj c49edf0963 CUDA: fuse FFN gate/up matmuls and GLU in MMQ with interleaved gate/up rows (#29948)
* cuda: fuse FFN gate/up and GLU in MMQ with interleaved gate/up rows

For prompt processing, run the gate and up matmuls of a dense FFN as one
mul_mat_q launch and apply the GLU in the write-back.

- The loaders take the gate weights as a second pointer and fill the weight
  tile with up and gate rows interleaved in chunks of 8, so each thread holds
  an up value and the matching gate value (rows r and r + 8 of an MMA tile).
- The MMA loop is unchanged: same tile, warp layout, vec_dot and one sum[]
  per thread. The fused write-back computes the GLU from registers and writes
  half the rows.
- The fused path uses the regular tiling (no stream-k) with the column tiles
  innermost so that neighbouring blocks share weight tiles.
- Enabled for Q4_K, Q5_K, Q6_K, Q8_0 and IQ4_XS on NVIDIA Turing and newer.

* cuda: assert the fused MMQ gate argument and trim redundant comments

* cuda: move the fused MMQ hint check to should_fuse and add TODOs

* cuda: pass prec_src1 to the fused MMQ write-back

* cuda: fuse only with efficient tiling and no fallback, merge the GLU write-back

* cuda: require has_fusion explicitly, drop the fusion-only src1 precision check

* cuda: launch the fused MMQ kernel through the stream-k kernel with one block per tile

* cuda: inline the MMQ tiling-efficiency check

* tests: one fused gate/up MMQ case per type and GLU op

* cuda: always launch the fused MMQ kernel with one block per tile

* tests: shorten the fused gate/up MMQ test comment

* cuda: select the fused MMQ loader and write-back in ggml_cuda_mmq_get_util_funcs

* cuda: run the fused MMQ kernel on the plain tiling path

* cuda: put the column tiles innermost in the MMQ plain tiling path
2026-10-25 11:24:38 +02:00
Aman Gupta a55c16e4ee ggml-cuda: fuse qwen MRoPE + set-rows (#31025) 2026-10-25 11:15:45 +02:00
Jeff Bolz e321bdad7e vulkan: Add a larger tile size for conv3d (coopmat2 only) (#29920)
* vulkan: Add a larger tile size for conv3d (coopmat2 only)

This adds a larger tile size, in practice it is only for coopmat2 where
we don't need shared memory to store the output (would use more than 64KB).

Add some additional representative backend tests. At least, this is what codex
came up with from looking at sd.cpp Wan VAE.

* only apply large tile size for newer driver versions
2026-10-25 11:13:48 +02:00
Ian McKellar 459a1ba549 vulkan : one warp tile per subgroup in the small matmul tiles (#30106)
WM of the small tiles was 32, or 8 for 8-wide subgroups, whatever the
tile's BLOCK_SIZE and WARP. With 16-wide subgroups (Mali) the second
subgroup of a workgroup wrote into the next tile's columns. With 8-wide
subgroups some _id tiles left rows uncomputed. Derive WM so there is one
warp tile per subgroup.

Tested on my Pixel 9 Android phone. Before the change there were 20
MUL_MAT and 6 MUL_MAT_ID failures. After the change they all pass.

Fixes #28637

Assisted-by: Claude Opus 5.5
2026-10-25 11:13:14 +02:00
Max Krasnyansky f0fb5984b0 hexagon: generic dims collapse/reshape for unary, binary and matmul ops (#31071)
* hex-tools: support for --compare option in the profile.py

* hex-unary: improve collapse logic to also do splits if nrows is small

* hex-colapse: handle all colapsing at the host and make it generic

* hex-colapse: remove leftover unary helper
2026-10-25 11:11:48 +02:00
lhez 9a4580b61c opencl: add topk and lightning indexer (#31026)
* opencl: add topk

* opencl: optimize topk

* opencl: add fp lightning indexer

* opencl: optimize lightning indexer

* opencl: refactor

* opencl: refactor tiled topk kernel

* opencl: adjust format
2026-10-25 11:11:15 +02:00
lhezandHongqiang Wang b083c384f7 opencl: add initial q2_k support (#30999)
* opencl: add `mul_mv_q2_k_f32`

* opencl: add q2_k SOA mv and tiled mm

* opencl: add q2_k noshuffle gemm/gemv for adreno

* opencl: add q2_k dp4a gemm for adreno

Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>

* opencl: tune q2_k kernels for adreno

---------

Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
2026-10-25 11:09:56 +02:00
Georgi Gerganov 344be0a600 ci : update CPU job assignments (#31136) 2026-10-25 10:35:11 +02:00
Yuri KhrustalevandXuan Son Nguyen b3fa69955e chat : reject JSON nested more than 128 levels before walking it (#30242)
* chat : reject JSON nested more than 128 levels before walking it

The schema builder, the jinja engine and the decision code walk request
JSON with recursion. A request nested deep enough overflows the stack of
the HTTP thread and llama-server dies (SIGBUS on macOS, SIGSEGV on Linux):
for a tool schema from a few hundred levels on macOS and about 10000 on
Linux.

Check the depth first and answer 400 past 128 levels: the chat inputs in
common_chat_templates_apply (tools, tool-call arguments,
chat_template_kwargs, json_schema), the json_schema of /completion, and the
whole /v1/systemone body. A gemma4 tool result that is too deep stays text.

* fix the root, not the call side

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-10-25 06:46:35 +01:00
Aman GuptaandDaniel Han 74bca39500 model: DSV4.1 flash (#31086)
* model : add DeepSeek-V4.1-Flash (deepseek41)

* deepseek41 : remove MXFP8, convert the Engram table to BF16

* deepseek41 : compute the compressed-KV read layers once in the cache

* deepseek41 : use get_cells() for the shared-history check, drop llama_kv_cache::seq_pos_tok_le

* deepseek41 : gather the Engram rows in-graph like qwen4exp, drop the host gather

* deepseek41 : keep the Engram hash constants in hparams.ple_*, one module per Engram layer

* deepseek41 : drop the LLAMA_DSV41_KV_MIRROR toggle

* deepseek41 : keep il last in build_hc_mixes

* deepseek41 : sort the top-k rows before the sparse gather

* tests : add deepseek41 to the Metal fusion baseline

---------

Co-authored-by: Daniel Han <danielhanchen@gmail.com>
2026-10-25 09:43:51 +08:00
Joseph Patricio d327bf1f07 jinja : fix select and selectattr argument handling (#31059)
* jinja : fix select and selectattr argument handling

* jinja : address review feedback

* jinja : Simplified test attr
2026-10-24 22:54:46 +02:00
Ehsan Bateni 78a8ae14be hexagon: KV-head team split for multicore HMX flash attention (#31055)
* hexagon: KV-head team split for HMX flash attention in multicore

* hexagon: merge FA_HEAD_SPLIT and FA_SPLIT into single GGML_HEXAGON_FA_SPLIT flag

0 = off (token-split only); 1 = head-split, plus team-split when
n_kv_heads < N_cores so stranded cores still participate (e.g. gemma-4
with 2 KV heads on 4 cores). Where heads divide evenly across cores,
plain head-split is used.
2026-10-24 12:19:32 -07:00
Xuan-Son Nguyen f3269ef96a server: refactor server entrypoint (#31113)
* server_role

* cont
2026-10-24 20:04:28 +02:00
Georgi Gerganov 3f03b17724 metal : remove mul_mm_id amax rescaling (#30924)
The src1 power-of-two rescale in kernel_mul_mm_id is superseded by the
per expert-token amax scaling done in the compute graph
(ref: https://github.com/ggml-org/llama.cpp/pull/30861), which keeps the
MoE down-projection activations within f16 range before the op runs.

Drop the two-stage reduction kernels, their scratch buffer and the
GGML_PREC_F32 dispatch added in https://github.com/ggml-org/llama.cpp/pull/26223,
as well as the test cases that drive MUL_MAT_ID src1 out of f16 range.

Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
2026-10-24 18:40:19 +03:00
Katara 9a0fde9967 rpc : add bounds check in rdma_recv (#30399) 2026-10-24 18:34:46 +03:00
Aleksander Grygier e168e041ec ui: Models Manager as Route + UI improvements (#30988)
* ui : move models manager to its own route, keep Discover as a dialog

Models now lives at #/models instead of a dialog. Discover opens as a
standalone dialog over it, and leaving the page returns to the chat it
was opened from.

Assisted by: Claude (Opus 5.5 Medium)

* ui : polish the models page layout and filters

Give the models page a page header with the Discover action, calmer
column labels, tinted section and family headers, and an empty state.
Load the model list on a direct visit, slide the detail pane over the
table below xl, and draw the filter toggles as one box matching the
context select, with room for their shadow.

Assisted by: Claude (Opus 5.5 Medium)

* ui : round the models table family headers

Assisted by: Claude (Opus 5.5 Medium)

* ui : fix models page navigation and Discover gating

Record the models page's way back after every navigation, so Back,
a typed URL or a reload no longer return to a stale chat. Escape
closes the detail pane, then leaves the page. Gate every Discover
entry point on one shared check, recheck the API key on /models,
skip the duplicate router fetch on a direct load, and share the
route load between pages.

Assisted by: Claude (Opus 5.5 Medium)

* ui : show model details in a drawer

Open the models manager's detail pane in a drawer, from the right on
desktop and from the bottom on a phone, instead of an inline side
panel and a full-screen cover. Pad the pane on both sides, drop its
close button on a phone, and assert that filtered rows are gone from
the page rather than merely hidden.

Assisted by: Claude (Opus 5.5 Medium)

* ui : make whole models table rows clickable

A click anywhere on a model or quant row opens it, unless it lands on
one of the row's own controls (load toggle, actions menu). The name
button stays the keyboard path to the same action.

Assisted by: Claude (Opus 5.5 Medium)
2026-10-24 18:34:11 +03:00
Aleksander Grygier ea1d7d66f3 fix(ui): align hover card and context popup styling with dropdown menu (#30978)
Use the dropdown menu surface (border, radius, background, shadow,
padding) for the hover card and the context gauge popup, and fix the
hover card open/close animation to use bits-ui data-state attributes.

Assisted-by: Claude (Opus 5.5 Medium)
2026-10-24 18:33:27 +03:00
Georgi Gerganov 71255fefc0 ci : pass per-runner results dir to self-hosted jobs on T4 runners (#31100)
The two T4 runners in the ggml-ci group (ggml-6-x86-vulkan-t4,
ggml-8-x86-nvidia-a10) share the same labels and can both pick up the
same jobs, so extend the per-runner results dir from #31046 to them.

Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
2026-10-24 17:17:04 +03:00
Satyam Thakurandorigin-labs-ai 73adf89e38 minimax-m3 : forward quant scales to the graph (fixes #31063) (#31097)
The generic scale pass in llama-model.cpp loads the per-expert
NVFP4 scale2 tensors (ffn_{up,gate,down}_exps_s) and the per-tensor
MXFP8 scales for the dense FFN and the shared expert, but
minimax-m3.cpp never forwarded any of them to the graph.

build_moe_ffn() defaults these to nullptr, so build_lora_mm_id()
skipped its w_s branch entirely and the routed-expert weights came
out global_scale times too large, swamping the residual (reported
G ~= 15123 for M3.1 Flash) and producing token soup.

Forward all scale tensors at the three call sites, matching the
pattern used by flint and the other NVFP4-capable MoE models.

Co-authored-by: origin-labs-ai <origin-labs-ai@users.noreply.github.com>
2026-10-24 16:09:02 +02:00
Alexey Kopytko df9beac715 tests : fix test-llama-context for large n_batch (#31083) 2026-10-24 14:14:58 +03:00
Georgi Gerganov 4d32b29aaa ci : exclude test-speculative-rejection-sampling from macos-latest-arm64 (#31088)
Add test-speculative-rejection-sampling to the ctest -E exclusion list of the macos-latest-arm64 CI job.

Assisted-by: dsh:llama.cpp/ggml-org/Flint-GGUF (RTX 5090 32GB, DDR5 128GB)
2026-10-24 14:14:31 +03:00
Georgi Gerganov 3032209758 llama : deny -sm tensor for arches with moe amax rescaling (#31081)
The amax POOL_1D reduction spans the whole matmul row, so a device cannot
compute it from an axis-0 split. The Meta split-state case reported
MIRRORED, which gives each device a partial max of its own slice. Drop
that case and deny -sm tensor for the arches which build the amax graph,
see [TAG_MOE_AMAX_SCALE].

Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
2026-10-24 13:48:32 +03:00
Xuan-Son Nguyenandyomaytk 701fd25b3a ggml : allow MUL_MAT_ID -1 index to skip the computation (#26631)
* ggml : allow MUL_MAT_ID -1 index to skip the computation

* nits

* update vulkan code

* ggml-webgpu: ensure all threads reach workgroup barriers in mul_mat_id_vec

(cherry picked from commit d8a4b9efb6e2b9abf3a866e847756b7e6bbc6b69)

* fix cuda

* openvino

* exclude CUDA from this PR

* add TODO tag

---------

Co-authored-by: yomaytk <yoshimura.masashi.frbs@gmail.com>
2026-10-24 12:15:36 +02:00
Daniel Bevenius 3642238cc1 tests : add speculative rejection draft sampling test (#30213)
* tests : add speculative rejection draft sampling test

This commit adds a test for speculative rejection sampling test that
excercises the speculative rejection sampling mechanism in
common/speculative.cpp.

The motivation for this is have a test that verifies that the
speculative draft models emitts tokens that  match the target models
distribution. So when rejection sampling occurs it should still
produce the same probability distribution as if we directly sampled
from the target model.

Running the test:
```console
$ ctest --test-dir build -R "^test-speculative-rejection-sampling$" \
    --n_trials 5000 -V
```
This will generate two models, one target and one draft model.

For subsequent runs use the "fixture exclude any" option to avoid
recreating the models:
```console
$ ctest --test-dir build-cpu-debug -R \
"^test-speculative-rejection-sampling$" \
    -FA "^speculative-models$" --n_trails 5000 -V
```

* remove draft model
2026-10-24 12:47:49 +03:00
Georgi Gerganov 6b9ba4f2e8 ci : fix data race between runners sharing the model cache (#31046)
* ci : fix data race between runners sharing the model cache

Two runners per mac mini share HOME, so concurrent jobs corrupt each
other downloads, converted models and venv under the shared MNT cache.

- gg_wget: download in a temp dir, publish with mv, so readers only
  ever see complete files (-N cannot be combined with -O)
- convert, quantize and wikitext unzip now produce under a per-runner
  scratch dir in OUT, published to MNT via hardlink when missing
- gguf-split and quantize tests get a per-runner TMP_DIR
- python venv moved from MNT to OUT
- metal, vulkan and webgpu mac jobs pass a per-runner OUT path

Assisted-by: dsh:llama.cpp/agent-qwen4 (RTX 5090 machine)

* ci : clean up scratch models after tests

Assisted-by: dsh:llama.cpp/agent-qwen4 (RTX 5090 machine)

* ci : fix q4_0 publish when a cached model exists

cp -n skips an existing file with exit code 1, so the ln fallback
aborted the qwen3 job under set -e. Publish through a unique temp
file with an atomic mv: readers never see a partial file and a
stale cached q4_0 gets replaced.

Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (RTX 5090 machine)

* ci : cache-first gg_wget to avoid re-downloads

HF sends no Last-Modified, so wget -N re-downloads the full files on
every run. Skip the download when a non-empty cache file exists: the
HF CI artifacts are static. Downloads still go through a temp dir and
an atomic mv, so concurrent runners never read partial files.

Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (RTX 5090 machine)
2026-10-24 09:21:24 +03:00
Ruben Ortlam 8e5a58df87 vulkan: ggml src1 precision support (#30656)
* vulkan: update accumulator precision selection

* handle fp32 overflows through scaling

* handle overflows in copy-to-fp16 activations

* add missing cases

* simplify code

* add tests

* add ggml prec get helpers, use in Vulkan backend

* clean up comments

* revert comment edit

* trim tests

* also handle fp32 src0, fix test scaling src0 values too

* don't scale if disabled

* remove fp16 scaling

* add check to mul_mat

* fix inconsistency

* rename to acc_prec
2026-10-24 06:05:05 +02:00
Arseniy Obolenskiy 1ddffe9f70 HIP: add byte permute fallback when __builtin_amdgcn_perm is unavailable (#30915)
* HIP: add byte permute fallback when __builtin_amdgcn_perm is unavailable

amdgcnspirv rejects target-specific builtins unless they are guarded by `__builtin_amdgcn_is_invocable`, so 66 of 145 HIP files failed to build

Wrap the perm calls in a helper that keeps v_perm_b32 when the target has it and falls back to a portable byte select otherwise

* move perm helper to vendors/hip.h
2026-10-24 12:04:05 +08:00
Xuan-Son Nguyen 7894e8b924 mtmd: move build_inp_temporal_merge to base class (#31065) 2026-10-24 03:38:25 +02:00
Yuri KhrustalevandXuan Son Nguyen edc353842c mtmd : lfm2 resize matches torchvision (#30943)
* mtmd : lfm2 bilinear resize matches torchvision

Lfm2VlImageProcessor and d1-omni vision.py resize uint8 images with
torchvision (antialias=True). That is Pillow's algorithm, except that the
weights are int16 with as many fractional bits as the largest weight allows,
where Pillow uses 22 bits. The Pillow port was 1-2 levels off on about 1% of
the values of the d1-omni and LFM2.5-VL crops.

RESIZE_ALGO_BILINEAR_TORCH rounds the weights of the Pillow port to that
precision and scales them back to 22 bits, which gives the output of
torchvision with the same passes, and "bilinear" maps to it for LFM2 and
d1-omni. Other models keep the Pillow algorithms.

* mtmd : lfm2 bicubic resize matches torchvision

Lfm2VlImageProcessor resizes with torchvision for the bicubic checkpoints too
(d1-3B, LFM2-VL), so "bicubic" maps to RESIZE_ALGO_BICUBIC_TORCH for LFM2: the
Pillow bicubic filter (a = -0.5, as torchvision with antialias) with the int16
weights of torchvision. The Pillow port was 1-2 levels off on about 0.25% of
the values of the d1-3B crops.

* use bitfield, add hparams setter

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-10-24 02:21:06 +02:00
Hans 9ae73e68f5 mtmd : add Soprano TTS support (#28607)
Add Soprano-80M and Soprano-1.1-80M (https://huggingface.co/ekwek/Soprano-1.1-80M)
as an audio generation model: a Qwen3 backbone plus a Vocos decoder that runs
as a gen-only mmproj.

- convert: detect Soprano checkpoints and export the backbone as qwen3 and the
  decoder.pth Vocos weights as the mmproj
- vocab: add the soprano pre-tokenizer (lowercase, whitespace collapse,
  per-digit split, [UNK] fallback)
- mtmd: add the Vocos graph and ISTFT, allow loading a gen-only mmproj for
  Soprano and include the gen-audio context in mtmd_get_memory_usage
- mtmd-helper: add the Soprano pipeline; the decoder runs over the whole
  utterance, so stream mode sends the audio in one chunk after flush()
- tts: document Soprano usage

Assisted-by: Codex
Assisted-by: Claude Opus 5.5
2026-10-24 01:09:50 +02:00
kurquharandMax Krasnyansky 0ae0e19766 hexagon: add SGN op (#30346)
* hexagon : add SGN support

Assisted-by: OpenCode

* hexagon : optimize SGN with sign bit

Assisted-by: OpenCode

* hex-sgn: add f16

* hex-unary: update SGN to use generic unary_colapse and relax colapse checks to apply to more conditions

* hex-unary: improve dims colapsing and run it in the host-side solver

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-10-23 14:58:44 -07:00
Adrien Gallouët 21ec307146 common : allow concurrent downloads across processes (#29312)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-10-23 23:52:56 +02:00
WalterCool 9145ddb955 hip: fix bf16 WMMA operand vector type for gfx11/gfx12 (#30885)
The __builtin_amdgcn_wmma_f32_16x16x16_bf16_w32 (and _gfx12) builtins
take their A/B operands as vectors of short (_ExtVector<16, short> /
_ExtVector<8, short>), not __bf16. Declaring the fragments as
ext_vector_type(N) __bf16 fails to compile:

  error: cannot initialize a parameter of type 'short
  __attribute__((ext_vector_type(16)))' with an lvalue of type 'const
  bf16x16_t' (vector of 16 '__bf16' values)

short and __bf16 are both 16-bit and the value is only ever passed to
the builtin via reinterpret_cast over raw bytes, so changing the element
type is bit-identical. The operand type has been short since the builtin
was introduced, so this works across LLVM/ROCm versions without a
version gate. Fixes HIP builds for RDNA3 (gfx1150/1151) and RDNA4
(gfx1200/1201).

Signed-off-by: Pablo Cholaky <waltercool@slash.cl>
2026-10-23 23:31:28 +02:00
thelittlefireman 1a24df1639 ggm-cuda: Enforce DP4A MMQ tile height multiple of warp size (#30837)
Added static assertion to enforce DP4A MMQ tile height requirement.
2026-10-23 21:37:20 +02:00
SreeramandJohannes Gäßler 9ff0d7e90f CUDA: fix build after chunked CUDA graphs (#30804) (#31053)
* CUDA: fix build after chunked CUDA graphs (#30804)

* Apply suggestion from @JohannesGaessler

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

* Fix newlines in common.cuh

---------

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
2026-10-23 20:47:20 +02:00
Christian Loehnert 06230cb877 allows the usage of 4-10 gpus for stepfun and laguna in tensor parallel (#24554) 2026-10-23 19:56:38 +02:00
Alexey Kopytko 5c31142682 qwen4exp : build the QSA selection without the kq_mask (#30965)
* qwen4exp : build the QSA selection without the kq_mask

* clean

* lang

* qwen4exp : reshape the fused QSA selection to the kq_mask streams

ggml_top_k_mask with a single keep row returns [n_kv, n_tokens], while
build_attn_qsa expects the kq_mask layout [n_kv, n_tokens/n_stream, 1, n_stream].
With several streams the shape assert failed at graph reserve.

* test the QSA optimization

* reword

* update build_fused_top_k_mask instead of handwiring

* simplify

* keep il at the end for consistency
2026-10-23 19:49:08 +02:00
Adrien Gallouët da2ee1e7a6 vendor : update LibreSSL to 4.4.0 [no ci] (#31049)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-10-23 19:12:35 +02:00
Aman Gupta 280141e6d4 CUDA: chunk graphs to amortize graph launch (#30804) 2026-10-24 00:26:16 +08:00
Max Krasnyansky 36e36e63a5 hexagon: add CEIL op (#31006)
* hex-ceil: add CEIL(f32) op

* hex-ceil: add f16 version

* hex-ops: update supported ops

* hex-ops: update docs with supported ops
2026-10-23 07:46:53 -07:00
Aleksander Grygier e1e3b2c409 ui : bump packages to clear npm audit (#31038)
Run npm audit fix, then bump the remaining vulnerable packages within
their current majors: storybook 10.6.2, vitest 4.1.12, vite 7.3.7,
@modelcontextprotocol/sdk 1.32.1, dompurify 3.4.17, mermaid 11.17.2,
@vite-pwa/assets-generator 1.0.4 and @tailwindcss/typography 0.5.20.

Override katex to 0.18.11 and postcss-selector-parser to 7.1.6 since
their dependents still pin vulnerable ranges, bump the sharp override
to 0.35.5, and pin @storybook/svelte so npm does not resolve the 11.0
alpha through the addon-svelte-csf peer range.

Storybook stays on 10.6.2 as 10.6.3 is newer than min-release-age.

Assisted by: Claude (Opus 5.5 Medium)
2026-10-23 16:33:13 +02:00
Kartik Gulia 4231b23dc2 ggml : fix time overflow in ggml_time_us (#30969)
* fix time overflow

* use double
2026-10-23 15:56:59 +03:00
Sigbjørn Skjæret f00c9083b2 ci : add behavior check workflow (#31036)
* remove python from tests

* add behavior and backend ctest labels

* add behavior check workflow

[no ci]
2026-10-23 14:53:44 +02:00
Raul Torres f5027fb213 CANN: missing const qualifiers added for consistency (#30933) 2026-10-23 15:34:20 +08:00
Titaniumtown f64a49819c sycl : pass device into mul_mat_uses_onednn (#31014) 2026-10-23 08:37:53 +02:00
Pranesh GonegandlaandPranesh Gonegandla 1ab09d632e Extend probabilistic sampling for DFlash2 drafts (#30663)
* spec : add probabilistic sampling for DFlash2 drafts

* spec : apply review suggestions

---------

Co-authored-by: Pranesh Gonegandla <pgonegandla@nvidia.com>
2026-10-23 11:15:36 +05:30
f9599d0cc8 vocab : fix Nemotron-3.5-Embed tokenization (#30974)
* vocab : fix Nemotron-3.5-Embed tokenization

* Update src/llama-vocab.h

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* rename

* Update src/llama-vocab.cpp

Co-authored-by: Daniel Bevenius <daniel.bevenius@gmail.com>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Daniel Bevenius <daniel.bevenius@gmail.com>
2026-10-23 07:31:30 +02:00
Max Krasnyansky 643b0cdacb hexagon: Op Fusion refactoring, optimizations and new fusions like BINARY_NX (back-to-back ADDs, etc) (#30843)
* hex-fusion: support for GELU+MUL, REPEAT+GET_ROWS and BINARY_NX (back-to-back ADD/MUL/...)

* hex-fusion: factor out fusion logic into separate .h and .cpp and update all related code

* hex-fusion: allow for RMS_NORM+MUL fusions for non-contig tensors

* hex-fusion: streamline fusion eligibility checks and cover missing corner cases

* hex-fusion: streamline fusion flow

* hex-opbatch: make n_tensors limit more scalable

* hex-bufs: adjust the trace buff sizing

* hex-glu: cleanup glu-ops interfaces and add kernel_params

* hex-kparams: clean up kparams handling in unary, ssm-conv, and other ops

* hex-kparams: update the rest of the ops PAD,DIAG,... to use kparams

* hex-unary: add support for collapsing dims

* hex-ops: cleanup redundant runtime checks

* hex-ops: consistent use of kernel_type in all kparams

* hex-fusion: remove redundant check in gdn+cpy

* hex-fusion: further streamlining of fusion rules and code

* hex-fusion: cleanup naming

* hex-opfmt: cleanup op formatting

* hex-fusion: bump fusion lookahead to 64

* hex-fusion: fuse SSM_CONV+SILU

* hex-fusion: fix hvx register spills in fused ssm_conv+silu

* hex-fusion: add more GLU fusions

* hex-fusion: remove duplicate checks and use ggml-common helpers

* hex-kparams: refactor kparams into htp-kparams.h/cpp

* hex-fusion: fix binary-nx fusion regression

* hex-kparams: another pass at cleaning up kparams (removing unused, etc)

* hex-kparams: streamling nameing and move _check_ wrappers into .h

* hex-kparams: templetize htp_kparams_check_<op> to reduce code bloat

* hex-ops: de-duplicate op dispatch switches

* hex-kparams: trim down exposed kparams calls

* hex-kparams: simplify type casting of kparams structs

* hex-fence: de-dup fence tensor construction

* hex-opbatch: simplify tensor shape init/checks

* hex-mm: cleanup leftovers of the original QK defines

* hex-fa: fix stack overflow on older npu firmware

* hex-ctx: move dma_caches into ops_ctx
2026-10-22 21:05:44 -07:00
Todd Malsbary 0a9ef00678 sycl: Replace host waits in fattn-mkl with explicit dependencies (#30684)
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
2026-10-22 22:43:12 -04:00
Todd Malsbary 7377bada8c sycl: Add graph record and replay (#28725)
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
2026-10-22 22:39:52 -04:00
pzhu 184f653602 server: Substitute rerank placeholder from the original template only. (#30864) 2026-10-23 00:38:03 +02:00
Xuan-Son NguyenandPascal d814cbb88c server: realtime API: send complete after committed audio drains (#30994)
* server: realtime API: send complete after committed audio drains

* server: realtime API: complete a commit at its end row

Completion is triggered when the stream runs dry, which is a timing
proxy: audio appended while the tail drains leaks into the committed
item, a final commit before the drain merges both items, and back to
back commits collapse into one.

mtmd_audio_stream_flush() records the row count at the end of the
flushed audio (mtmd_audio_stream_get_n_rows_flushed), next to the rows
produced so far (mtmd_audio_stream_get_n_rows). The slot keeps a FIFO
of these end rows and completes each commit once every row up to it is
decoded.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-10-23 00:37:47 +02:00
kurquharandLi He 970b51ce7c opencl: add depthwise conv2d (#30900)
* opencl: add Adreno depthwise conv2d

Assisted-by: OpenCode

* opencl: use local program for depthwise conv2d

Assisted-by: OpenCode

* opencl: use singed type for args

---------

Co-authored-by: Li He <lih@qti.qualcomm.com>
2026-10-22 10:54:24 -07:00
tc-mb a08ab75ae7 mtmd : add MiniCPM-OCR 1.0 support (#30979)
Signed-off-by: tc-mb <tianchi_cai@icloud.com>
2026-10-22 19:22:15 +02:00
Martin Kleinberger 19ab5a5e65 clef : include <algorithm> for std::find and std::copy (#30981)
[no ci]
2026-10-22 18:14:47 +02:00
Gaurav Garg 36cbf415f1 Fix !30793 2026-10-22 19:30:57 +05:30
Georgi Gerganov 5c19a5a756 glm5 : optimize inputs (#30932)
* glm5 : avoid sel_mask_2d input

* cont : avoid new_pool_flat input
2026-10-22 16:59:29 +03:00
Georgi Gerganov d9ee007a06 model : fix qwen4exp layer inputs for DFlash drafters (#30927)
qwen4exp published the hc-wide residual [n_embd, hc, n_tokens] as
t_layer_inp, but the drafter reads layer inputs as [n_embd, n_tokens]
rows: llama_context sizes each embd_layer_inp buffer as n_embd * n_batch
and the DFlash encoder input is n_extract * n_embd wide. Evaluating a
DFlash draft over a qwen4exp target therefore aborts on the first request
in llama_context::extract_layer_inputs.

Collapse the streams with a plain mean before exposing them, as deepseek4
already does for its hyper-connection residual, and build the mean only
for the layers a drafter actually requested.

Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
2026-10-22 16:49:38 +03:00
budiony d81d0d8fb9 Multi-column on Intel XMX matrix engines refactored (#29864)
* Multi-column on Intel XMX matrix engines refactored

Multi-column (2 to 80 activation columns) mul_mat_vec_q for reordered K-quant and Q8_0 weights on the Intel XMX matrix engines (int8 DPAS, ESIMD), for the SYCL backend. It targets speculative decoding verification (2 to 8 tokens per step) and short prompt batches (9 to 80 tokens). Three types also use it for a single column (Q2_K, Q3_K, Q8_0).

* Fixed regression bug

real bug found: the first version produced NaN whenever the row count wasn't a multiple of 4

* Removed GGML_TYPE_Q8_0 activation

* rewritten xmx_mul_mat

64-byte alignment and 16-byte pitch check on the activations

* Implemented fused gate/up kernel

One launch computes up * act(gate) for both FFN weights, reading the activations once and applying SWIGLU or GEGLU in the epilogue. It covers up to 16 columns and works for all six XMX types.

* Removed the temporary XMX_NOGLU environment switch

* Q2_K min-sum optimization

Q2_K weight is d·scl·q − dmin·mc, where scl (the 4-bit scale) and mc (the 4-bit min code) apply to each half of a 32-weight group

* Added dispatch check and GRF sweep

tg128 on Q4_K_M goes from 26.67 to 28.23 tok/s on the B70 (+5.9%).
Q6_K is unchanged.
MUL_MAT passes 1727/1727 and MUL_MAT_VEC_FUSION 1361/1361.
Perplexity at batch 1 is 5.2064 with XMX and 5.2058 without.

* Renamed variables

* Added 2 environment variables

GGML_SYCL_XMX_MAX_COLS (default 80, any value 0 or higher) and GGML_SYCL_XMX_GLU_MAX_COLS (default 16, range 0 to 16):  the largest batch for the fused gate/up kernel. It is clamped to 16 because that is the most the fused kernel is compiled for, so the variable can only lower it.

* Update esimd.hpp

1. All five quant structs were repeating the same boilerplate, so I moved it into shared pieces.

2. A new helper xmx_term does one DPAS, converts it to float and scales it per row. The four branches are now one or two lines each, and the shared Cb += z * d8r is written once.

The refactor passes all three op tests. MUL_MAT_VEC_FUSION is 1369/1369, MUL_MAT is 2281/2281 (the count includes your uncommitted f32 tests) and MUL_MAT_ID is 937/937.
2026-10-22 21:39:01 +08:00
Bill Sideris d77256999f vulkan: replace std::aligned_alloc with posix_memalign for termux compatibility (#30828)
std::aligned_alloc is unavailable on Android bionic below API 28 (Termux targets API 24); use posix_memalign like ggml.c does.
2026-10-22 14:40:13 +02:00
martin-cimminoandMartin Cimmino 0693dbfb91 nemotron : support non-standard GQA, BPE tokenizer and domyn-small pre-type (#23343)
* nemotron : support non-standard GQA, BPE tokenizer and domyn-small pre-type

* gguf-py : write Metaspace prepend_scheme as GGUF metadata

* vocab : parse metaspace prepend_scheme into enum, rename key

* vocab : use metaspace prepend_scheme in detokenize, add TODOs for other tokenizers

* vocab : use constant for metaspace escaped space

---------

Co-authored-by: Martin Cimmino <martin.cimmino@domyn.com>
2026-10-22 13:34:47 +02:00
Tim Wangandtimothywang21 a5d22276be server : support typed content input for RERANK endpoint (#29625)
* server : support typed content input for rerank endpoint

Assisted-by: Opencode/Qwen3.8 27B

* server : address rerank multimodal review findings

- reject multimodal rerank input when the model has no rerank chat template (images were silently dropped)
- translate handle_media download/base64 failures to 400 instead of 500
- reject negative top_n with 400 instead of letting resize throw
- fix comment typos, redundant comments, indentation and trailing whitespace
- README: document that TEI input accepts the same shape as Jina; drop stray blank line

Assisted-by: opencode

* Trimmed README

---------

Co-authored-by: timothywang21 <timothywang21@users.noreply.github.com>
2026-10-22 13:27:24 +02:00
Georgi Gerganov 5f5acfe250 ci : disable slow WebGPU tests (#30963) 2026-10-22 13:56:20 +03:00
Georgi Gerganov 1f4c2eeaa5 ci : be able to reconfigure HTTP port for server pytest workers (#30972) 2026-10-22 13:56:03 +03:00
Michael Taylor 7170cad4d0 sycl: fix CONV_2D wrong results for non-contiguous input (#30363)
* sycl: fix CONV_2D wrong results for non-contiguous input

The kernel indexes the `input` as a contiguous [W, H, C, N] tensor, but
supports_op claimed the op for any layout. A non-contiguous (e.g. `CWHN`
permuted) input was read with the wrong element offsets, silently
producing wrong results.

Gate the CONV_2D op on `input` and `kernel` contiguity, matching the
CUDA backend contract, and assert input contiguity in the op.

`test-backend-ops -b SYCL0 -o CONV_2D`: goes from 16 failures to 0,
none.

```text
  2006/2006 tests passed
  Backend SYCL0: OK
Backend 2/2: CPU
  Skipping
2/2 backends passed
OK
```

Assisted-by: pi (unsloth/Qwen3.8-27B-GGUF:Q4_K_M via llama.cpp)

* sycl : CONV_2D cwhn=1 support

Make the SYCL backend compute `GGML_OP_CONV_2D` correctly for `cwhn=1` (CWHN
channel-order) inputs and kernels, so the previously failing test-
backend-ops CONV_2D cases pass on SYCL instead of being skipped or
miscomputed.

- Implement CWHN layout support in `ggml/src/ggml-sycl/conv2d.cpp`
  (`ggml_sycl_op_conv2d`).
- Update SYCL `supports_op` for `GGML_OP_CONV_2D` in
  `ggml/src/ggml-sycl/ggml-sycl.cpp` to accept both layouts.

Assisted-by: pi (Greatjedi/Qwen3.8-27B-Intel-Arc-Tuned-GGUF via llama.cpp)

* sycl docs : update ops for SYCL
2026-10-22 16:19:58 +08:00
tkran2 d88040c704 vocab : add clean_spaces GGUF key (#30940) 2026-10-22 09:58:08 +02:00
tkran2 e197cef455 vocab : use Tekken pre-tokenizer for pixtral rather than the Llama 3 one (#30960) 2026-10-22 09:06:47 +02:00
Gabe Goodhart bf039c8496 models : Record layer inputs for all granite models (#30957)
Branch: GraniteSpeculatorSupport
AI-usage: full (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
2026-10-22 10:02:41 +03:00
Charles Xu dfeea52005 kleidiai : add Q8_0 SVE dotprod GEMV and SVE i8mm GEMM kernels (#30808) 2026-10-22 09:56:21 +03:00
Titaniumtown e3797b184d sycl: merge the fp16 and fp32 dequant switches (#30740) 2026-10-22 08:30:21 +03:00
Chanho Lee 0b1281aeef webgpu : add Q2_0 support (#30914)
* webgpu : add Q2_0 row dequantization

* webgpu : add Q2_0 matrix multiplication
2026-10-22 08:30:04 +03:00
Hrishith Thadicherla 4912ac24f1 dflash : load optional FC input scale (#30945)
Assisted-by: Codex
2026-10-22 08:24:12 +03:00
Aleksander Grygier 84746c5c94 ui: Add "Show tool call in progress" setting (#30681)
* feat(ui): add Show tool call in progress display setting

Add the showToolCallInProgress key and its checkbox to Settings > Display,
right next to its "always show" counterpart. Upstream dropped a setting of
this exact name in 43b5e63589, folding it into alwaysShowToolCallContent
because the old name promised expansion that outlives the call; this brings
back the narrower, in-progress-only meaning.

The entry is registry-driven, so SETTING_CONFIG_DEFAULT, the dialog field and
the ui_settings admin baseline pick it up without extra wiring. It ships
enabled, mirroring Show thought in progress, and the always-show help text is
reworded so the two toggles read as a pair rather than contradicting each
other.

Assisted by: llama-ui (ggml-org/Qwen3.8-Flash-Next:Q8_0)

* feat(ui): expand tool call blocks while they are in progress

Drive the default expansion of TOOL_CALL_STREAMING and TOOL_CALL_PENDING from
showToolCallInProgress and leave TOOL_CALL, the terminal state, to
alwaysShowToolCallContent. A finished call now folds back on its own once its
result lands, which is exactly how the reasoning block behaves when the
thought completes.

Manual toggling keeps precedence over both defaults in either direction:
isExpanded() only consults the settings while expandedStates holds no entry
for the section.

Client tests pin the truth table for the two toggles, including the automatic
fold-back and both manual overrides.

Assisted by: llama-ui (ggml-org/Qwen3.8-Flash-Next:Q8_0)

* fix(ui): keep running tool calls expanded

A call the agentic loop reports as running now outranks its section type.
exec_shell_command persists an empty result before reading any output, so a
live call is already TOOL_CALL and folded back, which hid the streaming
output the setting is meant to show.

Move the expansion default to isSectionExpandedByDefault so the running-id
rule is covered by a truth table, and test the pending -> finished transition
instead of only the finished state: a block that only ever got mounted
finished cannot prove it folds back on its own.

Assisted by: llama-ui (ggml-org/Qwen3.8-Flash-Next:Q8_0)

* feat(ui): stream the exec command from the first character

The exec parser read its args strictly, so while the model was still
typing the call the block title stayed empty and the body claimed
"Running...". Parse the args partially (the read_file / edit_file /
write_file pattern), keep a terminal cursor while the args JSON is
incomplete, and stand the conversation cwd in for the x-tool-cwd that
only arrives with the result message, so the prompt is there from the
first character. The body stays empty while the args stream, shows
"Executing command..." once the call is executing, and the wrapper
drops its padding for empty phases so the title sits flush with the
frame.

Assisted by: Delta (zai-org/GLM-5.3)

* fix(ui): stream file tool bodies even before a path arrives

write_file and edit_file refused to render until their args carried a
path, so a call that streams its body first (or a same-named tool with
other arg names) showed an empty block. Meta now carries optional
fileName/filePath, path alias reading moves to shared helpers, and the
blocks fall back to the raw args JSON when no known content key is
present.

Assisted by: Delta (zai-org/GLM-5.3)

* fix(ui): keep an empty write_file body instead of the raw args

A call carrying {"content":""} fell through to the raw-args fallback, so
a finished write of an empty file showed the args JSON. The parser now
leaves content undefined when the key is missing and the block tests
!== undefined, so an empty write renders as an empty code block with its
byte count.

Assisted by: DeepSeek-V4.1-Flash (deepseek-ai)
2026-10-22 08:14:47 +03:00
Hoang Vien Duy 0682a576a7 lora : support prism.hadamard folded weights (#30910)
* lora : support prism.hadamard folded weights

* tests : remove LoRA test from test-llama-archs
2026-10-22 08:12:17 +03:00
Aleksander Grygier a6b83d426f ui : key draft sidecar badges by model instead of repo (#30919)
Two quants of one repo pulled each other's draft: the badge read every
--model-draft arg of the repo, so each row showed both drafts.

Assisted-by: DeepSeek-V4.1-Flash-1017
2026-10-22 08:11:45 +03:00
kurquharandLi He c7fc509417 opencl: add F32-to-I32 ARGMAX (#30897)
* opencl: add F32-to-I32 ARGMAX

Assisted-by: OpenCode

* opencl: init to 0 to be consistent with cpu

---------

Co-authored-by: Li He <lih@qti.qualcomm.com>
2026-10-21 18:02:50 -07:00
alexey-rsiandSigbjørn Skjæret a37cc46104 models : add Flint model (FlintForCausalLM) (#30928)
* Add Flint model (FlintForCausalLM). Assisted by Codex

* cleanup

* one more

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-10-21 23:26:22 +02:00
Tarek Dakhran 99fb997184 jinja : fix namespace attributes shadowed by dict methods (#30941) 2026-10-21 23:06:56 +02:00
Maphist0andSigbjørn Skjæret 7775ffb7e4 model: Add support for SenseNova U1 model (#28919)
* model : add SenseNova U1 understanding support

Add support for the dense SenseNova U1 understanding branch, including
the U1.5-8B-MoT model.

- Add HF-to-GGUF conversion for NEOChatModel, excluding MoT generation
  tensors and concatenating temporal and spatial Q/K normalization
  weights.
- Register the sensenova_u1 GGUF architecture and custom spatial RoPE
  metadata.
- Implement temporal/spatial Q/K normalization with 4-axis RoPE.
- Keep U1-specific text position handling in the model implementation.
- Add synthetic architecture coverage for sensenova_u1.

Only the dense understanding branch is supported; generation
branches are intentionally rejected during conversion.

* conversion : treat one-expert SenseNova U1 configs as dense

* mtmd : add SenseNova U1 vision support

Add conversion support for the SenseNova U1 vision encoder and implement
its CPU mtmd projector graph.

The graph applies patch embedding, GELU, adjacent-pair 2D RoPE, spatial
downsampling, and dense projection for dynamic-resolution image inputs.

* mtmd : fix SenseNova U1 vision token layout

* model : clean up SenseNova U1 RoPE metadata

Replace the hard-coded spatial RoPE key with shared GGUF and LLM_KV
definitions used by conversion, model loading, and architecture tests.

Clarify why SenseNova U1 retains its custom text-position input: U1 uses
[t, 0, 0, 0], while the generic M-RoPE input uses [t, t, t, 0].

* mtmd : simplify SenseNova U1 projector integration

Map the SenseNova U1 dense projection to the standard mm.0 weight and
bias tensors and reuse the shared projector fields in the loader and graph.

Read vision parameters from the checkpoint configuration and remove the
U1-specific tensor names and model fields while preserving its strided
projection, bias, and row-major token layout.

* mtmd : fix SenseNova U1 image positions and attention

* fix spacing

* mtmd : restore generic test input chunks

* mtmd : restore SenseNova U1 integration after upstream merge

* Update conversion/sensenova_u1.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update src/models/sensenova-u1.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* model : remove SenseNova U1 BF16 activation casts

Use standard backend precision handling for QKV, FFN, and output
projections instead of casting inputs based on the output tensor type.

Assisted-by: Codex

* Update src/models/sensenova-u1.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-10-21 21:54:16 +02:00
Aman GuptaandPascal d454a204ef llama: create new fused op for QSA mask (#30640)
* llama: create new fused op for QSA scoring

* enroll glm5.3-next too

* tests : initialize the remaining tensors in test_kpool_mask

The harness sentinels were left uninitialized, so leftover memory could read as NaN and fail the NaN check.

* ggml: generalize the k-pool mask op into ggml_top_k_mask

The k-pool mask op only covered the pooled QSA selection of qwen4exp
and glm5-next, while the DSA models and the DeepSeek V4 CSA build the
same top-k mask by hand with fill, set_rows and add.

ggml_top_k_mask(mask, top_k, score, blocks, extra) keeps the mask
cells of the live picks and the extra cells, and sets the rest to
-inf. Every row tensor is 4D in the layout of top_k, and the mask is
broadcast to those rows, so a selection can also be made per head.
score, blocks and extra are optional: without score every pick is
live, without blocks a pick is a cell, and extra holds the cells a
row keeps without a pick.

llm_graph_context::build_fused_top_k_mask views the row tensors in
the layout of kq_mask and registers the fused node. The k-pool
models, the DSA attention of build_attn and the DeepSeek V4 CSA mask
use it when the probe enables the op, and keep their current path on
backends without it.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-10-22 01:02:22 +08:00
92es 0756e83fc1 sycl: q8 fast dequant (#30775) 2026-10-21 10:01:03 -07:00
alexey-rsi 9febe63d5a jinja: support hexadecimal string escapes (#30929) 2026-10-21 18:04:36 +02:00
kurquharandLi He 9b56ccf00d ggml-opencl : add IM2COL_3D support (#30895)
* ggml-opencl : add IM2COL_3D support

Assisted-by: OpenCode

* opencl: use same type for kernel and host

* opencl: fix for older adreno

---------

Co-authored-by: Li He <lih@qti.qualcomm.com>
2026-10-21 07:59:55 -07:00
Xuan-Son Nguyen 0a1ef309cd mtmd: add state object for mtmd_gen (#30902) 2026-10-21 12:59:54 +02:00
David Garcia 261700b896 server : forward llama.cpp sampling params on /v1/messages (#30856)
* server : forward llama.cpp sampling params on /v1/messages

* server : pass the request through on /v1/messages instead of copying listed keys

* NIT: Change comment from multiline to single line

* server : accept llama.cpp params under extra_params on /v1/messages

* server : remove test case for trivial functionality

* server : show extra_params in the /v1/messages README example
2026-10-21 12:58:52 +02:00
Alessandro de Oliveira Faria (A.K.A.CABELO) f849095365 vendor: update BoringSSL to 0.20261020.0 (#30889)
[no ci]
2026-10-21 11:15:54 +02:00
Evgen MykhailenkoandGeorgi Gerganov 824eaecf48 llama : optimize vocabulary loading (#30679)
* llama : optimize vocabulary loading

* Update src/unicode.h

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* CR : added reserve in gguf_kv_to_str

* CR : trailing space fix

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-10-21 12:03:27 +03:00
Aman Gupta eb182d85c0 llama : place d2t by its set_rows use so it can stay on the GPU (#30912) 2026-10-21 16:32:29 +08:00
Niklas Wenzel ea9e3446b5 metal : fix memory leaks and cleanup logic in GPU capture path (#30890) 2026-10-21 09:54:33 +02:00
Max Krasnyanskyandhubezhou 0efe8e4c6b hexagon: neon-vectorized and threaded repack (tiling) (#30901)
* add vectorized q4_0 repack

* apply to more types

* hex-repack: neon-vectorize the rest of the types q1/q2/f16/...

* hex-repack: add simple chunking / threading

---------

Co-authored-by: hubezhou <hubezhou@qti.qualcomm.com>
2026-10-21 10:48:39 +03:00
lhez b531e4c749 opencl: add dp4a GEMM variants to vector load activation (#30753)
* opencl: add dp4a GEMM variants with vector activation loads

* opencl: rename kernels and env vars
2026-10-21 10:48:13 +03:00
Neo Zhang 74b115ca6e sycl : fix fa crash for Query head dimension mismatch (neq0 != HSK / D=2048/4096) (#30286) 2026-10-21 10:46:43 +03:00
Titaniumtown 96caa80059 sycl: collapse duplicate need_check launch bodies in mmq (#30739) 2026-10-21 10:46:00 +03:00
bosh 479a52779f ggml-cpu : compute f32 gelu_quick and geglu_quick without the fp16 table (#30737)
* ggml-cpu : compute f32 gelu_quick and geglu_quick without the fp16 table

* ggml-cpu : keep the GGML_GELU_QUICK_FP16 table path

* ggml-cpu : add the VXE gelu_quick path

* tests : add test-gelu-quick for the f32 CPU kernels

* tests : drop inf/nan inputs from test-gelu-quick
2026-10-21 09:34:37 +02:00
Georgi Gerganov 8baabd53b7 llama : scale moe down projection per expert token (#30861)
* llama : scale moe down projection per expert token

Replace the global amax rescaling for the MoE down projection with a per-expert-token factor computed by a POOL_1D max reduction along the intermediate dimension. Undo the scale on the output with the same exact power-of-two factor. Handle POOL_1D in the Meta split-state logic so a split along axis 0 becomes a replicated per-device partial result, enabling tensor parallelism.

Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)

* cont : simplify comments

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* llama : refactor amax src1 scaling result

Return the rescaled activation and the optional scale tensor through a small struct instead of an output pointer.

Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)

* cont : add TODO + clean-up
2026-10-21 09:01:00 +03:00
Anant Shrivastava 493253fe5a Sycl iq3 crash fix (#30604) 2026-10-21 01:04:13 -04:00
Ben Ashbaugh 339f3242c7 opencl: fix rope program build error with newer Clang (#30893)
Newer versions of Clang diagnose an incompatible pointer type
error when assigning a pointer to float to a pointer to half.
2026-10-20 17:19:08 -07:00
Unbug Lee 9a3b91c680 server : fix usage.completion_tokens undercount with n > 1 (#29486)
Assisted-by: Copilot
2026-10-21 01:19:54 +02:00
Xuan-Son NguyenandPascal 5c0b49a23b server : add POST /tts endpoint for TTS models (#26603)
* mtmd: add audio out stream api

* add /tts endpoint

* add docs

* wire up mtmd_helper_model_can_chat

* upload speaker_ref via form-data

* expose seed to api

* fix header for pcm

* wip

* fix penalty_last_n

* fix server_output_limits

* fix pocket-tts stream

* wip

* add server_schema for tts

* add server routing

* server: coerce seed and stream from multipart form fields

Form fields arrive as strings, so -F seed=42 and -F stream=true were
rejected by the schema. seed joins the integer fields, parsed as a long
long so a full uint32 seed fits, and stream is read as a boolean when
it comes as a string.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-10-21 01:19:42 +02:00
Xuan-Son NguyenandPascal 1242be66f6 server: add realtime transcription API (#30636)
* server: add realtime transcription API

* drop transcription_sessions

* proxy ws

* add docs

* server: let the API key middleware guard the websocket

httplib runs the pre-routing handler before the websocket upgrade, so
the API key middleware answers 401 at handshake like any other route.
This covers the router proxy, whose children run keyless, and accepts
the same key forms as HTTP (Bearer, raw key, X-Api-Key).

* server: keep realtime slots out of the prompt loop

A realtime slot waiting for audio is GENERATING with nothing to decode,
so it fell through the prompt loop and took slot_batched. A request on
a higher slot then failed can_batch_with on the task type and starved
for the whole session. Realtime slots have no prompt: the loop sets
them up once and always leaves them.

* server: start the realtime session once its slot is live

The audio stream was created in the prompt loop, which skips a slot
that cannot batch with the one generating, and session.created went out
before the task was posted: early appends failed and the reader quit,
leaving the socket open and the slot held until the HTTP timeout.

The slot now sets up its stream at launch and signals it with the begin
result, session.created waits for it, and a reader that stops closes
the socket so the session ends with it.

* server: name the committed item in input_audio_buffer.committed

Per the Realtime spec, item_id is the item made from the committed audio
and previous_item_id the one committed before it, null for the first.
committed now comes before the transcript that completes that item, and
both go out with the item switch under one lock, so a delta cannot land
between them.

* server: use the GA transcription session shape

The session object follows the GA Realtime schema: type transcription,
with the input format, transcription model, turn detection and noise
reduction under audio.input. The rate stays the model sample rate, which
is what the client must send for now.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-10-21 00:04:15 +02:00
Tarek DakhranandGeorgi Gerganov 84e66d0321 refactor : move LFM2 shortconv and FFN blocks into a shared LFM base (#30874)
* llama : move LFM2 shortconv and FFN blocks into a shared LFM base

* Update src/models/lfm-base.cpp

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-10-20 23:17:13 +02:00
Takamichi Ohara 52ffa02745 model : support Nemotron-3(.5)-Embed (#30877) 2026-10-20 22:48:45 +02:00
Aman Gupta 839a572a4c qwen: reshape output to 2d instead of 3d (#30823)
* qwen: reshape output to 2d instead of 3d

* remove final reshape, fix comments
2026-10-20 22:09:56 +03:00
ead9f32058 server: Support logprobs with speculative decoding (#27196)
* return logprobs in spec path

* server: add regression test for logprobs with speculative decoding

Asserts accepted draft tokens report the target model's logprobs,
comparing a draft-enabled run against a no-draft baseline (#24271).
Fails on master (top_logprobs empty, logprob 0.0), passes with the
fix from #27196.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KARBWpPcijcBXzJEvttuwk

---------

Co-authored-by: Jeonghyun Lee <sig.jhl@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-10-20 19:36:14 +02:00
Tarek Dakhran b684b498b6 hexagon: fix VTCM size query on images that need layout out-params (#30655) 2026-10-20 10:22:08 -07:00
Yuri KhrustalevandXuan Son Nguyen 4a45de4f2c mtmd : lfm2 tiles and layout match transformers (#30261)
* mtmd : lfm2 tiles stretch the image to the grid

Lfm2VlImageProcessor.crop_image_to_patches resizes the image straight to
(512 * rows, 512 * cols) before it cuts the tiles. llama.cpp kept the aspect
ratio and added black bars for LFM2 (only d1-omni stretched), so the tiles of
d1-3B and every LFM2-VL / LFM2.5-VL model differed from transformers unless
the image had the aspect ratio of the grid. With PAD_NONE the tiles are the
same as in transformers.

* mtmd : lfm2 layout with Python rounding in double precision

Lfm2VlImageProcessor.smart_resize rounds with Python's round (half to even)
and computes beta and the tile grid aspect ratio in double. llama.cpp used
std::round and float, so 31926 of the 16.7M sizes in [1,4096]^2 got another
thumbnail size or grid than transformers (and d1-omni vision.py), e.g.
400x600 -> 416x608 instead of 384x608 and 720x1600 -> 2x5 tiles instead of
2x4. The lfm2 smart resize and grid choice now follow the Python code and
match it on every size of that range.

* fix

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-10-20 18:54:46 +02:00
7e3eeee048 model : add H2O-Lightning-4B decision model (#30361)
* model : add H2O-Lightning-4B decision model

Adds the h2o-lightning decision type for h2oai/h2o-lightning-4b
(Qwen3.5-4B with a merged decision LoRA).

None of the existing types match this model. It answers with " A"
rather than "A", uses up to 255 labels, and was trained on Python
json.dumps text, so floats like 12.0 must render as 12.0.

The converter writes the decision metadata and the systemone template
from the model's serve_config.json and refuses a config that doesn't
match. Tests skip unless the tiny test model is available locally.

Assisted-by: Claude Opus 5.5

* rm redundant code

* put labels to gguf

---------

Co-authored-by: Jonathan C. McKinney <jon.mckinney@h2o.ai>
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-10-20 18:54:27 +02:00
Xuan-Son Nguyen 5ba3b9fd3d mtmd: fix gemma 4 ffn act (#30882) 2026-10-20 18:33:32 +02:00
TANGBUDUandTANGBUDU 5a9a34d888 server : avoid repeated mmproj memory margin addition on sleep/resume (#28233)
Co-authored-by: TANGBUDU <TANGBUDU@users.noreply.github.com>
2026-10-20 18:32:12 +02:00
Tarek Dakhran 39da130fcd ggml-cuda : support ssm_conv kernel size 7 (#30873) 2026-10-20 17:47:44 +02:00
Me42019 4e201388c3 server: Router wasn't propagating the client disconnects fully to the child, the response wait could cause a delayed cancellation (#29707)
* Router wasn't propagating the client disconnects fully to the child, the response wait could cause a delayed cancellation

* tests: satisfy router disconnect type check

* server : remove timing-sensitive router cancellation test

Assisted-by: Codex
2026-10-20 17:43:45 +02:00
Renzis 1ba568c5ad server: fix missing output_index in Responses API SSE events for compatibility (#30863)
* server: include output_index in Responses API SSE events

* server: reuse Anthropic streaming code for fix output_index

* server: remove unnecessary generated_tool_call_ids and oai_resp_tool_calls_started
2026-10-20 16:17:06 +02:00
Tarek DakhranandMichael DM 36140dbee2 mtmd : read LFM2 tiling params from GGUF metadata (#30840)
* mtmd : read LFM2 tiling params from GGUF metadata

Write min_tiles, max_tiles, tile_size and max_pixels_tolerance from
processor_config.json when converting LFM2-VL mmproj, read them in clip.cpp
with the previous constants as defaults. do_image_splitting=false maps to
min_tiles=max_tiles=1 (no tiling).

Rebased version of #25524, original implementation by Michael DM (@michael-dm).

Co-authored-by: Michael DM <usable_siren180@simplelogin.com>

* fix test

---------

Co-authored-by: Michael DM <usable_siren180@simplelogin.com>
2026-10-20 16:05:30 +02:00
Georgi Gerganov df11702136 metal : optimize argmax and pool_1d (#30862)
* metal : optimize argmax and pool_1d

Add a two-pass partial reduction for large-row argmax and use cooperative threadgroup reductions for pool_1d. Allocate scratch space for the argmax partials through the Metal buffer allocator.

Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)

* tests : add pool_1d perf cases

Add representative POOL_1D max cases for the MoE down-projection amax shapes, reducing the whole intermediate dim.

Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)

* cont : minor fix
2026-10-20 16:46:00 +03:00
Georgi GerganovandSigbjørn Skjæret 9df0b73c70 model : drop the redundant precision overrides in the Kolibri-1 router (#30865)
* model : drop the redundant precision overrides in the Kolibri-1 router

Follow-up to the review in #30673: the router matmul works with the
default precision, so both ggml_prec_set_* calls can go away.

Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)

* cont : fix comment

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-10-20 15:58:09 +03:00
Sigbjørn Skjæret b38960f520 ci : remove superfluous webgpu jobs (#30868) 2026-10-20 14:27:55 +02:00
lhezandHongqiang Wang a01ce9ffc6 opencl: add several gemm variants for small batches and some fixes (#30850)
* opencl: fix OOB read in `mul_mv_mxfp4_f32_flat`

* opencl: remove large-N guard that forces CPU weight placement

* this condition claims that large batch produces incorrect results on
  adreno and cannot be reproduced
* model loader uses N512 to probe op support, resulting in cpu fallback
* thus this condition does not seem necessary

* opencl: cap num of subgroups by workgroup size for q4_0 mc3

Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>

* opencl: add small batch variant for q4_k dp4a GEMM

Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>

* opencl: add a narrow variant of mul_mm_f16_f32_l4_lm

Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>

* opencl: add a one subgroup variant for gemv_noshuffle_q1_0_f32 for big M

Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>

---------

Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
2026-10-20 14:37:28 +03:00
Neo Zhang 8115b4c31f sycl : replace memset due to not always ordered (#30812) 2026-10-20 14:35:48 +03:00
Bernard Ladenthin 119f68d7d4 ggml-cpu : select the NEON fp32->fp16 conversion (#30505)
The __ARM_NEON branch defines both GGML_CPU_COMPUTE_* macros but selects
only FP16_TO_FP32, so FP32_TO_FP16 falls through to the portable
software conversion while neon_compute_fp32_to_fp16, a single fcvt,
stays defined and referenced by nothing. __POWER9_VECTOR__ and
__riscv_zfhmin select both.

Add an #error pair that refuses a branch defining a COMPUTE macro
without selecting it. That is how this stayed unnoticed: the fallback
silently supplies a correct software implementation, so the only
symptom is a hardware instruction nothing ever reaches. The
FP16_TO_FP32 half carries a !defined(__F16C__) carve-out, because PR
30343 keeps that direction on the lookup table on purpose.

Replace the comment above the fallback: it singled out NEON and POWER9
and pointed at a selection that now happens in the branch itself.
2026-10-20 14:35:06 +03:00
uvos b684498579 HIP/CUDA: Fix reduction in mmf getting tile widths not devisiable by 64 (#30836) 2026-10-20 14:32:38 +03:00
Aaron Teo 522f2b6eed ggml-zdnn: introduce flash attention op (#30512)
* ggml-zdnn: add scratch buffer pool

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* ggml-zdnn: implement fattn, ditch scratch slots in favour of scratch buffer pool

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* docs: add note about fattn

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

---------

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
2026-10-20 14:31:35 +03:00
Peter SiderisandPetros Sideris 83c2d012a2 spec : save the dynamic ngram cache file (#22055)
* spec : save the dynamic ngram cache file

* save the cache at the common_speculative_impl_ngram_cache destruction
* remove save_static/save_dynamic flags as well as the relevant TODO
* if the dynamic cache file does not exist, start with an empty cache
  that is written on exit. if it exists but cannot be read, abort
  instead of overwriting it.

* spec : fix draft_one not updating the ngram cache history correctly

* common_ngram_cache_update received only the new tokens with none of
  the tokens before them causing ngrams ending at the new token to not
  being formed, leading the context cache to only ever hold the prompt

* fix: pass the full history to common_ngram_cache_update with nnew set
  to the number of new tokens the way llama-lookup does and also reset
  cache_size to 0 on each new request

---------

Co-authored-by: Petros Sideris <petros.sideris@nokia.com>
2026-10-20 14:31:10 +03:00
Bálint Bokros 13786fbb3a CUDA: Clear CUDA graphs after legacy memory pool flush (#26771)
Assisted-by: Claude Opus 5.5
2026-10-20 14:30:50 +03:00
Peter van VelzenandXuan Son Nguyen d80619397c server : pass id_slot through the Anthropic /v1/messages conversion (#28554)
* server : pass id_slot through the Anthropic /v1/messages conversion

server_chat_convert_anthropic_to_oai copies a small whitelist of common
params from the Anthropic request into the internal OAI-style body and
left id_slot out. Every other completion endpoint honours id_slot; a
proxy that pins a session to the slot holding its KV cache needs the
same on /v1/messages, or it silently falls back to automatic slot
selection and, on hybrid/recurrent models, pays a full re-prefill.

Adds a test asserting a request with id_slot set is served by that
slot, observed via GET /slots.

* server : shorten comments

* rm test

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-10-20 13:23:41 +02:00
Caladrel 704366d76f metal : fix missing MoE kernel variant for 9 experts per token (#30610) (#30699) 2026-10-20 13:02:05 +02:00
Christian Kastner 707616feb5 ci: Fix --strict mode for API check (#30623)
abi-compliance-checker was running in strict mode unconditionally,
rather than honoring the CLI flag.
2026-10-20 13:38:45 +03:00
tkran2 d8d92973c8 vocab : add \p{M} to lookaheads in Tekken and GPT-4o pre-tokenizer regexes (#30845) 2026-10-20 10:37:46 +02:00
Georgi Gerganov 7879491cf4 tests : make MoE router selection independent of the backend noise (#30818)
* tests : make MoE router selection independent of the backend noise

The router scores are in the range [0, 1), so a random draw can leave the
last expert (or expert group) still selected and the first one dropped tied
within the noise between the backends. They then route a token to different
experts and the logits differ by 4e-3 instead of the usual 5e-7.

Beam is the only arch here with a real top-k (8 experts, 2 used) and a group
top-k (4 groups, 2 used), so it is the only one that hits this: 2 of 20 seeds
failed on an M2 Ultra, the other MoE archs select 2 of 2 and cannot tie.

Fill the selection bias with a ramp of step 2 to give every expert a score
range of its own, which turns the top-k into a function of the expert index
alone. The bias affects only the selection, the weights still come from the
unbiased probs, so the MoE output keeps its magnitude. Beam now matches the
CPU reference within 4e-7 for all of 160 seeds.

Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)

* tests : tell the CPU and the GPU of an Apple SoC apart in the device table

Both devices report the same description ("Apple M2 Ultra" here), so the
device column of the test table had two indistinguishable rows and a failing
row did not say whether the Metal device or the host CPU produced it.

Add the device name, which is also what -b takes, unless the description
already is the name.

Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
2026-10-20 10:40:15 +03:00
Max Krasnyansky 8466395353 hexagon: update to snapdragon-toolchain v0.8 and enable hexagon v85 (#30854)
* hex-build: bump toolchain, enable v85

* hex-build: bump toolchain to v0.8 in the CI and build scripts

* hex-build: add a little python script for swiv needed on some iot devices

* hex-build: fix flake8 warnings

* hex-arch: update version cap to v85

* hex-inspect: flexible TOOLS_ROOT

* hex-build: update to snake_case naming

* hex-build: update leftover old toolchain refs

* hex-target: update build and run scripts to support ubuntu with adb transport
2026-10-19 23:49:59 -07:00
Michael Taylor dd002b91f5 sycl: fix compiler warnings - missing prototypes, dead code, extra semicolons (#30271) 2026-10-20 00:47:25 -04:00
aiqubits 1d88cb05f1 ui: hide the native clear button in SearchInput (#30719)
SearchInput already renders its own clear button.
Hide the browser's native search cancel button to avoid showing both controls.

Fixes #28627

Signed-off-by: aiqubits <aiqubits@hotmail.com>
2026-10-20 11:58:38 +08:00
Sam MurdochandSam 3bb63cbcbf sycl : fix get_rows on reordered quantized weights (#30430)
Co-authored-by: Sam <sam@garagellm.tail7dd858.ts.net>
2026-10-20 07:47:21 +08:00
bosh f711c8f303 mtmd : match the transformers Qwen-VL resize (#30736) 2026-10-20 01:14:40 +02:00
Mendy Berger 921257f500 ci, docs: add WASI CI and docs (#30754)
* ci, docs: add WASI CI and docs

* accept review feedback
2026-10-20 01:09:55 +02:00
Sigbjørn Skjæret 3ffb62935b ci : re-enable type/lint-check for skills (#30713) 2026-10-20 01:03:03 +02:00
7adf160c41 server: add --connect, allow p2p access to server outside LAN (#28277)
* init

* wip

* wire it up

* make it clear that it's experimental

* add props.connect_code

* nits

* fix windows build

* server: do not advertise a connect code without the tunnel

--connect-code on its own had no effect other than being served in
/props, so the Web UI showed a remote access code that no tunnel was
serving. Clear it when --connect is off, and say so once at startup.

The teardown comment claimed that terminate() closes the child stdout.
It does not: the log thread leaves fgets() only once every writer of
that pipe is gone, so a descendant of llama-connect would block the
join forever. Describe the real contract instead.

Also restore the --reasoning-preserve default in the generated docs.

* Update tools/connect/CMakeLists.txt

Co-authored-by: Adrien Gallouët <angt@huggingface.co>

* server: dial llama-connect on the bound port and a reachable host

Pass ctx_http.port so --port 0 and the GCP override work, pick the
dial host from --host (wildcards map to loopback, IPv6 is bracketed)
and refuse --connect when only unix sockets are bound.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
Co-authored-by: Adrien Gallouët <angt@huggingface.co>
2026-10-20 00:23:14 +02:00
A-Raphie ab7ed0d4b5 ui : fix attachment text preview clipped when a line is longer than the container (#30806)
In the attachment preview, SyntaxHighlightedCode's wrapper (overflow-auto) was not the element that ended up owning horizontal overflow: the inner code box kept its content clipped at the wrapper's width, so a single line longer than the preview pane was cut off with no scrollbar (scrollWidth == clientWidth while the code's own content measured tens of thousands of px).

Give the pre/code boxes their intrinsic width (width: max-content, min-width: 100%) so long-line overflow propagates to the scrolling wrapper. Verified in the running WebUI: with a 6000-char line, wrapper scrollWidth goes from 990 (clipped, unscrollable) to 50449 with a working horizontal scrollbar; short content still fills the container exactly as before (min-width: 100%).

Fixes #30796
2026-10-20 00:22:14 +02:00
Kashif RasulandSigbjørn Skjæret 4c3c5810f8 hybrid dna tokenizer fixes (#30825)
* vocab : support Carbon-14B HybridDNA tokenizer

Carbon-14B auto-tags untagged DNA runs and uses N-padded tail tokens
(tail=pad_n). Also strip whitespace inside <dna> tags and handle a stray
</dna> like the HF tokenizer.

* convert : fix special tokens and merges for older Carbon models

Carbon-500M/3B/8B don't register the Qwen added tokens and ship no
tokenizer.json, so take the token types, merges and special ids from the
base tokenizer.

* Update conversion/base.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* Update src/llama-vocab.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* Update src/llama-vocab.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* Update src/llama-vocab.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* Update conversion/base.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* Update conversion/base.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* Update conversion/base.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* address review comments

* Update src/llama-vocab.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
2026-10-19 23:01:40 +02:00
lhez 1b29621bac opencl: add cooperative-k dp4a gemm for q4_0, q4_k and q6_k (#30741)
* opencl: add cooperative-k dp4a GEMM for q4_0

* opencl: add cooperative-k dp4a GEMM for q4_k

* opencl: add cooperative-k dp4a GEMM for q6_k

* opencl: fix alignment
2026-10-19 12:30:39 -07:00
lhez 6fa4f6cd3e opencl: add get_rows for i32, cpy for i32 and f32 (#30756)
* opencl: support i32 GET_ROWS

* opencl: support i32 and f32 copy conversions
2026-10-19 12:22:34 -07:00
Aldehir Rojas 0ce8cd1141 jinja : implement unique filter (#30819)
* jinja : implement `unique` filter

* cont : inline key_of

* cont : fix alignment
2026-10-19 12:18:24 -05:00
Tarek Dakhran f174553e16 server : fix swa checkpoint invalidation logic (#30666)
* server : fix swa checkpoint invalidation logic

Assisted-by: Claude

* add testcase that fails with > n_past, but works with > n_prefix
2026-10-19 15:58:14 +02:00
Alessandro de Oliveira Faria (A.K.A.CABELO) 7951f601e8 vendor : update cpp-httplib to 0.62.0 (#30785) 2026-10-19 20:23:23 +08:00
Evan Huus 33d6135cbe moe-cache : replace MoE LRU cache with a 50/50 SLRU cache (#30637) 2026-10-19 13:28:50 +02:00
Georgi Gerganov 098ce526b5 metal : fix moe_reduce fusion race when dst aliases its inputs (#30795)
The fused moe_reduce kernel computes dst = sum(experts[e] * weights[e]) in a
single dispatch, reading experts and weights while writing dst. The graph
allocator computes tensor liveness on the unfused graph, where the inputs are
fully consumed by earlier nodes of the fused group, so it can legally recycle
the input memory as dst. In the fused kernel this becomes a
threadgroup-vs-threadgroup race that produces rare non-deterministic wrong
results (observed with test-llama-archs -a kolibri on the meta backend).

Reject the fusion when dst overlaps either input, same guard as the
mul_mat_add fusion. The unfused MUL + ADD path is always correct.

Assisted-by: pi:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (2x DGX Spark (RDMA, TP=2))
2026-10-19 11:55:02 +03:00
Raul Torres f3fb2fcdbc cann : refactor #ifdef blocks to avoid unreachable code after return (#25273) 2026-10-19 11:17:54 +03:00
Aaron Teo fda00f18cd tests: add big-endian conversion tests (#30705)
* ci: add quantize to big-endian tests

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* ci: remove incoherent data types from testing

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* tests: add verified hashes

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* tests: rename

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* tests: add result table

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* tests: swap hash and result column, add colours

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* tests: switch to perplexity test

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* tests: update output table

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

---------

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
2026-10-19 10:14:58 +02:00
Johannes Gäßler 4866db350d CUDA: fix warning for nonexistent extra semicolon (#30774) 2026-10-19 10:08:21 +02:00
Aaron Teo 42b70032b1 ggml-cpu: vectorize ggml_v_silu and ggml_v_expf (#30791)
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
2026-10-19 09:59:49 +02:00
JoelandJohannes Gäßler 4c1004c492 cuda: cap MMQ Stream-K blocks at available work (#30553)
* cuda: cap Stream-K blocks at the number of k-blocks

Stream-K normally launches one CUDA block per SM. For small matrices,
this can exceed the total number of output-tile/k-block pairs, leaving
blocks with no work and adding launch and fixup overhead. Cap the grid
at that count to remove these excess blocks without changing the useful
rounded K ranges or numerical results.

Assisted-by: Codex

* cuda: cap Stream-K blocks at the number of k iterations

Cap the grid at the number of K_vram iterations across all output tiles, rounding up partial iterations. This removes extra CUDA blocks left by the k-block cap.

Partial iterations can change FP32 summation grouping. MUL_MAT and MUL_MAT_ID backend tests pass on RTX 5090.

* Update ggml/src/ggml-cuda/mmq.cuh

---------

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
2026-10-19 09:41:14 +02:00
fzowlandSigbjørn Skjæret 412a363c65 qwen3 : bidirectional attention and dense_2 output projection (#30607)
* qwen3 : bidirectional attention and dense_2 output projection

Assisted-by: Grok

* convert : read use_bidirectional_attention in base

Assisted-by: Grok

* Update conversion/base.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-10-19 09:29:02 +03:00
Aleksander Grygier 2b57148b08 ui: Fix markdown tables and lists being cut off on phones (#30687)
* ui : keep markdown tables inside the chat column on phones

The phone rule made the table wrapper full-bleed with -1rem margins, so the
first and the last column sat under the screen edge. Keep the wrapper inside the
chat column and cap it at its width, so a wide table scrolls in place instead.

Assisted-by: llama-ui (ggml-org/Qwen3.8-Flash-Next:IQ4_NL)

* test : show a table-only model answer in the chat message story

Adds a story with an answer that is mostly a table, so the table layout can be
checked at a phone width.

Assisted-by: llama-ui (ggml-org/Qwen3.8-Flash-Next:IQ4_NL)

* ui : indent markdown lists by 2rem

1.5rem put the marker too close to the item text.

Assisted-by: llama-ui (ggml-org/Qwen3.8-Flash-Next:IQ4_NL)
2026-10-19 09:28:16 +03:00
ppAlexa 8f97d82da3 tools : reject truncated GGUF tensor data in split and export tools (#30497)
* The problem here in the repo is gguf-split can silently produce corrupted tensor output when the input GGUF file is truncated. The fix is to add tensor-data bounds checks and short-read checks in gguf-split.cpp and export-lora.cpp, and a regression test that checks truncated input is rejected and valid tensor data is preserved.#

* Bug : GGUF-split can silently produce corrupted tensor output when the input GGUF file is truncated. Fix : add short-read checks in gguf-split.cpp (in copy_file_to_file in split_strategy) and export-lora.cpp (in read_tensor_data of file_input) so that truncated input is rejected.
2026-10-19 09:27:46 +03:00
Georgi Gerganov 27af58168a tests : update Metal fusion baseline for beam (#30764)
* tests: add beam fusion rows to MTL baseline

Regenerate tests/fusion/MTL.csv with test-fusion --record. The newly
added beam architecture has no baseline rows, so test-fusion fails with
2 failing rows (MUL+ADD x6, RMS_NORM+MUL x25). All other arch counts are
unchanged.

Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)

* skills: add update-fusion-table skill

Add a skill for regenerating a per-device test-fusion baseline. It keeps
the workflow from tests/fusion/README.md and the parts that are easy to
get wrong: diff the recorded file before applying it, the distinct
meanings of a FAIL row, and never recording a partial run.

Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
2026-10-19 09:11:38 +03:00
Georgi Gerganov 7db0d3d781 pi : turn .pi/gg into a system prompt addendum (#30762)
.pi/gg/SYSTEM.md repeated the full agent ruleset. Move it to .pi/gg/APPEND_SYSTEM.md and keep only the llama.cpp-specific rules, so it can be appended to a main system prompt with --append-system-prompt.

Drop the .pi/SYSTEM.md symlink setup from wc2wt.sh and pr2wt.sh. Worktrees no longer need it now that the folder holds an addendum instead of a full prompt.

Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (RTX 5090 32GB, DDR5 128GB)
2026-10-19 09:11:13 +03:00
Paul MairhoferandJohannes Gäßler 99db719bdf CUDA: fix shared-memory WAR hazard in flash_attn_ext_vec (#30573)
* CUDA: fix KQ write-after-read race in flash_attn_ext_vec

Assisted-by: ChatGPT

* CUDA: apply review suggestions for KQ sync comments

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

---------

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
2026-10-19 13:25:04 +08:00
Max Krasnyansky 31288c9c66 hexagon: FLASH_ATTN_EXT improvements for longer context and SWA models (#30731)
* hex-fa: support for masked block skipping to reduce DMA fetching

* hex-fa: finetuning the hmx->hvx crossover thresholds (boosting gemma-4)

* hex-fa: update hmx gate to account for mdev.count
2026-10-18 18:08:11 -07:00
Jhen-Jie Hong fe8841a78a hexagon: allow multistream FA (#30218) 2026-10-18 18:07:53 -07:00
Aman Gupta ff5a233767 cuda : fuse qwen4exp's SCALE -> SIGMOID -> SCALE -> hc_post chain (#30602)
Port of the Vulkan HC_POST_GATE fusion (#29520). hc_combine builds the
hc_post weights as 2*sigmoid(inject/hc); dsv4_hc_post_f32 now takes a
gated template flag and computes scale_out*sigmoid(scale_in*p) from the
raw weights, so the chain runs as one kernel instead of four.
2026-10-19 01:48:47 +08:00
Andrea Rossetti 0878f7c304 ggml: support non-contiguous SUM (CPU, CUDA, Vulkan, WebGPU) (#28851)
* ggml: support non-contiguous SUM (CPU, CUDA)

* tests: add non-contiguous SUM cases

* tests: SUM over non-contiguous views

* vulkan: support non-contiguous SUM

* webgpu: support non-contiguous SUM
2026-10-18 18:57:36 +02:00
thelittlefireman 5aeac37a35 ggml-cuda: fattn do not check oob if k_VKQ_max = (k_VKQ_max - nbatch_fa) (#30572) 2026-10-18 10:59:06 +02:00
Todd Malsbary 547208dca2 Restore ESIMD kernels on FFN path (#30239)
* Reenable ESIMD on the FFN path

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>

* Extend ESIMD fusion on FFN path to include Q8_0

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>

* Remove comment explicitly referencing ESIMD in test-backend-ops

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>

---------

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
2026-10-18 00:02:38 -04:00
Todor BoinovskiandMax Krasnyansky 848bbda904 hexagon: add support for hadamard MUL_MAT and GET_ROW for flat and tiled Q1_0 and Q2_0 (#30627)
* hex-hadamard: add support for hadamard ops

* hex-mm: reduce pkt counts in the q8_1 quantizer and vec_dot_q1

* hex-q1/q2: add support for flat and tiled GET_ROWS for Q1_0 and Q2_0

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-10-17 19:17:33 -07:00
Todor BoinovskiandMax Krasnyansky 797de63b2f hexagon: add support for Q1_0 and Q2_0 (#30122)
* hexagon: add support for q1_0 quant

* hexagon: enable Q1_0 weights in the 2D HMX MUL_MAT path

* hex-q1: cleanup gates, generalize tiling and remove in-loop divs

* hex-tiling: document tiled formats in developer.md and update htp-tensor.h

* hex-q2: add support for Q2_0

* hex-q2: remove stale asserts and speacial cases

* hex-mm: fix MUL_MAT_NX assumption about square tiling, Q1/Q2 have non-square tiles

* hex-q1/2: minor optimizations (unrolls/un-unrolls/...)

* hex-q1: optimize Q1_0 layout to reduce unpack/vrmpy overhead

* hex-q1/q2_0: further optimize vec_dots

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-10-17 13:36:38 -07:00
thelittlefireman 9d1878a435 ggml-cuda: HIP: use __builtin_amdgcn_perm for q1_0 unpacking (#29927) 2026-10-17 19:04:39 +02:00
c0fd786264 ui : Discover & Download Models from HF Hub (#29584)
* common : resolve <quant>-<sidecar> download tags and list cached sidecars

A Q4_0-mtp style tag now resolves the sidecar file when no model file matches it, so a solo draft or mmproj download actually pulls the file. Cached sidecar files list as their own entries so the state survives a restart, and removing such a tag deletes only the sidecar.

Assisted-by: pi:zai-org/GLM-5.3

* ui : optional sanitized raw HTML in markdown

Add an allowHtml prop to MarkdownContent: raw HTML found in the markdown is
rendered after DOMPurify sanitization instead of being escaped as literal
text. Default stays escaped.

Assisted-by: pi:GLM-5.3-Flash

* ui : add the discover models components

The Discover surface: the curated-catalog list with its search and
skeletons, the details pane with its readme, metadata, Hub stats and
download options, and the standalone download progress bar that replaces
the plain one. The quant download button sizes its select with a new xs
trigger, so the select gains that size and the muted box look. The store
stays incomplete when a repo fails, so the next mount retries it.

Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash

* ui : put the discover view in the models manager

The manager dialog gains its second view: the Discover Models call to
action fades the table out, slides the title with its compass mark, and
fades the discover surface in; the arrow button returns to the table.

Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash

# Conflicts:
#	tools/ui/src/lib/components/ui/button/button.svelte

* ui : add the Enable Discover Models setting

The flag gates browsing and downloading HuggingFace GGUF models, so it is
declared on the discover layer; the repo-tree lookup in the manager layer
above reads it from here.

Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash

* ui : resume model downloads from the pause point

A stopped download keeps its partial file instead of deleting it, so
re-posting the tag resumes from disk. The first progress record now
carries the resume offset, the paused snapshot survives into the
resumed entry, and a pause settles as soon as it is requested.

Assisted-by: pi

* ui : lead the discover list with downloads

The discover sidebar lists running downloads above the suggested models,
and steps aside while a query is active. A paused download stays in the
manager table, where it is resumed or dropped.

Assisted-by: pi

* ui : make resuming a paused download work

Re-posting the tag of a paused download hit the router's "already
exists" check, because a paused download parks its entry as DOWNLOADED
until the next reload: a download entry that is not running is now
replaced and resumed. The UI keeps the paused row when the request still
fails, instead of dropping it and looking finished.

Assisted-by: pi

* ui : remember unfinished model downloads

A partial file is named after its content hash, so nothing on disk says
which model it belongs to once the router restarts. The router now keeps
the tags of the downloads it started, clears a tag when the download
finishes or is deleted, and lists what is left as still downloading
without starting anything. The UI adopts those tags as paused on every
list fetch, so a refresh or a fresh open shows an incomplete download and
can resume it from the files already on disk.

Assisted-by: pi

* ui : show a paused download's state on its chip

A paused quant chip reads as paused at rest, the way a running one shows
its spinner, and offers the resume action on hover. The selector's
download row gets a View in Discover action beside its cancel.

Assisted-by: pi

* ui : collapse the download options by default

The copyable download command starts collapsed so the option rows
lead; expanding it stays one click away.

Assisted-by: pi

* ui : keep decision models out of the hub lists

Decision models answer with calibrated probabilities instead of text,
so llama-ui cannot run them: the hub data layer drops them from the
catalog and every search, keyed by the decision-model tag or a
text-classification pipeline.

Assisted-by: pi

* ui : read download tags as quant and sidecar

A `<repo>:<quant>-<sidecar>` tag now parses into its quant and sidecar
instead of one opaque quant token, and ModelsService.draftSidecarsByRepo
collects the router listing's sidecar-only entries per repo, replacing
the inline copy in the models store.

Assisted-by: pi

* ui : cluster draft downloads under their main model

Tracked downloads group by repo: the main weights lead and the draft
sidecars ride inside their item as compact sub-lines, on the selector,
the discover list and the manager table. The main model's controls
pause, resume and cancel the whole cluster, and the manager's download
rows select into the configuration pane.

Assisted-by: pi

* ui : act on tracked downloads from the table rows

The table passes its clusters' draft sub-lines to the model rows. A
row's actions cascade pause and resume, an in-flight row offers Cancel
download while a paused one keeps Delete from disk, and the confirm
dialog's copy follows the state.

Assisted-by: pi

* ui : give downloads a configuration pane of their own

Selecting a tracked download opens the pane in a download mode: live
progress with pause, resume and cancel, and See in Discover, instead of
metadata and load actions that a model without files cannot offer.

Assisted-by: pi

* ui : make the discover details self-fetching and responsive

ModelsDiscoverDetailsContainer owns the details, tree and readme
fetches so hosts outside the discover view can render the pane, and the
details layout adapts to narrow panes with container queries instead of
viewport breakpoints.

Assisted-by: pi

* ui : download a matching draft with the main model

A main quant download now pulls the quant-matched draft of the most
preferred kind the repo ships (DFlash before MTP), fire-and-forget, so
speculative decoding is ready when the main lands. Manual draft chips
stay available; the options row also collapses to one column in narrow
panes.

Assisted-by: pi

* ui : keep discover off by default and couple it with the hub setting

Discover Models starts unchecked. Enabling it turns the Hugging Face
Hub setting on and disabling it turns it back off, so the checkbox
never lies; with discover off the hub setting stays available on its
own for the display data. Stored and reset state are corrected the same
way on load.

Assisted-by: pi

* ui : open the manage models dialog on the manager view

The dialog now opens on the manager instead of returning to the last
view, discover entries still land on the discover view while the
setting allows it, and the discover surfaces follow the setting's
state.

Assisted-by: pi

* ui : give the discover search room for its focus ring

The sticky search bar insets from the panel edges so the focus ring
stays inside the scroll container, and rows scroll behind an opaque bar
instead of through it.

Assisted-by: pi

* common: settle sidecar names on the listing and drop unfinished partials

An infix or trailing sidecar token is ambiguous: draft heads use it
(Model-MTP-BF16.gguf), so do full models with embedded heads
(Model-Q8D-MTP.gguf). Such a name is a sidecar only next to token-free
main weights of the same family, while mmproj or imatrix anywhere in the
name, or a leading token, decides alone (mmproj_f16.gguf).

A quant tag always carries its bit count, so a sidecar named after no
quant (mtp-Model-it.gguf) gets the bare tag: the listing, the plan and
remove all use it. An imatrix tag resolves into a plan slot of its own,
downloaded but never wired into the params.

remove resolves the tag the way a download does and deletes the partial
files of every planned blob, so cancelling a running download or
dropping one left unfinished by an earlier run leaves nothing on disk.
The router passes its token and releases its lock around the call, and
records the unfinished downloads with an atomic write.

llama download succeeds on a sidecar-only plan and prints its paths.

* ui : move dialog views and draft states into enums and constants

The models dialog's views become the ModelsDialogView enum, the draft
subline's states join ModelRowDownloadState, the view fade and the
copy-feedback reset move to ui constants, and the dialog's compass
marks come from DISCOVER_ICON.

Assisted-by: pi

* ui : drop the unused discover surface

The dead openDiscoverModels twin, the commented-out command divider,
the details pane's unused onClose and its header's close button, and a
redundant type cast leave; the command's copy feedback joins the shared
reset constant.

Assisted-by: pi

* ui : trim the download option comments and highlight the chat template

The comment verbosity the options panel and its chips carried shrinks
to the non-obvious why, and the chat template dialog renders through
SyntaxHighlightedCode as jinja instead of a plain pre block.

Assisted-by: pi

* common: only a .gguf file is a sidecar, and mmproj/imatrix as whole words

A non-GGUF file named like a sidecar (quantization-mmproj-Q8_0.json) is
picked as the mmproj, and a sidecar word inside a longer word of a model
name (Phi-2-OmniMatrix.i1-Q4_K_M.gguf) drops the model.

* ui: dedupe draft sidecar badges per quant

The draft sidecar badges deduped by kind and repo without the quant, so
with a draft cached at two quants only one badge survived, and a delete
of the other main quant found no badge to remove its sidecar with.
Also restores the getDownloadEntries doc lost when the clusters method
landed above it.

Assisted-by: pi (zai-org/GLM-5.3-Flash)

* ui: adopt the incomplete download status and drop the paused-ids registry

The store inferred an unfinished download from a downloading entry it
held no local state for, which misread a download running in another
tab as paused and offered a resume the server rejects. The listing
carries the "incomplete" status now, so the adoption keys off it
directly.

The paused ids no longer need to survive in localStorage: the server
remembers them across restarts, and the progress snapshot was stale
after a reload anyway. The stale localStorage key is removed.

Assisted-by: pi (zai-org/GLM-5.3-Flash)

* ui: quant-match the pause and cancel cascades

Pause, resume and cancel cascaded to every draft sidecar of the repo
while delete only took the quant-matched ones, so canceling a Q4_0
download also killed a Q8_0 draft that belongs to another quant's
main. Every cascade acts on the quant-matched drafts now, matching
what a delete takes with it.

Assisted-by: pi (zai-org/GLM-5.3-Flash)

* ui: collapse the duplicated draft subline markup

The downloaded and in-flight branches repeated the identical connector,
label and badge markup; one block with a conditional tail keeps the two
states in sync.

Assisted-by: pi (zai-org/GLM-5.3-Flash)

* ui: scrub positioned styles out of sanitized model card html

The style attribute stays allowed for the KaTeX output, which let a
third party card draw a fixed overlay over the app. The sanitize hook
drops position:fixed and position:absolute values and leaves the KaTeX
layout styles alone.

Assisted-by: pi (zai-org/GLM-5.3-Flash)

* ui: run the discover mount fetch once and reuse the shared repoOf helper

fetch() toggles its loading flag synchronously inside the mount effect,
which re-ran the fetch on every toggle and retried failed catalog repos
endlessly; the effect now untracks the call. The local repoOf copy and
the duplicate services import give way to the shared helpers.

Assisted-by: pi (zai-org/GLM-5.3-Flash)

* ui: narrow the spec-type map to the draft sidecars

SPEC_TYPE carried empty entries for the aux sidecars it is never
indexed with; the map and the command component now type the badge as
a draft sidecar through a draftBadgeOf helper instead of a cast.

Assisted-by: pi (zai-org/GLM-5.3-Flash)

* ui: name the decision-model tags and markdown cache keys

The decision-model hub tag and pipeline tag move into the huggingface
constants next to the other pipeline tags, and the markdown transform
cache prefixes get names instead of bare one-letter tokens.

Assisted-by: pi (zai-org/GLM-5.3-Flash)

* ui: read the discover setting through one truthiness check

Three places read the setting with !== false and one with Boolean();
all four read it the same way now.

Assisted-by: pi (zai-org/GLM-5.3-Flash)

* ui: compare the models dialog view against its enum

The template compared the view state against the 'manage' literal in
three spots while the script used the enum.

Assisted-by: pi (zai-org/GLM-5.3-Flash)

* nit: Remove `+` before draft sidecar badge

* server: list unfinished downloads as incomplete

A download that never finished was reported as "downloading", the same
status a live download carries, so a client could not tell a running
download from a dead one. The tags remembered in unfinished-downloads
now list under a dedicated "incomplete" status, without an instance.

Assisted-by: pi (zai-org/GLM-5.3-Flash)

* server: download a model with its sidecars as -hf does

Resolution goes back to master. POST /models takes the sidecar kinds to
pull with the model (mmproj by default) and the download child gets them
as -hf flags, so the server picks the files. /models lists real models
only, each with the sidecars it has on disk, and incomplete downloads
keep their sidecars across restarts. A stopped download keeps its
partial files for the next attempt, and DELETE purges them through the
same plan the download follows.

* ui: download sidecars as parameters of their model

The download options keep their chips, but a chip no longer posts a
sidecar tag: the store maps it to its model and the sidecar kinds to
pull, the quant chip takes its companion draft in the same download,
and draft badges come from the sidecars each cached model lists.

The server adds the sidecars a cached model lacks when it is posted
again, and DELETE removes a single sidecar of a cached model.

* ui : mirror the server's sidecar lookup in draft badges

The server resolves a sidecar by the case-sensitive `<kind>-` token in the same
directory as the weights. The UI matched the token loosely, so a badge could
offer a draft the server never pulls.

Assisted-by: DeepSeek-V4.1-Flash

* ui : post the model's sidecar kinds with its download

The store keeps one download per model: the chip's kinds ride along with the
model's own sidecars, and a download row no longer expands into one row per
sidecar. Draft badges carry no quant of their own yet, so a sidecar of a cached
model is never labelled with the model's quant.

Assisted-by: DeepSeek-V4.1-Flash

* ui : join a quant's draft into its download chip

One chip per quant, its draft badged inside it and its size covering both
files. The draft is the file the server would pick: the same directory, the
exact quant, else the closest bit depth, so a 4-bit model can pair with a
16-bit head.

Assisted-by: DeepSeek-V4.1-Flash

* ui : drop the draft download rows from the manager and selector

A download is one row per model again: the drafts ride with it, and the rows
read the sidecars on disk off the listing instead of pairing sidecar entries
with their model. Pause, resume and cancel act on the model's own download.

Assisted-by: DeepSeek-V4.1-Flash

* ui : keep the discover download list one row per model

The list leads with one row per downloading model, its sidecars as sub-lines.
A row badges only the draft kinds the server can resolve, so a badge never
outlives the file its lookup would pick.

Assisted-by: DeepSeek-V4.1-Flash

* ui : keep the discover chips readable in dark mode

The metadata chips, the chat template button and the View on Hugging Face link
carried a faint border and no surface over the dialog background, so they were
near invisible. They take the tertiary button surface now, and their icons size
with size-* so the variant's base rule leaves them alone.

Assisted-by: DeepSeek-V4.1-Flash

* ui : carry the draft through the download options

One chip per main quant, its draft badged on it and the chip's downloaded state
covering both files. The drafts sit behind an opt-in checkbox with an inline
type pick, and each row opens its own command: it pins the row's quant and only
names the draft type, so -hfd is gone.

Assisted-by: DeepSeek-V4.1-Flash

* ui : line the details skeleton up with the loaded pane

Same root padding and container context as the loaded view, so the container
queries resolve alike. The download options card pads its blocks instead of
itself, the readme gets its card, and a description line stands in for the
catalog paragraph.

Assisted-by: DeepSeek-V4.1-Flash

* common : drop the partials of a removed sidecar

Removing a sidecar alone left the .downloadInProgress files of the sidecars a
download had planned in blobs/. The removal purges them through the same plan
the download follows, and the test covers a finished draft beside an unfinished
one, whose partial goes while the model keeps its files.

Assisted-by: DeepSeek-V4.1-Flash

* server : name why a download is refused

A model that is already downloading, or that is running, answered "already
exists". It now says which: "is already downloading", or "is running, unload it
first", both before the resumable check. The router test covers the two.

Assisted-by: DeepSeek-V4.1-Flash

* ui : post a download once per click

Two quick clicks on the same chip posted twice, and the second answer was
"already exists". The store tracks the model as pending, returns early while it
is pending or in flight, and reads it as in flight before the server answers, so
the chip shows the download at once. The new client test pins the flows: one
POST per double click, the chip reading downloaded once the list reports the
model, a resume keeping its sidecars, a cancel dropping the whole download, and
a reload restoring the paused state.

Assisted-by: DeepSeek-V4.1-Flash

* ui : keep a plain download row the height of the others

The sub-lines wrapper of a download row was rendered even with nothing in it, so
its padding made the row taller than a plain one and the overlaid progress bar
sat in that empty strip. It renders only with sub-lines now, in the selector and
in the discover list, and the selector row takes the phone padding its
neighbours have.

Assisted-by: DeepSeek-V4.1-Flash

* ui : badge a repo's draft once

The args of a loaded model and the listing of a cached one report the same draft
in different words, one with the file's quant and one with the kind alone, so
the merge kept both and a repo row showed the same kind twice. The badges merge
per kind now, the args one winning as it knows the quant, and the test pins both
sources of a real repo.

Assisted-by: DeepSeek-V4.1-Flash

* server : report the sidecars of a preset or a download on its way

A model the preset also names is registered as a preset, so its drafts were
never reported and no row could badge them. A download on its way reported none
either, though the record of the kinds it pulls is right there: those win over
the plan now, the plan resolves through the preset's hf-repo, and a models dir
stays out of it.

Assisted-by: DeepSeek-V4.1-Flash

* ui : badge a download's drafts and open its model from the row

A draft still coming down badges the model id now, the way a landed one does,
while an on-disk draft keeps its checked sub-line. The row itself opens the
model, the status slot beside it pauses or resumes, and the manager resolves a
focus on a download row so its panel opens.

Assisted-by: DeepSeek-V4.1-Flash

* common : report a removed sidecar that only had a partial

Removing a sidecar alone dropped the result of the partial purge, so a sidecar
that only ever landed as a .downloadInProgress blob answered false and the UI
had no reason to refetch. The purge result is folded into the return now, and
the test covers a partial-only removal.

Assisted-by: DeepSeek-V4.1-Flash

* ui : ignore a pause while the download POST is still on its way

The chip reads as downloading before the server answers, so a quick second click
unloaded a model that had not started and left the store neither in flight nor
paused. A pause returns early while the POST is pending, before it touches the
feed or the paused marker. The client test holds the POST open and pins it.

Assisted-by: DeepSeek-V4.1-Flash

* ui : offer a missing draft as its own chip

A model already on disk without its draft had no way to add it: the quant chip
carried the draft, so it read as a download of both. The quant chip keeps to its
own state now, its joined draft shows only while it rides along or is on disk,
and a downloaded quant missing its draft gets a chip of its own - badge, size,
download - that posts the model with that kind, so the server keeps the files
already there and pulls the draft.

Assisted-by: DeepSeek-V4.1-Flash

* server : list a paused sidecar download by the files on disk

A model on disk whose sidecar download stops keeps its unfinished record,
and the listing reported the kinds that record plans as if they were on
disk: the UI checked a draft that never landed and offered no way to add it.
The planned kinds now stand in only while the download runs.

* chore: Clean up comments

* server : list sidecars only for a model that resolves from a HF repo

A preset on a local file has no HF repo, yet the listing resolved its name
as one: the plan threw on the invalid repo format and GET /models failed
for every model. The sidecars now come from the repo a preset names or a
cached model is listed under, and a local file lists without them.

* common : fix download cancel regression

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* common : preserve model partials when removing sidecars

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* server : one sidecar table and an unfinished downloads class

Address review from @ggerganov.

The sidecar kinds live in one table in common/download, mapping each name to
its download option and plan file; the server validates, builds its options
and lists cached sidecars through it instead of its own lists.

The unfinished downloads move into server_unfinished_downloads, a record with
its own mutex and a struct per entry, and server_models exposes them through
unfinished_downloads_get and unfinished_downloads_remove.

* common: one sidecar table, and removals that keep shared files

The resolution, the cached listing and the removal follow a single
table of sidecar kinds, so a kind is declared once with its file
keyword, whether it is a draft, its option and its plan file. The
server reads the same table for the kinds it accepts, the default
sidecars and the child flags.

Removing a tag keeps the files another cached tag still loads, such as
an mmproj named after the removed quant. A sidecar removed on its own
goes for every tag that resolves it.

An unfinished download records the exact paths of its partials when
it starts. A removal deletes those without asking the hub, and keeps a
partial another recorded download also writes. The record is taken
once the child is spawned, and the files go under the models lock, so
a download of the same tag cannot start while they are removed.

* server: drop stale partials and refuse a tag that resolves nothing

Recording a download again deletes the partials of the earlier record
that the new plan no longer writes, such as those of an older hub
revision or of a sidecar the new request leaves out.

A posted tag that resolves no file of the repo is refused before the
download starts. Removing a model follows a single path, whether the
reload erased its entry during the wait or not.

* server: fix the CI race between the router download tests

The pytest workers run in parallel on one shared cache, so a download
test could find the file another test had just fetched and see no
progress, or start from the unfinished downloads record another test
wrote. Each download test now runs on an empty cache of its own.

* server: refuse a DELETE of an unknown name and decide downloads in load()

A DELETE of a name that is neither a model nor a recorded download
went to common_download_remove, and without a tag it wiped the whole
repo cache, including the files of a loaded quant or of a download on
its way. Such a name is now not found, as it was before.

A POST that downloads over an existing entry checked its state outside
the lock, so a load of the same model in between got its instance
killed. The checks now run in load(), under the lock, right before the
spawn. The fake load time of the tests applies to model loads only,
since a download worker loads none.

* server: add incomplete_sidecars to the /models API

A sidecar added to a cached model and then paused kept its record, but
the listing showed only the complete model, so a fresh UI lost the
pause. The model entry now carries incomplete_sidecars, the kinds its
unfinished download has not written yet; a POST with the kind resumes
one and a DELETE with it drops its partial.

* common : drop a kept partial when a resume gets 416

Now that partials survive between attempts, an old or complete
partial makes every retry fail with the same 416.

Delete it so the next attempt can start over from zero.

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* common: drop a kept partial when a resume gets 200

A server that ignores the range answers 200 to a resume, and the kept
partial then fails every attempt the same way a 416 does. Drop it as
well, so the next attempt starts over without a range.

* server: refuse a download that shares a partial with one on its way

Two downloads of quants that share a file, the mmproj or an mtp picked
by the closest bits, appended to the same partial and renamed a file
with interleaved bytes. A download whose plan writes a partial that a
running download also writes is now refused until that one is done.

* common : check the saved etag before resuming a partial download

Before resuming, compare the etag saved with the partial to the one
the server now serves. If it is missing or different, start over,
so a resumed download never mixes bytes from two versions of the file.

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

---------

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
Co-authored-by: Pascal <admin@serveurperso.com>
Co-authored-by: Adrien Gallouët <angt@huggingface.co>
2026-10-17 17:20:51 +02:00
Aman Guptaandthelittlefireman 2744e48d1e CUDA: faster MMVQ for low-bit types (#30517)
* CUDA: faster MMVQ for low-bit types

* ggml-cuda: hip: small nwarps GCN fix on Q3_K(ncols=2,3), Q6_K(ncols=4)

---------

Co-authored-by: thelittlefireman <thelittlefireman3@gmail.com>
2026-10-17 15:35:30 +02:00
Pascal 10b8b75534 tests: give the random model generator a per-tensor filler table (#30674)
* tests: give the random model generator a per-tensor filler table

Muse Spark makes test-llama-context fail on state_load, cp_h and cp_d on about
six percent of the random seeds, with an NMSE of 1.0e-05 to 1.4e-05 against a
threshold of 1.0e-05. The sliding window is what exposes it: a restored state
keeps the window alone, so the attention of an SWA layer reduces over a
different cell layout and the continuation is not bit exact. Every SWA model
carries that, at 1e-08. Muse Spark carried it at 1e-05.

The generator is what turns 1e-08 into 1e-05. A norm preserving residual gate
holds alpha^2 + beta^2 == 1 and the conversion resolves both weights from one
raw weight, while the generator draws every tensor on its own. The pair then
scaled the stream and the branch independently, the stream lost an arbitrary
factor at each layer, and the model became an amplifier. The weights the test
ran on were not a Muse Spark.

Per-tensor rules lived inside the write path, where a flag picked the values of
a tensor inside each of the two branches that store them, so a new rule meant a
new flag threaded through both. The file asks to be rid of that in a TODO of its
own. Give each rule a filler, pick one from a table keyed on a substring of the
tensor name, and keep a single write path behind it. Mamba moves into the table
and Muse Spark joins it: the gate filler draws the raw weight once per block and
resolves alpha and beta the way the conversion does.

Over a thousand seeds on a CPU build the worst case drops from 1.36e-05 to
2.48e-06 and the median from 4.10e-06 to 5.48e-07, with no seed above the
threshold. The other 131 models come out identical to the bit, since no other
architecture carries these tensors. ctest is green on a CPU build and on a CUDA
build.

* tests: draw every tensor filler from fill_normal

The fillers repeated the generator and the distribution each time. The
normal draw now takes its key and the other fillers transform its
values, so the generator lives in one place. The generated models are
identical to the bit.
2026-10-17 16:07:55 +03:00
Adrien Gallouët 9d1e03616e cuda : fix -Wtautological-compare in fattn vec dispatch (#30519)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-10-17 14:36:17 +02:00
Sigbjørn Skjæret f2f667b02d cuda : add grouped experts top-k fusion (#29181)
* fuse grouped experts

* add tests

* hip fix

* add to graph_optimize

* apply suggestion from am17an

* hip fix

* fix hip fix

* fix fix fix

* gate on hip version instead
2026-10-17 13:55:33 +02:00
Xue ZhenliangandSigbjørn Skjæret 62318cd57d model : add Kolibri-1 (Aleph-Alpha) support (#30673)
* model : add Kolibri-1 (Aleph-Alpha) support

Kolibri-1 is a bilingual German/English MoE reasoning model with 78B
parameters and 3.46B active per token: 50 layers, 384 routed experts plus
one shared expert.

Attention interleaves four sliding layers at a 513 window with one full
layer. The sliding layers rotate with NeoX RoPE, the full layers are NoPE.
GQA runs 48 query heads against 4 KV heads at head_dim 128 with a
QK-RMSNorm, and every block carries four RMSNorms, so the attention and the
MLP output are normalized before they rejoin the residual stream. Routing
picks the experts from logits + expert_bias and weights them with the
sigmoid of the unbiased logits, without renorm.

The converter reuses AfmoeModel for the per-expert checkpoint layout and
the expert selection bias, and writes the shared expert and the per layer
sliding window pattern. The released FP8 checkpoint needs no special code,
it goes through the generic weight_scale_inv path. Three tensor names are
added to the mapping (post_attn_norm, post_ffn_norm,
moe.router.expert_bias) and the 128k vocabulary registers the qwen2
pre-tokenizer by hash. The arch is registered in test-llama-archs, with GQA
and the sliding window pattern.

Assisted-by: DeepSeek Harness (DeepSeek V4.1 Flash)

* Update src/models/kolibri1.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Add fusion records for kolibri1

* Update src/models/kolibri1.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-10-17 13:00:17 +02:00
Sandro Hanea 83f9389419 ci : added ABI reports as github artifact (#30619) 2026-10-17 11:41:53 +02:00
Gabe Goodhart 1268edac57 model : Granite switch decision 256 options (#30693)
* feat(convert): Attempt to come up with 256 single-token, semantic option markers

Each switch model needs to identify a collection of single-token option
markers that can be mapped to their respective logits for the output
probability distribution. The simple upper/lower/numbers set only yields 66
unique markers, but the model can support up to 255. The challenge is that
which combinations of characters result in single-token markers is model-
specific, so we do a discovery exercise at conversion time by looking
through the specific model's vocab for single-token sequences of 1+ plain
letter strings (eg `A`, `AA`, `BB`, etc).

Branch: GraniteSwitchDecisionFullOptions
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* docs: Note granite-switch with 256 option markers

Branch: GraniteSwitchDecisionFullOptions
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* style: Remove extra newline

Branch: GraniteSwitchDecisionFullOptions
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* feat(convert): Use the computed labels in the template so c++ side is source of truth

Branch: GraniteSwitchDecisionFullOptions
AI-usage: full (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* feat: Use LEV case for GRANITE_SWITCH w/ computed labels

Branch: GraniteSwitchDecisionFullOptions
AI-usage: full (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* docs: Consolidate g-s into 255 bucket

Branch: GraniteSwitchDecisionFullOptions
AI-usage: full (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
2026-10-17 09:14:32 +02:00
Marcus f74e2fa6a3 sycl: fix prompt-processing regression after Q4_0/Q8_0 weight reorder (#30374) 2026-10-16 22:03:41 -04:00
Xuan-Son Nguyen ea0c4c2c85 skills: add slice-model (#30677)
* skills: add slice-model

* exclude from py check

* add to agents.md
2026-10-17 02:11:56 +02:00
Zack Li 6420cf5dd5 ci: update device matrix for Snapdragon jobs by removing SM8750 (#30620)
* ci: update device matrix for Snapdragon jobs by removing SM8750

* ci: increase timeout for QDC device tests and update job timeout parameters
2026-10-16 16:25:05 -07:00
wes 27c7a769da hexagon: vectorize softplus with HVX (#30435)
softplus_f32 and tile_softplus_f32 ran a scalar logf(1.0f + expf(x)) loop.
Replace both with a shared hvx_softplus_f32_aa() helper built from the
backend's existing hvx_vec_exp_f32 / hvx_vec_log_f32 primitives, so no new
numerical approximation is introduced.

hvx_vec_exp_f32 clamps its input to (-88, 88) before exponentiating, so a
naive log(1 + exp(x)) composition would saturate to ~88.72 for large x
instead of returning x. The Q6_V_vmux_QVV select keeps the scalar
reference's x > 20 guard (ggml_compute_softplus_f32 in ggml-impl.h),
which is what makes the vector path match the CPU result.

Assisted-by: Claude
2026-10-16 15:20:43 -07:00
Shawn GuandLi He 2f2763db6e opencl: Add MxFP4 MoE bin kernel for Adreno X1 devices (#30691)
* opencl: add bin kernel `gemm_moe_mxfp4_q8_1_dp4a_ila_ldck` for X1

* opencl: fix whitespace

* opencl: undo unnecessary change

* opencl: enable dp4a clc for X1E

---------

Co-authored-by: Li He <lih@qti.qualcomm.com>
2026-10-16 15:15:04 -07:00
Daniel Han 01fc3aae84 model: add Beam (Reflection) (#30670) 2026-10-17 00:07:55 +02:00
Gabe Goodhart 23f7aabc16 model : Granite Switch Decision (#30676)
* feat(convert): Add decision.chat to support decision models that can also chat

Branch: GraniteSwitchDecision
AI-usage: full (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* feat(convert): Add conversion for Granite Switch as decision model

The template follows from the usage sample:
https://huggingface.co/ibm-granite/granite-switch-4.1-8b#guardian-and-decision-side-by-side

Branch: GraniteSwitchDecision
AI-usage: draft (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* feat(common): Add support for chat+decision models in common

Branch: GraniteSwitchDecision
AI-usage: full (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* feat(common): Add granite switch decision model enum

Branch: GraniteSwitchDecision
AI-usage: full (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* feat(server): Plumb through decision_chat bool to support decision+chat models

Branch: GraniteSwitchDecision
AI-usage: full (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* feat(server): Add Granite Switch decision support

Branch: GraniteSwitchDecision
AI-usage: draft (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* docs: Update server README for granite switch decisions / chat+decisions

Branch: GraniteSwitchDecision
AI-usage: draft (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* Revert "feat(convert): Add decision.chat to support decision models that can also chat"

This reverts commit 952ef567647e53ac9beda7a1f79d6927240d068d.

* Revert "feat(common): Add support for chat+decision models in common"

This reverts commit 93ed782e3a8b2fd5968732c6525c7ad29ad4bea2.

* Revert "feat(server): Plumb through decision_chat bool to support decision+chat models"

This reverts commit 97e9af60ddaccbd82f1eb0a488adff43b16ee095.

* fix: If Granite Switch supports decisions, it always supports text too

Branch: GraniteSwitchDecision
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* docs: Remove reference to decision.chat in server README

Branch: GraniteSwitchDecision
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
2026-10-16 20:00:41 +02:00
7182ef71e6 common: honor HTTP_PROXY/HTTPS_PROXY env vars in http client (#21752)
* common: honor HTTP_PROXY/HTTPS_PROXY env vars in http client

common_http_client() was ignoring standard proxy env vars, causing
downloads via -hf and similar flags to fail silently in environments
behind an HTTP proxy (corporate networks, air-gapped infra, etc.).

Checks HTTPS_PROXY/https_proxy for https:// URLs and HTTP_PROXY/http_proxy
for http:// URLs, in that order. If a proxy URL is set and parseable,
passes it to httplib's set_proxy() and set_proxy_basic_auth() as
appropriate. Malformed proxy URLs are silently ignored so existing
callers aren't affected.

Fixes #21694

* common: drop misleading comment about generic proxy fallback

The comment referenced a 'generic one' that doesn't exist in the code — both
HTTP_PROXY and HTTPS_PROXY are scheme-specific. Reviewer noted the mismatch.

Addresses review feedback from @ServeurpersoCom.

* common : warn when the proxy URL from the environment is malformed

* common : honor NO_PROXY when selecting the http proxy

* ci : retrigger checks (test-thread-safety flake)

* common : bypass the proxy for loopback destinations

llama cli talks to its spawned server on 127.0.0.1 through common_http_client, so with HTTP_PROXY set and no NO_PROXY the health polling went to the corporate proxy, which rejects loopback targets, and the wait loop never exits ("Loading model..." hangs forever).

A remote proxy can never reach the client machine's own loopback, so these destinations are now always contacted directly, before NO_PROXY is consulted: localhost, *.localhost, 127.0.0.0/8, ::1 and 0.0.0.0. Similar to Go's net/http. Proxy handling for downloads and remote --server-base endpoints is unchanged.

Signed-off-by: texasich <101962694+texasich@users.noreply.github.com>

* docs: move proxy tip from README to server docs

per review: the top level README stays lean; the proxy howto now lives with the server docs under Usage

Signed-off-by: texasich <101962694+texasich@users.noreply.github.com>

---------

Signed-off-by: texasich <101962694+texasich@users.noreply.github.com>
Co-authored-by: texasich <texasich@users.noreply.github.com>
Co-authored-by: texasich <texasich@gmail.com>
Co-authored-by: marecabo <23156476+marecabo@users.noreply.github.com>
2026-10-16 17:01:37 +02:00
aiqubits 2a667032dc convert : respect Mistral format for DeepseekV2Model (#30669)
Use the instance's is_mistral_format flag when DeepseekV2Model reloads
configuration, allowing Mistral MoE checkpoints to read params.json.

Fixes #29899

Signed-off-by: aiqubits <aiqubits@hotmail.com>
2026-10-16 15:17:30 +02:00
Piotr Wilkin (ilintar) f3363593f4 Add more benchmarks to llama-eval (#26174)
* llama-eval : add sandboxed execution for code suites

Adds a lightweight replacement for the Docker containers usually used to run
generated code, so a benchmark can be executed without a daemon, an image or
root.

A cached venv is provisioned per suite and reused; candidate programs run in a
throwaway working directory as their own process group, under the strongest
isolation the host offers:

  bwrap    read-only root, tmpfs /tmp and $HOME, no network
  unshare  no network, via --map-current-user
  rlimit   cpu/memory/file-size/process caps only

Candidate code must not run as uid 0, which is why "unshare -r" is a
last-resort tier that warns: root bypasses file permission bits, so a test
asserting that a write to a chmod 0444 file fails would instead see it succeed.

Two details worth noting for the bwrap tier. Only paths hidden by the $HOME
tmpfs are bound back, because bwrap cannot mount onto a symlink and
version-manager alias directories usually are one. And the venv's base
interpreter tree is resolved by walking the symlink chain, since a
version-managed interpreter is commonly reached through an unversioned alias
that startup resolves through.

Suites can declare provisioning steps, which run once at build time while the
network is still up, so test time never needs it.

Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>

* llama-eval : add HumanEval and ClassEval code suites

Adds two execution-graded suites alongside the existing text-graded ones. The
model's reply is parsed for a python code block, combined with the task's own
unit tests, and run in the sandbox.

  --dataset humaneval   164 function-level problems, scored pass@k
  --dataset classeval   100 class-level tasks, scored at class level (feeding
                        pass@k) and method level, matching upstream

pass@k reuses the existing chunk mechanism: --n_cases at a multiple of the
dataset size gives that many samples per problem, aggregated with the unbiased
estimator from the Codex paper.

Program construction and result interpretation are dataset-owned hooks, because
the two suites differ: one HumanEval problem is a function judged by a single
assert, whereas one ClassEval task is a class judged by many independent test
methods, and reducing that to pass/fail would throw away partial credit. Suites
also declare their own package set, provisioning steps, interpreter and timeout.

The package set is derived from the dataset's own imports rather than a
requirements file. HumanEval turns out to need no third-party packages at all;
ClassEval needs thirteen, two of which must be declared because no import
mentions them -- lxml is selected by string in ClassEval_44, and scipy is a
gensim requirement.

Notes on ClassEval specifically:

* gensim publishes no wheel past cp313, so 3.13 is preferred; on 3.14+ the two
  pure-Python APIs it uses are provided by a small shim instead.
* The three nltk corpora it needs are pre-seeded at venv build time. These are
  the modern names -- the dataset asks for the since-renamed punkt and
  averaged_perceptron_tagger.
* It defaults to a patched fork, ilintar/ClassEval, because three upstream
  tests cannot be passed by anything: a hard-coded 2024 date checked against
  datetime.now(), an assertion on the dataset author's own hostname, and an
  exact float comparison. Reference solutions go from 96/100 to 100/100 classes
  and 2182/2196 to 2196/2196 test methods. No skeleton is touched, so what the
  model is asked to write is unchanged; --dataset-source selects the original.

--exec-seed (default 0) seeds the RNG before each test, so suites whose tests
draw random inputs score reproducibly. ClassEval_58 fails its own reference
solution about 30% of the time without it.

The simulator gains both suites, so the harness can be exercised without a
model.

Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>

* llama-eval : fix silent isolation downgrade, generalise the sandbox runner

The isolation probe ran the interpreter without binding it back, so whenever
this tool is itself started from a venv under $HOME -- the usual case, since it
needs `datasets` -- bwrap's tmpfs hid the interpreter, the probe failed, and the
sandbox quietly fell back to a tier that does not hide $HOME at all. Nothing in
the output looked wrong when that happened, and the existing suites have been
running that way. The probe now uses the same interpreter and the same binds the
suite will actually get.

Two additions the agentic suite needs, neither of which changes the existing
suites:

* run_argv() runs an arbitrary command in a caller-owned directory that survives
  the call. run() is now a thin wrapper over it, so HumanEval and ClassEval take
  exactly the same path as before. A multi-step agent needs its edits to persist
  across many invocations, and a suite for a language other than Python needs to
  drive a linter and a test runner rather than a single source string.

* RLIMIT_AS is now per-sandbox, defaulting to what it was. It caps reserved
  address space rather than resident memory, which is the wrong instrument for a
  runtime that reserves a large region and commits little of it: node's
  TypeScript type-stripping goes through WebAssembly, reserves gigabytes per
  instance, and aborts at 4 GiB with an out-of-memory error that reads like a
  fault in the code under test.

HumanEval stays at 164/164 and ClassEval at 100/100.

Assisted-by: Claude Opus 5 <noreply@anthropic.com>

* llama-eval : add SACB, a multi-turn agentic coding suite

`--dataset agentic` is a different shape from the other code suites. Rather than
answering a single prompt, the model is given a small but real repository, a
narrow tool set and a bug report, drives its own multi-turn conversation, and is
graded by running a test suite it was never allowed to see.

The corpus is SACB (Simple Agent Coding Benchmark), 60 tasks over three
hand-written repositories in Python and TypeScript, hosted at ilintar/SACB.
Grading needs no container: a cached venv gives ruff and mypy for Python, and
node's own test runner plus tsc for TypeScript, so the whole dependency set is
four packages.

Three constraints are deliberate:

* No shell, and no way to run the tests. The only feedback channel is `lint`. A
  syntax or type error can be driven out mechanically; a logic error has to be
  reasoned about. A harness that lets an agent iterate against the test suite
  measures something closer to search than to understanding.
* Two edit tools, one line-addressed and one content-addressed. Which one a
  model reaches for, and whether it keeps line numbers straight after its own
  earlier edit, is itself a signal.
* The tests never touch the working tree. They are held outside it and copied
  into a throwaway copy only once the agent has stopped, so they can be neither
  read nor edited. Every tool path is resolved and confinement-checked against
  the repository root, so a symlink planted inside the tree cannot reach out of
  it.

A task counts as resolved when every test that was failing passes and nothing
that already passed broke. Both sets are derived by running the tests against
the defective tree and against the reference fix, never written by hand, so a
task whose defect no test exercises cannot enter the corpus. Many tasks carry
several independent defects, so the fraction of target tests passed is reported
alongside, separating "fixed two of the three faults" from "changed nothing".

An episode that arrives with no tool call and no `finish` is nudged rather than
ended. Models routinely narrate their analysis mid-task, and treating that as
completion ends the episode with no edits and scores a reasoning failure that
never happened; about one episode in four exits that way.

Nothing here is installed unless this suite is selected, and then only for the
language being run: agentic_eval is imported lazily and each toolchain is keyed
per language.

Assisted-by: Claude Opus 5 <noreply@anthropic.com>

* llama-eval : make the agentic hooks typed rather than discovered

`ty` flagged nine unresolved attributes, of which the seven in main() predate
the agentic suite: `EvalState.dataset` is Optional and was dereferenced without
narrowing.

* `loaded_dataset` turns the Optional into a value in one place, raising rather
  than handing back None, so callers do not each have to narrow it.
* `is_agentic`, `prepare()` and `run()` are declared on BaseDataset. They were
  reached through getattr with a default at three call sites, which no checker
  can see through; `run()` now raises a clear error naming the suite instead of
  failing as a missing attribute.
* The sandbox setup is one branch on `is_agentic` rather than two tests of
  `grader_type`, which also drops a duplicated condition.

No behaviour change: HumanEval stays at 164/164, and `ty check` on
examples/llama-eval is clean.

Assisted-by: Claude Opus 5 <noreply@anthropic.com>

* feat: API key

* fix reasoning and response length for agentic runs, add option to save entire transcripts

* llama-eval : fix ty unresolved-attribute on loaded_dataset.ae

Assisted-by: AI coding agent

* llama-eval : trim comments and remove non-ASCII from README

Assisted-by: AI coding agent
2026-10-16 11:52:18 +02:00
Jhen-Jie HongandMax Krasnyansky ceefd16daa hexagon: correct HMX FA sinks (#30581)
* hexagon: correct HMX FA sinks

* hex-fa: vectorize sinks in hmx kernels

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-10-16 16:51:14 +08:00
Mikhail Podvitskii 9a16f1dbe4 model : MTP support for Mellum2.1 (#30612)
* model : MTP support for Mellum

* model : inline rope construction in Mellum
2026-10-16 10:20:03 +02:00
Aaron Teo b84ea5cf5e convert : speed up model endian conversion and add more datatype support (#30647)
* convert: speed up model endian conversion

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* convert: explicitly state noop

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* convert: add missing data type support

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

---------

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
2026-10-16 10:18:29 +02:00
Michael Taylor a30fb4924d [SYCL]: fix memory errors in mul_mat, split buffer, host pool (#29889) 2026-10-16 03:28:50 -04:00
Sh1m13 495c5805ce gguf-py : add numpy Q1_0 and Q2_0 de/quantization support (#30630)
* gguf-py : add numpy Q1_0 and Q2_0 de/quantization support

* cont: added q1_0 / q2_0 to the test list removing the small test

Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>

---------

Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 09:26:14 +02:00
uvos 623c16393c CUDA: HIP: rms_norm: add variants that load float4 at a time and store x in registers (#29720)
* CUDA: HIP: rms_norm: load float4 at a time and store x in registers

* Add template paramter comments

* use ggml_cuda_is_aligned for src

* nits

* also use ggml_cuda_is_aligned for src in fused variants

* fix nits
2026-10-16 10:07:34 +03:00
hayder 13b8048920 common : read ngram cache parts by reference instead of copying (#30061) 2026-10-16 10:06:32 +03:00
FINAL-Bench 0f4376268f convert : support text-only checkpoints in pplx-decider (#30575)
* pplx-decider-text-only

convert : support text-only checkpoints in pplx-decider

* Apply suggestions from code review

* convert : remove unused import
2026-10-16 08:56:00 +02:00
Trivalik 3dbcb7fca5 cpu : reduce AVX2 tiled accumulators together (#30296)
Assisted-by: Codex
2026-10-16 09:48:23 +03:00
Aaron Teo c804754529 ggml-cpu: refactor s309x repack helper functions (#30588)
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
2026-10-16 09:45:35 +03:00
Neo Zhang c0083e3052 sycl : reduce the build threads to half (#30236) 2026-10-16 09:44:24 +03:00
Titaniumtown 13de11e78f sycl: wide stores for the q4_K/q5_K weight dequant (#29696) 2026-10-16 09:43:07 +03:00
Georgi Gerganov 88aaf325a7 tests : catch vanished rows in the fusion baseline (#30618)
* tests : update the Metal fusion baseline

Re-record the MTL fusion baseline with the current build: 276 rows over 131
dummy models.

Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)

* tests : fail test-fusion when a baseline row stops matching

The report was built from the counters the run produced, and the baseline was
only ever used as a lookup table, so a fusion that stopped matching produced no
row and was never compared. qwen4exp and mellum lost their fusions this way
while the Fusion / metal job stayed green.

Report a baseline row that the run does not reproduce as a regression. Scope
the pass to the arches that were actually run so --model keeps working.

Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)

* pi : never write inline comments unless the user asks

Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)

* tests : keep the mellum dummy model softmax-gated

mellum reads expert_gating_func from the GGUF since #30578, while the fixture
writes sigmoid for every MoE arch, so the Mellum dummy model stopped building
the softmax MoE routing fusion. Write softmax for mellum, matching the arch
default and the GGUFs that conversion/mellum.py produces.

Re-record the MTL baseline: the routing row is back, 277 ok / 0 failed.

Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)

* pi : use emoji in the AI stats disclosure

Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)

* pi : list the disclosure model after the hardware

Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)

* pi : use monochrome symbols in the AI stats disclosure

Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)

* tests : add muse-spark to the Metal fusion baseline

Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
2026-10-16 09:35:19 +03:00
Ruben Ortlam a8d9a7e1e5 vulkan: fix gpu-visible host memory issues (#30468)
* vulkan: fix log spam if host pointer import fails

* exclude MoltenVK from shared memory import

* remove unreachable code, assert instead. Shorten warnings

* disable host memory buffer sharing on AMD due to amdgpu perf issue
2026-10-16 08:29:23 +02:00
Karthik VijayanandKarthik Vijayan f12aaa344c Fix failing RocM tests (#30611)
Co-authored-by: Karthik Vijayan <kavijay@microsoft.com>
2026-10-16 12:46:55 +08:00
Mikhail Podvitskii 5e03a1ff45 graph : add precision handling for MELLUM architecture (#30584) 2026-10-16 11:32:26 +08:00
Max Krasnyansky b797b9ca81 hexagon: tensor repacking rework and support for tiled and flat quantized and fp weights (#30635)
* hex-repack: Remove need_repack hack and streamline the rules -- breaks GET_ROWS tests for K-Quants, will fix in the next round

Add support for GET_ROWS with tiled Q5_K
Add support for raw k-quant tensors

* hex-get-rows: vectorize raw k-quant ukernels

* hex-binary: refactor dma-loop thread functions (no need for macros, etc)

* hex-quant: move/cleanup flat and tiled dequant ukernels

* hex-repack: add support for repacking/tiling F16/F32/BF16 2D weights

* hex-get-rows: remove redundant cast

* hex-repack: proper filtering for BF16 that were leaking through during the tests

* hex-repack: streamling tiling rules (broken MM for now)

* hex-repack: fix mmid after tiling updates

* hex-repack: rename repacked to tiled in all flags/funcs/kernels/vars

* hex-mm: fix valid_rows calc in various kernels

* hex-mm: remove unused hvx_mm_nx_2d_flat

* hex-mm: cleanup and optimize hvx-tiled vec_dot_f16

* hex-mm: another pass at hvx-tiled f16 matmul

* hex-get-rows: optimize small src1

* hex-get-rows: one more pass at recovering get-rows perf regression

* hex-repack: export GGML_HEXAGON_RT_REPACK via run.py

* hex-mm-tiled: fix accuracy regression with hmx-tiled-f16 matmul

* hex-mm: fold tiled_f16 checks into the general flow

* hex-get-rows: remove dead checks for buffer_is_hexagon and tiling during supports_op

* hex-quant: proper tail handling in new dequant ukernels

* hex-get-rows: remove unused code and cleanup support checks

* hex-mm: cleanup ne1 padding/tiling checks

* hex-mm: cleanup fusion checks for tiled tensors

* hex-repack: add a little can_repack helper to reduce duplication

* hex-repack: never repack host buffers

* hex-repack: factor out repack functions into separate .h and .cpp

* hex-tensor: cleanup tensor ifaces for tiled/padded sizes

* hex-repack: update k-quant tiling comments and cleanup #defines

* hex-dcache: factor out l2/dcache dirty tracking and flushing

* hex-tensor: update docs and remove #defs duplication

* hex-tensor: remove unused/incorrect get_data interface

* hex-get-rows: cleanup vtcm layout build

* tests: disable huge MM tests

* hex-dcache: add missing header
2026-10-15 20:30:11 -07:00
Aman Gupta 6afa9a3134 llama : share the k-pool input views between layers (#30613)
The scheduler uploads each view of a host input on its own, so the per-layer reshape of new_pool_idxs (and of sel_mask in glm5-next) was copied once per layer.
2026-10-16 11:14:36 +08:00
Gaurav Garg b6334e37e6 Limit max CUDA pinned host buffer size to 4 GB (#30449) 2026-10-16 08:20:26 +05:30
Frosty40 e707790c73 sycl: support reordered Q8_0 MoE weights (#29337)
Preserve the existing MoE accumulation order.

Assisted-by: Codex
Assisted-by: Angel
2026-10-15 21:59:38 -04:00
Neo ZhangandSigbjørn Skjæret 43d57fb8ca [SYCL] Enable ssl in build script and CI (#30456)
* enable OPENSSL=ON as default in build script, update it in CI and dockerfile

* change to LLAMA_BUILD_BORINGSSL on windows

* refactor

* rm Download UI build

* Apply suggestion from @CISC

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Apply suggestion from @CISC

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Apply suggestion from @CISC

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-10-15 21:58:07 -04:00
Xuan-Son Nguyen 2851b3bcbf mtmd: add mtmd_audio_stream + support voxtral-realtime (#30629)
* mtmd: add mtmd_audio_stream + support voxtral-realtime

* address review
2026-10-16 01:46:42 +02:00
Nigel JonesandGabe Goodhart 59d5fad7c1 convert : fix Granite Switch block_count for MultiSwitch checkpoints (#30621)
* convert : fix Granite Switch block_count for MultiSwitch checkpoints

Signed-off-by: Nigel Jones <jonesn@uk.ibm.com>

* Update granite.py

Co-authored-by: Gabe Goodhart <ghart@us.ibm.com>

---------

Signed-off-by: Nigel Jones <jonesn@uk.ibm.com>
Co-authored-by: Gabe Goodhart <ghart@us.ibm.com>
2026-10-16 00:41:01 +02:00
Matt Rutkowski e287bd8729 convert : fix regression in mapping of Granite speech tensors (#30628)
Signed-off-by: Matt Rutkowski <mrutkows@us.ibm.com>
2026-10-16 00:31:14 +02:00
Said Taghadouini 2f729cfa4c mtmd: lightonocr does not require longest edge to always be 1540, native resolution is the default (#30600)
Assisted-by: Codex
2026-10-15 23:53:02 +02:00
ba3c05f71d model: add Muse Spark Open (text, vision and audio) (#30616)
* model: add Muse Spark Open (text, vision and audio)

Muse Spark Open is a latent space MoE: the residual stream stays at 8192
while the 256 experts live in 4096, with a projection down and up around
them, top-8 sigmoid routing with a correction bias. Attention interleaves
three sliding layers at a 2048 window with rope for one global layer
without rope, with a sigmoid output gate and a weightless QK-norm.
Residuals are norm preserving, their gate is folded at conversion time.

Vision reuses the Muse Glimmer tower. Audio is a causal convolutional
pyramid with no attention and no spectrogram, four stages folding 16, 6,
5 and 4 samples into the channel dim, one embedding per 80 ms of 24 kHz.

The quantizer keeps the latent projections and the attention gate at
Q8_0, a little over 1% of the weights, since the whole residual stream
passes through them and the low-bit types turned the output into a
repeating loop. The model saver writes the post norm epsilon, and the
arch is registered in test-llama-archs.

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>

* Muse Spark: drop the redundant forward expand

* Muse Spark: drop the redundant forward expand and tighten the comments

* Update src/llama-quant.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update src/llama-graph.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update conversion/muse_spark.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update conversion/muse_spark.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update conversion/muse_spark.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update conversion/muse_spark.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-10-15 23:27:12 +02:00
Georgi Gerganov b76b13379d pi : expand AI disclosure with hardware and stats (#30615)
* pi : expand AI disclosure with hardware and stats

Prefer PI_HARDWARE for the hardware field, spell out the disclosure block
template (model link, harness, hardware, stats) and source [STATS] from
~/.pi/stats.sh. Record the hardware in the Assisted-by commit tag.

Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)

* pi : drop square brackets from placeholders

Brackets inside a markdown link text break the link on GitHub. Describe the
disclosure bullets in words instead of embedding a bracketed template.

Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
2026-10-15 20:32:52 +03:00
Georgi Gerganov 620d2225cc llama.cpp : bump version to 0.7.0 (#30603) 2026-10-15 16:48:22 +03:00
Kartik Gulia cd0a5b6a18 models : add Nemotron Super 3.5 VL and Nemotron Bolt 74B A9B support (#30520)
* add support

* remove unneeded bos fix

* add video flag

* fix flake error

* remove video-specific budgeting

* convert : fix lilicorr-dflash bid clash

* spec : enforce lilicorr n_tokens condition

* spec : fix lilicorr walk

[no release]
2026-10-15 16:34:56 +03:00
Georgi Gerganov a11b6530b5 scripts : add ci-status.sh to list active CI runs (#30586)
Small helper around the GitHub Actions API: lists the workflow runs that are not
finished yet (in_progress, queued, pending, waiting, requested), grouped by status.

Supports selecting runs by status, by branch (server-side, to reduce API traffic)
or by a regex on the branch name, and can cancel the selected runs via --kill.

Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
2026-10-15 16:28:09 +03:00
Pascal 61ad80bf93 server: fix the off-by-two in the SWA reuse threshold (#30580)
The first evaluated token attends to the positions above
pos - n_swa, so a SWA state starting at pos - n_swa + 1 holds the
whole window, but the reuse threshold asked for two more cells.
A saved SWA state keeps only the cells inside its window, so a
state read back from a slot file or from the RAM prompt cache was
always rejected and the slot rolled back to a checkpoint.

The SWA threshold now matches the window, while a state without SWA
keeps the exact bound. The test harness gains override_kv, used to
give the SWA test model a window smaller than the test context.

[no release]
2026-10-15 16:25:19 +03:00
Mikhail PodvitskiiandSigbjørn Skjæret a6b91d8c5e model : Mellum - read MoE gating params from GGUF, unify RoPE (#30578)
* model : mellum - read MoE gating params from GGUF, unify RoPE

- read/write expert_weights_norm and expert_gating_func instead of hardcoded values (defaults unchanged)
- use per-layer rope params via get_rope_freq_base/scale
- check n_expert once, outside the layer loop

* Update conversion/mellum.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

[no release]
2026-10-15 16:24:47 +03:00
Pascal ea736c1d5c server: save the draft state in the slot file (#30574)
The slot file held the target context and the context checkpoints,
but never the draft context, so a restored slot drafted with an
empty or stale draft KV until the cache refilled.

The file now ends with the live draft state, the full draft context
and the speculative state, in the same appendix style as the
checkpoints. A draft state that does not fit the draft context is
dropped and the draft starts empty, while older files and older
servers keep working.

[no release]
2026-10-15 16:24:19 +03:00
Pascal b0d239164b server: prefer empty slots and keep states overwritten by child tasks (#30567)
Slot selection takes free slots by id or by last use only, so child
tasks often land on a slot holding a parent state while empty slots
are idle, and overwrite it without saving it to the prompt cache.

Free slots are now ordered empty first, then least recently used, for
both the LRU fallback and the child slots, and a child task saves the
state it overwrites. With 4 grouped decision requests in flight and
12 states in rotation, prompt cache reads go from 5 to 36.

[no release]
2026-10-15 16:24:01 +03:00
Georgi Gerganov c0cb53ac4e sync : ggml 2026-10-15 16:20:14 +03:00
Georgi Gerganov 4ed1c8507e ggml : bump version to 0.27.0 2026-10-15 16:20:14 +03:00
Sigbjørn Skjæret 711780ffc7 ci : bump ty to 0.0.88 (#30587)
* bump ty to 0.0.88

* arange -> linspace.tolist() to pass expected type
2026-10-15 14:15:55 +02:00
Aleksander Grygier fd55cb98fe fix(ui): take the agentic turns default from DEFAULT_AGENTIC_CONFIG (#30582)
The registry hard-coded 10 for AGENTIC_MAX_TURNS while DEFAULT_AGENTIC_CONFIG
said 100. The registry seeds SETTING_CONFIG_DEFAULT, so the loop's
`|| DEFAULT_AGENTIC_CONFIG.maxTurns` fallback never fires and users got 10
turns. Reference the shared constant, as the adjacent MCP timeout does.

Assisted by: llama-ui (ggml-org/Qwen3.8-Flash-Next:IQ4_NL)
2026-10-15 12:36:06 +02:00
Karthik VijayanandKarthik Vijayan 3e4dfa8f39 models : add maion-coder (#29778)
* Add Support to maion-coder arch

* iAdd Support to support Draft models

* Address comments

* Remove maion context shift function

---------

Co-authored-by: Karthik Vijayan <kavijay@microsoft.com>
2026-10-15 12:18:46 +03:00
William Tan 08698c81ed jinja : improve context copy constructor performance (#30562)
Assisted-by: Codex
2026-10-15 11:00:57 +02:00
Yuri Khrustalev 82e1e2fd84 jinja : write nan and inf in tojson as json.dumps does (#30483)
tojson wrote nan, inf and -inf for these floats, as values and as dict
keys; json.dumps writes NaN, Infinity and -Infinity. The string and safe
filters of a float keep nan and inf, as Python's str() does, so they no
longer use the tojson function. float_to_json is renamed float_repr, as
it now gives str(); float_json gives the json.dumps text.

Also say in a comment why a last digit 9 is not bumped at a power of 2.
2026-10-15 10:49:40 +02:00
losewayy 47f5a64ae1 vulkan : fix concurrent first device init (#29766)
* vulkan : fix concurrent first device init

* tests : load models in parallel in test-thread-safety

Loading them serially never exercised concurrent first backend init, so the crash reported in #29764 was invisible to this test.

* vulkan : create fence before retaining device in buffer context
2026-10-15 10:37:35 +02:00
Neo Zhang e36df60470 sycl : replace oneAPI package by DLE+oneDNN package (#30131)
* replace oneAPI package by DLE+oneDNN package

* support to install oneDNN in windows, in release and docker

* for internal test

* remove test code

* Update intel.Dockerfile
2026-10-15 09:26:01 +02:00
lhezandHongqiang Wang 9166ceaf16 opencl: fix OOB tail access for adreno noshuffle gemv kernels (#30421)
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
2026-10-15 10:21:18 +03:00
William Tan 2130ca8476 jinja : fix loop body variable scoping (#30533)
Assisted-by: Codex
2026-10-15 09:14:01 +02:00
Adrien Gallouët 23f83e056d ggml-cpu : optimize pad_f32 (#30526)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-10-15 09:13:18 +02:00
Simone IacominoandSigbjørn Skjæret 2ce20d22fd unicode : reject overlong UTF-8 and out-of-range codepoints (#30485)
* unicode : reject overlong UTF-8 and out-of-range codepoints

* Update tests/test-unicode.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* tests: document byte-wise UTF-8 error recovery

* break out of first test on failure

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-10-15 10:12:56 +03:00
Adrien Gallouët 68ac765726 llama : split llama_prefetch() into merge and prefetch steps (#30532)
* llama : split llama_prefetch() into merge and prefetch steps

Merge the ranges first, then call the system functions.

This fixes -Wunreachable-code-return on Windows.

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Use const_cast

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

---------

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-10-15 09:12:45 +02:00
lhez 77c1f39929 opencl: apply flash attn softcap before mask (#30546) 2026-10-15 10:11:51 +03:00
Nathan WilsonandRuben Ortlam 733f57a999 vulkan : copy strided f16 KV once on AMD so reads spread across memory channels (#27703)
* vulkan : copy strided f16 KV once so reads spread across memory channels

Assisted-by: Claude (Opus 4.8 / Opus 5)

* tests : cover the f16 KV copy to scratch

Assisted-by: Claude (Opus 4.8 / Opus 5)

* vulkan : gate the f16 KV scratch copy to AMD

Assisted-by: Claude (Opus 4.8 / Opus 5)

* vulkan : reuse the generic copy shader for the f16 KV scratch fill

The f16 clause needs only a strided copy, not a dequant, so drop the
dedicated shader and dispatch the existing copy pipeline instead. The
copy is described in source memory order so the strided access lands on
the writes, which stream, instead of the reads, which would concentrate
on a few memory channels.

Assisted-by: Claude (Opus 4.8 / Opus 5 / Fable 5)

* vulkan : extend the strided KV scratch copy to bf16

Assisted-by: Claude Opus 5

* vulkan : clean up comments

Assisted-by: Claude Opus 5

* consolidate copy shaders

* disable on RDNA4

---------

Co-authored-by: Ruben Ortlam <rortlam@redhat.com>
2026-10-15 10:09:02 +03:00
cqderekandravi9 5a638aedb7 cpu: add F16 SWIGLU_OAI reference implementation and validation (#29200)
* cpu: add F16 SWIGLU_OAI reference implementation and F16 activation op test coverage

Adds ggml_compute_forward_swiglu_oai_f16() to the CPU backend, mirroring
the existing F32 path, and extends test-backend-ops with F16 test cases
for SWIGLU_OAI so downstream backends(e.g. Hexagon) can be CPU-diffed
for correctness.

* ggml-openvino: fix f16 SWIGLU_OAI type mismatch in alpha/one constants

translate_glu_swiglu_oai creates alpha_const and one as f32 constants
unconditionally, so f16 inputs hit a type mismatch in Swish/Add on GPU
("Swish inputs must have the same type but they are: f16 and f32").
translate_glu_swiglu_clamp already upcasts to f32 and converts back at
the end; apply the same pattern here.

---------

Co-authored-by: ravi9 <ravi.panchumarthy@intel.com>
2026-10-15 10:07:45 +03:00
Ethan Guo 6e66cb5d4e metal : add BF16 variant of the tensor API flash attention kernel (#30491)
* metal : add BF16 variant of the tensor API flash attention kernel

* cont : move tensor FA kernels to kernels/fa_tensor.metal
2026-10-15 09:39:47 +03:00
PascalandGeorgi Gerganov e467e5b5e5 kv-cache: honor PARTIAL_ONLY in the plain KV cache (#30514)
* llama: honor PARTIAL_ONLY in the plain KV cache

A non-SWA llama_kv_cache rolls back with seq_rm, so it holds no
partial state, yet state_write ignored the flag and serialized every
cell. A draft context made of a plain KV cache (MTP head, dense -md
draft) therefore copied its whole KV into each server checkpoint,
growing it by the draft KV size per token.

The cache now writes and reads nothing for PARTIAL_ONLY unless it is
SWA, and the iSWA wrapper no longer gates its base cache since it
filters itself.

* Update src/llama-kv-cache.cpp

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* llama: delegate PARTIAL_ONLY to the caches of every memory wrapper

The hybrid, hybrid-idx and dsa-iswa wrappers skipped their attention
caches on PARTIAL_ONLY, which also dropped an SWA window that is
partial state. Each cache now applies the flag itself, so the
wrappers only delegate. The indexer is marked stale only when a
restore rewrites cells.

test-llama-context gains a case checking that the partial state is
not larger than the full state and, without SWA, does not grow with
the tokens.

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-10-15 07:39:18 +03:00
Pascal 1ddc262558 graph: keep the graph non empty without embeddings (#30522)
An embedding-only graph has no logits, and build_pooling() returns
early when embeddings are off, so since the models no longer expand
t_embd the graph ends up empty and its inputs are never allocated.
Expand t_embd in that case.
2026-10-15 06:38:12 +02:00
Pascal 2639a3b4e0 server: save only restorable slot states to the prompt cache (#30498)
* server: save only restorable slot states to the prompt cache

The idle-slot save loop stores every idle slot whenever a task starts,
while only completion tasks restore from the prompt cache. Rerank,
embedding, decision and infill states fill the cache up to its limit
without ever being read back, and slots already cached are offered
again at every task start.

Each slot now tracks with cache_pending whether its state is restorable
and missing from the cache: a completion release or an explicit slot
restore sets it, prompt_save consumes it, prompt_clear resets it. Both
save paths share this rule, and the LRU path restores only for
completion tasks.

* server: test that decision slots skip the prompt cache

Decision tasks never mark their slot as cacheable, so starting new
tasks must not save their idle slots to the RAM prompt cache, with or
without a unified KV cache.

* server: restore any reusing task from the prompt cache

Embedding and rerank tasks never reuse their state, so their slots are
cleared on release like child slots and the cache only holds states
that a later task can restore. Infill and decision tasks now restore
from the cache too, which replaces the cache_pending flag.

Decision models disable the prompt cache unless --cache-ram is given,
since their states seldom repeat.
2026-10-15 06:30:17 +02:00
Kartik GuliaandGeorgi Gerganov 843e7dcc15 spec : add lilicorr support (#30506)
* spec : add lilicorr support

* convert : use convention for hparam assignment

* add todo for hparam use in llama-batch.cpp

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* typo fix

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-10-15 05:49:25 +02:00
Ruben Ortlam 3e390af18d vulkan: Use L2-friendly workgroup scheduling for cm2 mulmat (#30215) 2026-10-15 05:26:10 +02:00
Hongqiang Wang a0e6dc7384 opencl: support CLAMP on non-contiguous views (#30186)
The clamp kernel indexed source and destination as one flat contiguous array, so
a strided view as input (the new CLAMP v=1 case) read the wrong elements. Add a
strided kernel_clamp_nc, as the other unary ops have, and take it whenever the
source is not contiguous.
2026-10-14 13:02:27 -07:00
stormy 23deec0255 devops: incorrect COPY instruction (#30529) 2026-10-15 02:09:28 +08:00
Sigbjørn Skjæret 683bbc0dce gguf-py : detect nfc normalization (#30540) 2026-10-14 20:08:09 +02:00
Anant Shrivastava 529ac893e7 sycl: IQ3 code reorder (#29107)
* sycl: persistent IQ3_S/IQ3_XXS weight reorder for SYCL decode and prefill

Add persistent reordered weight layouts for IQ3_S and IQ3_XXS. Weights
are reordered once at load time; the reordered MMVQ dot products then
walk dense super-blocks with dp4a and SWAR sign expansion instead of
gathering bits from the interleaved standard layout. Reorder-aware
dequantization feeds the batched prefill GEMMs so prompt processing
sees the same benefit.

Also harden the oneDNN build check: a missing DNNL package or a
CPU-only DNNL build now fails at configure time instead of silently
producing a degraded backend.

Measured on Intel Arc Pro B70 with Qwen3.8-27B-GSQ-RCO-IQ3_S (MTP),
-ngl 999 -b 4096 -ub 1024 -fa on:

tg256 @ 0 ctx      12.92 -> 23.54 tok/s (+82%)
tg256 @ 9,216      12.33 -> 21.69 tok/s (+76%)
tg256 @ 64,000      9.76 -> 14.91 tok/s (+53%)
pp8192             833.9  -> 833.6 tok/s (flat)

Qwen3.5-9B Q4_K_M / Q3_K_M are unchanged within noise (largest delta
-0.5% pp, tg flat), so standard quants are not affected.

The MMVQ GLU fusion widening and rms_norm+scale fusion are
intentionally not part of this change; master already carries them
via #28931.

* removing wait as per review request

* sycl : fix fusion.cpp line ending
2026-10-14 09:41:23 -07:00
Leebr Data Consulting 685f70894b ggml-cpu : keep the FP16->FP32 lookup table on x86 with F16C (#30342) (#30343)
* ggml-cpu : keep the FP16->FP32 lookup table on x86 with F16C (#30342)

#30270 routed both scalar conversions through F16C. For FP16->FP32, Clang 20.1.8
lowers the intrinsic to a vpinsrw merging into the live accumulator register of
the quantized dot products, which serialises the loop (19 vs 4 cycles/iteration
estimated, 45-65% slower Q8_0/Q4_0 MUL_MAT_ID kernels measured). Keep F16C for
FP32->FP16 as intended and the table for FP16->FP32.

* ggml-cpu : comment out the F16C FP16->FP32 define instead of removing it (review)
2026-10-14 19:36:06 +03:00
Aaron Teo c114bd4c58 docs: update IBM zDNN backend docs (#30527)
* docs: update zdnn lib install + add notes on running models

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* docs: change title

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* docs: add note about minimum version

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

---------

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
2026-10-14 19:17:50 +03:00
Aman Gupta 5cdadfc326 ggml-cpu : size set_rows conversion buffer for all threads (#30454)
* ggml-cpu : size set_rows conversion buffer for all threads

SET_ROWS is planned with n_tasks = 1, but every thread runs it and uses its own row of the F16 -> F32 conversion buffer, so threads > 0 wrote past the work buffer when the graph had another multi-threaded op.

* move set_rows to n_task = n_threads group
2026-10-14 18:31:53 +03:00
62df1815bc cuda, hip: GDN chunked kernel (#29353)
* cuda, hip: GDN chunked kernel

Ported-from: https://github.com/tetherto/qvac-fabric-llm.cpp/pull/257
Source-commit: 53f0027d8070715fe9ad2131f3201bb0001d4eed
Co-authored-by: Aman Gupta <aman.gupta@tether.io>

* fix older builds

* review fixes

* cuda: enable GDN MMA on tested Ampere and Ada GPUs

Enable SM86 and SM89 after RTX 3090 and RTX 4090 correctness and prefill benchmarks. Keep other untested architectures on the existing fallback.

Assisted-by: Codex

* cuda : use refactored swizzle API in GDN MMA

* ggml : GDN only guarantees the final state when n_tokens > K

* cuda : add short GDN MMA phase comments

---------

Co-authored-by: gianni-cor <gianfranco.cordella@tether.io>
Co-authored-by: Aman Gupta <aman.gupta@tether.io>
2026-10-14 22:00:58 +08:00
Aleksander Grygier 37d6d653e4 skill: Update code-review skill for UI development (#30509)
* skill : add tools/ui review checklist to code-review skill

Adds a WebUI (tools/ui) section distilled from a year of review
feedback on tools/ui PRs: constants/enums placement, layering
(stores/services/utils/hooks), DRY and dead code, Svelte 5 idioms,
comment hygiene, streaming perf, frontend security, and PR hygiene
(static build, demos, scope discipline). Scoped to tools/ui only.

* skill : expand tools/ui constants/enums ruleset

Replaces the short constants bullet list with a decision ruleset
distilled from review outcomes: when a literal becomes a string
enum vs an enum-keyed map vs a declarative config vs a grouped
as-const object vs a scalar constant, naming and suffix
conventions (_REGEX, _MS, _PX, semantic separators), placement
and composition rules, and what legitimately stays inline.

* skill : drop stale static-build-commit requirement for tools/ui

Static build output is no longer stored in the repo - the UI is
built by CI and bundled at release time. Keep the requirement that
npm run build succeeds and the UI build job is green.

* skill : address copilot review of tools/ui checklist

- scope the video-demo requirement to user-facing interactive/visual
  changes instead of every tools/ui PR
- narrow the constants trigger to reused/domain-significant literals,
  keeping the inline boundary as the escape hatch
- carve out independent literals for persisted/wire-contract test
  assertions so they don't become tautological via shared constants
- allow simple presentation-local $derived in components; only shared
  business logic, orchestration and parsing leave the component
- match the existing barrel documentation style instead of requiring
  per-export JSDoc everywhere
- keep non-obvious why/rationale comments; move only examples and
  edge-case enumerations into tests
- point at the actual interpolate-size/grid-template-rows transition
  pattern in CollapsibleRegion.svelte instead of calc-size()

* skill : use spaces for nested list indentation

EditorConfig CI requires space indentation for all files; the
scalar-constants sub-bullets used tabs. Matches the 2-space nested
list convention used in docs/.
2026-10-14 15:20:27 +02:00
Pascal 6868a52cb3 ggml: fix conv_1d result layout for batched input (#28738)
* ggml: fix conv_1d result layout for batched input

The GEMM behind conv_1d folds the batch into its rows, so the product
sits in memory as [OL, N, OC] while the reshape claims [OL, OC, N].
The shape is right and the contents are transposed, so nothing fires:
no assert, no abort, the graph just runs and the channels come out
interleaved with the batches. With N == 1 the two readings describe
the same buffer, which is why the three vocoders in the tree never
hit it, they all squeeze to 2d right after the call.

Reshape to the real layout and permute OC and N back. The N == 1 path
keeps its direct reshape, so the current callers build the same graph
and pay for no extra copy.

tests/test-conv-1d.cpp compares one batched convolution against N
single batch convolutions built from the same weight and the same
input slices. Five of the seven geometries fail before this change and
all seven pass after. N == 1 and OC == 1 pass either way, since
swapping two dimensions when one of them is 1 changes nothing.

The batched path matters to downstream audio projects where several
synthesis lanes share a single graph. Nothing in mtmd batches a
vocoder today, each one streams one flow and carries its left context
between calls, but the op is correct now if one of them grows a lane
dimension.

* ggml: build the same conv_1d graph for every batch size

Address review from ggerganov: drop the N == 1 branch of ggml_conv_1d.
The GEMM result is always reshaped to [OC, N, OL] and permuted back,
as ggml_conv_2d does, so every batch size builds the same graph. For
N == 1 this adds a cont on an already ordered tensor.
2026-10-14 14:16:08 +03:00
Georgi Gerganov 3fb47677ea common : add --spec-draft-n-ubatch (#30459)
* common : add --spec-draft-n-ubatch

Allow configuring the physical batch size of the draft context. The
drafters feed it at most the drafted tokens per sequence and chunk larger
batches by its own n_ubatch, so it does not need the ubatch of the target.
A smaller value shrinks the draft compute buffer and leaves more VRAM for
the target KV cache (the draft is fitted together with the target).

The default 0 keeps the current behavior of following --n-ubatch.

Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next

* cont : remove comment

[no ci]
2026-10-14 13:58:05 +03:00
Francisco Herrera 318a471a62 docs : clarify installation recommendation for Intel oneAPI Toolkit (#30407)
Added that the oneapi toolkit includes the sycl dependencies. This makes it clear for first time users, that the toolkit includes all necessary dependencies, so that the user is guided to install the toolkit if they do not already have the dependencies separately
2026-10-14 12:19:10 +02:00
Georgi Gerganov 8c2134a875 ggml : add note about mul_mat_id + disable mul_mat_id_dup tests (#30501) 2026-10-14 12:36:52 +03:00
Pascal df18bee11e model: split layers evenly when no device reports free memory (#29677)
When every device reports zero free memory the split points are
normalized by a zero sum, the NaN split sends layers to a device index
past the end and loading aborts with vector::_M_range_check.
2026-10-14 09:30:00 +03:00
Adrien Gallouët 3eb101360f vendor : update cpp-httplib to 0.61.0 (#30494)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-10-14 08:28:24 +02:00
Georgi Gerganov af0579c78a ggml : improve RPC tensor transfer logs (#30481)
- add rpc_log_format_size() and print byte counts in KiB/MiB/GiB across the
  RPC client, server and transport traces
- add rpc_log_format_tensor() and include tensor name, op, type and shape in
  the set/get/memset/copy/init tensor logs
- fix rpc_cmd_name() returning UNKNOWN for SET_TENSOR_2D, GET_TENSOR_2D,
  COMM_INIT, COMM_ALLREDUCE and COMM_FREE
- move the log helpers to ggml-rpc/log.{h,cpp} under a common rpc_log_ prefix

Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
2026-10-14 09:23:44 +03:00
An Longandthelittlefireman 62bacfa011 cuda : use SIMT implicit GEMM conv2d on AMD without WMMA (#30325)
* cuda : use SIMT implicit GEMM conv2d on AMD without WMMA

Assisted-by: Claude Opus

* cuda : update conv2d SIMT comment

* cuda : skip SIMT conv2d on AMD for short reductions

Co-authored-by: thelittlefireman <5165783+thelittlefireman@users.noreply.github.com>

---------

Co-authored-by: thelittlefireman <5165783+thelittlefireman@users.noreply.github.com>
2026-10-14 09:22:11 +03:00
Yuri Khrustalev 77390baa18 jinja : write floats in tojson as Python does (#30362)
tojson printed floats through a C++ stream (6 significant digits, no
".0"), so 48.858844 became 48.8588 and 1.0 became 1 in tool definitions,
tool-call arguments and other template output. Write the shortest digits
that read back as the same value, as json.dumps does. The string and safe
filters of a float use the same code.
2026-10-14 09:18:38 +03:00
lhez c5a997624a opencl: use buf+offset instead of subbuffer for adreno moe expert sort (#30418) 2026-10-14 09:17:09 +03:00
lhezandHongqiang Wang 21a19cd3be opencl: fix q8_0 dp4a mul_mat_id and mark PREC_F32 mul_mat_id unsupported for adreno (#30365)
* opencl: fix min handling for moe q8_0 dp4a GEMM

Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>

* opencl: reject PREC_F32 MUL_MAT_ID for adreno

---------

Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
2026-10-14 09:16:54 +03:00
Pascal 9cf967a04d models: drop the t_embd expand of the embedding graphs (#30461)
These graphs only produce embeddings, and build_pooling() already
expands its output for every pooling type, so the expand at the end of
the model graph is redundant.
2026-10-14 08:15:08 +02:00
liangliangchang 4dd6dd4b96 HIP: use 64-row MMQ tiles for some quant types on RDNA3.5 (#28195)
The RDNA3.5 config table selects a 64-row / 128-thread tile up to J=32 and a
128-row / 256-thread tile above that. On gfx115x the smaller tile leaves enough
LDS free for a second workgroup per CU, while the larger one does not.

Extend the 64-row tile to the (type, J) pairs where it measured faster on
gfx1151, and leave every other entry untouched. J=96 is deliberately excluded
because it did not benefit.

test-backend-ops perf on gfx1151, 100 MUL_MAT shapes over the affected types
(N in 48..512, M both padded and unpadded), 3 rounds: median +2.1%, best
+26.0%, no shape outside run-to-run noise. 5 MUL_MAT_ID shapes: median +8.0%.

Assisted-by: Claude Opus 5
2026-10-14 08:10:29 +02:00
Ruben Ortlam 14310190cd vulkan: fix compile error (#30495) 2026-10-14 09:06:09 +03:00
Georgi Gerganov e7268bf6a9 llama : build the k-pool selection in the causal mask layout (#30480)
* llama : build the k-pool selection in the causal mask layout

- the glm5-next indexer folded the causal mask in through a [n_kv, n_tokens]
  view: one reshape of the mask ahead of the add, and one of the result ahead
  of every layer reading the selection, including the shared-indexer layers
- build the selection directly in the mask layout, as qwen4exp already does,
  and drop the now-degenerate reshape in its attention as well
- assert the selection width matches the input it is taken from

Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next

* cont : remove comment
2026-10-14 08:45:55 +03:00
Masato Nakasaka cdce056a3e vulkan: Add workaround for device free memory > total memory on Intel Windows (#29835)
* add workaround for Intel Windows driver issue

* applying safeguard to all GPUs

also a minor refactor

* partial revert
2026-10-14 08:45:38 +03:00
Matt Corallo d524bb660b vulkan: Define the shader count for Intel Arc Pro B65 + B70 (#30431) 2026-10-14 08:44:22 +03:00
Ruben Ortlam 7ec420bc09 vulkan: support mulmat bf16 activations (#29069)
* vulkan: support bf16 src1 in mul_mat

* clean up mul_mat b-type selection

* fix conditions
2026-10-14 08:36:32 +03:00
Sandro Hanea 10da8c463f unicode : add opt-in NFC normalization for Unigram tokenizers (#30390)
* unicode : add opt-in NFC normalization for Unigram tokenizers

Note: used help from Codex, Sol 6.1 Medium

* Fixing initial bot comments + linter error (using existing out instead of print)

* removed test_nfc_tokenizer
2026-10-14 08:33:24 +03:00
Takamichi Ohara eca5458ce0 convert : export YaRN truncation config (#30288)
* convert : export YaRN truncation config

* rename
2026-10-14 08:31:40 +03:00
Raffaele Di DonnaandRaffaele Di Donna 091b6a7fc5 Enable OpenSSL in FP32/FP16 SYCL builds to restore HF HTTPS downloads (#30330)
Co-authored-by: Raffaele Di Donna <235519540+raffaeleedidonna@users.noreply.github.com>
2026-10-13 22:25:26 -04:00
Masashi Yoshimura fcac19079f webgpu: (Fix CI) fix mul_mat_id with duplicate expert ids (#30451)
* Fix mm-id of webgpu in the case of ids duplication

* formatting

* fix comments to avoid breaking up sentences in comments
2026-10-14 08:24:23 +09:00
Yuri Khrustalev 04f7226a2c server : escape the special tokens without std::regex (#30243)
The d1-omni and kev prompts escape "<|name|>" in the request text with
std::regex_replace. libstdc++ recurses for each char of the name, so a
state of "<|" and 30000 letters overflows the stack of the HTTP thread
and llama-server dies with SIGSEGV.

server_decision_escape_special_tokens() writes the same text with a loop.
test-server-component compares it with the regex on every string of up to
6 chars of "<|>a-".
2026-10-14 01:17:11 +02:00
Xuan-Son Nguyen ff026f8c0a chat: replace output string with parts (#30389)
* chat: replace output string with parts

* reserve first
2026-10-14 01:03:56 +02:00
Adrien Gallouët 1b70350c92 ggml : simplify backend search path error handling (#28367)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-10-13 21:22:19 +03:00
Aaron Teo d71e324a44 docs : update zDNN supported ops (#30472)
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
2026-10-13 20:20:41 +02:00
Charles Xu 45451c366d kleidiai: add SME2 Q4_K and Q6_K matmul support (#29219)
Change-Id: Ib4aa4c9814c010f9cf084783cc80612e364f9a27
Change-Id: I406a6606c3fea51f8cd32331c1ed71466845d363
Change-Id: Ib4aa4c9814c010f9cf084783cc80612e364f9a27

Signed-off-by: Charles Xu <charles.xu@arm.com>
2026-10-13 09:37:48 -07:00
Jeff Bolz 77c4540ce4 vulkan: Optimize pool_2d when the pooling window is large (#30119)
When the window is larger than a subgroup, use a whole subgroup to
compute each output.
2026-10-13 17:17:59 +02:00
Igor Matuszewski ecdd29e8b2 meta: recreate all subgraphs when the arena is reallocated (#30453)
* meta: recreate all subgraphs when the arena is reallocated

* chore: revert added test cases
2026-10-13 17:58:35 +03:00
Pranav DhinakarandMax Krasnyansky ec0bfac179 hexagon: add configurable HTP DCVS/HMX power mode (#30423)
* hexagon: add configurable HTP DCVS/HMX power mode

* hexagon: address PR review feedback (drop CLI flag, align QAIRT power modes, comment/log cleanup)

* hexagon: add --hex-power-mode flag to run.py

Hooks GGML_HEXAGON_POWER_MODE into the unified run.py runner, following
the existing --hex-nhvx pattern. The env var must be set before the
process touches the Hexagon backend registry (see comment in
ggml_hexagon_init), so this is exposed as a runner flag rather than a
common/arg.cpp CLI option.

* hexagon: keep HTP awake for DCVS power profiles

* Apply suggestion from @max-krasnyansky

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-10-13 07:47:55 -07:00
Pascal ace20f679e graph: pad the KQ bias of the no-cache attention (#30446)
The no-cache attention pads K, V and the mask to 256, but T5 also adds
a relative position bias of [n_tokens, n_tokens], so the add with KQ
fails unless n_tokens divides the padded width. Pad the bias with
zeros up to the KV dimension of the mask, like K and V.
2026-10-13 16:34:10 +02:00
Georgi Gerganov b9f3599fb1 llama : do not map foreign ranges into a backend buffer (#30405)
* llama : do not map foreign ranges into a backend buffer

A buffer created from a host pointer pins everything that it wraps. When
several contexts share the same file (lazy tensors, MoE tensors kept on the
CPU with -cmoe), the single buffer that spans the first and the last tensor
of a context also wraps the ranges owned by the other contexts, pinning data
that the backend never reads.

For models with per-layer token embeddings this pins the whole lazy tensor
into the GPU buffer (27 GiB for Qwen3.8-Flash-Next IQ4_NL), defeating the
point of reading it lazily.

Instead, compute the file ranges of each context and subtract the ranges
owned by the other contexts, then create one buffer per range. Weights that
are mapped in more than one context (e.g. token_embd.weight) keep being part
of the range of every context that uses them.

Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next

* llama : enable prefetch on macOS

The prefetch hint of the tensors marked as prefetchable is a no-op on macOS,
so their pages are faulted in one by one while the graph runs. madvise
(MADV_WILLNEED) behaves as expected here, so add the platform to the guard.

With the lazy per-layer token embeddings of Qwen3.8-Flash-Next IQ4_NL read
from a cold file cache, prompt processing goes from 356 / 375 / 475 t/s to
735 / 749 / 788 t/s for pp512 / pp2048 / pp4096.

Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next

* llama : use a plain buffer list per context

Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
2026-10-13 17:28:15 +03:00
Adam CaldwellandAdam Caldwell f441d2d9db model: load YaRN beta metadata for laguna models (#30331)
Assisted-by: Codex

Co-authored-by: Adam Caldwell <accaldwell@users.noreply.github.com>
2026-10-13 22:18:01 +08:00
Pascal cb55ffe6e2 graph: skip the LM head for pooled embeddings (#30445)
* graph: skip the LM head for pooled embeddings

A context that extracts pooled embeddings outputs one vector per
sequence, but every model still expanded its LM head, so the graph
computed n_vocab logits per token and the context kept a host buffer
for them.

The models now only declare t_logits, and set_outputs() expands it
unless the context extracts embeddings with a pooling type other than
NONE. Since the toggle changes the graph shape for a pooled context,
set_embeddings() schedules a re-reserve in that case.

* graph: drop the redundant t_embd expand

Address review from @CISC: with embeddings on, build_pooling() already
expands its output for every pooling type, and a decoder graph without
embeddings does not read t_embd, so set_outputs() only expands
t_logits.

* graph: add build_logits() for the LM head output

Address review from @ggerganov: the LM head is expanded by a dedicated
build_logits(), called from llama_model::build_graph() like
build_pooling(), and set_outputs() only marks the outputs again.

* graph: call build_logits() before build_sampling()

Address review from @ggerganov: build_sampling() reads t_logits, so the
LM head is built first. A pooled context then skips the sampling layers
too, since it has no logits to sample from.

* nit
2026-10-13 16:16:14 +02:00
Georgi Gerganov a17e94d229 tests : consolidate context state tests into test-llama-context (#30023)
* tests : consolidate the context state tests into test-llama-context

test-save-load-state, test-recurrent-state-rollback and
test-state-restore-fragmented check the same object from three places:
the KV cache and the recurrent state of a llama_context, saved, loaded,
copied, rolled back. Each had its own main, its own loop over the
models, its own copy of the token and the comparison helpers, and a run
of all three loaded the same models several times.

They are one binary now, and the case is the unit of the test:

  tests/llama-context/test-case.h   the interface a case implements
  tests/llama-context/cases-kv.cpp  the cases of the KV cache
  tests/llama-context/cases-rs.cpp  the cases of the recurrent state
  tests/llama-context/test-util.cpp tokens, generation, comparison
  tests/llama-context/test-runner.cpp the models of a case, on threads
  tests/llama-context/test-log.cpp  the log file of a model

A run is a table: a row per model, a column per case, a cell holds the
verdict of that pair. The models load on several threads (-j, --threads,
min(cores, 4) by default), the table is printed when it is complete and
the log of a model that failed is printed after it, in the order of the
rows. A case returns a verdict and the reason for it, and the runner is
the only one who reports - the test does not use tests/testing.h, a run
over 130 models is a matrix of verdicts and not a tree of subtests.

Staging no longer travels through the environment: llama_model_set_attn_rot
replaces the LLAMA_ATTN_ROT_DISABLE variable, which a model loaded in one
thread used to change for the models the other threads were loading, and
llama_memory_get_buffers is added for the cases that copy the buffers of
a memory. common_log_set_stdo keeps the console quiet while the table is
being built.

ggml_aligned_malloc allocates with posix_memalign when the thread
sanitizer is on. On macOS it allocates with the Mach VM, which the
sanitizer does not intercept, so it never sees a page range change hands
and reports the first write into recycled memory as a race - 30 of them
in this test, and every one of them false.

Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next

* cont : run single-threaded by default

* cont : try to build the tests on all platforms

* cont : trim down some body fat

* cont : fix

* cont : fix the fix
2026-10-13 16:21:55 +03:00
Georgi Gerganov b476d9a762 llama : fix log message (#30452)
[no ci]
2026-10-13 16:07:54 +03:00
Ruben Ortlam e0bcaccbe5 vulkan: allow multiple devices to access gpu-visible host memory (#29741)
* vulkan: allow multiple devices to access gpu-visible host memory

* clean up erroneous newlines

* add Windows support

* handle nullptr

* fix enable condition
2026-10-13 15:05:10 +02:00
homeofeandSigbjørn Skjæret 0a731cced9 ggml-cpu : add AVX2 dot product for Q2_0 (#30447)
* ggml-cpu : add AVX2 dot product for Q2_0

* fix comment [no ci]

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-10-13 14:36:47 +02:00
Georgi Gerganov f851e244d0 cuda : add GGML_CUDA_GRAPH_DEBUG and read env vars once (#30327)
* ggml-cuda: gate CUDA graph debug logs behind GGML_CUDA_GRAPH_DEBUG

* ggml-cuda: cache env var lookups in static const instead of re-reading per call

* ggml-cuda: read env vars exactly once via ggml_cuda_env_flag/ggml_cuda_env_int

* ggml-cuda: consolidate env flag reads into single static accessors

* ggml-cuda : drop env helper comments

Assisted-by: pi:llama.cpp/Qwen3.8-27B

* cont : remove NDEBUG gates for CUDA graph logs
2026-10-13 15:24:34 +03:00
Adrien Gallouët bfe65e1dfa cuda : fix compiler warnings in mmq-load-tiles.cuh (#30443)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-10-13 13:48:49 +02:00
losewayy 411d548515 vulkan : split matmul dispatch when it would exceed maxComputeWorkGroupCount (#29533)
* vulkan : split matmul dispatch when it would exceed maxComputeWorkGroupCount

* vulkan : fix narrowing in chunked matmul dispatch

* vulkan : zero fill the A side of mul_mmq for k steps past the end

The A side skipped the shared memory write for the k steps past end_k, while the B side was already told the bound and zero filled. The A registers are read for all BK_STEP steps, so the stale scale went into the dot product and produced NaN at k = 32.

Pass the bound to block_a_to_shmem and write zeros when it is false, for every DATA_A type, mirroring the B side.

Also assert that the B and D subbuffer sizes are not VK_WHOLE_SIZE before they are adjusted per chunk, and add two batched cases of the chunked shape.

Signed-off-by: losewayy <104015127+losewayy@users.noreply.github.com>

* vulkan : remove redundant matmul tile adjustment

---------

Signed-off-by: losewayy <104015127+losewayy@users.noreply.github.com>
2026-10-13 13:03:47 +02:00
Oliver Simons 296538adea spec : remove tokens assert from dsv4 graph_mtp (#30444)
cont of #29622, where we forgot to remove token assert
2026-10-13 12:39:10 +02:00
Oliver Simons 7f51953d7d model : allow Qwen3.8 Flash Next to accept image embds for MTP (#30397)
Cont of #30257, closes #30307
2026-10-13 12:34:47 +02:00
Adrien Gallouët 882d67166d cuda : fix warnings from dead breaks (#30441)
* cuda : fix warnings from dead breaks

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Remove useless braces

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

---------

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-10-13 12:29:27 +02:00
Adrien GallouëtandSigbjørn Skjæret 30f7df1a83 cuda : warnings cleanup (#30420)
* cuda : warnings cleanup

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Update ggml/src/ggml-cuda/common.cuh

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-10-13 11:44:37 +02:00
Chanho Lee 9d7d504980 convert : fix LoRA conversion with --outtype auto (#30433)
* convert : fix automatic LoRA output type

* convert : remove LoRA test and CI changes
2026-10-13 10:29:09 +02:00
Animesh ed019cd8b9 llama : skip logits buffer allocation when graph produces no logits (#30255) 2026-10-13 11:24:20 +03:00
Georgi Gerganov 9880157881 graph : avoid reallocations in the "no cache" path (#30439) 2026-10-13 11:22:19 +03:00
Aaron Teo af50f2f248 ggml-zdnn : introduce Q8_0 mat mul (#29836)
* ggml-zdnn: initial prototype of q8_0 matmul

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* docs: update zdnn docs

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

---------

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
2026-10-13 09:53:48 +02:00
Muhammad SaadandSigbjørn Skjæret 071b553fb1 models : remove redundant ggml_cont calls now supported by all unary kernels (#30198)
* models: remove redundant ggml_cont calls now supported by CUDA unary kernels

* models: restore cont_2d after sigmoid for broadcast mul, clean dflash cont

* Update src/models/qwen3next.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* models: restore cont_2d before sigmoid in qwen3next

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-10-13 09:26:52 +02:00
kurquharandMax Krasnyansky 4413a19241 hexagon: extend scalar DMA broadcasting support for binary ops (#30245)
* tests: cover weighted scalar broadcasts

Assisted-by: OpenCode

* ggml : support extended scalar DMA binary ops

Assisted-by: OpenCode

* hex-binary: remove all non-dma paths

* hex-binary: add support for non-contig strides and cleanup dma preable a bit

* hex-binary: cleanup the gating and simplify dma retry

* hex-geglu-erf: add a gate for F32 only, we dont support F16 for this op

* tests: cover weighted repeat broadcast

Assisted-by: OpenCode

* hex-binary: update REPEAT and COMPLEX kernels to use dma as well (missed them last time)

* hex-add-id: add dma support for src2 (idx)

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-10-12 20:33:05 -07:00
Pratyush Kumar 8f574b0ad4 ci : build OpenVINO releases with GGML_NATIVE=OFF and CPU backend variants (#30324) 2026-10-13 10:22:05 +08:00
Pratyush Kumar a49f58dcc3 ci: fix OpenVINO download links (#30401) 2026-10-13 10:20:04 +08:00
Xuan-Son Nguyen cbe28bb3cb model: fix vllm d3 multi seq case (#30408)
* model: fix vllm d3 multi seq case

* add TODO
2026-10-13 00:23:01 +02:00
Pascal 358a420ec4 graph: pad the KV dimension of the no-cache attention to 256 (#30416)
The no-cache path builds a [n_tokens, n_tokens] mask and passes K and V
unpadded, so FLASH_ATTN_EXT sees K->ne[1] == n_tokens. The CUDA kernels
for head size 512 require a multiple of FATTN_KQ_STRIDE, so the op falls
back to the CPU for most lengths, and the -fa auto probe (1 token)
disables flash attention for such models.

Pad the mask like llama_kv_cache::get_n_kv and pad K and V with zeros up
to the KV dimension of the mask. clef builds its own causal mask and
pads it the same way.
2026-10-12 23:00:27 +02:00
48e9857567 metal: fix metal tensor kernel_mul_mm error for large tensor (#28748)
* metal: fix metal tensor kernel_mul_mm error for large tensor

Assisted-by: Codex

* Update ggml/src/ggml-metal/kernels/mul_mm.metal

Co-authored-by: YiChen Lv <63285796+forforever73@users.noreply.github.com>

---------

Co-authored-by: 向锡铭 <xiangximing@xiangximings-MacBook-Air.local>
Co-authored-by: YiChen Lv <63285796+forforever73@users.noreply.github.com>
2026-10-12 22:57:26 +03:00
8bb3cf0d21 hexagon: optimize CPY/CONCAT — round 3, short-rows edition (#30341)
* hexagon: move the converting CPY in blocks of rows per DMA descriptor so short rows are not DMA-latency bound

* hex-cpy: cleanup cpy cursor handling

* hex-cpy-concat: introduce threaded dma/hvx gather-row helper

Co-authored-by: Ruslan Karymov <67752306+karusrus@users.noreply.github.com>

* hex-cpy: remove leftover rejections for extended buffers

* hex-cpy: relax the gate for fast-path

* hex-cpy: remove scalar divs and vtcm access and use explicit gather_sync

* hex-cpy: add proper support for mdev work splitting to fastpath

* hex-cpy: optimize dense gather-row tasks

* hex-cpy: unroll gather-rows loops

* hex-cpy: add fastpath for f16

* hex-cpy-concat: support for colapsing 3/4D dims

* hex-cpy: double-buffer support

* hex-cpy: factor out gather-rows into more of an op by itself

* hex-cpy-concat: dedup kernel-params

* hex-cpy-concat: step-down nr to fit instead of hard fail

* hex-gather: replace div with a mask

* hex-gather-rows: cleanup dma pipeline and remove the stalls

* hex-gather: specialize compute funcs

* hex-gather: reduce scalar reg.spills and func calls

* hex-gather: do not recompute rows-per-thread on the npu

* hex-build: add support for building/installing just hexagon

* hex-cpy-concat: remove redundant barrier in concat and move cpy barrier under fence

* hex-gather: annotations for dma-out

* hex-mdev: make all key vtcm_acquire sync state atomic (fixes random hangs in mdev setups)

---------

Co-authored-by: Yiwei Shao <yiwei@aizip.ai>
Co-authored-by: Ruslan Karymov <67752306+karusrus@users.noreply.github.com>
2026-10-12 07:39:55 -07:00
Alex Morgan 0f701c8829 vulkan: pack FMA for f16 accumulation, no dot product op (#29877)
* vulkan: pack FMA for f16 accumulation, no dot product op

* vulkan: dot_product packs two rows per f16 FMA

* vulkan: rename dot_product to dot_product_v2
2026-10-12 16:19:20 +02:00
Xuan-Son Nguyen efe3626b11 model: support vllm d3 models (#30383)
* models: support pplx-decider

* vllm d2

* update

* clarify naming rule

* stricter validation

* d3
2026-10-12 15:01:52 +02:00
Xuan-Son Nguyen 097cf33d94 model: support vllm decision 2 model (#30378)
* models: support pplx-decider

* vllm d2

* update

* clarify naming rule

* stricter validation

* fix ci

* rm redundant filter_tensors
2026-10-12 13:39:55 +02:00
Georgi Gerganov ccdbe29053 tests : link each test against only the libraries it uses (#30299)
Give llama_build() and llama_build_and_test() a LINK keyword (default:
llama-common) so call sites can declare what a test actually needs:

- ggml-only tests now link just ggml: test-opt, test-metal-graph-optimize,
  test-backend-ops, test-tiled-mulmat, test-barrier, test-quantize-fns,
  test-quantize-perf, test-rope, test-col2im-1d, test-gguf, test-alloc
- llama-only tests drop llama-common: test-unicode, test-sampling,
  test-grammar-parser, test-llama-grammar, test-batch-alloc,
  test-gbnf-validator, test-quant-type-selection
- both mtmd tests link mtmd only, dropping the LLAMA_TEST_NAME /
  target_link_libraries workarounds

Both helpers now set cxx_std_17 explicitly, since the tests no longer
inherit it from llama-common's PUBLIC interface.

tests/testing.h no longer includes common.h; the only symbol it used from
there (string_join) is inlined.

Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
2026-10-12 13:32:56 +03:00
Georgi Gerganov 073b74034b metal : fix mul_mat_id with duplicate ids (#30373)
* metal : fix mul_mat_id limits and kernel name

* metal : fix mul_mat_id with duplicate ids

kernel_mul_mm_id_map0 assumed at most one match per token and a fixed
capacity of ne21 entries per expert, which produced wrong or dropped
output rows and could overflow into the slice of the next expert when
the ids contain duplicates.

build a compact per-expert list of the (token, slot) entries instead:

- pass 1 counts the entries of every expert, thread 0 computes the
  offsets of the expert slices (tpe now holds [base, count])
- pass 2 writes the entries with one unconditional store per token
  (conditional stores of divergent lanes are very slow), the dummy
  value of the tokens without a match lands on a padding slot of the
  slice that is never read - duplicates are handled by a rare branch
- kernel_mul_mm_id reads the base/count of its expert and loops over
  the entry tiles that do not fit into the grid

The ids buffer is sized as ne20*ne21 + ne02 (entries + padding).

883/883 test-backend-ops MUL_MAT_ID tests pass, llama-bench is within
run-to-run variance of the previous behavior.

Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-MOPD

* metal : make mul_mat_id map0 ne02 a function constant

use FOR_UNROLL for the id loops of kernel_mul_mm_id_map0 and turn ne02
into the function constant FC_mul_mm_id_map0_ne02 (FC_MUL_MM + 7), so
that the threadgroup offsets of cnt/bas/sids and the prefix-sum bound
fold at compile time.

the pipeline name now encodes ne02 (same convention as the ssm_conv
nc/silu constants) and the map0 pipeline is compiled with the constant
value set. ne02 is no longer part of ggml_metal_kargs_mul_mm_id_map0 -
the dispatch grid still comes from the tensor ne[2] on the host.

no measurable perf change: 883/883 test-backend-ops MUL_MAT_ID tests
pass, op-level and llama-bench results are within run variance.

Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-MOPD

* cont : clean-up + TODO

* metal : dispatch mul_mat_id entry tiles from the prepass

follow the approach of the vulkan implementation (#29998): instead of
launching a grid over experts x ne21 entry tiles and letting most of
the threadgroups early exit, kernel_mul_mm_id_map0 now also emits a
row map - the total tile count plus one (expert, first entry, valid
rows) descriptor per 32-entry tile that has work.

kernel_mul_mm_id is launched with a static upper bound on the number
of tiles (full tiles over all entries plus at most one partial tile
per expert), reads its descriptor and processes a single tile - the
tail early exits. This removes the round loop that was needed to cover
duplicate ids past the grid (register pressure: th_max 768 -> back to
the plain mul_mm level) and duplicates are now handled implicitly,
since they only produce more descriptors within the bound.

The grid shrinks by ~8-16x on typical shapes (e.g. 128 experts,
8 used, 512 tokens: 128 tiles instead of 2048 entry tiles, of which
94% only launched to exit) and ne21 is no longer needed by the mm
kernel kargs.

883/883 test-backend-ops MUL_MAT_ID tests pass. llama-bench
(Qwen3.6-35B-A3B Q8_0) is now faster than master on all pp sizes
(pp2048/4096/8192 +2%, pp512 +0.5%), tg32 unchanged.

Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-MOPD

* cont : add smem assert
2026-10-12 13:31:41 +03:00
Daniel Bevenius 77757a3467 examples : fix draft sampling and rejection handling (#30095)
* examples : fix draft sampling and rejection handling

This commit fixes two issues in the speculative example which are
described in detail in the linked issue.

Resolves: https://github.com/ggml-org/llama.cpp/issues/29975

* examples : add kv-unified requirement for parallel > 1
2026-10-12 12:02:24 +02:00
Georgi Gerganov 972b5fcc19 llama : report compute graph reallocations during inference (#30329)
Add an optional ggml_backend_sched_realloc_callback that user code registers with
ggml_backend_sched_set_realloc_callback to be notified when the buffers reserved for a
graph are not sufficient and have to be reserved again. The callback reports whether the
backend assignments and the graph size changed since the previous scheduling.

llama_context registers the callback when creating the scheduler (wrapped into the new
llama_context::sched_new helper) and treats the reallocations that follow a change of the
backend assignments as expected, counting the rest by graph size - unchanged or changed
from the previously scheduled graph. Both counters are reported on one line in the perf
stats.

Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
2026-10-12 11:53:58 +03:00
shamnadxyz 2f160ba59a ui: fix text attachment preview overflow (#29898)
* ui: fix text attachment preview overflow

Set the max height to 80dvh and the max width to full on screens
below medium and 80vw from medium onward.

* style: run formatter
2026-10-12 10:20:16 +02:00
Frosty40 f9a95aeb20 sycl: guard DMMV reads at row tails (#29338) 2026-10-12 04:06:27 -04:00
Amadeus dyw 08ffba21e5 vulkan: use ternary expressions to avoid switch lowering for CM2 quant block size (#30164)
* vulkan: specialize quant block size for CM2 matmul

* vulkan: use ternary expressions to avoid switch lowering for CM2 quant block size

* vulkan: clarify CM2 quant block size comment
2026-10-12 08:00:26 +02:00
Christian Kastner a05e58c981 ggml: Change default visibility of symbols to hidden (#30014)
See discussion in #29997. This prevents a handful of internal symbols
leaking via GGML_API.
2026-10-12 08:50:12 +03:00
miyan b2f036c0f1 vulkan: fix buffer logging on 32-bit platforms after source split (#30326)
Move the vk::Buffer stream operator to the shared header so it is visible to ggml-vulkan-debug.cpp.
2026-10-12 06:51:24 +02:00
Jhen-Jie Hong e937b17a2d hexagon: optimize q2_k bpw and activation (#30035) 2026-10-11 20:52:01 -07:00
kurquharandMax Krasnyansky b13a93a3d5 hexagon: add IM2COL_3D (#30305)
* hexagon: add IM2COL_3D support

Assisted-by: OpenCode

* hexagon: add DMA paths for IM2COL_3D

Assisted-by: OpenCode

* hexagon: avoid scalar fp16 conversions in IM2COL

Assisted-by: OpenCode

* remove in-loop spills in pre-v81 devices

Assisted-by: OpenCode

* hex-im2col: re-write to use canonical dma dbl-buff pipeline, remove divs and very inefficient fp conversions

* hex-im2col: move kernel_params and vtcm layout precompute into the host

* hex-im2col: remove explicit tracking of number of dma-pushes

* hex-im2col: remove incorrect uint64_t casts

* hex-im2col: no need for macros for dma loops and pass compute func ptr via local context

* hex-im2col: use cursor pattern, tracing and proper n_threads handling

* hex-im2col: support non-contigue views

* hex-im2col: minor reformatting

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-10-11 20:24:47 -07:00
lhezandHongqiang Wang f2cc7282ce opencl: improve fa, allow dk512 for gemma-4, improve dk64 (#30266)
* opencl: enable Gemma-4 E4B GPU decode

Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>

* opencl: extend Gemma-4 GPU decode to E2B

Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>

* opencl: optimize DK64 GQA8 decode

Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>

* opencl: optimize DK128 GQA4 decode

Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>

---------

Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
2026-10-11 11:49:25 -07:00
lhezandHongqiang Wang c83f3058b6 opencl: fix image limit for q4_k dense bin kernels, q4_0 bcast and rms_norm (#30310)
* opencl: fallback when Q4_K weight images exceed device limits

Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>

* opencl: avoid incomplete subgroup in rms_norm

* opencl: refine conditions for dense bin kernels and fix bcast for q4_0

Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>

---------

Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
2026-10-11 09:41:29 -07:00
Georgi Gerganov 5c8854fd88 rpc : turn GGML_RPC_DEBUG into a verbosity level and add logs (#29544)
* rpc : turn GGML_RPC_DEBUG into a verbosity level and add logs

GGML_RPC_DEBUG is now parsed as a number: 0/unset disables debug logs,
1-3 emit increasingly detailed output (events, per-command trace,
transport detail). Non-numeric values fall back to 1.

The duplicated env/macro blocks in ggml-rpc.cpp and transport.cpp are
replaced by a shared log.h, which becomes the single choke point for
all logging of the RPC backend: LOG_ERROR/LOG_WARN/LOG_INFO for
unconditional severity logs and LOG_DBG/LOG_DBG2/LOG_DBG3 for the
verbosity-gated ones. The transport files no longer need ggml-impl.h,
and the server banner now also goes through the ggml logger (stderr).

Missing logs are added on both the client (handshake, buffer ops,
tensor transfers, graph computes, cache decisions) and the server
(per-command dispatch, graph nodes), including the negotiated
transport via the new socket_t::transport_name().

Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-RL

* rpc : align log levels with the documented verbosity semantics

The first pass introduced GGML_RPC_DEBUG levels, but many call sites did not
follow the documented semantics and some logs bypassed the macros entirely:

- Route the remaining raw GGML_LOG_* call sites through the local LOG_* macros
  so that log.h stays the single choke point for RPC logging
- Emit per-tensor transfers and per-command handler traces at level 2, and
  one-shot lifecycle events (backend and buffer type creation, buffer
  allocation, graph compute, graph cache eviction) at level 1
- Drop the duplicated enqueue trace in rpc_dispatcher::send, work() already
  logs every dispatched command together with its round-trip timing
- Promote degraded-operation events to LOG_WARN: remote allocation failure,
  RDMA falling back to TCP, unexpected peer disconnect
- Log graph cache eviction on both sides, since client and server must stay
  in sync for the incremental graph update to be valid
- Distinguish Unknown command (opcode out of range) from Unhandled command
  (valid opcode with no handler)
- Fix format specifiers (%zu for size_t, %u for device ids, 0x for hex
  pointers) and a missing newline in an error log
- Document that llama.cpp applications drop ggml debug records unless the
  verbosity threshold is raised with -lv 5

Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
2026-10-11 17:43:35 +03:00
bri-prismandGeorgi Gerganov a9d27ac693 model : support for Prism Bonsai 2 27B (#29600)
* Runtime support for Prism Bonsai 2 27B

Assisted-by: Claude Code

* address prism hadamard runtime feedback

* move hadamard tensors into method, define folded weight

* load_* hadamard fixes

* cont : clean-up

* conversion cleanup

Assisted-by: Claude Code

* prism hadamard key and methods for converter

Assisted-by: Claude Code

* address review bot feedback

Assisted-by: Claude Code

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-10-11 14:39:16 +02:00
Ploo 1cf7767765 cpu: fix AVX2 CPUs not using F16C when available (#30270) 2026-10-11 15:24:41 +03:00
Jeff BolzandAnjielon 966baae76b vulkan: handle mul_mat_id duplicates in the prepass rather than looping (#29998)
* vulkan: compute every row of an expert in mul_mm_id when ids repeat

* vulkan: handle mul_mat_id duplicates in the prepass rather than looping

Extend the "hoist row ids" optimization to always be enabled and to emit a
compact list of tile descriptions that need to run, and to emit multiple tiles
when needed. Then we can launch a tighter upper bound on the number of
workgroups and the tail can trivially early exit.

---------

Co-authored-by: Anjielon <[EMAIL_REDACTED]>
2026-10-11 14:02:57 +02:00
Xuan-Son Nguyen 62a6f74bab server: add component tests (#30312)
* server: add component tests

* nits

* add codeowners
2026-10-11 12:51:07 +02:00
Pascal 23b0202a18 server: leave a busy slot untouched when a request pins it (#30295)
A request asking for a busy id_slot still ran the prompt cache update
on that slot before being deferred. When the RAM cache held a better
match, it was loaded into the slot while another request was still
generating there, and that generation continued on the wrong context.

The busy slot is now returned as is and the request waits for it.
2026-10-10 21:25:38 +02:00
69f201a205 model : support MiniCPM-V 4.7 (#29416)
* mtmd : add MiniCPM-V 4.7 support

Signed-off-by: tc-mb <tianchi_cai@icloud.com>

* model : allow mrope time from an extra position slot

Signed-off-by: tc-mb <tianchi_cai@icloud.com>

* Update conversion/minicpm.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Slim down comments

Signed-off-by: tc-mb <tianchi_cai@icloud.com>

* fix for "do not hand-wrap comments"

Signed-off-by: tc-mb <tianchi_cai@icloud.com>

* fix ci

Signed-off-by: tc-mb <tianchi_cai@icloud.com>

* rm 3d repo for pr one

Signed-off-by: tc-mb <tianchi_cai@icloud.com>

* gguf: add rope.section_order metadata

* fix comments

* handle grid layout

* allow compat

---------

Signed-off-by: tc-mb <tianchi_cai@icloud.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-10-10 20:41:59 +02:00
Aaron Teo abee0c8476 cmake(s390x): disable z17 target for unsupported compilers (#30297) 2026-10-10 20:00:56 +02:00
Mikolaj Kucharski aa94f20861 args: add LLAMA_ARG_SLOT_SAVE_PATH env for --slot-save-path (#30272)
Allow configuring --slot-save-path via LLAMA_ARG_SLOT_SAVE_PATH so
it can be set from an EnvironmentFile in a systemd unit file.
2026-10-10 15:34:40 +02:00
Xuan-Son Nguyen 781dbc5ac9 spec: properly handle mtmd input for mtp (#30257)
* spec: properly handle mtmd input for mtp

* nits
2026-10-10 11:22:38 +02:00
Xuan-Son Nguyen 0fd868cbca mtmd: add build_inp_attn_mask (#30259) 2026-10-10 11:22:12 +02:00
David Friehs 1623d8ce47 cuda: always use MMVQ for MUL_MAT_ID on sm_60 (#27828) 2026-10-10 09:15:42 +03:00
uvos 1bb2b9fcbe CUDA/HIP: fix race in flash_attn_ext_f16_process_tile when nbatch_combine != DKQ/2 (#30103)
Suggested-by: Johannes Gäßler <johannesg@5d6.de>
2026-10-10 09:14:51 +03:00
Aaron Teo 2bbca8f202 ggml-cpu: vectorize fp32 to fp16 conversion (#30157)
ggml-cpu(s390x): rename ulong to uint64_t sized types



ggml-cpu(s390x): rm comment

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
2026-10-10 09:08:52 +03:00
Amadeus dyw 404f557b5b vulkan: use 4 rows for NVIDIA MUL_MAT_ID MMVQ (#29274)
Keep the existing RDNA3/4 policy unchanged and update only rm_id to use 4 rows for NVIDIA except pre-Turing.
2026-10-10 09:08:19 +03:00
Aaron Teo b797c82c7d ggml: fix s390x all cpu build (#30140)
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
2026-10-10 09:07:52 +03:00
Shawn Gu f2918cabbf opencl: add bin kernels kernel_gemm_moe_q4_k_q8_1_dp4a_bin, kernel_gemm_moe_q6_k_q8_1_dp4a_bin (#30187) 2026-10-09 21:02:11 -07:00
Captain-Tripps 1e6f04a75e sycl : accelerate MXFP4 MoE with arithmetic decoding and weight reordering (#29809) 2026-10-09 22:56:23 -04:00
Xuan-Son Nguyen 10a60cf303 vendor: apply deep nested json patch from upstream (#30253) 2026-10-10 00:07:44 +02:00
Dante 79e2e74eb1 CUDA: fix round issue, under MSVC the CPU and GPU agree (#30229) 2026-10-09 19:59:50 +02:00
Georgi Gerganov 8e2d31e0eb graph : reorder get_rows for embeddings (#30160)
* graph : reorder get_rows for embeddings

* cont : fix gemma4 and improve input embedding construction logic

* cont : add TODO for lora

* cont : fix raw embeddings path

* gemma4 : avoid ple cast in embeddings path
2026-10-09 20:43:05 +03:00
Aleksander GrygierandPascal baef3ed9a1 ui: Models Manager Follow-up Improvements (#30228)
* common : read a GGUF's trained context from its metadata

common_get_gguf_n_ctx_train opens only the file's metadata (no_alloc,
like common_get_decision_type) and reads <arch>.context_length, so a
caller can learn the trained context without loading the model. It
accepts both u32 and u64 values and returns 0 when the file is missing,
unreadable, invalid, or reports no context length.

Assisted-by: pi:zai-org/GLM-5.3-Flash

* server : report the trained context in the models listing

update_caps already resolves the model file offline to read its
modalities, so it now reads the trained context from the same GGUF
metadata, and GET /models reports it as context_length when it is
known. A router listing then carries the context without any Hub
request, which lets the UI sort and filter by it offline.

Assisted-by: pi:zai-org/GLM-5.3-Flash

* ui : take the trained context from the models listing

The router now reports context_length per model, so the option mapping
fills contextLength from it and the manager reads it before the Hub
record. The Context column, the context sort and the context filter
then work with the Hugging Face Hub API turned off. A browser suite
guards the sort and the search, the Hub-cache driven context filter and
the re-sort when details arrive after the sort was clicked.

Assisted-by: pi:zai-org/GLM-5.3-Flash

* ui : mark favorite models with a heart

A favorited model shows a rose heart in the selector even before its
row is hovered, and the crossed heart takes its place on hover, so
unfavoriting stays one hover away. The manager table marks its
favorited rows with the same heart after the badges and capabilities.

Assisted-by: pi:zai-org/GLM-5.3-Flash

* fix: UI text nit

* fix: UI nits

* fix: Favorite models grouping in models table

* feat: Remove sorting from Status column in Models Table

* server: read the GGUF metadata once per model

Read the decision type and the trained context in a single GGUF open,
accept only a UINT32 context length like the model loader, and reset
n_ctx_train with the other caps so a failed refresh drops it.

* fix: Post-review fixes

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-10-09 19:28:54 +02:00
Martin Emrich f39148a953 llama-bench: respect -fitc if bigger than required benchmark size (#28331)
Assisted-By: opencode,llama.cpp,Qwen3.6-35B-A3B,Qwen3.8-27B
2026-10-09 19:14:30 +02:00
Anav Prasad 50e3e3e480 CUDA: Remove redundant CUDA copies after SSM_SCAN (#29807)
* CUDA: fuse copy of updated state snapshots into recurrent cache with ssm_scan

* CUDA: remove redundant cuda copies with K==1 (non spec-dec) scenario as well
2026-10-09 18:23:28 +02:00
jingzhou 64df9183f5 opencl: fix kernel compilation for a6x GPUs (#30176)
* opencl: skip kernel_cpy_f32_f32_pack on A6X to avoid shader compiler crash

* The A6x compiler backend found in iot device with a623 (E031.50.31.01)
  cannot handle kernels with a large number of arguments. Skip this
  kernel for A6x to avoid compiler crash

* opencl: A6X constant-fold workaround for get_local_size in GEMV kernels

* opencl: add Adreno 623 to A6X GPU detection list
2026-10-09 09:19:20 -07:00
bosh 6184e92c57 model : use exact GELU for ModernBERT encoders (#30108)
* model : use exact GELU for ModernBERT encoders

Assisted-by: Codex

* model : keep tanh GELU aliases on ggml_geglu

Assisted-by: Claude Opus 5.5

* model : map gelu_python to ggml_geglu_erf

Assisted-by: Claude Opus 5.5
2026-10-09 23:26:56 +08:00
Aldehir Rojas 8b54361025 chat : refactor API (#30210) 2026-10-09 10:06:22 -05:00
Pascal a518119d30 llama: keep the backend sampling graph static across ubatches (#30223)
The reserve builds n_outputs_max_per_seq sampling chains per sampler,
while a decode built one per output row, so the graph changed its
topology after the reserve and GGML_SCHED_NO_REALLOC builds aborted on
the next same sized graph. Every sampler now builds
n_outputs_max_per_seq chains, the ones without a row of the ubatch on
the padding row and not selected, and graph_max_nodes counts them.
2026-10-09 17:20:25 +03:00
Jasmine-tim 8ae386707b ggml: fix OOB write in ggml_acc with negative offset (#30135)
ggml_acc_impl narrowed a size_t offset to int32_t without checking that it
fits, so a large offset could truncate to a negative int32_t. The forward
then sign-extended it to a huge size_t and the bounds assertion wrapped,
allowing an OOB write below the dst buffer. Check the offset before the
narrowing, matching the existing check in ggml_set_impl.
2026-10-09 16:57:33 +03:00
Georgi Gerganov e60eff95fd meta : handle host views (#30217)
* meta : handle views of tensors allocated on the host

A view shares the memory of its view_src, so ggml-alloc never allocates a view in
the buffer of the split it lands in - the scheduler copies the source into the
split and the ops that use the view read that copy. The view node itself is a noop
and does not need a split of its own, but the meta backend asserted when one was
left inside a meta split:

  - ggml_backend_meta_get_split_state() dereferenced tensor->buffer->context
  - the graph rebuild mapped every node with ggml_backend_meta_buffer_simple_tensor()

Accept such nodes when they are views of host tensors, which also generalizes the
previous s_copy_main workaround. This fixes the assert hit by KV cache views when
using --split-mode tensor with partial offload.

Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next

* archs : re-enable sm tensor for K2 Horizon

* cont : add TODO and reference
2026-10-09 16:01:04 +03:00
Masashi Yoshimura ba6439a6b5 webgpu: use 2D workgroup dispatch for all the ops which use 1D dispatch (e.g., rms_norm) (#30219)
* remove 1D workgroups dispatching

* formatting
2026-10-09 16:00:37 +03:00
Ruben Ortlam 5e4878e978 vulkan: fix rms_norm workgroup count overflow (#30145) 2026-10-09 15:00:14 +02:00
ynankani 609290be6b convert : support compressed-tensor mixed-precision NVFP4 checkpoint (#28636)
Signed-off-by: ynankani <ynankani@nvidia.com>
2026-10-09 13:54:58 +02:00
R0CKSTAR 86a2835320 musa: drop mp_21 from the default architectures (#30203)
* musa: drop mp_21 from the default architectures

Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com>

* Address review comments

Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com>

* Address review comments

Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com>

---------

Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com>
2026-10-09 12:51:24 +02:00
Pascal e94acad853 ci: fix Models Backend webgpu by supporting GGML_OP_DUP (#30216)
The mixed batch path of PR-29622 writes the token rows with set_rows
into a dup of the embeddings. WebGPU did not support DUP, so the dup
ran on the CPU while the set_rows writing into it was scheduled on
WebGPU, which then bound a CPU buffer and crashed. DUP is the same copy
as CPY and CONT and now goes through the same path.
2026-10-09 12:27:32 +02:00
Sigbjørn Skjæret 5e1d74043e ci : set run-name for publish release (#30214) 2026-10-09 11:12:24 +02:00
Xuan-Son Nguyen b42b7e6d30 server: use port 9931 by default (#30159)
* server: use port 9931 by default

* revert unrelated changes
2026-10-09 10:40:03 +02:00
Aleksander Grygier b013e56a71 ui : add the models manager (#29583)
* ui : add the drawer, sheet and grouped list primitives

Add the drawer and sheet overlay components, the toggle and toggle
group, the shared searchable input, the collapsible sections and the
grouped list, and the near viewport helper they position with.

Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash

* ui : add the shared model components and data layer

Add the model row pieces shared by the selector and the manager, the
models store with its download status feed, the huggingface and
migration services, and the model utils, enums and constants they
read through.

Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash

* ui : rework the models selector around its providers

Group the selector components under their own folder, rework the list
around the provider grouping, add the mobile trigger and the download
item, and derive the selection state in one hook.

Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash

* ui : add the models manager

Add the manager table with its repo, quant and status rows, the
filters and the toolbar, the row actions in a drawer, the downloads
section, the manage models dialog and the stories and tests.

Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash

* ui : rework the chat form actions and the mobile experience

Move the add actions into a drawer and a sheet, float the model
actions in a bar on a phone, open the context panel in a drawer, and
derive the attachment, reasoning and tools menus in hooks.

Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash

* ui : move the mcp servers to the sidebar rail and tidy the dialogs

Move the mcp servers dialog to the sidebar rail with its menu
entries, even out the dialogs on a phone, and group the mcp and
settings stores behind their own modules.

Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash

* ui : polish the shell

The secondary button gets its own look back and the chat add button its
own light surface. Pressable elements get a pointer cursor again, the
font rendering smooths on the app shell, and the agent skills stay out
of prettier's way.

The root layout props probe rework that came with this polish reads the
providers api url and stays with the providers change instead.

Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash

* ui : wrap the model id classes

Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash

* ui : load the model the new chat CTA picks

Start a new chat selected the model but left it unloaded, so the chat
opened against a server with nothing in memory. Both CTAs share a load
helper now.

Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash

* refactor: Post-review fixes
2026-10-09 10:34:45 +02:00
Pascal 2c564f43df ci: fix HIP quality check by ignoring the 320/256 FA kernel spill (#30212) 2026-10-09 10:29:22 +02:00
Pascal 1f8fa52318 metal: add the 128/96 flash attention kernels, accept every compiled pair (#30209) 2026-10-09 10:15:32 +02:00
ynankani 8a1a9b5126 CUDA: pass src1 precision to host MMQ config helpers (#30168)
* CUDA: pass src1 precision to host MMQ config helpers

Signed-off-by: ynankani <ynankani@nvidia.com>

* make src1 prec explicit

Signed-off-by: ynankani <ynankani@nvidia.com>

---------

Signed-off-by: ynankani <ynankani@nvidia.com>
2026-10-09 12:02:52 +05:30
Mark TodorovichandMax Krasnyansky d4d82d67f4 hexagon: fix IM2COL patch-embed DMA ring overflow (#30189)
* hexagon: fix IM2COL patch-embed DMA ring overflow

The exact-tiling (stride == kernel, no pad/dilation) IM2COL DMA kernel
issues IC*KH DDR->VTCM descriptors per output row without checking the
return value of dma_queue_push(), and then pops IC*KH times. The per-thread
DMA ring holds 256 entries and a push into a full ring returns false
and drops the transfer, so for IC*KH > 255 the remaining rows of the
VTCM staging buffer were never written and stale data (often NaN/inf)
leaked into the output.

Solution is to retire the oldest descriptor when the ring is full, just as the blocked
kernel in the same file already does, and wait with dma_queue_flush().

For testing, added exact-tiling test cases with IC*KH > 256 (2D 1x1, 2D 2x2 patch
embed and 1D, F16 and F32 dst), which fail on HTP without this fix.

* Apply suggestion from @max-krasnyansky

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-10-09 08:04:09 +02:00
Nicolas Mowen 3d65c90d04 sycl : Q5_K reorder-layout MMVQ and fused GLU (#29375) 2026-10-08 22:19:32 -04:00
3102 changed files with 406568 additions and 137148 deletions

No files matched your search

+11 -6
View File
@@ -19,12 +19,16 @@ FROM docker.io/node:$NODE_VERSION AS web
ARG APP_VERSION
WORKDIR /app/tools/ui
WORKDIR /app/ui/apps/chat
COPY tools/ui/package.json tools/ui/package-lock.json ./
COPY ui/apps/chat/package.json ui/apps/chat/package-lock.json ./
RUN npm ci
COPY tools/ui/ ./
COPY ui/apps/chat/ ./
# the app compiles Llama UI and Llama Voice from their sources next to it
COPY ui/lib/ui/package.json /app/ui/lib/ui/package.json
COPY ui/lib/ui/src/lib/ /app/ui/lib/ui/src/lib/
COPY ui/lib/voice/src/ /app/ui/lib/voice/src/
RUN LLAMA_BUILD_NUMBER="$APP_VERSION" npm run build
FROM ${CANN_BASE_IMAGE} AS build
@@ -40,7 +44,7 @@ WORKDIR /app
# -- Copy project files --
COPY . .
COPY --from=web /app/tools/ui/dist tools/ui/dist
COPY --from=web /app/ui/apps/chat/dist ui/apps/chat/dist
# -- Set CANN environment variables (required for compilation) --
# Using ENV instead of `source` allows environment variables to persist across the entire image layer
@@ -145,7 +149,7 @@ ENTRYPOINT ["/app/tools.sh"]
# ==============================================================================
FROM base AS light
COPY --from=build /app/full/llama /app/full/llama-cli /app/full/llama-completion /app
COPY --from=build /app/full/llama /app/full/llama-cli /app/full/llama-completion /app/
ENTRYPOINT [ "/app/llama-cli" ]
@@ -155,8 +159,9 @@ ENTRYPOINT [ "/app/llama-cli" ]
FROM base AS server
ENV LLAMA_ARG_HOST=0.0.0.0
ENV LLAMA_ARG_PORT=8080
COPY --from=build /app/full/llama /app/full/llama-server /app
COPY --from=build /app/full/llama /app/full/llama-server /app/
HEALTHCHECK --interval=5m CMD [ "curl", "-f", "http://localhost:8080/health" ]
+13 -10
View File
@@ -1,4 +1,4 @@
ARG UBUNTU_VERSION=24.04
ARG UBUNTU_VERSION=26.04
ARG BUILD_DATE=N/A
ARG APP_VERSION=N/A
ARG APP_REVISION=N/A
@@ -9,12 +9,16 @@ FROM docker.io/node:$NODE_VERSION AS web
ARG APP_VERSION
WORKDIR /app/tools/ui
WORKDIR /app/ui/apps/chat
COPY tools/ui/package.json tools/ui/package-lock.json ./
COPY ui/apps/chat/package.json ui/apps/chat/package-lock.json ./
RUN npm ci
COPY tools/ui/ ./
COPY ui/apps/chat/ ./
# the app compiles Llama UI and Llama Voice from their sources next to it
COPY ui/lib/ui/package.json /app/ui/lib/ui/package.json
COPY ui/lib/ui/src/lib/ /app/ui/lib/ui/src/lib/
COPY ui/lib/voice/src/ /app/ui/lib/voice/src/
RUN LLAMA_BUILD_NUMBER="$APP_VERSION" npm run build
FROM docker.io/ubuntu:$UBUNTU_VERSION AS build
@@ -22,15 +26,13 @@ FROM docker.io/ubuntu:$UBUNTU_VERSION AS build
ARG TARGETARCH
RUN apt-get update && \
apt-get install -y gcc-14 g++-14 build-essential git cmake libssl-dev
ENV CC=gcc-14 CXX=g++-14
apt-get install -y build-essential git cmake libssl-dev
WORKDIR /app
COPY . .
COPY --from=web /app/tools/ui/dist tools/ui/dist
COPY --from=web /app/ui/apps/chat/dist ui/apps/chat/dist
RUN if [ "$TARGETARCH" = "amd64" ] || [ "$TARGETARCH" = "arm64" ]; then \
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=OFF -DLLAMA_BUILD_TESTS=OFF -DGGML_BACKEND_DL=ON -DGGML_CPU_ALL_VARIANTS=ON; \
@@ -104,7 +106,7 @@ ENTRYPOINT ["/app/tools.sh"]
### Light, CLI only
FROM base AS light
COPY --from=build /app/full/llama /app/full/llama-cli /app/full/llama-completion /app
COPY --from=build /app/full/llama /app/full/llama-cli /app/full/llama-completion /app/
WORKDIR /app
@@ -114,8 +116,9 @@ ENTRYPOINT [ "/app/llama-cli" ]
FROM base AS server
ENV LLAMA_ARG_HOST=0.0.0.0
ENV LLAMA_ARG_PORT=8080
COPY --from=build /app/full/llama /app/full/llama-server /app
COPY --from=build /app/full/llama /app/full/llama-server /app/
WORKDIR /app
+14 -6
View File
@@ -17,12 +17,16 @@ FROM docker.io/node:$NODE_VERSION AS web
ARG APP_VERSION
WORKDIR /app/tools/ui
WORKDIR /app/ui/apps/chat
COPY tools/ui/package.json tools/ui/package-lock.json ./
COPY ui/apps/chat/package.json ui/apps/chat/package-lock.json ./
RUN npm ci
COPY tools/ui/ ./
COPY ui/apps/chat/ ./
# the app compiles Llama UI and Llama Voice from their sources next to it
COPY ui/lib/ui/package.json /app/ui/lib/ui/package.json
COPY ui/lib/ui/src/lib/ /app/ui/lib/ui/src/lib/
COPY ui/lib/voice/src/ /app/ui/lib/voice/src/
RUN LLAMA_BUILD_NUMBER="$APP_VERSION" npm run build
FROM ${BASE_CUDA_DEV_CONTAINER} AS build
@@ -40,7 +44,7 @@ WORKDIR /app
COPY . .
COPY --from=web /app/tools/ui/dist tools/ui/dist
COPY --from=web /app/ui/apps/chat/dist ui/apps/chat/dist
RUN if [ "${CUDA_DOCKER_ARCH}" != "default" ]; then \
export CMAKE_ARGS="-DCMAKE_CUDA_ARCHITECTURES=${CUDA_DOCKER_ARCH}"; \
@@ -84,6 +88,9 @@ RUN apt-get update \
&& find /var/cache/apt/archives /var/lib/apt/lists -not -name lock -type f -delete \
&& find /var/cache -type f -delete
# Some hosts (e.g. GKE) only mount the driver here, and CUDA 12.8.1-12.9.1 base images don't set these paths
ENV LD_LIBRARY_PATH=/usr/local/nvidia/lib64:/usr/local/nvidia/lib:${LD_LIBRARY_PATH}
COPY --from=build /app/lib/ /app
### Full
@@ -113,7 +120,7 @@ ENTRYPOINT ["/app/tools.sh"]
### Light, CLI only
FROM base AS light
COPY --from=build /app/full/llama /app/full/llama-cli /app/full/llama-completion /app
COPY --from=build /app/full/llama /app/full/llama-cli /app/full/llama-completion /app/
WORKDIR /app
@@ -123,8 +130,9 @@ ENTRYPOINT [ "/app/llama-cli" ]
FROM base AS server
ENV LLAMA_ARG_HOST=0.0.0.0
ENV LLAMA_ARG_PORT=8080
COPY --from=build /app/full/llama /app/full/llama-server /app
COPY --from=build /app/full/llama /app/full/llama-server /app/
WORKDIR /app
+33 -21
View File
@@ -1,4 +1,4 @@
ARG ONEAPI_VERSION=2026.1.1-devel-ubuntu24.04
ARG ONEAPI_VERSION=2026.2.0-devel-ubuntu24.04
ARG BUILD_DATE=N/A
ARG APP_VERSION=N/A
ARG APP_REVISION=N/A
@@ -11,32 +11,41 @@ FROM docker.io/node:$NODE_VERSION AS web
ARG APP_VERSION
WORKDIR /app/tools/ui
WORKDIR /app/ui/apps/chat
COPY tools/ui/package.json tools/ui/package-lock.json ./
COPY ui/apps/chat/package.json ui/apps/chat/package-lock.json ./
RUN npm ci
COPY tools/ui/ ./
COPY ui/apps/chat/ ./
# the app compiles Llama UI and Llama Voice from their sources next to it
COPY ui/lib/ui/package.json /app/ui/lib/ui/package.json
COPY ui/lib/ui/src/lib/ /app/ui/lib/ui/src/lib/
COPY ui/lib/voice/src/ /app/ui/lib/voice/src/
RUN LLAMA_BUILD_NUMBER="$APP_VERSION" npm run build
FROM docker.io/intel/oneapi-toolkit:$ONEAPI_VERSION AS build
FROM docker.io/intel/deep-learning-essentials:$ONEAPI_VERSION AS build
RUN echo "Download oneDNN installation package" && \
wget https://registrationcenter-download.intel.com/akdlm/IRC_NAS/46452a4a-7b7f-49f2-be4c-602a900180bb/intel-onednn-2026.1.0.625_offline.sh -O intel-onednn_offline.sh && \
echo "Install oneDNN components" && \
bash intel-onednn_offline.sh -s -a --silent --eula accept
ARG GGML_SYCL_F16=ON
ARG LEVEL_ZERO_VERSION=1.28.2
ARG LEVEL_ZERO_VERSION=1.34.0
ARG LEVEL_ZERO_UBUNTU_VERSION=u24.04
RUN apt-get update && \
apt-get install -y git libssl-dev wget ca-certificates && \
cd /tmp && \
wget -q "https://github.com/oneapi-src/level-zero/releases/download/v${LEVEL_ZERO_VERSION}/level-zero_${LEVEL_ZERO_VERSION}%2B${LEVEL_ZERO_UBUNTU_VERSION}_amd64.deb" -O level-zero.deb && \
wget -q "https://github.com/oneapi-src/level-zero/releases/download/v${LEVEL_ZERO_VERSION}/level-zero-devel_${LEVEL_ZERO_VERSION}%2B${LEVEL_ZERO_UBUNTU_VERSION}_amd64.deb" -O level-zero-devel.deb && \
apt-get -o Dpkg::Options::="--force-overwrite" install -y ./level-zero.deb ./level-zero-devel.deb && \
rm -f /tmp/level-zero.deb /tmp/level-zero-devel.deb
wget -q "https://github.com/oneapi-src/level-zero/releases/download/v${LEVEL_ZERO_VERSION}/libze1_${LEVEL_ZERO_VERSION}%2B${LEVEL_ZERO_UBUNTU_VERSION}_amd64.deb" -O libze1.deb && \
wget -q "https://github.com/oneapi-src/level-zero/releases/download/v${LEVEL_ZERO_VERSION}/libze-dev_${LEVEL_ZERO_VERSION}%2B${LEVEL_ZERO_UBUNTU_VERSION}_amd64.deb" -O libze-dev.deb && \
apt-get -o Dpkg::Options::="--force-overwrite" install -y ./libze1.deb ./libze-dev.deb && \
rm -f /tmp/libze1.deb /tmp/libze-dev.deb
WORKDIR /app
COPY . .
COPY --from=web /app/tools/ui/dist tools/ui/dist
COPY --from=web /app/ui/apps/chat/dist ui/apps/chat/dist
RUN if [ "${GGML_SYCL_F16}" = "ON" ]; then \
echo "GGML_SYCL_F16 is set" \
@@ -45,7 +54,7 @@ RUN if [ "${GGML_SYCL_F16}" = "ON" ]; then \
fi && \
echo "Building with dynamic libs" && \
cmake -B build -DGGML_NATIVE=OFF -DGGML_SYCL=ON -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx -DGGML_BACKEND_DL=ON -DGGML_CPU_ALL_VARIANTS=ON -DLLAMA_BUILD_TESTS=OFF ${OPT_SYCL_F16} && \
cmake --build build --config Release -j$(nproc)
cmake --build build --config Release -j$((($(nproc)+1)/2))
RUN mkdir -p /app/lib && \
find build -name "*.so*" -exec cp -P {} /app/lib \;
@@ -59,7 +68,7 @@ RUN mkdir -p /app/full \
&& cp requirements.txt /app/full \
&& cp .devops/tools.sh /app/full/tools.sh
FROM docker.io/intel/oneapi-toolkit:$ONEAPI_VERSION AS base
FROM docker.io/intel/deep-learning-essentials:$ONEAPI_VERSION AS base
ARG BUILD_DATE=N/A
ARG APP_VERSION=N/A
@@ -74,6 +83,8 @@ LABEL org.opencontainers.image.created=$BUILD_DATE \
org.opencontainers.image.url=$IMAGE_URL \
org.opencontainers.image.source=$IMAGE_SOURCE
COPY --from=build /opt/intel/oneapi/dnnl /opt/intel/oneapi/dnnl
#Following versions are for multiple GPUs, since 26.x has known issue:
# https://github.com/ggml-org/llama.cpp/issues/21747,
# https://github.com/intel/compute-runtime/issues/921.
@@ -84,11 +95,11 @@ LABEL org.opencontainers.image.created=$BUILD_DATE \
#ARG IGDGMM_VERSION=22.8.2
ARG IGC_VERSION=v2.34.4
ARG IGC_VERSION_FULL=2_2.34.4+21428
ARG COMPUTE_RUNTIME_VERSION=26.18.38308.1
ARG COMPUTE_RUNTIME_VERSION_FULL=26.18.38308.1-0
ARG IGDGMM_VERSION=22.10.0
ARG IGC_VERSION=v2.42.6
ARG IGC_VERSION_FULL=2_2.42.6+23070
ARG COMPUTE_RUNTIME_VERSION=26.40.40196.8
ARG COMPUTE_RUNTIME_VERSION_FULL=26.40.40196.8-0
ARG IGDGMM_VERSION=22.10.2
RUN mkdir /tmp/neo/ && cd /tmp/neo/ \
&& wget https://github.com/intel/intel-graphics-compiler/releases/download/$IGC_VERSION/intel-igc-core-${IGC_VERSION_FULL}_amd64.deb \
&& wget https://github.com/intel/intel-graphics-compiler/releases/download/$IGC_VERSION/intel-igc-opencl-${IGC_VERSION_FULL}_amd64.deb \
@@ -102,7 +113,7 @@ RUN mkdir /tmp/neo/ && cd /tmp/neo/ \
&& dpkg --install *.deb
RUN apt-get update \
&& apt-get install -y libgomp1 curl ffmpeg \
&& apt-get install -y libgomp1 curl ffmpeg openssl libssl-dev \
&& apt autoremove -y \
&& apt clean -y \
&& rm -rf /tmp/* /var/tmp/* \
@@ -141,7 +152,7 @@ ENTRYPOINT ["/app/tools.sh"]
FROM base AS light
COPY --from=build /app/lib/ /app
COPY --from=build /app/full/llama /app/full/llama-cli /app/full/llama-completion /app
COPY --from=build /app/full/llama /app/full/llama-cli /app/full/llama-completion /app/
WORKDIR /app
@@ -151,9 +162,10 @@ ENTRYPOINT [ "/app/llama-cli" ]
FROM base AS server
ENV LLAMA_ARG_HOST=0.0.0.0
ENV LLAMA_ARG_PORT=8080
COPY --from=build /app/lib/ /app
COPY --from=build /app/full/llama /app/full/llama-server /app
COPY --from=build /app/full/llama /app/full/llama-server /app/
WORKDIR /app
+11 -6
View File
@@ -15,12 +15,16 @@ FROM docker.io/node:$NODE_VERSION AS web
ARG APP_VERSION
WORKDIR /app/tools/ui
WORKDIR /app/ui/apps/chat
COPY tools/ui/package.json tools/ui/package-lock.json ./
COPY ui/apps/chat/package.json ui/apps/chat/package-lock.json ./
RUN npm ci
COPY tools/ui/ ./
COPY ui/apps/chat/ ./
# the app compiles Llama UI and Llama Voice from their sources next to it
COPY ui/lib/ui/package.json /app/ui/lib/ui/package.json
COPY ui/lib/ui/src/lib/ /app/ui/lib/ui/src/lib/
COPY ui/lib/voice/src/ /app/ui/lib/voice/src/
RUN LLAMA_BUILD_NUMBER="$APP_VERSION" npm run build
FROM ${BASE_MUSA_DEV_CONTAINER} AS build
@@ -45,7 +49,7 @@ WORKDIR /app
COPY . .
COPY --from=web /app/tools/ui/dist tools/ui/dist
COPY --from=web /app/ui/apps/chat/dist ui/apps/chat/dist
RUN if [ "${MUSA_DOCKER_ARCH}" != "default" ]; then \
export CMAKE_ARGS="-DMUSA_ARCHITECTURES=${MUSA_DOCKER_ARCH}"; \
@@ -120,7 +124,7 @@ ENTRYPOINT ["/app/tools.sh"]
### Light, CLI only
FROM base AS light
COPY --from=build /app/full/llama /app/full/llama-cli /app/full/llama-completion /app
COPY --from=build /app/full/llama /app/full/llama-cli /app/full/llama-completion /app/
WORKDIR /app
@@ -130,8 +134,9 @@ ENTRYPOINT [ "/app/llama-cli" ]
FROM base AS server
ENV LLAMA_ARG_HOST=0.0.0.0
ENV LLAMA_ARG_PORT=8080
COPY --from=build /app/full/llama /app/full/llama-server /app
COPY --from=build /app/full/llama /app/full/llama-server /app/
WORKDIR /app
+138 -41
View File
@@ -22,12 +22,19 @@
spirv-headers,
nodejs,
importNpmLock,
syclToolkit ? null,
level-zero,
ocl-icd,
intel-graphics-compiler,
intel-compute-runtime,
makeBinaryWrapper,
useBlas ?
builtins.all (x: !x) [
useCuda
useMetalKit
useRocm
useVulkan
useSycl
]
&& blas.meta.available,
useCuda ? config.cudaSupport,
@@ -38,11 +45,28 @@
rocmGpuTargets ? builtins.concatStringsSep ";" rocmPackages.clr.gpuTargets,
useVulkan ? false,
useRpc ? false,
useSycl ? false,
# Rebuild, not a runtime switch. cmake default is off.
syclF16 ? false,
# null keeps JIT. AOT makes icpx run ocloc, which DPC++ does not ship.
syclDeviceArch ? null,
syclMaxParallelLinkJobs ? null,
llamaVersion ? "0.0.0", # Arbitrary version, substituted by the flake
# It's necessary to consistently use backendStdenv when building with CUDA support,
# otherwise we get libstdc++ errors downstream.
effectiveStdenv ? if useCuda then cudaPackages.backendStdenv else stdenv,
# CUDA needs its backend stdenv or libstdc++ mismatches downstream.
# SYCL needs icpx which is provided by the oneAPI stdenv
effectiveStdenv ?
if useCuda && useSycl then
throw "llama-cpp: useCuda and useSycl are mutually exclusive"
else if useCuda then
cudaPackages.backendStdenv
else if useSycl then
if syclToolkit == null then
throw "llama-cpp: useSycl requires syclToolkit, consider setting it to intel-oneapi-toolkit"
else
syclToolkit.stdenv
else
stdenv,
enableStatic ? effectiveStdenv.hostPlatform.isStatic,
precompileMetalShaders ? false,
useWebUi ? true,
@@ -65,7 +89,8 @@ let
++ lib.optionals useMetalKit [ "MetalKit" ]
++ lib.optionals useMpi [ "MPI" ]
++ lib.optionals useRocm [ "ROCm" ]
++ lib.optionals useVulkan [ "Vulkan" ];
++ lib.optionals useVulkan [ "Vulkan" ]
++ lib.optionals useSycl [ "SYCL" ];
pnameSuffix =
strings.optionalString (suffices != [ ])
@@ -108,6 +133,26 @@ let
shaderc
spirv-headers
];
syclAot = useSycl && syclDeviceArch != null && syclDeviceArch != "";
# source ocloc binary from intel-compute-runtime
oclocAot = runCommand "ocloc-aot" { nativeBuildInputs = [ makeBinaryWrapper ]; } ''
mkdir -p "$out/bin"
shopt -s nullglob
exact=(${lib.getBin intel-compute-runtime}/bin/ocloc)
vers=(${lib.getBin intel-compute-runtime}/bin/ocloc-*)
if [ ''${#exact[@]} -eq 1 ]; then
src=''${exact[0]}
elif [ ''${#vers[@]} -eq 1 ]; then
src=''${vers[0]}
else
echo "expected one ocloc binary from intel-compute-runtime" >&2
exit 1
fi
makeWrapper "$src" "$out/bin/ocloc" \
--prefix LD_LIBRARY_PATH : ${lib.makeLibraryPath [ intel-graphics-compiler ]}
'';
in
effectiveStdenv.mkDerivation (finalAttrs: {
@@ -133,11 +178,13 @@ effectiveStdenv.mkDerivation (finalAttrs: {
src = lib.cleanSource ../../.;
};
# Builds the webui locally, taking care not to require updating any sha256 hash.
# Builds Llama Chat locally, taking care not to require updating any sha256 hash.
webui = stdenvNoCC.mkDerivation {
pname = "webui";
version = llamaVersion;
src = lib.cleanSource ../../tools/ui;
# the app compiles Llama UI from its source in ../../lib/ui, so the source is ui/ as a whole
src = lib.cleanSource ../../ui;
sourceRoot = "source/apps/chat";
nativeBuildInputs = [
nodejs
@@ -146,7 +193,7 @@ effectiveStdenv.mkDerivation (finalAttrs: {
# no sha256 required when using buildNodeModules
npmDeps = importNpmLock.buildNodeModules {
npmRoot = ../../tools/ui;
npmRoot = ../../ui/apps/chat;
inherit nodejs;
};
@@ -156,8 +203,8 @@ effectiveStdenv.mkDerivation (finalAttrs: {
};
postPatch = lib.optionalString useWebUi ''
cp -r ${finalAttrs.webui} tools/ui/dist
chmod -R u+w tools/ui/dist
cp -r ${finalAttrs.webui} ui/apps/chat/dist
chmod -R u+w ui/apps/chat/dist
'';
# With PR#6015 https://github.com/ggml-org/llama.cpp/pull/6015,
@@ -168,6 +215,13 @@ effectiveStdenv.mkDerivation (finalAttrs: {
# see https://github.com/ggml-org/llama.cpp/pull/6118 for discussion
__noChroot = effectiveStdenv.hostPlatform.isDarwin && useMetalKit && precompileMetalShaders;
# Do not fortify under SYCL as it breaks compiled memcpy invocations
# _FORTIFY_SOURCE rewrites memcpy -> __memcpy_chk
hardeningDisable = optionals useSycl [
"fortify"
"fortify3"
];
nativeBuildInputs =
[
cmake
@@ -181,7 +235,12 @@ effectiveStdenv.mkDerivation (finalAttrs: {
autoAddDriverRunpath
]
++ optionals (effectiveStdenv.hostPlatform.isGnu && enableStatic) [ glibc.static ]
++ optionals (effectiveStdenv.hostPlatform.isDarwin && useMetalKit && precompileMetalShaders) [ xcrunHost ];
++ optionals (effectiveStdenv.hostPlatform.isDarwin && useMetalKit && precompileMetalShaders) [ xcrunHost ]
++ optionals useSycl [
autoAddDriverRunpath
makeBinaryWrapper
]
++ optionals syclAot [ oclocAot ];
buildInputs =
optionals effectiveStdenv.hostPlatform.isDarwin darwinBuildInputs
@@ -190,39 +249,53 @@ effectiveStdenv.mkDerivation (finalAttrs: {
++ optionals useRocm rocmBuildInputs
++ optionals useBlas [ blas ]
++ optionals useVulkan vulkanBuildInputs
++ optionals useSycl [
level-zero
ocl-icd
intel-graphics-compiler
]
++ [ openssl ];
cmakeFlags =
[
(cmakeBool "LLAMA_BUILD_SERVER" true)
(cmakeBool "LLAMA_BUILD_WEBUI" useWebUi)
(cmakeBool "BUILD_SHARED_LIBS" (!enableStatic))
(cmakeBool "CMAKE_SKIP_BUILD_RPATH" true)
(cmakeBool "GGML_NATIVE" false)
(cmakeBool "GGML_BLAS" useBlas)
(cmakeBool "GGML_CUDA" useCuda)
(cmakeBool "GGML_HIP" useRocm)
(cmakeBool "GGML_METAL" useMetalKit)
(cmakeBool "GGML_VULKAN" useVulkan)
(cmakeBool "GGML_STATIC" enableStatic)
(cmakeBool "GGML_RPC" useRpc)
]
++ optionals useCuda [
(
with cudaPackages.flags;
cmakeFeature "CMAKE_CUDA_ARCHITECTURES" (
builtins.concatStringsSep ";" (map dropDot cudaCapabilities)
)
cmakeFlags = [
(cmakeBool "LLAMA_BUILD_SERVER" true)
(cmakeBool "LLAMA_BUILD_WEBUI" useWebUi)
(cmakeBool "BUILD_SHARED_LIBS" (!enableStatic))
(cmakeBool "CMAKE_SKIP_BUILD_RPATH" true)
(cmakeBool "GGML_NATIVE" false)
(cmakeBool "GGML_BLAS" useBlas)
(cmakeBool "GGML_CUDA" useCuda)
(cmakeBool "GGML_HIP" useRocm)
(cmakeBool "GGML_METAL" useMetalKit)
(cmakeBool "GGML_VULKAN" useVulkan)
(cmakeBool "GGML_STATIC" enableStatic)
(cmakeBool "GGML_RPC" useRpc)
]
++ optionals useCuda [
(
with cudaPackages.flags;
cmakeFeature "CMAKE_CUDA_ARCHITECTURES" (
builtins.concatStringsSep ";" (map dropDot cudaCapabilities)
)
]
++ optionals useRocm [
(cmakeFeature "CMAKE_HIP_COMPILER" "${rocmPackages.llvm.clang}/bin/clang")
(cmakeFeature "CMAKE_HIP_ARCHITECTURES" rocmGpuTargets)
]
++ optionals useMetalKit [
(lib.cmakeFeature "CMAKE_C_FLAGS" "-D__ARM_FEATURE_DOTPROD=1")
(cmakeBool "GGML_METAL_EMBED_LIBRARY" (!precompileMetalShaders))
];
)
]
++ optionals useRocm [
(cmakeFeature "CMAKE_HIP_COMPILER" "${rocmPackages.llvm.clang}/bin/clang")
(cmakeFeature "CMAKE_HIP_ARCHITECTURES" rocmGpuTargets)
]
++ optionals useMetalKit [
(lib.cmakeFeature "CMAKE_C_FLAGS" "-D__ARM_FEATURE_DOTPROD=1")
(cmakeBool "GGML_METAL_EMBED_LIBRARY" (!precompileMetalShaders))
]
++ optionals useSycl [
(cmakeBool "GGML_SYCL" true)
(cmakeBool "GGML_SYCL_F16" syclF16)
]
++ optionals syclAot [
(cmakeFeature "GGML_SYCL_DEVICE_ARCH" syclDeviceArch)
]
++ optionals (syclAot && syclMaxParallelLinkJobs != null) [
(cmakeFeature "GGML_SYCL_MAX_PARALLEL_LINK_JOBS" (toString syclMaxParallelLinkJobs))
];
# Environment variables needed for ROCm
env = optionalAttrs useRocm {
@@ -237,11 +310,35 @@ effectiveStdenv.mkDerivation (finalAttrs: {
cp $src/include/llama.h $out/include/
'';
# Fix SYCL binary calling missing important libraries
# Also use Level Zero + sysman INSTEAD of OpenCL
# due to OpenCL reporting free memory as total memory,
# breaking `-fit` functionality
postFixup = lib.optionalString useSycl ''
driverLibs=${
lib.makeLibraryPath [
intel-graphics-compiler
ocl-icd
]
}
for b in "$out"/bin/*; do
[ -f "$b" ] && [ -x "$b" ] || continue
patchelf --print-interpreter "$b" >/dev/null 2>&1 || continue
patchelf --add-needed libOpenCL.so.1 --add-needed libigc.so.2 \
--add-rpath "$driverLibs" "$b"
wrapProgram "$b" \
--set-default ONEAPI_DEVICE_SELECTOR level_zero:gpu \
--set-default ZES_ENABLE_SYSMAN 1
done
'';
meta = {
# Configurations we don't want even the CI to evaluate. Results in the
# "unsupported platform" messages. This is mostly a no-op, because
# cudaPackages would've refused to evaluate anyway.
badPlatforms = optionals useCuda lib.platforms.darwin;
badPlatforms =
optionals useCuda lib.platforms.darwin
++ optionals useSycl (lib.subtractLists [ "x86_64-linux" ] lib.platforms.all);
# Configurations that are known to result in build failures. Can be
# overridden by importing Nixpkgs with `allowBroken = true`.
-4
View File
@@ -22,10 +22,6 @@ let
gitpython
tabulate
# for examples/pydantic-models-to-grammar-examples.py
docstring-parser
pydantic
];
llama-python-test-deps = with python3Packages; [
+14 -9
View File
@@ -3,11 +3,11 @@ ARG OPENVINO_VERSION_FULL=2026.4.1.22982.07f9c262b05
ARG UBUNTU_VERSION=24.04
# Intel GPU driver versions. https://github.com/intel/compute-runtime/releases
ARG IGC_VERSION=v2.41.5
ARG IGC_VERSION_FULL=2_2.41.5+22716
ARG COMPUTE_RUNTIME_VERSION=26.35.39758.10
ARG COMPUTE_RUNTIME_VERSION_FULL=26.35.39758.10-0
ARG IGDGMM_VERSION=22.10.0
ARG IGC_VERSION=v2.42.6
ARG IGC_VERSION_FULL=2_2.42.6+23070
ARG COMPUTE_RUNTIME_VERSION=26.40.40196.8
ARG COMPUTE_RUNTIME_VERSION_FULL=26.40.40196.8-0
ARG IGDGMM_VERSION=22.10.2
# Intel NPU driver versions. https://github.com/intel/linux-npu-driver/releases
ARG NPU_DRIVER_VERSION=v1.38.0
@@ -28,12 +28,16 @@ FROM docker.io/node:$NODE_VERSION AS web
ARG APP_VERSION
WORKDIR /app/tools/ui
WORKDIR /app/ui/apps/chat
COPY tools/ui/package.json tools/ui/package-lock.json ./
COPY ui/apps/chat/package.json ui/apps/chat/package-lock.json ./
RUN npm ci
COPY tools/ui/ ./
COPY ui/apps/chat/ ./
# the app compiles Llama UI and Llama Voice from their sources next to it
COPY ui/lib/ui/package.json /app/ui/lib/ui/package.json
COPY ui/lib/ui/src/lib/ /app/ui/lib/ui/src/lib/
COPY ui/lib/voice/src/ /app/ui/lib/voice/src/
RUN LLAMA_BUILD_NUMBER="$APP_VERSION" npm run build
## Build Image
@@ -83,7 +87,7 @@ WORKDIR /app
COPY . .
COPY --from=web /app/tools/ui/dist tools/ui/dist
COPY --from=web /app/ui/apps/chat/dist ui/apps/chat/dist
# Build Stage
RUN bash -c "source ${OpenVINO_DIR}/setupvars.sh && \
@@ -227,6 +231,7 @@ ENTRYPOINT [ "/app/llama-cli" ]
FROM base AS server
ENV LLAMA_ARG_HOST=0.0.0.0
ENV LLAMA_ARG_PORT=8080
COPY --from=build /app/full/llama /app/full/llama-server /app/
+11 -6
View File
@@ -17,12 +17,16 @@ FROM docker.io/node:$NODE_VERSION AS web
ARG APP_VERSION
WORKDIR /app/tools/ui
WORKDIR /app/ui/apps/chat
COPY tools/ui/package.json tools/ui/package-lock.json ./
COPY ui/apps/chat/package.json ui/apps/chat/package-lock.json ./
RUN npm ci
COPY tools/ui/ ./
COPY ui/apps/chat/ ./
# the app compiles Llama UI and Llama Voice from their sources next to it
COPY ui/lib/ui/package.json /app/ui/lib/ui/package.json
COPY ui/lib/ui/src/lib/ /app/ui/lib/ui/src/lib/
COPY ui/lib/voice/src/ /app/ui/lib/voice/src/
RUN LLAMA_BUILD_NUMBER="$APP_VERSION" npm run build
### Build image
@@ -52,7 +56,7 @@ WORKDIR /app
COPY . .
COPY --from=web /app/tools/ui/dist tools/ui/dist
COPY --from=web /app/ui/apps/chat/dist ui/apps/chat/dist
RUN HIPCXX="$(hipconfig -l)/clang" HIP_PATH="$(hipconfig -R)" \
cmake -S . -B build \
@@ -126,7 +130,7 @@ ENTRYPOINT ["/app/tools.sh"]
### Light, CLI only
FROM base AS light
COPY --from=build /app/full/llama /app/full/llama-cli /app/full/llama-completion /app
COPY --from=build /app/full/llama /app/full/llama-cli /app/full/llama-completion /app/
WORKDIR /app
@@ -136,8 +140,9 @@ ENTRYPOINT [ "/app/llama-cli" ]
FROM base AS server
ENV LLAMA_ARG_HOST=0.0.0.0
ENV LLAMA_ARG_PORT=8080
COPY --from=build /app/full/llama /app/full/llama-server /app
COPY --from=build /app/full/llama /app/full/llama-server /app/
WORKDIR /app
+1
View File
@@ -133,6 +133,7 @@ ENTRYPOINT [ "/llama.cpp/bin/llama-cli" ]
FROM base AS server
ENV LLAMA_ARG_HOST=0.0.0.0
ENV LLAMA_ARG_PORT=8080
WORKDIR /llama.cpp/bin
+16 -17
View File
@@ -9,12 +9,16 @@ FROM docker.io/node:$NODE_VERSION AS web
ARG APP_VERSION
WORKDIR /app/tools/ui
WORKDIR /app/ui/apps/chat
COPY tools/ui/package.json tools/ui/package-lock.json ./
COPY ui/apps/chat/package.json ui/apps/chat/package-lock.json ./
RUN npm ci
COPY tools/ui/ ./
COPY ui/apps/chat/ ./
# the app compiles Llama UI and Llama Voice from their sources next to it
COPY ui/lib/ui/package.json /app/ui/lib/ui/package.json
COPY ui/lib/ui/src/lib/ /app/ui/lib/ui/src/lib/
COPY ui/lib/voice/src/ /app/ui/lib/voice/src/
RUN LLAMA_BUILD_NUMBER="$APP_VERSION" npm run build
FROM docker.io/ubuntu:$UBUNTU_VERSION AS build
@@ -31,7 +35,7 @@ WORKDIR /app
COPY . .
COPY --from=web /app/tools/ui/dist tools/ui/dist
COPY --from=web /app/ui/apps/chat/dist ui/apps/chat/dist
RUN cmake -B build -DGGML_NATIVE=OFF -DGGML_VULKAN=ON -DLLAMA_BUILD_TESTS=OFF -DGGML_BACKEND_DL=ON -DGGML_CPU_ALL_VARIANTS=ON && \
cmake --build build --config Release -j$(nproc)
@@ -82,20 +86,14 @@ COPY --from=build /app/full /app
WORKDIR /app
ENV PATH="/root/.venv/bin:/root/.local/bin:${PATH}"
# Flag for compatibility with pip
ARG UV_INDEX_STRATEGY="unsafe-best-match"
RUN apt-get update \
&& apt-get install -y \
build-essential \
curl \
git \
ca-certificates \
&& curl -LsSf https://astral.sh/uv/install.sh | sh \
&& uv python install 3.13 \
&& uv venv --python 3.13 /root/.venv \
&& uv pip install --python /root/.venv/bin/python -r requirements.txt \
python3 \
python3-pip \
python3-wheel \
&& pip install --break-system-packages --upgrade setuptools \
&& pip install --break-system-packages -r requirements.txt \
&& apt autoremove -y \
&& apt clean -y \
&& rm -rf /tmp/* /var/tmp/* \
@@ -107,7 +105,7 @@ ENTRYPOINT ["/app/tools.sh"]
### Light, CLI only
FROM base AS light
COPY --from=build /app/full/llama /app/full/llama-cli /app/full/llama-completion /app
COPY --from=build /app/full/llama /app/full/llama-cli /app/full/llama-completion /app/
WORKDIR /app
@@ -117,8 +115,9 @@ ENTRYPOINT [ "/app/llama-cli" ]
FROM base AS server
ENV LLAMA_ARG_HOST=0.0.0.0
ENV LLAMA_ARG_PORT=8080
COPY --from=build /app/full/llama /app/full/llama-server /app
COPY --from=build /app/full/llama /app/full/llama-server /app/
WORKDIR /app
+11 -6
View File
@@ -9,12 +9,16 @@ FROM docker.io/node:$NODE_VERSION AS web
ARG APP_VERSION
WORKDIR /app/tools/ui
WORKDIR /app/ui/apps/chat
COPY tools/ui/package.json tools/ui/package-lock.json ./
COPY ui/apps/chat/package.json ui/apps/chat/package-lock.json ./
RUN npm ci
COPY tools/ui/ ./
COPY ui/apps/chat/ ./
# the app compiles Llama UI and Llama Voice from their sources next to it
COPY ui/lib/ui/package.json /app/ui/lib/ui/package.json
COPY ui/lib/ui/src/lib/ /app/ui/lib/ui/src/lib/
COPY ui/lib/voice/src/ /app/ui/lib/voice/src/
RUN LLAMA_BUILD_NUMBER="$APP_VERSION" npm run build
FROM docker.io/ubuntu:$UBUNTU_VERSION AS build
@@ -28,7 +32,7 @@ WORKDIR /app
COPY . .
COPY --from=web /app/tools/ui/dist tools/ui/dist
COPY --from=web /app/ui/apps/chat/dist ui/apps/chat/dist
RUN cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=OFF -DLLAMA_BUILD_TESTS=OFF -DGGML_BACKEND_DL=ON -DGGML_CPU_ALL_VARIANTS=ON -DGGML_ZENDNN=ON && \
cmake --build build -j $(nproc)
@@ -97,7 +101,7 @@ ENTRYPOINT ["/app/tools.sh"]
### Light, CLI only
FROM base AS light
COPY --from=build /app/full/llama /app/full/llama-cli /app/full/llama-completion /app
COPY --from=build /app/full/llama /app/full/llama-cli /app/full/llama-completion /app/
WORKDIR /app
@@ -107,8 +111,9 @@ ENTRYPOINT [ "/app/llama-cli" ]
FROM base AS server
ENV LLAMA_ARG_HOST=0.0.0.0
ENV LLAMA_ARG_PORT=8080
COPY --from=build /app/full/llama /app/full/llama-server /app
COPY --from=build /app/full/llama /app/full/llama-server /app/
WORKDIR /app
+1 -1
View File
@@ -10,7 +10,7 @@
build*/
tools/ui/node_modules/
ui/apps/chat/node_modules/
models/*
+4 -1
View File
@@ -45,7 +45,10 @@ insert_final_newline = unset
trim_trailing_whitespace = unset
insert_final_newline = unset
[tools/ui/**]
[vendor/**.patch]
trim_trailing_whitespace = unset
[ui/**]
indent_style = unset
indent_size = unset
end_of_line = unset
+2
View File
@@ -5,6 +5,8 @@ exclude =
# Do not traverse examples and tools
examples,
tools,
# Do not traverse .venv directories
.venv*,
# Do not include package initializers
__init__.py,
# No need to traverse our git directory
+3 -1
View File
@@ -14,7 +14,9 @@ runs:
run: |
BUILD_NUMBER="$(git rev-list --count HEAD)"
SHORT_HASH="$(git rev-parse --short=7 HEAD)"
if [[ "${{ env.BRANCH_NAME }}" == "master" || "${{ env.BRANCH_NAME }}" == "b${BUILD_NUMBER}" ]]; then
if [[ "$GITHUB_REF_TYPE" == "tag" && "${{ env.BRANCH_NAME }}" =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
echo "name=${{ env.BRANCH_NAME }}" >> $GITHUB_OUTPUT
elif [[ "${{ env.BRANCH_NAME }}" == "master" || "${{ env.BRANCH_NAME }}" == "b${BUILD_NUMBER}" ]]; then
echo "name=b${BUILD_NUMBER}" >> $GITHUB_OUTPUT
else
SAFE_NAME=$(echo "${{ env.BRANCH_NAME }}" | tr '/' '-')
+56 -46
View File
@@ -25,7 +25,9 @@ runs:
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/visual_studio_integration/windows-x86_64/visual_studio_integration-windows-x86_64-11.7.91-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_nvprof/windows-x86_64/cuda_nvprof-windows-x86_64-11.7.101-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_cccl/windows-x86_64/cuda_cccl-windows-x86_64-11.7.91-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_nvml_dev/windows-x86_64/cuda_nvml_dev-windows-x86_64-11.7.91-archive.zip"
unzip '*.zip' -d "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7"
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7\cuda_nvml_dev-windows-x86_64-11.7.91-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7\cuda_cudart-windows-x86_64-11.7.99-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7\cuda_nvcc-windows-x86_64-11.7.99-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7\cuda_nvrtc-windows-x86_64-11.7.99-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7" /E /I /H /Y
@@ -54,7 +56,9 @@ runs:
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/visual_studio_integration/windows-x86_64/visual_studio_integration-windows-x86_64-12.4.127-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_nvprof/windows-x86_64/cuda_nvprof-windows-x86_64-12.4.127-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_cccl/windows-x86_64/cuda_cccl-windows-x86_64-12.4.127-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_nvml_dev/windows-x86_64/cuda_nvml_dev-windows-x86_64-12.4.127-archive.zip"
unzip '*.zip' -d "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4"
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4\cuda_nvml_dev-windows-x86_64-12.4.127-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4\cuda_cudart-windows-x86_64-12.4.127-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4\cuda_nvcc-windows-x86_64-12.4.131-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4\cuda_nvrtc-windows-x86_64-12.4.127-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4" /E /I /H /Y
@@ -85,7 +89,9 @@ runs:
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_profiler_api/windows-x86_64/cuda_profiler_api-windows-x86_64-13.1.80-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/visual_studio_integration/windows-x86_64/visual_studio_integration-windows-x86_64-13.1.68-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_cccl/windows-x86_64/cuda_cccl-windows-x86_64-13.1.78-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_nvml_dev/windows-x86_64/cuda_nvml_dev-windows-x86_64-13.1.68-archive.zip"
unzip '*.zip' -d "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.1"
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.1\cuda_nvml_dev-windows-x86_64-13.1.68-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.1" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.1\cuda_crt-windows-x86_64-13.1.80-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.1" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.1\cuda_cudart-windows-x86_64-13.1.80-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.1" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.1\cuda_nvcc-windows-x86_64-13.1.80-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.1" /E /I /H /Y
@@ -100,56 +106,60 @@ runs:
echo "CUDA_PATH=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.1" | Out-File -FilePath $env:GITHUB_ENV -Append -Encoding utf8
echo "CUDA_PATH_V13_1=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.1" | Out-File -FilePath $env:GITHUB_ENV -Append -Encoding utf8
- name: Install Cuda Toolkit 13.4 for x64
if: ${{ inputs.cuda_version == '13.4' && inputs.cuda_arch == 'x64' }}
- name: Install Cuda Toolkit 13.5 for x64
if: ${{ inputs.cuda_version == '13.5' && inputs.cuda_arch == 'x64' }}
shell: pwsh
run: |
mkdir -p "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.4"
mkdir -p "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5"
choco install unzip -y
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_crt/windows-x86_64/cuda_crt-windows-x86_64-13.4.59-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_cudart/windows-x86_64/cuda_cudart-windows-x86_64-13.4.49-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_nvcc/windows-x86_64/cuda_nvcc-windows-x86_64-13.4.59-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_nvrtc/windows-x86_64/cuda_nvrtc-windows-x86_64-13.4.59-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/libcublas/windows-x86_64/libcublas-windows-x86_64-13.7.0.27-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/libnvvm/windows-x86_64/libnvvm-windows-x86_64-13.4.59-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_nvtx/windows-x86_64/cuda_nvtx-windows-x86_64-13.4.49-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_profiler_api/windows-x86_64/cuda_profiler_api-windows-x86_64-13.4.49-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/visual_studio_integration/windows-x86_64/visual_studio_integration-windows-x86_64-13.4.49-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cccl/windows-x86_64/cccl-windows-x86_64-13.3.4.2.1-archive.zip"
unzip '*.zip' -d "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.4"
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.4\cuda_crt-windows-x86_64-13.4.59-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.4" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.4\cuda_cudart-windows-x86_64-13.4.49-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.4" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.4\cuda_nvcc-windows-x86_64-13.4.59-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.4" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.4\cuda_nvrtc-windows-x86_64-13.4.59-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.4" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.4\libcublas-windows-x86_64-13.7.0.27-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.4" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.4\libnvvm-windows-x86_64-13.4.59-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.4" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.4\cuda_nvtx-windows-x86_64-13.4.49-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.4" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.4\cuda_profiler_api-windows-x86_64-13.4.49-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.4" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.4\visual_studio_integration-windows-x86_64-13.4.49-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.4" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.4\cccl-windows-x86_64-13.3.4.2.1-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.4" /E /I /H /Y
echo "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.4\bin" | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append
echo "CUDA_PATH=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.4" | Out-File -FilePath $env:GITHUB_ENV -Append -Encoding utf8
echo "CUDA_PATH_V13_4=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.4" | Out-File -FilePath $env:GITHUB_ENV -Append -Encoding utf8
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_crt/windows-x86_64/cuda_crt-windows-x86_64-13.5.37-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_cudart/windows-x86_64/cuda_cudart-windows-x86_64-13.5.37-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_nvcc/windows-x86_64/cuda_nvcc-windows-x86_64-13.5.37-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_nvrtc/windows-x86_64/cuda_nvrtc-windows-x86_64-13.5.37-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/libcublas/windows-x86_64/libcublas-windows-x86_64-13.10.0.20-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/libnvvm/windows-x86_64/libnvvm-windows-x86_64-13.5.37-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_nvtx/windows-x86_64/cuda_nvtx-windows-x86_64-13.5.25-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_profiler_api/windows-x86_64/cuda_profiler_api-windows-x86_64-13.5.25-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/visual_studio_integration/windows-x86_64/visual_studio_integration-windows-x86_64-13.5.25-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cccl/windows-x86_64/cccl-windows-x86_64-13.3.5.0.6-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_nvml_dev/windows-x86_64/cuda_nvml_dev-windows-x86_64-13.5.37-archive.zip"
unzip '*.zip' -d "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5"
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5\cuda_nvml_dev-windows-x86_64-13.5.37-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5\cuda_crt-windows-x86_64-13.5.37-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5\cuda_cudart-windows-x86_64-13.5.37-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5\cuda_nvcc-windows-x86_64-13.5.37-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5\cuda_nvrtc-windows-x86_64-13.5.37-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5\libcublas-windows-x86_64-13.10.0.20-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5\libnvvm-windows-x86_64-13.5.37-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5\cuda_nvtx-windows-x86_64-13.5.25-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5\cuda_profiler_api-windows-x86_64-13.5.25-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5\visual_studio_integration-windows-x86_64-13.5.25-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5\cccl-windows-x86_64-13.3.5.0.6-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5" /E /I /H /Y
echo "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5\bin" | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append
echo "CUDA_PATH=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5" | Out-File -FilePath $env:GITHUB_ENV -Append -Encoding utf8
echo "CUDA_PATH_V13_5=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5" | Out-File -FilePath $env:GITHUB_ENV -Append -Encoding utf8
- name: Install Cuda Toolkit 13.4 for ARM64
if: ${{ inputs.cuda_version == '13.4' && inputs.cuda_arch == 'arm64' }}
- name: Install Cuda Toolkit 13.5 for ARM64
if: ${{ inputs.cuda_version == '13.5' && inputs.cuda_arch == 'arm64' }}
shell: pwsh
run: |
mkdir -p "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.4"
mkdir -p "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5"
choco install unzip -y
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cccl/windows-x86_64/cccl-windows-x86_64-13.3.4.2.1-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_crt/windows-x86_64/cuda_crt-windows-x86_64-13.4.59-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_nvcc/windows-x86_64/cuda_nvcc-windows-x86_64-13.4.59-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/libnvvm/windows-x86_64/libnvvm-windows-x86_64-13.4.59-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_cudart/windows-arm64/cuda_cudart-windows-arm64-13.4.49-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/libcublas/windows-arm64/libcublas-windows-arm64-13.7.0.27-archive.zip"
unzip '*.zip' -d "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.4"
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.4\cccl-windows-x86_64-13.3.4.2.1-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.4" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.4\cuda_crt-windows-x86_64-13.4.59-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.4" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.4\cuda_nvcc-windows-x86_64-13.4.59-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.4" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.4\libnvvm-windows-x86_64-13.4.59-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.4" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.4\cuda_cudart-windows-arm64-13.4.49-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.4" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.4\libcublas-windows-arm64-13.7.0.27-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.4" /E /I /H /Y
echo "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.4\bin" | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append
echo "CUDA_PATH=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.4" | Out-File -FilePath $env:GITHUB_ENV -Append -Encoding utf8
echo "CUDA_PATH_V13_4=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.4" | Out-File -FilePath $env:GITHUB_ENV -Append -Encoding utf8
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cccl/windows-x86_64/cccl-windows-x86_64-13.3.5.0.6-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_crt/windows-x86_64/cuda_crt-windows-x86_64-13.5.37-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_nvcc/windows-x86_64/cuda_nvcc-windows-x86_64-13.5.37-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/libnvvm/windows-x86_64/libnvvm-windows-x86_64-13.5.37-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_cudart/windows-arm64/cuda_cudart-windows-arm64-13.5.37-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/libcublas/windows-arm64/libcublas-windows-arm64-13.10.0.20-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_nvml_dev/windows-arm64/cuda_nvml_dev-windows-arm64-13.5.37-archive.zip"
unzip '*.zip' -d "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5"
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5\cuda_nvml_dev-windows-arm64-13.5.37-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5\cccl-windows-x86_64-13.3.5.0.6-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5\cuda_crt-windows-x86_64-13.5.37-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5\cuda_nvcc-windows-x86_64-13.5.37-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5\libnvvm-windows-x86_64-13.5.37-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5\cuda_cudart-windows-arm64-13.5.37-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5\libcublas-windows-arm64-13.10.0.20-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5" /E /I /H /Y
echo "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5\bin" | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append
echo "CUDA_PATH=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5" | Out-File -FilePath $env:GITHUB_ENV -Append -Encoding utf8
echo "CUDA_PATH_V13_5=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5" | Out-File -FilePath $env:GITHUB_ENV -Append -Encoding utf8
+1 -2
View File
@@ -48,7 +48,6 @@ examples:
- all-globs-to-all-files:
- '!tools/server/**'
- '!tools/mtmd/**'
- '!tools/ui/**'
testing:
- changed-files:
- any-glob-to-any-file:
@@ -72,7 +71,7 @@ android:
server/ui:
- changed-files:
- any-glob-to-any-file:
- tools/ui/**
- ui/**
server:
- changed-files:
- any-glob-to-any-file:
+177
View File
@@ -0,0 +1,177 @@
name: Behavior Check
on:
workflow_dispatch: # allows manual triggering
push:
branches:
- master
paths: [
'.github/workflows/behavior-check.yml',
'**/CMakeLists.txt',
'**/.cmake',
'**/*.h',
'**/*.hpp',
'**/*.c',
'**/*.cpp',
'**/*.py',
'**/requirements*.txt',
]
pull_request:
types: [opened, synchronize, reopened]
paths: [
'.github/workflows/behavior-check.yml',
'**/CMakeLists.txt',
'**/.cmake',
'**/*.h',
'**/*.hpp',
'**/*.c',
'**/*.cpp',
'**/*.py',
'**/requirements*.txt',
]
cache-mode: none
permissions:
contents: read
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}
cancel-in-progress: true
env:
GGML_NLOOP: 3
GGML_N_THREADS: 1
GGML_SCHED_DEBUG_REALLOC: 1
LLAMA_ARG_LOG_COLORS: 1
LLAMA_ARG_LOG_PREFIX: 1
LLAMA_ARG_LOG_TIMESTAMPS: 1
jobs:
ubuntu:
runs-on: ubuntu-26.04-arm
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: ccache
uses: ggml-org/ccache-action@v1.2.24
with:
restore: false
save: false
- name: Build Dependencies
id: build_depends
run: |
sudo apt update
sudo apt install -y build-essential libssl-dev git-lfs libjpeg-dev python3 python3-venv python3-pip
- name: Python Dependencies
id: python_depends
run: |
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements-convert_hf_to_gguf.txt
- name: ccache-buckets-restore
uses: ./.github/actions/ccache-buckets
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CI }}
with:
key: behavior-check-ubuntu-26.04-arm
folder: llama.cpp
hf_bucket: ggml-org/cache
- name: Build
id: cmake_build
run: |
cmake -B build \
-DGGML_NATIVE=OFF \
-DGGML_BACKEND_DL=OFF \
-DLLAMA_FATAL_WARNINGS=ON
time cmake --build build --config Release -j $(nproc)
- name: ccache-buckets-save
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
uses: ./.github/actions/ccache-buckets
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CACHE_OUTPUT }}
with:
key: behavior-check-ubuntu-26.04-arm
folder: llama.cpp
evict-old-files: 1d
hf_bucket: ggml-org/cache
save: true
- name: Test
id: cmake_test
run: |
source .venv/bin/activate
cd build
ctest -L behavior -LE 'model|backend' --verbose --timeout 900
windows:
runs-on: windows-2025
cache-mode: write
permissions:
actions: write
contents: read
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: ccache
uses: ggml-org/ccache-action@v1.2.24
with:
key: behavior-check-windows-2025
evict-old-files: 1d
save: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
- name: Python Dependencies
id: python_depends
shell: bash
run: |
python3 -m venv .venv
source .venv/Scripts/activate
pip install -r requirements/requirements-convert_hf_to_gguf.txt
- name: Install LLVM
run: |
choco install llvm --version=22.1.8
echo "C:\Program Files\LLVM\bin" >> $env:GITHUB_PATH
- name: Build
id: cmake_build
shell: cmd
run: |
call "C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64
cmake -B build -G "Ninja Multi-Config" ^
-DCMAKE_TOOLCHAIN_FILE=cmake/x64-windows-llvm.cmake ^
-DCMAKE_BUILD_TYPE=Release ^
-DGGML_NATIVE=OFF ^
-DGGML_BACKEND_DL=OFF
set /A NINJA_JOBS=%NUMBER_OF_PROCESSORS%-1
cmake --build build --config Release -j %NINJA_JOBS%
- name: Test
id: cmake_test
shell: bash
run: |
source .venv/Scripts/activate
cd build
ctest -C Release -L behavior -LE 'model|backend' --verbose --timeout 900
- name: ccache-clear
uses: ./.github/actions/ccache-clear
env:
GH_TOKEN: ${{ github.token }}
with:
key: behavior-check-windows-2025
older: 5m
min: 1
dry-run: ${{ github.event_name != 'push' || github.ref != 'refs/heads/master' }}
+2 -2
View File
@@ -31,8 +31,8 @@ env:
LLAMA_ARG_LOG_TIMESTAMPS: 1
jobs:
ubuntu-24-llguidance:
runs-on: ${{ 'ubuntu-24.04-arm' || 'ubuntu-24.04' }}
ubuntu-26-llguidance:
runs-on: ${{ 'ubuntu-26.04-arm' || 'ubuntu-26.04' }}
steps:
- name: Clone
@@ -33,9 +33,9 @@ concurrency:
jobs:
android-ndk-snapdragon:
runs-on: ubuntu-24.04 # previously ubuntu-latest
runs-on: ubuntu-26.04
container:
image: 'ghcr.io/snapdragon-toolchain/arm64-android:v0.7'
image: 'ghcr.io/snapdragon-toolchain/arm64-android:v0.8'
defaults:
run:
shell: bash
@@ -67,9 +67,9 @@ jobs:
path: pkg-snapdragon/llama.cpp
linux-iot-snapdragon:
runs-on: ubuntu-24.04 # previously ubuntu-latest
runs-on: ubuntu-26.04
container:
image: 'ghcr.io/snapdragon-toolchain/arm64-linux:v0.7'
image: 'ghcr.io/snapdragon-toolchain/arm64-linux:v0.8'
defaults:
run:
shell: bash
@@ -103,12 +103,12 @@ jobs:
test-snapdragon-qdc:
name: Test on QDC Device (${{ matrix.device }})
needs: [android-ndk-snapdragon, linux-iot-snapdragon]
runs-on: ubuntu-24.04-arm
timeout-minutes: 90
runs-on: ubuntu-26.04-arm
timeout-minutes: 180
strategy:
fail-fast: false
matrix:
device: [SM8750, SM8850, QCS9075M]
device: [SM8850, QCS9075M]
permissions:
actions: read
@@ -155,7 +155,7 @@ jobs:
--pkg-dir pkg-snapdragon/llama.cpp \
--model-url "https://huggingface.co/bartowski/Llama-3.2-1B-Instruct-GGUF/resolve/main/Llama-3.2-1B-Instruct-Q4_0.gguf" \
--device ${{ matrix.device }} \
${{ startsWith(matrix.device, 'QCS') && '--retries 2 --retry-delay 300' || '' }}
${{ startsWith(matrix.device, 'QCS') && '--job-timeout 5400 --retries 2 --retry-delay 300' || '' }}
env:
QDC_API_KEY: ${{ secrets.QDC_API_KEY }}
+4 -3
View File
@@ -37,7 +37,7 @@ env:
jobs:
default:
runs-on: ubuntu-24.04 # previously ubuntu-latest
runs-on: ubuntu-26.04
steps:
- name: Clone
@@ -56,6 +56,7 @@ jobs:
uses: android-actions/setup-android@be39fa834029ff78f1a44aa3bb0819b8fc2bd8fd # v4.0.4
with:
log-accepted-android-sdk-licenses: false
packages: platform-tools cmake;3.31.6
- name: Build
run: |
@@ -63,7 +64,7 @@ jobs:
./gradlew build --no-daemon
ndk:
runs-on: ubuntu-24.04 # previously ubuntu-latest
runs-on: ubuntu-26.04
container:
image: 'ghcr.io/snapdragon-toolchain/arm64-android:v0.3'
defaults:
@@ -101,7 +102,7 @@ jobs:
path: pkg-adb/llama.cpp
arm64:
runs-on: ubuntu-24.04 # previously ubuntu-latest
runs-on: ubuntu-26.04
env:
NDK_VERSION: "29.0.14206865"
+160
View File
@@ -0,0 +1,160 @@
name: CI (apple - daily)
on:
workflow_dispatch: # allows manual triggering
schedule:
- cron: '0 0 * * *'
cache-mode: none
permissions:
contents: read
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}
cancel-in-progress: true
env:
GGML_NLOOP: 3
GGML_N_THREADS: 1
LLAMA_ARG_LOG_COLORS: 1
LLAMA_ARG_LOG_PREFIX: 1
LLAMA_ARG_LOG_TIMESTAMPS: 1
jobs:
macos-latest-ios-xcode:
runs-on: macos-latest
permissions:
actions: write
contents: read
steps:
- name: Checkout code
uses: actions/checkout@v6
- name: Setup Xcode
uses: ggml-org/setup-xcode@v1
with:
xcode-version: latest-stable
- name: Build
id: cmake_build
run: |
sysctl -a
cmake -B build -G Xcode \
-DGGML_METAL_EMBED_LIBRARY=ON \
-DLLAMA_OPENSSL=OFF \
-DLLAMA_BUILD_APP=OFF \
-DLLAMA_BUILD_EXAMPLES=OFF \
-DLLAMA_BUILD_TOOLS=OFF \
-DLLAMA_BUILD_TESTS=OFF \
-DLLAMA_BUILD_SERVER=OFF \
-DCMAKE_SYSTEM_NAME=iOS \
-DCMAKE_OSX_DEPLOYMENT_TARGET=14.0 \
-DCMAKE_XCODE_ATTRIBUTE_DEVELOPMENT_TEAM=ggml
cmake --build build --config Release -j $(sysctl -n hw.logicalcpu) -- CODE_SIGNING_ALLOWED=NO
- name: xcodebuild for swift package
id: xcodebuild
run: |
./build-xcframework.sh
- name: Upload xcframework artifact
uses: actions/upload-artifact@v6
with:
name: llama-xcframework
path: build-apple/llama.xcframework/
retention-days: 1
- name: Build Xcode project
run: |
xcodebuild -downloadPlatform iOS
xcodebuild -project examples/llama.swiftui/llama.swiftui.xcodeproj -scheme llama.swiftui -sdk iphoneos CODE_SIGNING_REQUIRED=NO CODE_SIGN_IDENTITY= -destination 'generic/platform=iOS' FRAMEWORK_FOLDER_PATH=./build-ios build
macos-latest-tvos:
runs-on: macos-latest
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: Build
id: cmake_build
run: |
sysctl -a
cmake -B build -G Xcode \
-DGGML_METAL_EMBED_LIBRARY=ON \
-DLLAMA_BUILD_COMMON=OFF \
-DLLAMA_BUILD_APP=OFF \
-DLLAMA_BUILD_EXAMPLES=OFF \
-DLLAMA_BUILD_TOOLS=OFF \
-DLLAMA_BUILD_TESTS=OFF \
-DLLAMA_BUILD_SERVER=OFF \
-DCMAKE_SYSTEM_NAME=tvOS \
-DCMAKE_OSX_DEPLOYMENT_TARGET=14.0 \
-DCMAKE_XCODE_ATTRIBUTE_DEVELOPMENT_TEAM=ggml
cmake --build build --config Release -j $(sysctl -n hw.logicalcpu) -- CODE_SIGNING_ALLOWED=NO
macos-latest-visionos:
runs-on: macos-latest
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: Build
id: cmake_build
run: |
sysctl -a
cmake -B build -G Xcode \
-DGGML_METAL_EMBED_LIBRARY=ON \
-DLLAMA_BUILD_COMMON=OFF \
-DLLAMA_BUILD_APP=OFF \
-DLLAMA_BUILD_EXAMPLES=OFF \
-DLLAMA_BUILD_TOOLS=OFF \
-DLLAMA_BUILD_TESTS=OFF \
-DLLAMA_BUILD_SERVER=OFF \
-DCMAKE_SYSTEM_NAME=visionOS \
-DCMAKE_OSX_DEPLOYMENT_TARGET=1.0 \
-DCMAKE_XCODE_ATTRIBUTE_DEVELOPMENT_TEAM=ggml
cmake --build build --config Release -j $(sysctl -n hw.logicalcpu) -- CODE_SIGNING_ALLOWED=NO
macos-latest-swift:
runs-on: macos-latest
needs: macos-latest-ios-xcode
permissions:
actions: read
contents: read
strategy:
matrix:
destination: ['generic/platform=macOS', 'generic/platform=iOS', 'generic/platform=tvOS']
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: Download xcframework artifact
uses: actions/download-artifact@v7
with:
name: llama-xcframework
path: build-apple/llama.xcframework/
- name: Build llama.cpp with CMake
id: cmake_build
run: |
sysctl -a
cmake -B build -G Xcode \
-DGGML_METAL_EMBED_LIBRARY=ON \
-DLLAMA_OPENSSL=OFF \
-DLLAMA_BUILD_APP=OFF \
-DLLAMA_BUILD_EXAMPLES=OFF \
-DLLAMA_BUILD_TOOLS=OFF \
-DLLAMA_BUILD_TESTS=OFF \
-DLLAMA_BUILD_SERVER=OFF \
-DCMAKE_OSX_ARCHITECTURES="arm64;x86_64"
cmake --build build --config Release -j $(sysctl -n hw.logicalcpu)
+1 -194
View File
@@ -105,198 +105,5 @@ jobs:
cd build
# Metal Paravirtual devices are difficult to support -> disable
# ref: https://github.com/ggml-org/llama.cpp/pull/19802#issuecomment-4013704023
ctest -L main -E "test-llama-archs|test-save-load-state|test-recurrent-state-rollback" --verbose --timeout 900
ctest -L main -E "test-llama-archs|test-llama-context|test-speculative-rejection-sampling" --verbose --timeout 900
macos-latest-x64:
runs-on: macos-15-intel
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: ccache
uses: ggml-org/ccache-action@v1.2.24
with:
restore: false
save: false
- name: ccache-buckets-restore
uses: ./.github/actions/ccache-buckets
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CI }}
with:
key: apple-x64
folder: llama.cpp
hf_bucket: ggml-org/cache
- name: Build
id: cmake_build
run: |
sysctl -a
# Metal is disabled due to intermittent failures with Github runners not having a GPU:
# https://github.com/ggml-org/llama.cpp/actions/runs/8635935781/job/23674807267#step:5:2313
cmake -B build \
-DCMAKE_BUILD_RPATH="@loader_path" \
-DLLAMA_FATAL_WARNINGS=ON \
-DLLAMA_BUILD_BORINGSSL=ON \
-DGGML_METAL=OFF \
-DGGML_RPC=ON \
-DCMAKE_OSX_DEPLOYMENT_TARGET=13.3
time cmake --build build --config Release -j $(sysctl -n hw.logicalcpu)
- name: ccache-buckets-save
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
uses: ./.github/actions/ccache-buckets
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CACHE_OUTPUT }}
with:
key: apple-x64
folder: llama.cpp
evict-old-files: 1d
hf_bucket: ggml-org/cache
save: true
- name: Test
id: cmake_test
run: |
cd build
ctest -L main --verbose --timeout 900
macos-latest-ios-xcode:
runs-on: macos-latest
permissions:
actions: write
contents: read
steps:
- name: Checkout code
uses: actions/checkout@v6
- name: Setup Xcode
uses: ggml-org/setup-xcode@v1
with:
xcode-version: latest-stable
- name: Build
id: cmake_build
run: |
sysctl -a
cmake -B build -G Xcode \
-DGGML_METAL_EMBED_LIBRARY=ON \
-DLLAMA_OPENSSL=OFF \
-DLLAMA_BUILD_APP=OFF \
-DLLAMA_BUILD_EXAMPLES=OFF \
-DLLAMA_BUILD_TOOLS=OFF \
-DLLAMA_BUILD_TESTS=OFF \
-DLLAMA_BUILD_SERVER=OFF \
-DCMAKE_SYSTEM_NAME=iOS \
-DCMAKE_OSX_DEPLOYMENT_TARGET=14.0 \
-DCMAKE_XCODE_ATTRIBUTE_DEVELOPMENT_TEAM=ggml
cmake --build build --config Release -j $(sysctl -n hw.logicalcpu) -- CODE_SIGNING_ALLOWED=NO
- name: xcodebuild for swift package
id: xcodebuild
run: |
./build-xcframework.sh
- name: Upload xcframework artifact
uses: actions/upload-artifact@v6
with:
name: llama-xcframework
path: build-apple/llama.xcframework/
retention-days: 1
- name: Build Xcode project
run: |
xcodebuild -downloadPlatform iOS
xcodebuild -project examples/llama.swiftui/llama.swiftui.xcodeproj -scheme llama.swiftui -sdk iphoneos CODE_SIGNING_REQUIRED=NO CODE_SIGN_IDENTITY= -destination 'generic/platform=iOS' FRAMEWORK_FOLDER_PATH=./build-ios build
macos-latest-tvos:
runs-on: macos-latest
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: Build
id: cmake_build
run: |
sysctl -a
cmake -B build -G Xcode \
-DGGML_METAL_EMBED_LIBRARY=ON \
-DLLAMA_BUILD_COMMON=OFF \
-DLLAMA_BUILD_APP=OFF \
-DLLAMA_BUILD_EXAMPLES=OFF \
-DLLAMA_BUILD_TOOLS=OFF \
-DLLAMA_BUILD_TESTS=OFF \
-DLLAMA_BUILD_SERVER=OFF \
-DCMAKE_SYSTEM_NAME=tvOS \
-DCMAKE_OSX_DEPLOYMENT_TARGET=14.0 \
-DCMAKE_XCODE_ATTRIBUTE_DEVELOPMENT_TEAM=ggml
cmake --build build --config Release -j $(sysctl -n hw.logicalcpu) -- CODE_SIGNING_ALLOWED=NO
macos-latest-visionos:
runs-on: macos-latest
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: Build
id: cmake_build
run: |
sysctl -a
cmake -B build -G Xcode \
-DGGML_METAL_EMBED_LIBRARY=ON \
-DLLAMA_BUILD_COMMON=OFF \
-DLLAMA_BUILD_APP=OFF \
-DLLAMA_BUILD_EXAMPLES=OFF \
-DLLAMA_BUILD_TOOLS=OFF \
-DLLAMA_BUILD_TESTS=OFF \
-DLLAMA_BUILD_SERVER=OFF \
-DCMAKE_SYSTEM_NAME=visionOS \
-DCMAKE_OSX_DEPLOYMENT_TARGET=1.0 \
-DCMAKE_XCODE_ATTRIBUTE_DEVELOPMENT_TEAM=ggml
cmake --build build --config Release -j $(sysctl -n hw.logicalcpu) -- CODE_SIGNING_ALLOWED=NO
macos-latest-swift:
runs-on: macos-latest
needs: macos-latest-ios-xcode
permissions:
actions: read
contents: read
strategy:
matrix:
destination: ['generic/platform=macOS', 'generic/platform=iOS', 'generic/platform=tvOS']
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: Download xcframework artifact
uses: actions/download-artifact@v7
with:
name: llama-xcframework
path: build-apple/llama.xcframework/
- name: Build llama.cpp with CMake
id: cmake_build
run: |
sysctl -a
cmake -B build -G Xcode \
-DGGML_METAL_EMBED_LIBRARY=ON \
-DLLAMA_OPENSSL=OFF \
-DLLAMA_BUILD_APP=OFF \
-DLLAMA_BUILD_EXAMPLES=OFF \
-DLLAMA_BUILD_TOOLS=OFF \
-DLLAMA_BUILD_TESTS=OFF \
-DLLAMA_BUILD_SERVER=OFF \
-DCMAKE_OSX_ARCHITECTURES="arm64;x86_64"
cmake --build build --config Release -j $(sysctl -n hw.logicalcpu)
+4 -4
View File
@@ -40,11 +40,11 @@ jobs:
# path: ./spacemit_toolchain
# version: ${{ env.SPACEMIT_IME_TOOLCHAIN_VERSION }}
ubuntu-24-openvino-cache:
runs-on: ubuntu-24.04
ubuntu-26-openvino-cache:
runs-on: ubuntu-26.04
env:
# Sync versions in build-openvino.yml, build-self-hosted.yml, release.yml, build-cache.yml, .devops/openvino.Dockerfile
# Sync versions in build-openvino.yml, ci-self-hosted-openvino.yml, release.yml, release-publish.yml, build-cache.yml, .devops/openvino.Dockerfile, docs/backend/OPENVINO.md
OPENVINO_VERSION_MAJOR: "2026.4.1"
OPENVINO_VERSION_FULL: "2026.4.1.22982.07f9c262b05"
@@ -72,7 +72,7 @@ jobs:
runs-on: windows-2022
env:
# Sync versions in build.yml, build-self-hosted.yml, release.yml, build-cache.yml, .devops/openvino.Dockerfile
# Sync versions in build-openvino.yml, ci-self-hosted-openvino.yml, release.yml, release-publish.yml, build-cache.yml, .devops/openvino.Dockerfile, docs/backend/OPENVINO.md
OPENVINO_VERSION_MAJOR: "2026.4.1"
OPENVINO_VERSION_FULL: "2026.4.1.22982.07f9c262b05"
-57
View File
@@ -1,57 +0,0 @@
name: Build relocatable cmake package
on:
workflow_dispatch:
workflow_call:
cache-mode: none
permissions:
contents: read
jobs:
linux:
runs-on: [self-hosted, Linux, CPU]
steps:
- uses: actions/checkout@v6
with:
fetch-depth: 0
- name: Build
run: |
PREFIX="$(pwd)"/inst
cmake -S . -B build \
-DCMAKE_PREFIX_PATH="$PREFIX" \
-DLLAMA_OPENSSL=OFF \
-DLLAMA_BUILD_TESTS=OFF \
-DLLAMA_BUILD_TOOLS=OFF \
-DLLAMA_BUILD_EXAMPLES=OFF \
-DLLAMA_BUILD_APP=OFF \
-DLLAMA_BUILD_IS_DEV=OFF \
-DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j $(nproc)
cmake --install build --prefix "$PREFIX" --config Release
export LLAMA_CONFIG="$PREFIX"/lib/cmake/llama/llama-config.cmake
build_commit=$(git rev-parse --short HEAD | xargs)
build_number=$(git rev-list --count HEAD | xargs)
major=$(grep -oE "set\(LLAMA_VERSION_MAJOR[[:space:]]+[0-9]+" CMakeLists.txt | grep -oE "[0-9]+$")
minor=$(grep -oE "set\(LLAMA_VERSION_MINOR[[:space:]]+[0-9]+" CMakeLists.txt | grep -oE "[0-9]+$")
patch=$(grep -oE "set\(LLAMA_VERSION_PATCH[[:space:]]+[0-9]+" CMakeLists.txt | grep -oE "[0-9]+$")
build_version="$major.$minor.$patch"
checks=("set\(LLAMA_VERSION[[:space:]]+$build_version\)"
"set\(LLAMA_BUILD_COMMIT[[:space:]]+$build_commit\)"
"set\(LLAMA_BUILD_NUMBER[[:space:]]+$build_number\)")
for check in "${checks[@]}"; do
if ! grep -qE "$check" "$LLAMA_CONFIG"; then
echo "Checking llama-config.cmake version... \"$check\" failed!"
exit 1
fi
done
echo "Checking llama-config.cmake version... success."
cd examples/simple-cmake-pkg
cmake -S . -B build -DCMAKE_PREFIX_PATH="$PREFIX"/lib/cmake
cmake --build build -j $(nproc)
+84 -15
View File
@@ -7,7 +7,6 @@ on:
- master
paths: [
'.github/workflows/build-cpu.yml',
'.github/workflows/build-cmake-pkg.yml',
'**/CMakeLists.txt',
'**/.cmake',
'**/*.h',
@@ -20,7 +19,6 @@ on:
types: [opened, synchronize, reopened]
paths: [
'.github/workflows/build-cpu.yml',
'.github/workflows/build-cmake-pkg.yml',
'ggml/src/ggml-rpc/**',
'**/CMakeLists.txt',
'**/.cmake',
@@ -47,17 +45,14 @@ env:
LLAMA_ARG_LOG_TIMESTAMPS: 1
jobs:
build-cmake-pkg:
uses: ./.github/workflows/build-cmake-pkg.yml
ubuntu:
strategy:
matrix:
include:
- build: 'x64'
os: ubuntu-22.04
os: ubuntu-26.04
- build: 'arm64'
os: ubuntu-24.04-arm
os: ubuntu-26.04-arm
runs-on: ${{ matrix.os }}
@@ -81,13 +76,6 @@ jobs:
libjpeg-dev build-essential libssl-dev \
git-lfs
- name: Toolchain workaround (GCC 14)
if: ${{ contains(matrix.os, 'ubuntu-24.04') }}
run: |
sudo apt-get install -y gcc-14 g++-14
echo "CC=gcc-14" >> "$GITHUB_ENV"
echo "CXX=g++-14" >> "$GITHUB_ENV"
- name: Python Dependencies
id: python_depends
run: |
@@ -129,7 +117,7 @@ jobs:
id: cmake_test
run: |
cd build
ctest -L 'main|python' --verbose --timeout 900
ctest -L main --verbose --timeout 900
- name: Test llama2c conversion
id: llama2c_test
@@ -199,6 +187,11 @@ jobs:
run: |
choco install ninja
- name: Install LLVM
run: |
choco install llvm --version=22.1.8
echo "C:\Program Files\LLVM\bin" >> $env:GITHUB_PATH
- name: Build
id: cmake_build
run: |
@@ -242,3 +235,79 @@ jobs:
older: 5m
min: 1
dry-run: ${{ github.event_name != 'push' || github.ref != 'refs/heads/master' }}
build-cmake-pkg:
runs-on: ubuntu-24.04
steps:
- name: Clone
uses: actions/checkout@v6
with:
fetch-depth: 0
- name: ccache
uses: ggml-org/ccache-action@v1.2.24
with:
restore: false
save: false
- name: ccache-buckets-restore
uses: ./.github/actions/ccache-buckets
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CI }}
with:
key: build-cmake-pkg
folder: llama.cpp
hf_bucket: ggml-org/cache
- name: Build
run: |
PREFIX="$(pwd)"/inst
cmake -S . -B build \
-DCMAKE_PREFIX_PATH="$PREFIX" \
-DLLAMA_OPENSSL=OFF \
-DLLAMA_BUILD_TESTS=OFF \
-DLLAMA_BUILD_TOOLS=OFF \
-DLLAMA_BUILD_EXAMPLES=OFF \
-DLLAMA_BUILD_APP=OFF \
-DLLAMA_BUILD_IS_DEV=OFF \
-DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j $(nproc)
cmake --install build --prefix "$PREFIX" --config Release
export LLAMA_CONFIG="$PREFIX"/lib/cmake/llama/llama-config.cmake
build_commit=$(git rev-parse --short HEAD | xargs)
build_number=$(git rev-list --count HEAD | xargs)
major=$(grep -oE "set\(LLAMA_VERSION_MAJOR[[:space:]]+[0-9]+" CMakeLists.txt | grep -oE "[0-9]+$")
minor=$(grep -oE "set\(LLAMA_VERSION_MINOR[[:space:]]+[0-9]+" CMakeLists.txt | grep -oE "[0-9]+$")
patch=$(grep -oE "set\(LLAMA_VERSION_PATCH[[:space:]]+[0-9]+" CMakeLists.txt | grep -oE "[0-9]+$")
build_version="$major.$minor.$patch"
checks=("set\(LLAMA_VERSION[[:space:]]+$build_version\)"
"set\(LLAMA_BUILD_COMMIT[[:space:]]+$build_commit\)"
"set\(LLAMA_BUILD_NUMBER[[:space:]]+$build_number\)")
for check in "${checks[@]}"; do
if ! grep -qE "$check" "$LLAMA_CONFIG"; then
echo "Checking llama-config.cmake version... \"$check\" failed!"
exit 1
fi
done
echo "Checking llama-config.cmake version... success."
cd examples/simple-cmake-pkg
cmake -S . -B build -DCMAKE_PREFIX_PATH="$PREFIX"/lib/cmake
cmake --build build -j $(nproc)
- name: ccache-buckets-save
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
uses: ./.github/actions/ccache-buckets
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CACHE_OUTPUT }}
with:
key: build-cmake-pkg
folder: llama.cpp
evict-old-files: 1d
hf_bucket: ggml-org/cache
save: true
+4 -4
View File
@@ -163,7 +163,7 @@ jobs:
# cmake --build build --config Release -j $(nproc)
debian-13-loongarch64-cpu-cross:
runs-on: ${{ 'ubuntu-24.04-arm' || 'ubuntu-24.04' }}
runs-on: ${{ 'ubuntu-26.04-arm' || 'ubuntu-26.04' }}
container: debian@sha256:653dfb9f86c3782e8369d5f7d29bb8faba1f4bff9025db46e807fa4c22903671
steps:
@@ -218,7 +218,7 @@ jobs:
cmake --build build --config Release -j $(nproc)
debian-13-loongarch64-vulkan-cross:
runs-on: ${{ 'ubuntu-24.04-arm' || 'ubuntu-24.04' }}
runs-on: ${{ 'ubuntu-26.04-arm' || 'ubuntu-26.04' }}
container: debian@sha256:653dfb9f86c3782e8369d5f7d29bb8faba1f4bff9025db46e807fa4c22903671
steps:
@@ -276,8 +276,8 @@ jobs:
cmake --build build --config Release -j $(nproc)
ubuntu-24-riscv64-cpu-spacemit-ime-cross:
runs-on: ubuntu-24.04
ubuntu-26-riscv64-cpu-spacemit-ime-cross:
runs-on: ubuntu-26.04
env:
# Make sure this is in sync with build-cache.yml
+1 -1
View File
@@ -98,7 +98,7 @@ jobs:
hip:
runs-on: ubuntu-22.04
container: rocm/dev-ubuntu-22.04:6.1.2
container: rocm/dev-ubuntu-22.04:6.3.4
steps:
- name: Clone
+4 -4
View File
@@ -40,12 +40,12 @@ jobs:
- cuda: '12.4'
arch: x64
defines: '-DGGML_CUDA_CCCL_VERSION=v3.4.3'
- cuda: '13.4'
- cuda: '13.5'
arch: x64
defines: '-DGGML_CUDA_CCCL_VERSION=v3.4.3'
- cuda: '13.4'
defines: ''
- cuda: '13.5'
arch: arm64
defines: '-DGGML_CUDA_CCCL_VERSION=v3.4.3 -DCMAKE_TOOLCHAIN_FILE=cmake/arm64-windows-msvc-cuda.cmake'
defines: '-DCMAKE_TOOLCHAIN_FILE=cmake/arm64-windows-msvc-cuda.cmake'
steps:
- name: Clone
+22 -36
View File
@@ -20,7 +20,11 @@ on:
paths: [
'.github/workflows/build-ibm.yml',
'ggml/src/ggml-cpu/**',
'ggml/src/ggml-zdnn/**'
'ggml/src/ggml-zdnn/**',
'ggml/src/ggml-quants.c',
'src/llama-quant.cpp',
'gguf-py/gguf/scripts/gguf_convert_endian.py',
'tests/test-quantize-be.sh'
]
cache-mode: none
@@ -39,9 +43,9 @@ env:
LLAMA_ARG_LOG_TIMESTAMPS: 1
jobs:
ubuntu-24-s390x:
name: ubuntu-24-s390x (VXE ${{ matrix.vxe }})
runs-on: ubuntu-24.04-s390x
ubuntu-26-s390x:
name: ubuntu-26-s390x (VXE ${{ matrix.vxe }})
runs-on: ubuntu-26.04-s390x
strategy:
fail-fast: false
@@ -62,12 +66,6 @@ jobs:
libjpeg-dev build-essential libssl-dev \
git-lfs
- name: Toolchain workaround (GCC 14)
run: |
sudo apt-get install -y gcc-14 g++-14
echo "CC=gcc-14" >> "$GITHUB_ENV"
echo "CXX=g++-14" >> "$GITHUB_ENV"
- name: Python Dependencies
id: python_depends
run: |
@@ -102,35 +100,29 @@ jobs:
run: |
cd build
echo "Fetch llama2c big-endian model"
wget https://huggingface.co/ggml-org/models/resolve/main/tinyllamas/stories260K-be.gguf
wget https://huggingface.co/ggml-org-dev/stories260K/resolve/main/stories260K-be.gguf
./bin/llama-completion -m stories260K-be.gguf -p "One day, Lily met a Shoggoth" -n 500 -c 256
- name: Test quantize (s390x)
id: quantize_test_s390x
run: |
./tests/test-quantize-be.sh ./build/bin/llama-quantize ./build/bin/llama-perplexity 2>&1
ubuntu-26-zdnn-s390x:
name: ubuntu-26-zdnn-s390x
runs-on: ubuntu-24.04-s390x
container: ubuntu:26.04 # required to get GCC 15.1 and binutils 2.44
defaults:
run:
shell: bash
runs-on: ubuntu-26.04-s390x
steps:
- name: Build Dependencies
id: build_depends
run: |
apt-get update
apt-get install -y --no-install-recommends \
build-essential cmake git ca-certificates \
libssl-dev libzdnn-dev
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: Toolchain workaround (GCC 15)
- name: Build Dependencies
id: build_depends
run: |
apt-get install -y gcc-15 g++-15
echo "CC=gcc-15" >> "$GITHUB_ENV"
echo "CXX=g++-15" >> "$GITHUB_ENV"
sudo apt-get update
sudo apt-get install -y --no-install-recommends \
build-essential libssl-dev libzdnn-dev
- name: Build with zDNN Backend
id: cmake_build
@@ -145,8 +137,8 @@ jobs:
-DCMAKE_CXX_FLAGS="-march=arch15"
time cmake --build build --config Release -j $(nproc)
ubuntu-24-ppc64le:
runs-on: ubuntu-24.04-ppc64le
ubuntu-26-ppc64le:
runs-on: ubuntu-26.04-ppc64le
steps:
- name: Clone
@@ -162,12 +154,6 @@ jobs:
libjpeg-dev build-essential libssl-dev \
git-lfs
- name: Toolchain workaround (GCC 14)
run: |
sudo apt-get install -y gcc-14 g++-14
echo "CC=gcc-14" >> "$GITHUB_ENV"
echo "CXX=g++-14" >> "$GITHUB_ENV"
- name: Python Dependencies
id: python_depends
run: |
+5
View File
@@ -61,6 +61,11 @@ jobs:
run: |
choco install ninja
- name: Install LLVM
run: |
choco install llvm --version=22.1.8
echo "C:\Program Files\LLVM\bin" >> $env:GITHUB_PATH
- name: Install OpenCL Headers and Libs
id: install_opencl
run: |
+2 -57
View File
@@ -37,64 +37,9 @@ env:
LLAMA_ARG_LOG_PREFIX: 1
LLAMA_ARG_LOG_TIMESTAMPS: 1
# TODO: fix failing tests on OpenVINO backend
CTEST_EXCLUDE: "test-llama-archs|^test-recurrent-state-|test-save-load-state"
CTEST_EXCLUDE: "test-llama-archs|test-llama-context|test-fp8-kv"
jobs:
ubuntu-24-openvino:
runs-on: [self-hosted, Linux, Intel, OpenVINO]
env:
# Sync versions in build-openvino.yml, build-self-hosted.yml, release.yml, build-cache.yml, .devops/openvino.Dockerfile
OPENVINO_VERSION_MAJOR: "2026.4.1"
OPENVINO_VERSION_FULL: "2026.4.1.22982.07f9c262b05"
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: Dependencies
id: depends
run: |
sudo apt-get update
sudo apt-get install -y build-essential libssl-dev libtbb12 cmake ninja-build python3-pip
sudo apt-get install -y ocl-icd-opencl-dev opencl-headers opencl-clhpp-headers intel-opencl-icd
- name: Setup OpenVINO Toolkit
uses: ./.github/actions/linux-setup-openvino
with:
path: ./openvino_toolkit
version_major: ${{ env.OPENVINO_VERSION_MAJOR }}
version_full: ${{ env.OPENVINO_VERSION_FULL }}
- name: Install OpenVINO dependencies
run: |
cd ./openvino_toolkit
chmod +x ./install_dependencies/install_openvino_dependencies.sh
echo "Y" | sudo -E ./install_dependencies/install_openvino_dependencies.sh
- name: Build
id: cmake_build
run: |
source ./openvino_toolkit/setupvars.sh
cmake -B build/ReleaseOV -G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DGGML_OPENVINO=ON
time cmake --build build/ReleaseOV --config Release --parallel
- name: Test (CPU)
id: cmake_test_cpu
run: |
cd ${{ github.workspace }}
ctest --test-dir build/ReleaseOV -L main -E "${{ env.CTEST_EXCLUDE }}" --verbose --timeout 3000
- name: Test (GPU)
id: cmake_test_gpu
run: |
cd ${{ github.workspace }}
export GGML_OPENVINO_DEVICE=GPU
ctest --test-dir build/ReleaseOV -L main -E "${{ env.CTEST_EXCLUDE }}" --verbose --timeout 3000
openvino-windows-2022:
runs-on: windows-2022
@@ -104,7 +49,7 @@ jobs:
contents: read
env:
# Sync versions in build-openvino.yml, build-self-hosted.yml, release.yml, build-cache.yml, .devops/openvino.Dockerfile
# Sync versions in build-openvino.yml, ci-self-hosted-openvino.yml, release.yml, release-publish.yml, build-cache.yml, .devops/openvino.Dockerfile, docs/backend/OPENVINO.md
OPENVINO_VERSION_MAJOR: "2026.4.1"
OPENVINO_VERSION_FULL: "2026.4.1.22982.07f9c262b05"
+29 -25
View File
@@ -43,13 +43,12 @@ jobs:
strategy:
matrix:
include:
# thread and address doesn't run properly on some self hosted machines, so run it on Github instead
- sanitizer: ADDRESS
machine: ubuntu-24.04
machine: ubuntu-26.04
- sanitizer: THREAD
machine: ubuntu-24.04
machine: ubuntu-26.04
- sanitizer: UNDEFINED
machine: [self-hosted, X64, Linux]
machine: ubuntu-26.04
runs-on: ${{ matrix.machine }}
@@ -58,31 +57,24 @@ jobs:
id: checkout
uses: actions/checkout@v6
# - name: ccache
# uses: ggml-org/ccache-action@v1.2.24
# if: ${{ matrix.sanitizer != 'UNDEFINED' }}
# with:
# key: ctest-${{ matrix.sanitizer }}-ubuntu-24.04
# variant: ccache
# evict-old-files: 1d
# save: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
- name: ccache
uses: ggml-org/ccache-action@v1.2.24
with:
restore: false
save: false
# with UNDEFINED sanitizer, we have to build in Debug to avoid GCC 13 false-positive warnings
- name: Build (undefined)
id: cmake_build_undefined
if: ${{ matrix.sanitizer == 'UNDEFINED' }}
run: |
cmake -B build \
-DCMAKE_BUILD_TYPE=Debug \
-DLLAMA_FATAL_WARNINGS=ON \
-DLLAMA_SANITIZE_${{ matrix.sanitizer }}=ON \
-DGGML_SANITIZE_${{ matrix.sanitizer }}=ON
cmake --build build --config Debug -j $(nproc)
- name: ccache-buckets-restore
uses: ./.github/actions/ccache-buckets
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CI }}
with:
key: ctest-sanitizer-${{ matrix.sanitizer }}
folder: llama.cpp
hf_bucket: ggml-org/cache
- name: Build
id: cmake_build
if: ${{ matrix.sanitizer == 'ADDRESS' }}
if: ${{ matrix.sanitizer == 'ADDRESS' || matrix.sanitizer == 'UNDEFINED' }}
run: |
cmake -B build \
-DCMAKE_BUILD_TYPE=RelWithDebInfo \
@@ -103,6 +95,18 @@ jobs:
cmake --build build --config RelWithDebInfo -j $(nproc)
- name: ccache-buckets-save
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
uses: ./.github/actions/ccache-buckets
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CACHE_OUTPUT }}
with:
key: ctest-sanitizer-${{ matrix.sanitizer }}
folder: llama.cpp
evict-old-files: 1d
hf_bucket: ggml-org/cache
save: true
- name: Test
id: cmake_test
run: |
+29 -14
View File
@@ -38,7 +38,7 @@ env:
LLAMA_ARG_LOG_TIMESTAMPS: 1
jobs:
ubuntu-24-sycl:
ubuntu-26-sycl:
strategy:
matrix:
build: [fp32, fp16]
@@ -48,12 +48,12 @@ jobs:
- build: fp16
fp16: ON
runs-on: ubuntu-24.04
runs-on: ubuntu-26.04
env:
ONEAPI_ROOT: /opt/intel/oneapi/
ONEAPI_INSTALLER_VERSION: "2026.1"
LEVEL_ZERO_VERSION: "1.33.1"
ONEAPI_INSTALLER_VERSION: "2026.2"
LEVEL_ZERO_VERSION: "1.34.0"
LEVEL_ZERO_UBUNTU_VERSION: "u24.04"
continue-on-error: true
@@ -67,8 +67,15 @@ jobs:
shell: bash
run: |
cd /tmp
wget https://registrationcenter-download.intel.com/akdlm/IRC_NAS/5996e26b-f48a-42b1-8db0-b002ad0bd8d7/intel-oneapi-toolkit-2026.1.1.33_offline.sh -O intel-oneapi-toolkit_offline.sh
sudo bash intel-oneapi-toolkit_offline.sh -s -a --silent --eula accept
echo "Download oneAPI Deep Learning Essentials installation package"
wget https://registrationcenter-download.intel.com/akdlm/IRC_NAS/aef232d2-aac5-4cd6-bceb-911019bbb6a8/intel-deep-learning-essentials-2026.2.0.640_offline.sh -O intel-deep-learning-essentials_offline.sh
echo "Install oneAPI Deep Learning Essentials components"
sudo bash intel-deep-learning-essentials_offline.sh -s -a --silent --eula accept
echo "Download oneDNN installation package"
wget https://registrationcenter-download.intel.com/akdlm/IRC_NAS/46452a4a-7b7f-49f2-be4c-602a900180bb/intel-onednn-2026.1.0.625_offline.sh -O intel-onednn_offline.sh
echo "Install oneDNN components"
sudo bash intel-onednn_offline.sh -s -a --silent --eula accept
- name: Install Level Zero SDK
shell: bash
@@ -79,6 +86,12 @@ jobs:
wget -q "https://github.com/oneapi-src/level-zero/releases/download/v${LEVEL_ZERO_VERSION}/libze-dev_${LEVEL_ZERO_VERSION}%2B${LEVEL_ZERO_UBUNTU_VERSION}_amd64.deb" -O libze-dev.deb
sudo apt-get install -y ./libze1.deb ./libze-dev.deb
- name: Dependencies
id: depends
run: |
sudo apt-get update
sudo apt-get install -y libssl-dev
- name: ccache
uses: ggml-org/ccache-action@v1.2.24
with:
@@ -90,7 +103,7 @@ jobs:
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CI }}
with:
key: sycl-ubuntu-24-${{ matrix.build }}
key: sycl-ubuntu-26-${{ matrix.build }}
folder: llama.cpp
hf_bucket: ggml-org/cache
@@ -104,7 +117,6 @@ jobs:
-DGGML_SYCL=ON \
-DCMAKE_C_COMPILER=icx \
-DCMAKE_CXX_COMPILER=icpx \
-DLLAMA_OPENSSL=OFF \
-DGGML_NATIVE=OFF \
-DGGML_SYCL_F16=${{ matrix.fp16 }} \
-DGGML_SYCL_SUPPORT_LEVEL_ZERO_API=ON \
@@ -119,7 +131,7 @@ jobs:
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CACHE_OUTPUT }}
with:
key: sycl-ubuntu-24-${{ matrix.build }}
key: sycl-ubuntu-26-${{ matrix.build }}
folder: llama.cpp
evict-old-files: 1d
hf_bucket: ggml-org/cache
@@ -138,11 +150,13 @@ jobs:
shell: bash
env:
WINDOWS_BASEKIT_URL: https://registrationcenter-download.intel.com/akdlm/IRC_NAS/0cb67a0d-67f6-410b-868b-f4a0a17ff0cf/intel-oneapi-toolkit-2026.1.1.32_offline.exe
WINDOWS_DPCPP_MKL: intel.oneapi.win.cpp-dpcpp-common:intel.oneapi.win.mkl.devel:intel.oneapi.win.dnnl:intel.oneapi.win.tbb.devel
LEVEL_ZERO_SDK_URL: https://github.com/oneapi-src/level-zero/releases/download/v1.33.1/level-zero-win-sdk-1.33.1.zip
WINDOWS_DLE_URL: https://registrationcenter-download.intel.com/akdlm/IRC_NAS/b177a006-5cd9-4cb2-807f-8199cc73d53c/intel-deep-learning-essentials-2026.2.0.638_offline.exe
WINDOWS_ONEDNN_URL: https://registrationcenter-download.intel.com/akdlm/IRC_NAS/8f905666-c85c-43fb-9bd0-2618c7ecd33b/intel-onednn-2026.1.0.622_offline.exe
WINDOWS_DPCPP_MKL: intel.oneapi.win.cpp-dpcpp-common:intel.oneapi.win.mkl.devel:intel.oneapi.win.tbb.devel
WINDOWS_DNNL: intel.oneapi.win.dnnl
LEVEL_ZERO_SDK_URL: https://github.com/oneapi-src/level-zero/releases/download/v1.34.0/level-zero-win-sdk-1.34.0.zip
ONEAPI_ROOT: "C:/Program Files (x86)/Intel/oneAPI"
ONEAPI_INSTALLER_VERSION: "2026.1"
ONEAPI_INSTALLER_VERSION: "2026.2"
steps:
- name: Clone
id: checkout
@@ -151,7 +165,8 @@ jobs:
- name: Download & Install oneAPI
shell: bash
run: |
scripts/install-oneapi.bat $WINDOWS_BASEKIT_URL $WINDOWS_DPCPP_MKL
scripts/install-oneapi.bat $WINDOWS_DLE_URL $WINDOWS_DPCPP_MKL
scripts/install-oneapi.bat $WINDOWS_ONEDNN_URL $WINDOWS_DNNL
- name: Install Level Zero SDK
shell: pwsh
+2 -2
View File
@@ -31,8 +31,8 @@ concurrency:
cancel-in-progress: true
jobs:
ubuntu-24-virtgpu:
runs-on: ${{ 'ubuntu-24.04-arm' || 'ubuntu-24.04' }}
ubuntu-26-virtgpu:
runs-on: ${{ 'ubuntu-26.04-arm' || 'ubuntu-26.04' }}
steps:
- name: Clone
+12 -9
View File
@@ -42,7 +42,7 @@ env:
jobs:
ubuntu-arm64:
runs-on: ubuntu-24.04-arm
runs-on: ubuntu-26.04-arm
steps:
- name: Clone
@@ -53,9 +53,7 @@ jobs:
id: depends
run: |
sudo apt-get update
sudo apt-get install -y gcc-14 g++-14 build-essential glslc libvulkan-dev spirv-headers libssl-dev ninja-build
echo "CC=gcc-14" >> "$GITHUB_ENV"
echo "CXX=g++-14" >> "$GITHUB_ENV"
sudo apt-get install -y build-essential glslc libvulkan-dev spirv-headers libssl-dev ninja-build
- name: ccache
uses: ggml-org/ccache-action@v1.2.24
@@ -69,7 +67,7 @@ jobs:
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CI }}
with:
key: vulkan-ubuntu-24.04-arm
key: vulkan-ubuntu-26.04-arm
folder: llama.cpp
hf_bucket: ggml-org/cache
@@ -92,14 +90,14 @@ jobs:
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CACHE_OUTPUT }}
with:
key: vulkan-ubuntu-24.04-arm
key: vulkan-ubuntu-26.04-arm
folder: llama.cpp
evict-old-files: 1d
hf_bucket: ggml-org/cache
save: true
ubuntu-llvmpipe:
runs-on: ubuntu-24.04
runs-on: ubuntu-26.04
steps:
- name: Clone
@@ -137,7 +135,7 @@ jobs:
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CI }}
with:
key: vulkan-ubuntu-24.04-llvmpipe
key: vulkan-ubuntu-26.04-llvmpipe
folder: llama.cpp
hf_bucket: ggml-org/cache
@@ -156,7 +154,7 @@ jobs:
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CACHE_OUTPUT }}
with:
key: vulkan-ubuntu-24.04-llvmpipe
key: vulkan-ubuntu-26.04-llvmpipe
folder: llama.cpp
evict-old-files: 1d
hf_bucket: ggml-org/cache
@@ -215,6 +213,11 @@ jobs:
run: |
choco install ninja
- name: Install LLVM
run: |
choco install llvm --version=22.1.8
echo "C:\Program Files\LLVM\bin" >> $env:GITHUB_PATH
- name: Build
id: cmake_build
run: |
+142
View File
@@ -0,0 +1,142 @@
name: CI (wasi)
on:
workflow_dispatch: # allows manual triggering
push:
branches:
- master
paths: [
'.github/workflows/build-wasi.yml',
'**/CMakeLists.txt',
'**/.cmake',
'**/*.h',
'**/*.hpp',
'**/*.c',
'**/*.cpp',
'**/*.wgsl',
'**/*.tmpl',
'ggml/src/ggml-webgpu/wgsl-shaders/embed_wgsl.py'
]
pull_request:
types: [opened, synchronize, reopened]
paths: [
'.github/workflows/build-wasi.yml',
'**/CMakeLists.txt',
'**/.cmake',
'**/*.h',
'**/*.hpp',
'**/*.c',
'**/*.cpp',
'**/*.wgsl',
'**/*.tmpl',
'ggml/src/ggml-webgpu/wgsl-shaders/embed_wgsl.py'
]
cache-mode: none
permissions:
contents: read
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}
cancel-in-progress: true
env:
WASI_SDK_VERSION: 34
DAWN_VERSION: v20260908.214631
DAWN_COMMIT: 94c3c9cc0d5fb2e85aebb370fa8d37b71aa34655
WASI_WEBGPU_HEADERS_COMMIT: 917765edd7cb8f8cf95bda1217514c78f71b582b
jobs:
ubuntu-webgpu:
runs-on: ubuntu-26.04-arm
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: ccache
uses: ggml-org/ccache-action@v1.2.24
with:
restore: false
save: false
- name: Install wasi-sdk
uses: bytecodealliance/setup-wasi-sdk-action@77d09f3df1e8da344d8403cf8631af8977938b4c # v1.0.0
with:
version: ${{ env.WASI_SDK_VERSION }}
add-to-path: false
- name: wasi:webgpu Dependency
run: |
mkdir -p wasi-webgpu/dawn-headers
curl -fL "https://github.com/google/dawn/releases/download/${DAWN_VERSION}/dawn-headers-${DAWN_COMMIT}.tar.gz" |
tar -xz -C wasi-webgpu/dawn-headers --strip-components=1
git init -q wasi-webgpu/wasi-webgpu-headers
cd wasi-webgpu/wasi-webgpu-headers
git fetch -q --depth 1 https://github.com/wasi-gfx/wasi-webgpu-headers.git ${WASI_WEBGPU_HEADERS_COMMIT}
git checkout -q FETCH_HEAD
git submodule update -q --init --depth 1
cd ../..
# ggml-webgpu does find_package(Dawn) and links dawn::webgpu_dawn. There is no Dawn on WASI,
# so this target instead compiles wasi-webgpu-headers (webgpu.h over wasi:webgpu imports)
# and exposes Dawn's webgpu_cpp.h on top of it.
cat > wasi-webgpu/DawnConfig.cmake << 'EOF'
set(_wasi_webgpu "${CMAKE_CURRENT_LIST_DIR}/wasi-webgpu-headers")
add_library(dawn::webgpu_dawn INTERFACE IMPORTED)
set_target_properties(dawn::webgpu_dawn PROPERTIES
INTERFACE_SOURCES "${_wasi_webgpu}/webgpu.c;${_wasi_webgpu}/async_futures.c;${_wasi_webgpu}/imports.c;${_wasi_webgpu}/imports_component_type.o"
INTERFACE_INCLUDE_DIRECTORIES "${CMAKE_CURRENT_LIST_DIR}/dawn-headers/include/webgpu_upstream;${_wasi_webgpu}")
EOF
- name: ccache-buckets-restore
uses: ./.github/actions/ccache-buckets
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CI }}
with:
key: webgpu-ubuntu-26.04-arm-wasi
folder: llama.cpp
hf_bucket: ggml-org/cache
# llama-common does not build for WASI, so the executables are linked by hand
- name: Build
id: cmake_build
run: |
WASI_FLAGS="-msimd128 -D_WASI_EMULATED_SIGNAL -fwasm-exceptions -mllvm -wasm-use-legacy-eh=false"
cmake -B build-wasi \
-G "Ninja" \
-DCMAKE_TOOLCHAIN_FILE=${WASI_SDK_PATH}/share/cmake/wasi-sdk-p3.cmake \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_C_FLAGS="${WASI_FLAGS}" \
-DCMAKE_CXX_FLAGS="${WASI_FLAGS}" \
-DCMAKE_HAVE_LIBC_PTHREAD=1 \
-DBUILD_SHARED_LIBS=OFF \
-DGGML_NATIVE=OFF \
-DGGML_OPENMP=OFF \
-DGGML_WEBGPU=ON \
-DDawn_DIR=$PWD/wasi-webgpu \
-DLLAMA_BUILD_COMMON=OFF
time cmake --build build-wasi --config Release --target llama -j $(nproc)
LIBS="build-wasi/ggml/src/libggml.a build-wasi/ggml/src/ggml-webgpu/libggml-webgpu.a build-wasi/ggml/src/libggml-cpu.a build-wasi/ggml/src/libggml-base.a"
LDFLAGS="-lunwind -ldl -Wl,-z,stack-size=8388608"
CXX="${WASI_SDK_PATH}/bin/clang++ --target=wasm32-wasip3 -O3 -std=c++17 ${WASI_FLAGS} -I include -I ggml/include"
$CXX examples/simple/simple.cpp build-wasi/src/libllama.a $LIBS $LDFLAGS -o build-wasi/llama-simple.wasm
$CXX tests/test-backend-ops.cpp $LIBS $LDFLAGS -o build-wasi/test-backend-ops.wasm
- name: ccache-buckets-save
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
uses: ./.github/actions/ccache-buckets
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CACHE_OUTPUT }}
with:
key: webgpu-ubuntu-26.04-arm-wasi
folder: llama.cpp
evict-old-files: 1d
hf_bucket: ggml-org/cache
save: true
+3 -3
View File
@@ -50,7 +50,7 @@ env:
jobs:
ubuntu-webgpu:
runs-on: ubuntu-24.04-arm
runs-on: ubuntu-26.04-arm
steps:
- name: Clone
@@ -84,7 +84,7 @@ jobs:
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CI }}
with:
key: webgpu-ubuntu-24.04-arm-wasm
key: webgpu-ubuntu-26.04-arm-wasm
folder: llama.cpp
hf_bucket: ggml-org/cache
@@ -107,7 +107,7 @@ jobs:
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CACHE_OUTPUT }}
with:
key: webgpu-ubuntu-24.04-arm-wasm
key: webgpu-ubuntu-26.04-arm-wasm
folder: llama.cpp
evict-old-files: 1d
hf_bucket: ggml-org/cache
+3 -143
View File
@@ -7,15 +7,7 @@ on:
- master
paths: [
'.github/workflows/build-webgpu.yml',
'**/CMakeLists.txt',
'**/.cmake',
'**/*.h',
'**/*.hpp',
'**/*.c',
'**/*.cpp',
'**/*.wgsl',
'**/*.tmpl',
'ggml/src/ggml-webgpu/wgsl-shaders/embed_wgsl.py'
'ggml/src/ggml-webgpu/**'
]
pull_request:
@@ -42,7 +34,7 @@ env:
jobs:
format:
runs-on: ubuntu-24.04
runs-on: ubuntu-26.04
steps:
- name: Clone
@@ -53,7 +45,7 @@ jobs:
wget -qO- https://apt.llvm.org/llvm-snapshot.gpg.key |
sudo tee /etc/apt/trusted.gpg.d/apt.llvm.org.asc > /dev/null
sudo add-apt-repository -y \
"deb http://apt.llvm.org/noble/ llvm-toolchain-noble-22 main"
"deb http://apt.llvm.org/resolute/ llvm-toolchain-resolute-22 main"
sudo apt-get update
sudo apt-get install -y clang-format-22
@@ -63,135 +55,3 @@ jobs:
-type f \( -name '*.cpp' -o -name '*.hpp' -o -name '*.h' \) \
-print0 |
xargs -0 clang-format-22 --dry-run --Werror
macos:
runs-on: macos-latest
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: ccache
uses: ggml-org/ccache-action@v1.2.24
with:
restore: false
save: false
- name: Dawn Dependency
id: dawn-depends
run: |
DAWN_VERSION="v20260908.214631"
DAWN_OWNER="google"
DAWN_REPO="dawn"
DAWN_ASSET_NAME="Dawn-94c3c9cc0d5fb2e85aebb370fa8d37b71aa34655-macos-latest-Release"
echo "Fetching release asset from https://github.com/google/dawn/releases/download/${DAWN_VERSION}/${DAWN_ASSET_NAME}.tar.gz"
curl -L -o artifact.tar.gz \
"https://github.com/google/dawn/releases/download/${DAWN_VERSION}/${DAWN_ASSET_NAME}.tar.gz"
mkdir dawn
tar -xvf artifact.tar.gz -C dawn --strip-components=1
- name: ccache-buckets-restore
uses: ./.github/actions/ccache-buckets
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CI }}
with:
key: webgpu-macos-latest
folder: llama.cpp
hf_bucket: ggml-org/cache
- name: Build
id: cmake_build
run: |
export CMAKE_PREFIX_PATH=dawn
cmake -B build -G "Ninja" -DCMAKE_BUILD_TYPE=Release -DGGML_WEBGPU=ON -DGGML_METAL=OFF -DGGML_BLAS=OFF
time cmake --build build --config Release -j $(sysctl -n hw.logicalcpu)
- name: ccache-buckets-save
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
uses: ./.github/actions/ccache-buckets
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CACHE_OUTPUT }}
with:
key: webgpu-macos-latest
folder: llama.cpp
evict-old-files: 1d
hf_bucket: ggml-org/cache
save: true
- name: Test
id: cmake_test
run: |
cd build
ctest -L main --verbose --timeout 900
ubuntu:
runs-on: ubuntu-24.04
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: ccache
uses: ggml-org/ccache-action@v1.2.24
with:
restore: false
save: false
- name: Dependencies
id: depends
run: |
sudo add-apt-repository -y ppa:kisak/kisak-mesa
sudo apt-get update -y
sudo apt-get install -y build-essential mesa-vulkan-drivers \
libxcb-xinput0 libxcb-xinerama0 libxcb-cursor-dev libssl-dev
- name: Dawn Dependency
id: dawn-depends
run: |
sudo apt-get install -y libxrandr-dev libxinerama-dev libxcursor-dev mesa-common-dev libx11-xcb-dev libxi-dev
DAWN_VERSION="v20260908.214631"
DAWN_OWNER="google"
DAWN_REPO="dawn"
DAWN_ASSET_NAME="Dawn-94c3c9cc0d5fb2e85aebb370fa8d37b71aa34655-ubuntu-latest-Release"
echo "Fetching release asset from https://github.com/google/dawn/releases/download/${DAWN_VERSION}/${DAWN_ASSET_NAME}.tar.gz"
curl -L -o artifact.tar.gz \
"https://github.com/google/dawn/releases/download/${DAWN_VERSION}/${DAWN_ASSET_NAME}.tar.gz"
mkdir dawn
tar -xvf artifact.tar.gz -C dawn --strip-components=1
- name: ccache-buckets-restore
uses: ./.github/actions/ccache-buckets
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CI }}
with:
key: webgpu-ubuntu-24.04
folder: llama.cpp
hf_bucket: ggml-org/cache
- name: Build
id: cmake_build
run: |
export Dawn_DIR=dawn/lib64/cmake/Dawn
cmake -B build \
-DGGML_WEBGPU=ON
time cmake --build build --config Release -j $(nproc)
- name: ccache-buckets-save
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
uses: ./.github/actions/ccache-buckets
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CACHE_OUTPUT }}
with:
key: webgpu-ubuntu-24.04
folder: llama.cpp
evict-old-files: 1d
hf_bucket: ggml-org/cache
save: true
- name: Test
id: cmake_test
run: |
cd build
ctest -L main --verbose --timeout 900
+39 -2
View File
@@ -46,17 +46,54 @@ env:
jobs:
cpu-x64-high-perf:
runs-on: [self-hosted, Linux, X64]
runs-on: hf-jobs-cpu-upgrade:ubuntu26_04
# note: this runner has 8 CPUs, but std::thread::hardware_concurrency reports the cores of the
# host (https://github.com/ggml-org/llama.cpp/pull/31439#issuecomment-6683386059)
env:
TEST_ARG_HARDWARE_CONCURRENCY: 8
TEST_ARG_PARALLEL_WORKERS: 4
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: Dependencies
id: depends
run: |
sudo apt update
sudo apt install -y build-essential cmake libssl-dev time unzip wget python3 python3-venv python3-dev
- name: ccache
uses: ggml-org/ccache-action@v1.2.24
with:
restore: false
save: false
- name: ccache-buckets-restore
uses: ./.github/actions/ccache-buckets
with:
key: self-hosted-cpu-x64-high-perf
folder: llama.cpp
hf_bucket: ggml-org/cache
- name: Test
id: ggml-ci
run: |
LLAMA_ARG_THREADS=$(nproc) GG_BUILD_HIGH_PERF=1 GG_BUILD_EXTRA_TESTS_0=1 bash ./ci/run.sh ~/results/llama.cpp ~/mnt/llama.cpp
LLAMA_ARG_THREADS=$(nproc) GG_BUILD_HIGH_PERF=1 GG_BUILD_EXTRA_TESTS_0=1 bash ./ci/run.sh ~/results/llama.cpp/$RUNNER_NAME ~/mnt/llama.cpp
- name: ccache-buckets-save
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
uses: ./.github/actions/ccache-buckets
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CACHE_OUTPUT }}
with:
key: self-hosted-cpu-x64-high-perf
folder: llama.cpp
evict-old-files: 1d
hf_bucket: ggml-org/cache
save: true
cpu-arm64-high-perf-graviton4:
runs-on: ah-ubuntu_24_04-c8g_8x
+77 -3
View File
@@ -93,13 +93,72 @@ jobs:
save: true
gpu-rocm:
runs-on: [self-hosted, Linux, AMD]
strategy:
fail-fast: false
matrix:
include:
# Strix Halo (iGPU, RDNA3.5)
- gpu: gfx1151
runner: [self-hosted, Linux, strix-halo, devlab-dispatch]
# async-execution correctness workaround, gfx1151 only (see Test step)
hip_launch_blocking: "1"
# Radeon AI PRO R9700 (dGPU, RDNA4)
- gpu: gfx1201
runner: [self-hosted, Linux, radeon-r9700, devlab-dispatch]
hip_launch_blocking: "0"
# HIP-graph capture fails on gfx1201 (hipMemset -> hipErrorStreamCaptureImplicit)
disable_hip_graphs: true
name: gpu-rocm (${{ matrix.gpu }})
runs-on: ${{ matrix.runner }}
timeout-minutes: 120
env:
ROCM_VERSION: "10.0.0"
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: Verify GPU device
run: |
ls /dev/kfd /dev/dri/renderD* || { echo "ERROR: no amdgpu compute device"; exit 1; }
# same TheRock wheels as the ubuntu-24-rocm release build, so the runner does not need a ROCm install
- name: Setup TheRock with Wheels
run: |
python3 -m venv "$RUNNER_TEMP/rocm"
"$RUNNER_TEMP/rocm/bin/python" -m pip install --index-url https://stable.repo.amd.com/rocm/whl-next/ "rocm[libraries,devel,device-${{ matrix.gpu }}]==${ROCM_VERSION}"
ROCM_PATH=$("$RUNNER_TEMP/rocm/bin/rocm-sdk" path --root)
{
echo "ROCM_PATH=$ROCM_PATH"
echo "HIP_PATH=$ROCM_PATH"
echo "CMAKE_PREFIX_PATH=$("$RUNNER_TEMP/rocm/bin/rocm-sdk" path --cmake)"
echo "LD_LIBRARY_PATH=$ROCM_PATH/lib:${LD_LIBRARY_PATH:-}"
} >> "$GITHUB_ENV"
"$RUNNER_TEMP/rocm/bin/rocm-sdk" path --bin >> "$GITHUB_PATH"
- name: ccache
uses: ggml-org/ccache-action@v1.2.24
with:
restore: false
save: false
# ROCm is installed again on each run, so hash the compiler content, not its mtime
- name: Tune ccache for reinstalled ROCm toolchain
run: |
ccache --set-config=compiler_check=content
ccache --set-config=sloppiness=time_macros,include_file_mtime,include_file_ctime
ccache --set-config=depend_mode=true
- name: ccache-buckets-restore
uses: ./.github/actions/ccache-buckets
with:
key: self-hosted-gpu-rocm-${{ matrix.gpu }}
folder: llama.cpp
hf_bucket: ggml-org/cache
- name: Test
id: ggml-ci
# HIP_LAUNCH_BLOCKING=1: workaround for an async-execution correctness
@@ -107,10 +166,25 @@ jobs:
# incorrect output (perplexity ~88 vs ~9.4). Serializing kernel launches
# restores correctness. Remove once the underlying ROCm/HIP issue is fixed.
env:
HIP_LAUNCH_BLOCKING: "1"
HIP_LAUNCH_BLOCKING: ${{ matrix.hip_launch_blocking }}
run: |
rocminfo
GG_BUILD_ROCM=1 GG_BUILD_AMDGPU_TARGETS=gfx1151 bash ./ci/run.sh ~/results/llama.cpp ~/mnt/llama.cpp
# Exported rather than set in env:, because ggml only checks that the
# variable is defined: an empty value would disable graphs on gfx1151 too.
if [ "${{ matrix.disable_hip_graphs }}" = "true" ]; then export GGML_CUDA_DISABLE_GRAPHS=1; fi
GG_BUILD_ROCM=1 GG_BUILD_AMDGPU_TARGETS=${{ matrix.gpu }} bash ./ci/run.sh "$RUNNER_TEMP/results/llama.cpp" "$RUNNER_TEMP/mnt/llama.cpp"
- name: ccache-buckets-save
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
uses: ./.github/actions/ccache-buckets
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CACHE_OUTPUT }}
with:
key: self-hosted-gpu-rocm-${{ matrix.gpu }}
folder: llama.cpp
evict-old-files: 1d
hf_bucket: ggml-org/cache
save: true
# TODO: provision AMD GPU machine
# amd-rocm:
+1 -1
View File
@@ -60,4 +60,4 @@ jobs:
- name: Test
id: ggml-ci
run: |
GG_BUILD_METAL=1 bash ./ci/run.sh ~/results/llama.cpp ~/mnt/llama.cpp
GG_BUILD_METAL=1 bash ./ci/run.sh ~/results/llama.cpp/$RUNNER_NAME ~/mnt/llama.cpp
@@ -21,7 +21,9 @@ on:
paths: [
'.github/workflows/ci-self-hosted-openvino.yml',
'ci/run.sh',
'**/CMakeLists.txt',
'CMakeLists.txt',
'ggml/CMakeLists.txt',
'tests/CMakeLists.txt',
'**/.cmake',
'ggml/src/*',
'ggml/src/ggml-cpu/**',
@@ -50,7 +52,7 @@ jobs:
runs-on: [self-hosted, Linux, Intel, OpenVINO]
env:
# Sync versions in build.yml, build-self-hosted.yml, release.yml, build-cache.yml, .devops/openvino.Dockerfile
# Sync versions in build-openvino.yml, ci-self-hosted-openvino.yml, release.yml, release-publish.yml, build-cache.yml, .devops/openvino.Dockerfile, docs/backend/OPENVINO.md
OPENVINO_VERSION_MAJOR: "2026.4.1"
OPENVINO_VERSION_FULL: "2026.4.1.22982.07f9c262b05"
+3 -3
View File
@@ -79,7 +79,7 @@ jobs:
id: ggml-ci
run: |
vulkaninfo --summary
GG_BUILD_VULKAN=1 GGML_VK_DISABLE_COOPMAT2=1 bash ./ci/run.sh ~/results/llama.cpp ~/mnt/llama.cpp
GG_BUILD_VULKAN=1 GGML_VK_DISABLE_COOPMAT2=1 bash ./ci/run.sh ~/results/llama.cpp/$RUNNER_NAME ~/mnt/llama.cpp
# - name: ccache-buckets-save
# if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
@@ -124,7 +124,7 @@ jobs:
id: ggml-ci
run: |
vulkaninfo --summary
GG_BUILD_VULKAN=1 bash ./ci/run.sh ~/results/llama.cpp ~/mnt/llama.cpp
GG_BUILD_VULKAN=1 bash ./ci/run.sh ~/results/llama.cpp/$RUNNER_NAME ~/mnt/llama.cpp
# - name: ccache-buckets-save
# if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
@@ -150,7 +150,7 @@ jobs:
id: ggml-ci
run: |
vulkaninfo --summary
GG_BUILD_VULKAN=1 bash ./ci/run.sh ~/results/llama.cpp ~/mnt/llama.cpp
GG_BUILD_VULKAN=1 bash ./ci/run.sh ~/results/llama.cpp/$RUNNER_NAME ~/mnt/llama.cpp
gpu-vulkan-intel-linux:
runs-on: [self-hosted, Linux, Intel]
+1 -1
View File
@@ -131,4 +131,4 @@ jobs:
id: ggml-ci
run: |
GG_BUILD_WEBGPU=1 GG_BUILD_WEBGPU_DAWN_PREFIX="$GITHUB_WORKSPACE/dawn" \
bash ./ci/run.sh ~/results/llama.cpp ~/mnt/llama.cpp
bash ./ci/run.sh ~/results/llama.cpp/$RUNNER_NAME ~/mnt/llama.cpp
+1 -1
View File
@@ -16,7 +16,7 @@ cache-mode: none
jobs:
# The job MUST be called `copilot-setup-steps` or it will not be picked up by Copilot.
copilot-setup-steps:
runs-on: ubuntu-24.04
runs-on: ubuntu-26.04
# Set the permissions to the lowest permissions possible needed for your steps.
# Copilot will be given its own token for its operations.
+31 -16
View File
@@ -16,6 +16,10 @@ on:
description: "Skip the s390x build target (useful for fast test runs that do not need the IBM Z runner)"
type: boolean
default: false
skip_ref_check:
description: "Skip the requirement to run from a tag (allows branch refs)"
type: boolean
default: false
schedule:
# Rebuild daily rather than on every push because it is expensive
- cron: '12 4 * * *'
@@ -38,6 +42,14 @@ jobs:
source_tag: ${{ steps.srctag.outputs.name }}
steps:
- name: Check manual run ref
if: ${{ github.event_name == 'workflow_dispatch' && !inputs.skip_ref_check }}
run: |
if [[ "$GITHUB_REF_TYPE" != "tag" ]]; then
echo "::error::Manual runs of this workflow must be started from a tag"
exit 1
fi
- name: Clone
id: checkout
uses: actions/checkout@v6
@@ -70,7 +82,7 @@ jobs:
prepare_matrices:
name: Prepare Docker matrices
runs-on: ubuntu-24.04
runs-on: ubuntu-26.04
outputs:
build_matrix: ${{ steps.matrices.outputs.build_matrix }}
merge_matrix: ${{ steps.matrices.outputs.merge_matrix }}
@@ -87,19 +99,19 @@ jobs:
# Keep all build targets in one place and derive merge targets from it.
cat > build-matrix.json <<'JSON'
[
{ "tag": "cpu", "dockerfile": ".devops/cpu.Dockerfile", "platforms": "linux/amd64", "full": true, "light": true, "server": true, "free_disk_space": false, "runs_on": "ubuntu-24.04" },
{ "tag": "cpu", "dockerfile": ".devops/cpu.Dockerfile", "platforms": "linux/arm64", "full": true, "light": true, "server": true, "free_disk_space": false, "runs_on": "ubuntu-24.04-arm" },
{ "tag": "cpu", "dockerfile": ".devops/s390x.Dockerfile", "platforms": "linux/s390x", "full": true, "light": true, "server": true, "free_disk_space": false, "runs_on": "ubuntu-24.04-s390x", "prebuilt_ui": true },
{ "tag": "cuda cuda12", "dockerfile": ".devops/cuda.Dockerfile", "cuda_version": "12.8.1", "platforms": "linux/amd64", "full": true, "light": true, "server": true, "free_disk_space": true, "runs_on": "ubuntu-24.04" },
{ "tag": "cuda cuda12", "dockerfile": ".devops/cuda.Dockerfile", "cuda_version": "12.8.1", "platforms": "linux/arm64", "full": true, "light": true, "server": true, "free_disk_space": true, "runs_on": "ubuntu-24.04-arm" },
{ "tag": "cuda13", "dockerfile": ".devops/cuda.Dockerfile", "cuda_version": "13.4.1", "platforms": "linux/amd64", "full": true, "light": true, "server": true, "free_disk_space": true, "runs_on": "ubuntu-24.04" },
{ "tag": "cuda13", "dockerfile": ".devops/cuda.Dockerfile", "cuda_version": "13.4.1", "platforms": "linux/arm64", "full": true, "light": true, "server": true, "free_disk_space": true, "runs_on": "ubuntu-24.04-arm" },
{ "tag": "musa", "dockerfile": ".devops/musa.Dockerfile", "platforms": "linux/amd64", "full": true, "light": true, "server": true, "free_disk_space": true, "runs_on": "ubuntu-24.04" },
{ "tag": "intel", "dockerfile": ".devops/intel.Dockerfile", "platforms": "linux/amd64", "full": true, "light": true, "server": true, "free_disk_space": true, "runs_on": "ubuntu-24.04" },
{ "tag": "vulkan", "dockerfile": ".devops/vulkan.Dockerfile", "platforms": "linux/amd64", "full": true, "light": true, "server": true, "free_disk_space": false, "runs_on": "ubuntu-24.04" },
{ "tag": "vulkan", "dockerfile": ".devops/vulkan.Dockerfile", "platforms": "linux/arm64", "full": true, "light": true, "server": true, "free_disk_space": false, "runs_on": "ubuntu-24.04-arm" },
{ "tag": "rocm", "dockerfile": ".devops/rocm.Dockerfile", "platforms": "linux/amd64", "full": true, "light": true, "server": true, "free_disk_space": true, "runs_on": "ubuntu-24.04" },
{ "tag": "openvino", "dockerfile": ".devops/openvino.Dockerfile", "platforms": "linux/amd64", "full": true, "light": true, "server": true, "free_disk_space": false, "runs_on": "ubuntu-24.04" }
{ "tag": "cpu", "dockerfile": ".devops/cpu.Dockerfile", "platforms": "linux/amd64", "full": true, "light": true, "server": true, "free_disk_space": false, "runs_on": "ubuntu-26.04" },
{ "tag": "cpu", "dockerfile": ".devops/cpu.Dockerfile", "platforms": "linux/arm64", "full": true, "light": true, "server": true, "free_disk_space": false, "runs_on": "ubuntu-26.04-arm" },
{ "tag": "cpu", "dockerfile": ".devops/s390x.Dockerfile", "platforms": "linux/s390x", "full": true, "light": true, "server": true, "free_disk_space": false, "runs_on": "ubuntu-26.04-s390x", "prebuilt_ui": true },
{ "tag": "cuda cuda12", "dockerfile": ".devops/cuda.Dockerfile", "cuda_version": "12.8.1", "platforms": "linux/amd64", "full": true, "light": true, "server": true, "free_disk_space": true, "runs_on": "ubuntu-26.04" },
{ "tag": "cuda cuda12", "dockerfile": ".devops/cuda.Dockerfile", "cuda_version": "12.8.1", "platforms": "linux/arm64", "full": true, "light": true, "server": true, "free_disk_space": true, "runs_on": "ubuntu-26.04-arm" },
{ "tag": "cuda13", "dockerfile": ".devops/cuda.Dockerfile", "cuda_version": "13.5.0", "platforms": "linux/amd64", "full": true, "light": true, "server": true, "free_disk_space": true, "runs_on": "ubuntu-26.04" },
{ "tag": "cuda13", "dockerfile": ".devops/cuda.Dockerfile", "cuda_version": "13.5.0", "platforms": "linux/arm64", "full": true, "light": true, "server": true, "free_disk_space": true, "runs_on": "ubuntu-26.04-arm" },
{ "tag": "musa", "dockerfile": ".devops/musa.Dockerfile", "platforms": "linux/amd64", "full": true, "light": true, "server": true, "free_disk_space": true, "runs_on": "ubuntu-26.04" },
{ "tag": "intel", "dockerfile": ".devops/intel.Dockerfile", "platforms": "linux/amd64", "full": true, "light": true, "server": true, "free_disk_space": true, "runs_on": "ubuntu-26.04" },
{ "tag": "vulkan", "dockerfile": ".devops/vulkan.Dockerfile", "platforms": "linux/amd64", "full": true, "light": true, "server": true, "free_disk_space": false, "runs_on": "ubuntu-26.04" },
{ "tag": "vulkan", "dockerfile": ".devops/vulkan.Dockerfile", "platforms": "linux/arm64", "full": true, "light": true, "server": true, "free_disk_space": false, "runs_on": "ubuntu-26.04-arm" },
{ "tag": "rocm", "dockerfile": ".devops/rocm.Dockerfile", "platforms": "linux/amd64", "full": true, "light": true, "server": true, "free_disk_space": true, "runs_on": "ubuntu-26.04" },
{ "tag": "openvino", "dockerfile": ".devops/openvino.Dockerfile", "platforms": "linux/amd64", "full": true, "light": true, "server": true, "free_disk_space": false, "runs_on": "ubuntu-26.04" }
]
JSON
@@ -108,6 +120,9 @@ jobs:
mv build-matrix.json.tmp build-matrix.json
fi
jq 'map(select(.tag != "musa"))' build-matrix.json > build-matrix.json.tmp
mv build-matrix.json.tmp build-matrix.json
BUILD_MATRIX="$(jq -c . build-matrix.json)"
MERGE_MATRIX="$(jq -c '
reduce .[] as $entry ({}; .[$entry.tag] |= (
@@ -170,7 +185,7 @@ jobs:
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
name: llama-ui.zip
path: tools/ui/dist
path: ui/apps/chat/dist
# - name: Set up QEMU
# if: ${{ contains(matrix.config.platforms, 'linux/amd64') }}
@@ -400,7 +415,7 @@ jobs:
merge_arch_tags:
name: Create shared tags from digests
needs: [prepare_matrices, push_to_registry, create_tag]
runs-on: ubuntu-24.04
runs-on: ubuntu-26.04
permissions:
contents: read
packages: write
+1 -1
View File
@@ -24,7 +24,7 @@ permissions:
jobs:
deploy:
runs-on: ubuntu-24.04 # previously ubuntu-latest
runs-on: ubuntu-26.04
steps:
- uses: actions/checkout@v6
+10 -1
View File
@@ -39,7 +39,7 @@ permissions:
jobs:
make-release:
runs-on: ubuntu-24.04 # previously ubuntu-latest
runs-on: ubuntu-26.04
steps:
- name: Checkout
@@ -61,8 +61,17 @@ jobs:
RELEASE_BRANCH: ${{ github.ref_name }}
SKIP_APIABI_CHECK: ${{ github.event.inputs.skip_apiabi_check }}
APIABI_COMPARE_TAG: ${{ github.event.inputs.apiabi_compare_tag }}
APIABI_REPORT_DIR: ${{ runner.temp }}/apiabi-reports
REQUIRE_DOCKER: ${{ github.event.inputs.require_docker }}
- name: Upload API/ABI reports
if: ${{ always() && github.event.inputs.skip_apiabi_check != 'true' }}
uses: actions/upload-artifact@v7
with:
name: apiabi-reports
path: ${{ runner.temp }}/apiabi-reports
if-no-files-found: ignore
- name: Create release tag
if: ${{ github.event.inputs.dry_run == 'false' }}
run: |
+14 -2
View File
@@ -11,7 +11,8 @@ on:
'tests/fusion/**',
'tests/test-fusion.cpp',
'tests/test-llama-archs.cpp',
'src/llama-graph.cpp',
'src/llama-graph*',
'src/llama-kv-cache*',
'src/llama-model*',
'src/models/**'
]
@@ -24,7 +25,8 @@ on:
'tests/fusion/**',
'tests/test-fusion.cpp',
'tests/test-llama-archs.cpp',
'src/llama-graph.cpp',
'src/llama-graph*',
'src/llama-kv-cache*',
'src/llama-model*',
'src/models/**'
]
@@ -145,6 +147,11 @@ jobs:
rm -rf build-ci-models && mkdir -p build-ci-models
./build/bin/test-llama-archs -o build-ci-models
- name: Test KV shift
id: test_kv_shift
run: |
./build/bin/test-llama-archs --kv-shift
- name: Test fusion
id: test_fusion
run: |
@@ -293,6 +300,11 @@ jobs:
# run: |
# ./build/bin/test-fusion --models build-ci-models --device Vulkan0 --check tests/fusion/Vulkan.csv
- name: Test KV shift
id: test_kv_shift
run: |
./build/bin/test-llama-archs --kv-shift
- name: Test archs
id: test_archs
run: |
+37
View File
@@ -0,0 +1,37 @@
name: "PR video demo label"
on:
pull_request_target:
types: [opened, edited, reopened, synchronize]
cache-mode: none
permissions:
pull-requests: write
issues: write
jobs:
video-demo-label:
runs-on: ubuntu-slim
steps:
- name: Add or remove the video demo label
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
PR_URL: ${{ github.event.pull_request.html_url }}
PR_NUMBER: ${{ github.event.pull_request.number }}
# passed through env, so the body is only matched, never run, and is safe to read from a fork
BODY: ${{ github.event.pull_request.body }}
HAS_LABEL: ${{ contains(github.event.pull_request.labels.*.name, 'video demo') }}
run: |
# sections written by demo:pr (see ui/apps/chat/demo/cli.ts): the managed comment
# markers, or a preview linking to a demo video uploaded for this PR
has_demo=false
if { grep -qF '<!-- demo:start -->' <<< "$BODY" && grep -qF '<!-- demo:end -->' <<< "$BODY"; } ||
grep -qE "/resolve/videos/pr/$PR_NUMBER/[0-9a-f]{10}(-[0-9]+)?\.mp4" <<< "$BODY"; then
has_demo=true
fi
if [ "$has_demo" = true ] && [ "$HAS_LABEL" = false ]; then
gh pr edit "$PR_URL" --add-label 'video demo'
elif [ "$has_demo" = false ] && [ "$HAS_LABEL" = true ]; then
gh pr edit "$PR_URL" --remove-label 'video demo'
fi
@@ -24,7 +24,7 @@ concurrency:
jobs:
python-check-requirements:
runs-on: ${{ 'ubuntu-24.04-arm' || 'ubuntu-24.04' }}
runs-on: ${{ 'ubuntu-26.04-arm' || 'ubuntu-26.04' }}
name: check-requirements
steps:
- name: Check out source repository
+1 -1
View File
@@ -35,7 +35,7 @@ jobs:
uses: actions/setup-python@v6
with:
python-version: "3.11"
pip-install: -r requirements/requirements-all.txt ty==0.0.84
pip-install: -r requirements/requirements-all.txt ty==0.0.100
# - name: Type-check with Pyright
# uses: jakebailey/pyright-action@v2
# with:
+11 -7
View File
@@ -9,6 +9,8 @@ on:
branches:
- master
run-name: "Publish ${{ github.event.workflow_run.display_title }}"
cache-mode: none
permissions:
actions: read
@@ -169,14 +171,16 @@ jobs:
**Linux:**
- [Ubuntu x64 (CPU)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-ubuntu-x64.tar.gz)
- [Ubuntu arm64 (CPU)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-ubuntu-arm64.tar.gz)
- [Ubuntu s390x (CPU)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-ubuntu-s390x.tar.gz)
- [Ubuntu 24.04 s390x (CPU)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-ubuntu-s390x.tar.gz)
- [Ubuntu 26.04 s390x (CPU)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-ubuntu-26.04-s390x.tar.gz)
- [Ubuntu 26.04 s390x (zDNN)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-ubuntu-zdnn-s390x.tar.gz)
- [Ubuntu x64 (Vulkan)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-ubuntu-vulkan-x64.tar.gz)
- [Ubuntu arm64 (Vulkan)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-ubuntu-vulkan-arm64.tar.gz)
- [Ubuntu x64 (CUDA 12)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-ubuntu-cuda-12.8-x64.tar.gz) - [CUDA 12.8 libraries](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/cudart-llama-${{ steps.tag.outputs.name }}-bin-ubuntu-cuda-12.8-x64.tar.gz)
- [Ubuntu x64 (CUDA 13)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-ubuntu-cuda-13.4-x64.tar.gz) - [CUDA 13.4 libraries](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/cudart-llama-${{ steps.tag.outputs.name }}-bin-ubuntu-cuda-13.4-x64.tar.gz)
- [Ubuntu arm64 (CUDA 13)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-ubuntu-cuda-13.4-arm64.tar.gz) - [CUDA 13.4 libraries](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/cudart-llama-${{ steps.tag.outputs.name }}-bin-ubuntu-cuda-13.4-arm64.tar.gz)
- [Ubuntu x64 (CUDA 13)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-ubuntu-cuda-13.5-x64.tar.gz) - [CUDA 13.5 libraries](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/cudart-llama-${{ steps.tag.outputs.name }}-bin-ubuntu-cuda-13.5-x64.tar.gz)
- [Ubuntu arm64 (CUDA 13)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-ubuntu-cuda-13.5-arm64.tar.gz) - [CUDA 13.5 libraries](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/cudart-llama-${{ steps.tag.outputs.name }}-bin-ubuntu-cuda-13.5-arm64.tar.gz)
- [Ubuntu x64 (ROCm 10.0)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-ubuntu-rocm-10.0-x64.tar.gz)
- [Ubuntu x64 (OpenVINO)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-ubuntu-openvino-${{ needs.ubuntu-24-openvino.outputs.openvino_version }}-x64.tar.gz)
- [Ubuntu x64 (OpenVINO)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-ubuntu-openvino-2026.4.1-x64.tar.gz)
- [Ubuntu x64 (SYCL FP32)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-ubuntu-sycl-fp32-x64.tar.gz)
- [Ubuntu x64 (SYCL FP16)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-ubuntu-sycl-fp16-x64.tar.gz)
- [Linux arm64 (Snapdragon: CPU, Adreno GPU, Hexagon NPU)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-linux-arm64-snapdragon.tar.gz) - [setup guide](https://github.com/ggml-org/llama.cpp/blob/master/docs/backend/snapdragon/linux.md)
@@ -190,11 +194,11 @@ jobs:
- [Windows arm64 (CPU)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-win-cpu-arm64.zip)
- [Windows arm64 (OpenCL Adreno)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-win-opencl-adreno-arm64.zip)
- [Windows x64 (CUDA 12)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-win-cuda-12.4-x64.zip) - [CUDA 12.4 DLLs](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/cudart-llama-bin-win-cuda-12.4-x64.zip)
- [Windows x64 (CUDA 13)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-win-cuda-13.4-x64.zip) - [CUDA 13.4 DLLs](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/cudart-llama-bin-win-cuda-13.4-x64.zip)
- [Windows arm64 (CUDA 13)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-win-cuda-13.4-arm64.zip) - [CUDA 13.4 DLLs](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/cudart-llama-bin-win-cuda-13.4-arm64.zip)
- [Windows x64 (CUDA 13)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-win-cuda-13.5-x64.zip) - [CUDA 13.5 DLLs](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/cudart-llama-bin-win-cuda-13.5-x64.zip)
- [Windows arm64 (CUDA 13)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-win-cuda-13.5-arm64.zip) - [CUDA 13.5 DLLs](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/cudart-llama-bin-win-cuda-13.5-arm64.zip)
- [Windows x64 (Vulkan)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-win-vulkan-x64.zip)
- [Windows arm64 (Vulkan)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-win-vulkan-arm64.zip)
- [Windows x64 (OpenVINO)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-win-openvino-${{ needs.windows-openvino.outputs.openvino_version }}-x64.zip)
- [Windows x64 (OpenVINO)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-win-openvino-2026.4.1-x64.zip)
- [Windows x64 (SYCL)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-win-sycl-x64.zip)
- [Windows x64 (ROCm 10.0)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-win-rocm-10.0-x64.zip)
+149 -71
View File
@@ -35,7 +35,7 @@ permissions:
env:
GH_TOKEN: ${{ github.token }}
BRANCH_NAME: ${{ github.head_ref || github.ref_name }}
CMAKE_ARGS: "-DLLAMA_BUILD_EXAMPLES=OFF -DLLAMA_BUILD_TESTS=OFF -DLLAMA_BUILD_TOOLS=ON -DLLAMA_BUILD_SERVER=ON -DGGML_RPC=ON"
CMAKE_ARGS: "-DLLAMA_BUILD_EXAMPLES=OFF -DLLAMA_BUILD_TESTS=OFF -DLLAMA_BUILD_TOOLS=ON -DLLAMA_BUILD_SERVER=ON -DGGML_RPC=ON -DLLAMA_CONNECT=ON"
# note: run this workflow one at a time for better cache reuse
concurrency:
@@ -130,7 +130,7 @@ jobs:
uses: actions/download-artifact@v8
with:
name: llama-ui.zip
path: tools/ui/dist
path: ui/apps/chat/dist
- name: ccache
uses: ggml-org/ccache-action@v1.2.24
@@ -180,6 +180,8 @@ jobs:
os: ubuntu-24.04-arm
- build: 's390x'
os: ubuntu-24.04-s390x
- build: '26.04-s390x'
os: ubuntu-26.04-s390x
runs-on: ${{ matrix.os }}
@@ -194,13 +196,13 @@ jobs:
uses: actions/download-artifact@v8
with:
name: llama-ui.zip
path: tools/ui/dist
path: ui/apps/chat/dist
- name: Dependencies
id: depends
run: |
sudo apt-get update
sudo apt-get install build-essential libssl-dev
sudo apt-get install -y build-essential libssl-dev
- name: Toolchain workaround (GCC 14)
if: ${{ contains(matrix.os, 'ubuntu-24.04') }}
@@ -210,7 +212,7 @@ jobs:
echo "CXX=g++-14" >> "$GITHUB_ENV"
- name: ccache
if: ${{ matrix.build != 's390x' }}
if: ${{ !contains(matrix.os, 's390x') }}
uses: ggml-org/ccache-action@v1.2.24
with:
key: release-${{ matrix.os }}-cpu
@@ -242,7 +244,7 @@ jobs:
archive: false
- name: ccache-clear
if: ${{ matrix.build != 's390x' }}
if: ${{ !contains(matrix.os, 's390x') }}
uses: ./.github/actions/ccache-clear
with:
key: release-${{ matrix.os }}-cpu
@@ -272,7 +274,7 @@ jobs:
uses: actions/download-artifact@v8
with:
name: llama-ui.zip
path: tools/ui/dist
path: ui/apps/chat/dist
- name: Dependencies
id: depends
@@ -343,14 +345,14 @@ jobs:
defines: '-DGGML_CUDA_CCCL_VERSION=v3.4.3'
- build: 'x64'
os: ubuntu-24.04
cuda: '13.4.1'
label: '13.4'
defines: '-DGGML_CUDA_CCCL_VERSION=v3.4.3'
cuda: '13.5.0'
label: '13.5'
defines: ''
- build: 'arm64'
os: ubuntu-24.04-arm
cuda: '13.4.1'
label: '13.4'
defines: '-DGGML_CUDA_CCCL_VERSION=v3.4.3'
cuda: '13.5.0'
label: '13.5'
defines: ''
runs-on: ${{ matrix.os }}
container: nvidia/cuda:${{ matrix.cuda }}-devel-ubuntu24.04
@@ -379,7 +381,7 @@ jobs:
uses: actions/download-artifact@v8
with:
name: llama-ui.zip
path: tools/ui/dist
path: ui/apps/chat/dist
- name: Dependencies
id: depends
@@ -461,7 +463,7 @@ jobs:
needs: [check-release, ui-build]
if: ${{ needs.check-release.outputs.should_release == 'true' }}
runs-on: ubuntu-24.04 # previously ubuntu-latest
runs-on: ubuntu-24.04
env:
NDK_VERSION: "29.0.14206865"
@@ -477,7 +479,7 @@ jobs:
uses: actions/download-artifact@v8
with:
name: llama-ui.zip
path: tools/ui/dist
path: ui/apps/chat/dist
- name: Set up JDK
uses: actions/setup-java@v5
@@ -546,8 +548,8 @@ jobs:
needs: [check-release, ui-build]
if: ${{ needs.check-release.outputs.should_release == 'true' }}
runs-on: ubuntu-latest
container: 'ghcr.io/snapdragon-toolchain/arm64-android:v0.7'
runs-on: ubuntu-24.04
container: 'ghcr.io/snapdragon-toolchain/arm64-android:v0.8'
defaults:
run:
@@ -569,7 +571,7 @@ jobs:
uses: actions/download-artifact@v8
with:
name: llama-ui.zip
path: tools/ui/dist
path: ui/apps/chat/dist
- name: Build
id: cmake_build
@@ -599,8 +601,8 @@ jobs:
needs: [check-release, ui-build]
if: ${{ needs.check-release.outputs.should_release == 'true' }}
runs-on: ubuntu-latest
container: 'ghcr.io/snapdragon-toolchain/arm64-linux:v0.7'
runs-on: ubuntu-24.04
container: 'ghcr.io/snapdragon-toolchain/arm64-linux:v0.8'
defaults:
run:
@@ -622,7 +624,7 @@ jobs:
uses: actions/download-artifact@v8
with:
name: llama-ui.zip
path: tools/ui/dist
path: ui/apps/chat/dist
- name: Build
id: cmake_build
@@ -654,19 +656,12 @@ jobs:
runs-on: ubuntu-24.04
outputs:
openvino_version: ${{ steps.openvino_version.outputs.value }}
env:
# Sync versions in build-openvino.yml, build-self-hosted.yml, release.yml, build-cache.yml, .devops/openvino.Dockerfile
# Sync versions in build-openvino.yml, ci-self-hosted-openvino.yml, release.yml, release-publish.yml, build-cache.yml, .devops/openvino.Dockerfile, docs/backend/OPENVINO.md
OPENVINO_VERSION_MAJOR: "2026.4.1"
OPENVINO_VERSION_FULL: "2026.4.1.22982.07f9c262b05"
steps:
- name: Set OpenVINO version output
id: openvino_version
run: echo "value=${{ env.OPENVINO_VERSION_MAJOR }}" >> $GITHUB_OUTPUT
- name: Clone
id: checkout
uses: actions/checkout@v6
@@ -677,7 +672,7 @@ jobs:
uses: actions/download-artifact@v8
with:
name: llama-ui.zip
path: tools/ui/dist
path: ui/apps/chat/dist
- name: ccache
uses: ggml-org/ccache-action@v1.2.24
@@ -721,6 +716,9 @@ jobs:
-DGGML_OPENVINO=ON \
-DCMAKE_INSTALL_RPATH='$ORIGIN' \
-DCMAKE_BUILD_WITH_INSTALL_RPATH=ON \
-DGGML_BACKEND_DL=ON \
-DGGML_NATIVE=OFF \
-DGGML_CPU_ALL_VARIANTS=ON \
${{ env.CMAKE_ARGS }}
cmake --build build/ReleaseOV --config Release --parallel
@@ -759,38 +757,26 @@ jobs:
with:
key: release-ubuntu-24.04-openvino-release-no-preset-v1
# note: builds only the ggml-openvino backend - llama-server and the CPU backend are injected from the
# windows-cpu zip during the release "Merge artifacts" step
windows-openvino:
needs: [check-release, ui-build]
needs: [check-release]
if: ${{ needs.check-release.outputs.should_release == 'true' }}
runs-on: windows-2022
outputs:
openvino_version: ${{ steps.openvino_version.outputs.value }}
env:
# Sync versions in build-openvino.yml, build-self-hosted.yml, release.yml, build-cache.yml, .devops/openvino.Dockerfile
# Sync versions in build-openvino.yml, ci-self-hosted-openvino.yml, release.yml, release-publish.yml, build-cache.yml, .devops/openvino.Dockerfile, docs/backend/OPENVINO.md
OPENVINO_VERSION_MAJOR: "2026.4.1"
OPENVINO_VERSION_FULL: "2026.4.1.22982.07f9c262b05"
steps:
- name: Set OpenVINO version output
id: openvino_version
shell: bash
run: echo "value=${{ env.OPENVINO_VERSION_MAJOR }}" >> $GITHUB_OUTPUT
- name: Clone
id: checkout
uses: actions/checkout@v6
with:
fetch-depth: 0
- name: Download UI build
uses: actions/download-artifact@v8
with:
name: llama-ui.zip
path: tools/ui/dist
- name: ccache
uses: ggml-org/ccache-action@v1.2.24
with:
@@ -838,11 +824,13 @@ jobs:
-A x64 ^
-DCMAKE_BUILD_TYPE=Release ^
-DGGML_OPENVINO=ON ^
-DGGML_BACKEND_DL=ON ^
-DGGML_NATIVE=OFF ^
-DGGML_CPU=OFF ^
-DLLAMA_BUILD_BORINGSSL=ON ^
-DCMAKE_TOOLCHAIN_FILE=C:\vcpkg\scripts\buildsystems\vcpkg.cmake ^
${{ env.CMAKE_ARGS }}
-DCMAKE_TOOLCHAIN_FILE=C:\vcpkg\scripts\buildsystems\vcpkg.cmake
cmake --build build\ReleaseOV --config Release -- /m
cmake --build build\ReleaseOV --config Release --target ggml-openvino -- /m
- name: Pack artifacts
id: pack_artifacts
@@ -870,12 +858,12 @@ jobs:
Copy-Item -Path (Join-Path $OPENVINO_ROOT 'docs\licensing\*') -Destination $licensingDest -Recurse -Force
Copy-Item LICENSE $dest
7z a -snl llama-${{ needs.check-release.outputs.tag_name }}-bin-win-openvino-${{ env.OPENVINO_VERSION_MAJOR }}-x64.zip $dest\*
7z a -snl llama-bin-win-openvino-${{ env.OPENVINO_VERSION_MAJOR }}-x64.zip $dest\*
- name: Upload artifacts
uses: actions/upload-artifact@v7
with:
path: llama-${{ needs.check-release.outputs.tag_name }}-bin-win-openvino-${{ env.OPENVINO_VERSION_MAJOR }}-x64.zip
path: llama-bin-win-openvino-${{ env.OPENVINO_VERSION_MAJOR }}-x64.zip
archive: false
- name: ccache-clear
@@ -906,12 +894,17 @@ jobs:
uses: actions/download-artifact@v8
with:
name: llama-ui.zip
path: tools/ui/dist
path: ui/apps/chat/dist
- name: Install Ninja
run: |
choco install ninja
- name: Install LLVM
run: |
choco install llvm --version=22.1.8
echo "C:\Program Files\LLVM\bin" >> $env:GITHUB_PATH
- name: ccache
uses: ggml-org/ccache-action@v1.2.24
with:
@@ -982,6 +975,11 @@ jobs:
evict-old-files: 1d
max-size: "1G"
- name: Tune ccache for ROCm
run: |
# since ROCm 10.1, clang -E output has no GPU code, so the preprocessor fallback can reuse objects with old GPU code
ccache --set-config=depend_mode=true
# - name: Cache ROCm Installation
# id: cache-rocm
# uses: actions/cache@v5
@@ -1153,6 +1151,11 @@ jobs:
run: |
choco install ninja
- name: Install LLVM
run: |
choco install llvm --version=22.1.8
echo "C:\Program Files\LLVM\bin" >> $env:GITHUB_PATH
# TODO: these jobs need to use llvm toolchain in order to utilize the ccache
#- name: ccache
# uses: ggml-org/ccache-action@v1.2.24
@@ -1218,12 +1221,12 @@ jobs:
- cuda: '12.4'
arch: x64
defines: '-DGGML_CUDA_CCCL_VERSION=v3.4.3'
- cuda: '13.4'
- cuda: '13.5'
arch: x64
defines: '-DGGML_CUDA_CCCL_VERSION=v3.4.3'
- cuda: '13.4'
defines: ''
- cuda: '13.5'
arch: arm64
defines: '-DGGML_CUDA_CCCL_VERSION=v3.4.3 -DCMAKE_TOOLCHAIN_FILE=cmake/arm64-windows-msvc-cuda.cmake'
defines: '-DCMAKE_TOOLCHAIN_FILE=cmake/arm64-windows-msvc-cuda.cmake'
steps:
- name: Clone
@@ -1246,6 +1249,7 @@ jobs:
with:
key: release-windows-2022-${{ matrix.arch }}-cuda-${{ matrix.cuda }}
evict-old-files: 1d
max-size: "1G"
- name: Build
id: cmake_build
@@ -1314,11 +1318,14 @@ jobs:
shell: bash
env:
WINDOWS_BASEKIT_URL: https://registrationcenter-download.intel.com/akdlm/IRC_NAS/0cb67a0d-67f6-410b-868b-f4a0a17ff0cf/intel-oneapi-toolkit-2026.1.1.32_offline.exe
WINDOWS_DPCPP_MKL: intel.oneapi.win.cpp-dpcpp-common:intel.oneapi.win.mkl.devel:intel.oneapi.win.dnnl:intel.oneapi.win.tbb.devel
LEVEL_ZERO_SDK_URL: https://github.com/oneapi-src/level-zero/releases/download/v1.33.1/level-zero-win-sdk-1.33.1.zip
WINDOWS_DLE_URL: https://registrationcenter-download.intel.com/akdlm/IRC_NAS/b177a006-5cd9-4cb2-807f-8199cc73d53c/intel-deep-learning-essentials-2026.2.0.638_offline.exe
WINDOWS_ONEDNN_URL: https://registrationcenter-download.intel.com/akdlm/IRC_NAS/8f905666-c85c-43fb-9bd0-2618c7ecd33b/intel-onednn-2026.1.0.622_offline.exe
WINDOWS_DPCPP_MKL: intel.oneapi.win.cpp-dpcpp-common:intel.oneapi.win.mkl.devel:intel.oneapi.win.tbb.devel
WINDOWS_DNNL: intel.oneapi.win.dnnl
LEVEL_ZERO_SDK_URL: https://github.com/oneapi-src/level-zero/releases/download/v1.34.0/level-zero-win-sdk-1.34.0.zip
ONEAPI_ROOT: "C:/Program Files (x86)/Intel/oneAPI"
ONEAPI_INSTALLER_VERSION: "2026.1"
ONEAPI_INSTALLER_VERSION: "2026.2"
steps:
- name: Clone
@@ -1328,7 +1335,8 @@ jobs:
- name: Download & Install oneAPI
shell: bash
run: |
scripts/install-oneapi.bat $WINDOWS_BASEKIT_URL $WINDOWS_DPCPP_MKL
scripts/install-oneapi.bat $WINDOWS_DLE_URL $WINDOWS_DPCPP_MKL
scripts/install-oneapi.bat $WINDOWS_ONEDNN_URL $WINDOWS_DNNL
- name: Install Level Zero SDK
shell: pwsh
@@ -1366,11 +1374,12 @@ jobs:
cp "${{ env.ONEAPI_ROOT }}/mkl/latest/bin/mkl_def.3.dll" ./build/bin
cp "${{ env.ONEAPI_ROOT }}/mkl/latest/bin/mkl_avx2.3.dll" ./build/bin
cp "${{ env.ONEAPI_ROOT }}/mkl/latest/bin/mkl_tbb_thread.3.dll" ./build/bin
cp "${{ env.ONEAPI_ROOT }}/mkl/latest/bin/mkl_intel_thread.3.dll" ./build/bin
cp "${{ env.ONEAPI_ROOT }}/mkl/latest/bin/libimalloc.dll" ./build/bin
cp "${{ env.ONEAPI_ROOT }}/compiler/latest/bin/ur_adapter_level_zero.dll" ./build/bin
cp "${{ env.ONEAPI_ROOT }}/compiler/latest/bin/ur_adapter_level_zero_v2.dll" ./build/bin
cp "${{ env.ONEAPI_ROOT }}/compiler/latest/bin/ur_adapter_opencl.dll" ./build/bin
cp "${{ env.ONEAPI_ROOT }}/compiler/latest/bin/ur_loader.dll" ./build/bin
cp "${{ env.ONEAPI_ROOT }}/compiler/latest/bin/ur_loader_shared.dll" ./build/bin
cp "${{ env.ONEAPI_ROOT }}/compiler/latest/bin/ur_win_proxy_loader.dll" ./build/bin
ZE_LOADER_DLL=$(find "${{ env.ONEAPI_ROOT }}" "$LEVEL_ZERO_V1_SDK_PATH" -iname ze_loader.dll -print -quit 2>/dev/null || true)
if [ -n "$ZE_LOADER_DLL" ]; then
@@ -1424,8 +1433,8 @@ jobs:
env:
ONEAPI_ROOT: /opt/intel/oneapi/
ONEAPI_INSTALLER_VERSION: "2026.1"
LEVEL_ZERO_VERSION: "1.33.1"
ONEAPI_INSTALLER_VERSION: "2026.2"
LEVEL_ZERO_VERSION: "1.34.0"
LEVEL_ZERO_UBUNTU_VERSION: "u24.04"
steps:
@@ -1439,8 +1448,15 @@ jobs:
shell: bash
run: |
cd /tmp
wget https://registrationcenter-download.intel.com/akdlm/IRC_NAS/5996e26b-f48a-42b1-8db0-b002ad0bd8d7/intel-oneapi-toolkit-2026.1.1.33_offline.sh -O intel-oneapi-toolkit_offline.sh
sudo bash intel-oneapi-toolkit_offline.sh -s -a --silent --eula accept
echo "Download oneAPI Deep Learning Essentials installation package"
wget https://registrationcenter-download.intel.com/akdlm/IRC_NAS/aef232d2-aac5-4cd6-bceb-911019bbb6a8/intel-deep-learning-essentials-2026.2.0.640_offline.sh -O intel-deep-learning-essentials_offline.sh
echo "Install oneAPI Deep Learning Essentials components"
sudo bash intel-deep-learning-essentials_offline.sh -s -a --silent --eula accept
echo "Download oneDNN installation package"
wget https://registrationcenter-download.intel.com/akdlm/IRC_NAS/46452a4a-7b7f-49f2-be4c-602a900180bb/intel-onednn-2026.1.0.625_offline.sh -O intel-onednn_offline.sh
echo "Install oneDNN components"
sudo bash intel-onednn_offline.sh -s -a --silent --eula accept
- name: Install Level Zero SDK
shell: bash
@@ -1454,7 +1470,13 @@ jobs:
uses: actions/download-artifact@v8
with:
name: llama-ui.zip
path: tools/ui/dist
path: ui/apps/chat/dist
- name: Dependencies
id: depends
run: |
sudo apt-get update
sudo apt-get install -y libssl-dev
- name: ccache
uses: ggml-org/ccache-action@v1.2.24
@@ -1474,8 +1496,8 @@ jobs:
-DCMAKE_CXX_COMPILER=icpx \
-DCMAKE_INSTALL_RPATH='$ORIGIN' \
-DCMAKE_BUILD_WITH_INSTALL_RPATH=ON \
-DLLAMA_OPENSSL=OFF \
-DGGML_NATIVE=OFF \
-DLLAMA_CONNECT=ON \
-DGGML_SYCL_F16=${{ matrix.fp16 }}
time cmake --build build --config Release -j $(nproc)
@@ -1520,7 +1542,7 @@ jobs:
uses: actions/download-artifact@v8
with:
name: llama-ui.zip
path: tools/ui/dist
path: ui/apps/chat/dist
- name: Free up disk space
uses: ggml-org/free-disk-space@v1.3.1
@@ -1542,6 +1564,8 @@ jobs:
# builds hit.
ccache --set-config=compiler_check=content
ccache --set-config=sloppiness=time_macros,include_file_mtime,include_file_ctime
# since ROCm 10.1, clang -E output has no GPU code, so the preprocessor fallback can reuse objects with old GPU code
ccache --set-config=depend_mode=true
- name: Dependencies
id: depends
@@ -1616,6 +1640,60 @@ jobs:
with:
key: release-ubuntu-24.04-rocm-${{ matrix.ROCM_VERSION }}-${{ matrix.build }}
ubuntu-26-zdnn:
needs: [check-release, ui-build]
if: ${{ needs.check-release.outputs.should_release == 'true' }}
runs-on: ubuntu-26.04-s390x
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
with:
fetch-depth: 0
- name: Download UI build
uses: actions/download-artifact@v8
with:
name: llama-ui.zip
path: ui/apps/chat/dist
- name: Dependencies
id: depends
run: |
sudo apt-get update
sudo apt-get install -y --no-install-recommends \
build-essential libssl-dev libzdnn-dev
- name: Build
id: cmake_build
run: |
cmake -B build \
-DCMAKE_INSTALL_RPATH='$ORIGIN' \
-DCMAKE_BUILD_WITH_INSTALL_RPATH=ON \
-DGGML_BACKEND_DL=ON \
-DGGML_NATIVE=OFF \
-DGGML_VXE=ON \
-DGGML_ZDNN=ON \
-DCMAKE_C_FLAGS="-march=arch15" \
-DCMAKE_CXX_FLAGS="-march=arch15" \
-DLLAMA_FATAL_WARNINGS=ON \
${{ env.CMAKE_ARGS }}
cmake --build build --config Release -j $(nproc)
- name: Pack artifacts
id: pack_artifacts
run: |
cp LICENSE ./build/bin/
tar -czvf llama-${{ needs.check-release.outputs.tag_name }}-bin-ubuntu-zdnn-s390x.tar.gz --transform "s,^\.,llama-${{ needs.check-release.outputs.tag_name }}," -C ./build/bin .
- name: Upload artifacts
uses: actions/upload-artifact@v7
with:
path: llama-${{ needs.check-release.outputs.tag_name }}-bin-ubuntu-zdnn-s390x.tar.gz
archive: false
ios-xcode:
needs: [check-release]
if: ${{ needs.check-release.outputs.should_release == 'true' }}
+2
View File
@@ -42,6 +42,8 @@ env:
LLAMA_ARG_LOG_PREFIX: 1
LLAMA_ARG_LOG_TIMESTAMPS: 1
LLAMA_ARG_LOG_VERBOSITY: 10
# [TAG_MTMD_NO_REALLOC] must match -DGGML_SCHED_NO_REALLOC=ON below, skips tests that are known to realloc
GGML_SCHED_NO_REALLOC: ON
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}-${{ github.head_ref || github.run_id }}
+2
View File
@@ -39,6 +39,8 @@ env:
LLAMA_ARG_LOG_PREFIX: 1
LLAMA_ARG_LOG_TIMESTAMPS: 1
LLAMA_ARG_LOG_VERBOSITY: 10
# [TAG_MTMD_NO_REALLOC] must match -DGGML_SCHED_NO_REALLOC=ON below, skips tests that are known to realloc
GGML_SCHED_NO_REALLOC: ON
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}-${{ github.head_ref || github.run_id }}
+100
View File
@@ -0,0 +1,100 @@
name: Server (speculative decoding)
on:
workflow_dispatch:
inputs:
sha:
description: 'Commit SHA1 to build'
required: false
type: string
spec_methods:
description: 'Speculative methods to test'
required: true
default: 'compact'
type: choice
options:
- compact
- nightly
- all
push:
branches:
- master
paths: [
'.github/workflows/server-speculative.yml',
'common/speculative.*',
'src/models/dflash*.cpp',
'src/models/dspark.cpp',
'src/models/eagle3.cpp',
'src/models/lilicorr.cpp',
'src/models/qwen35.cpp',
'tools/server/tests/unit/test_spec_methods.py'
]
pull_request:
types: [opened, synchronize, reopened]
paths: [
'.github/workflows/server-speculative.yml',
'common/speculative.*',
'src/models/dflash*.cpp',
'src/models/dspark.cpp',
'src/models/eagle3.cpp',
'src/models/lilicorr.cpp',
'src/models/qwen35.cpp',
'tools/server/tests/unit/test_spec_methods.py'
]
cache-mode: none
permissions:
contents: read
env:
LLAMA_ARG_LOG_COLORS: 1
LLAMA_ARG_LOG_PREFIX: 1
LLAMA_ARG_LOG_TIMESTAMPS: 1
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}-${{ github.head_ref || github.run_id }}
cancel-in-progress: true
jobs:
server-speculative:
runs-on: [self-hosted, Linux, gb10, perf]
container: "nvidia/cuda:13.4.3-devel-ubuntu26.04"
timeout-minutes: 60
defaults:
run:
shell: bash
steps:
- name: Clone
uses: actions/checkout@v6
with:
fetch-depth: 0
ref: ${{ github.event.inputs.sha || github.event.pull_request.head.sha || github.sha || github.head_ref || github.ref_name }}
- name: Install dependencies
run: |
apt update
apt install -y cmake libssl-dev python3 python3-venv python3-pip
- name: Build
run: |
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_COMPILER=/usr/local/cuda/bin/nvcc
cmake --build build --config Release -j $(nproc) --target llama-server
- name: Python setup
run: |
cd tools/server/tests
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
- name: Test speculative decoding
env:
SPEC_METHODS: ${{ github.event.inputs.spec_methods || 'compact' }}
# pull requests from forks do not get repository variables
SPEC_MODEL_DIR: ${{ vars.SPEC_METHOD_MODEL_DIR || '/mnt/cache/spec-models' }}
N_GPU_LAYERS: 99
run: |
cd tools/server/tests
source venv/bin/activate
PYTEST_WORKERS=1 ./tests.sh -s -v unit/test_spec_methods.py
+8 -3
View File
@@ -59,7 +59,7 @@ concurrency:
jobs:
ubuntu:
runs-on: ubuntu-24.04-arm
runs-on: ubuntu-26.04-arm
steps:
- name: Dependencies
@@ -94,7 +94,7 @@ jobs:
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CI }}
with:
key: server-ubuntu-24.04-arm
key: server-ubuntu-26.04-arm
folder: llama.cpp
hf_bucket: ggml-org/cache
@@ -111,7 +111,7 @@ jobs:
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CACHE_OUTPUT }}
with:
key: server-ubuntu-24.04-arm
key: server-ubuntu-26.04-arm
folder: llama.cpp
evict-old-files: 1d
hf_bucket: ggml-org/cache
@@ -175,6 +175,11 @@ jobs:
evict-old-files: 1d
save: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
- name: Install LLVM
run: |
choco install llvm --version=22.1.8
echo "C:\Program Files\LLVM\bin" >> $env:GITHUB_PATH
- name: Build
id: cmake_build
shell: cmd
+4 -4
View File
@@ -23,20 +23,20 @@ jobs:
with:
node-version: "24"
# cache: "npm"
# cache-dependency-path: "tools/ui/package-lock.json"
# cache-dependency-path: "ui/apps/chat/package-lock.json"
package-manager-cache: false
- name: Install dependencies
run: npm ci
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Build application
run: npm run build
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Upload built UI
uses: actions/upload-artifact@v6
with:
name: llama-ui.zip
path: tools/ui/dist/
path: ui/apps/chat/dist/
retention-days: 1
+6 -6
View File
@@ -15,7 +15,7 @@ permissions:
jobs:
build:
runs-on: ubuntu-slim
runs-on: ubuntu-26.04-arm
steps:
- name: Checkout code
@@ -37,26 +37,26 @@ jobs:
with:
node-version: "24"
# cache: "npm"
# cache-dependency-path: "tools/ui/package-lock.json"
# cache-dependency-path: "ui/apps/chat/package-lock.json"
package-manager-cache: false
- name: Install dependencies
run: npm ci
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Build application
env:
LLAMA_BUILD_NUMBER: ${{ steps.version.outputs.ui_version }}
run: npm run build
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Run PWA unit tests (versioned build output)
run: npx vitest --project=unit --run tests/unit/pwa.spec.ts
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Upload built UI
uses: actions/upload-artifact@v7
with:
name: llama-ui.zip
path: tools/ui/dist/
path: ui/apps/chat/dist/
retention-days: 1
+183
View File
@@ -0,0 +1,183 @@
name: UI Chat Publish
# Publishes Llama Chat to npm: @ggml-org/llama-chat (ui/apps/chat, the app and its CLI) and
# llama-chat (ui/apps/chat/alias, the short name for npx), always at the same version.
#
# The version in ui/apps/chat/package.json is the release, set by hand, as for @ggml-org/llama-ui
# in ui-lib-publish.yml. A commit that reaches master with a version npm does not have builds the
# app and publishes both packages; any other commit publishes nothing. Bump both package.json
# files in the PR that should ship.
#
# Auth: no npm token is stored here. The job exchanges its OIDC identity for one (npm trusted
# publishing) and signs the tarballs with provenance. One-time setup, before this can publish: on
# each of the two packages, Publishing settings, Trusted Publisher, GitHub Actions, with
# organization ggml-org, repository llama.cpp, workflow file ui-chat-publish.yml and Environment
# left blank. Until then the publish step fails on master; re-running the job publishes it.
on:
workflow_dispatch:
inputs:
dry_run:
description: 'Build and pack, but do not publish'
required: false
type: boolean
default: false
push:
branches:
- master
paths: [
'.github/workflows/ui-chat-publish.yml',
'ui/apps/chat/package.json',
'ui/apps/chat/alias/package.json'
]
# a dry run for PRs that change what goes into the packages
pull_request:
types: [opened, synchronize, reopened]
paths: [
'.github/workflows/ui-chat-publish.yml',
'ui/apps/chat/package.json',
'ui/apps/chat/alias/**',
'ui/apps/chat/bin/**',
'ui/apps/chat/cli/**'
]
cache-mode: none
permissions:
contents: read
id-token: write
concurrency:
# never cancel a run mid-publish: a version half-published cannot be taken back
group: ${{ github.workflow }}
cancel-in-progress: false
env:
APP_DIR: ui/apps/chat
ALIAS_DIR: ui/apps/chat/alias
VERSION_CHECK: ui/lib/ui/scripts/npm-version-check.mjs
jobs:
publish:
name: Publish @ggml-org/llama-chat and llama-chat
if: github.repository == 'ggml-org/llama.cpp'
runs-on: ubuntu-26.04-arm
steps:
- name: Checkout code
uses: actions/checkout@v6
with:
fetch-depth: 0
- name: Setup Node.js
uses: actions/setup-node@v6
with:
node-version: "24"
registry-url: 'https://registry.npmjs.org'
package-manager-cache: false
- name: Check npm can publish with an OIDC token
run: |
node -e '
const v = process.argv[1];
const [major, minor] = v.split(".").map(Number);
if (major < 11 || (major === 11 && minor < 5)) {
console.error(`npm ${v} cannot publish with trusted publishing: 11.5.1 or newer is needed`);
process.exit(1);
}
' "$(npm -v)"
- name: Install dependencies
run: npm ci
working-directory: ${{ env.APP_DIR }}
- name: Build the app
# the same build.json version as the UI that llama-server embeds, for the app's update check.
# This folder's .npmrc sets ignore-scripts, so npm never builds it: this step does.
run: LLAMA_BUILD_NUMBER="b$(git rev-list --count HEAD)" npm run build
working-directory: ${{ env.APP_DIR }}
- name: Test the CLI and the alias version
run: npx vitest --project=unit --run tests/unit/cli.test.ts
working-directory: ${{ env.APP_DIR }}
- name: Check what goes into the package
run: |
npm pack --dry-run --json > "$RUNNER_TEMP/pack.json"
node -e '
const [{ files }] = JSON.parse(require("fs").readFileSync(process.argv[1], "utf8"));
const paths = files.map((file) => file.path);
const maps = paths.filter((path) => path.endsWith(".map"));
if (!paths.includes("dist/index.html")) {
console.error("dist/index.html is not in the package: nothing would be served");
process.exit(1);
}
if (maps.length > 0) {
console.error(`source maps in the package: ${maps.join(", ")}`);
process.exit(1);
}
console.log(`${paths.length} files, no source maps`);
' "$RUNNER_TEMP/pack.json"
working-directory: ${{ env.APP_DIR }}
- name: Compare @ggml-org/llama-chat with npm
id: app
continue-on-error: ${{ github.event_name == 'pull_request' }}
run: node "$VERSION_CHECK" "$APP_DIR"
- name: Compare llama-chat with npm
id: alias
continue-on-error: ${{ github.event_name == 'pull_request' }}
run: node "$VERSION_CHECK" "$ALIAS_DIR"
# master only: npm's trusted publisher matches the repository and the workflow file, not the
# branch, so this condition is what keeps a manual run on a PR branch from publishing.
# The app goes first, since the alias depends on it at the same version.
- name: Publish @ggml-org/llama-chat
if: github.ref == 'refs/heads/master' && steps.app.outputs.should_publish == 'true' && inputs.dry_run != true
env:
TAG: ${{ steps.app.outputs.tag }}
run: npm publish --access public --provenance --ignore-scripts --tag "$TAG"
working-directory: ${{ env.APP_DIR }}
- name: Publish llama-chat
if: github.ref == 'refs/heads/master' && steps.alias.outputs.should_publish == 'true' && inputs.dry_run != true
env:
TAG: ${{ steps.alias.outputs.tag }}
run: npm publish --access public --provenance --ignore-scripts --tag "$TAG"
working-directory: ${{ env.ALIAS_DIR }}
- name: Report
if: always()
# values reach the shell through env, so a name from a PR's package.json cannot inject here
env:
APP: ${{ steps.app.outputs.name }}@${{ steps.app.outputs.version }}
APP_VERDICT: ${{ steps.app.outputs.verdict || 'not reached' }}
APP_NOTE: ${{ steps.app.outputs.note }}
ALIAS: ${{ steps.alias.outputs.name }}@${{ steps.alias.outputs.version }}
ALIAS_VERDICT: ${{ steps.alias.outputs.verdict || 'not reached' }}
ALIAS_NOTE: ${{ steps.alias.outputs.note }}
DRY: ${{ inputs.dry_run }}
EVENT: ${{ github.event_name }}
run: |
{
echo "### Llama Chat on npm"
echo
echo "| package | verdict | note |"
echo "| --- | --- | --- |"
echo "| \`$APP\` | \`$APP_VERDICT\` | $APP_NOTE |"
echo "| \`$ALIAS\` | \`$ALIAS_VERDICT\` | $ALIAS_NOTE |"
echo
if [ "$EVENT" = "pull_request" ] || [ "$DRY" = "true" ]; then echo "Dry run: nothing was published."; fi
} >> "$GITHUB_STEP_SUMMARY"
- name: Mind the verdict
# a version going backwards is a mistake; an unchanged version is a package.json edit that
# is not a release, such as a dependency update
if: always() && github.event_name != 'pull_request'
env:
VERDICTS: ${{ steps.app.outputs.verdict }} ${{ steps.alias.outputs.verdict }}
run: |
case " $VERDICTS " in
*" stale "*)
echo "::error::a version in package.json is older than the one on npm"
exit 1 ;;
esac
+230
View File
@@ -0,0 +1,230 @@
name: UI Lib Publish
# Publishes @ggml-org/llama-ui, the library in ui/lib/ui, to npm.
#
# The version in ui/lib/ui/package.json is the release. A commit that reaches master with a
# version npm does not have builds the library and publishes it; a commit that changes the
# library without bumping it publishes nothing and says so on the run, which is the drift that
# let 0.3.0 sit a day behind master. Bump the version in the PR whose change should ship, and
# this workflow does the rest.
#
# Auth: no npm token is stored here. The job exchanges its OIDC identity for one (npm trusted
# publishing) and signs the tarball with a provenance attestation. release-publish.yml has done
# the same exchange on every push to master since it landed, its actions/attest step being the
# proof that this repository can be issued a token.
#
# One-time setup, before this can publish: on the package, Publishing settings, Trusted
# Publisher, GitHub Actions, with organization ggml-org, repository llama.cpp, workflow file
# ui-lib-publish.yml, and Environment left blank - this job names no environment, since creating
# one needs repository admin. Merging this workflow before that exists leaves the publish step
# red on master: the release is late rather than wrong, and re-running the job publishes it.
#
# The unscoped llama-ui is an alias: once the scoped package is out, the same build is renamed
# and published again at the same version, so the two never drift. It is a package of its own
# on npm, and needs the same trusted publisher set on it.
on:
workflow_dispatch:
inputs:
dry_run:
description: 'Build and pack, but do not publish'
required: false
type: boolean
default: false
push:
branches:
- master
paths: [
'.github/workflows/ui-lib-publish.yml',
'ui/lib/ui/**.*'
]
# the advisory half, so a library change without a bump is annotated on the PR rather than
# discovered after the merge: the path filter keeps it off PRs that touch only Llama Chat
pull_request:
types: [opened, synchronize, reopened]
paths: [
'.github/workflows/ui-lib-publish.yml',
'ui/lib/ui/**.*'
]
cache-mode: none
permissions:
contents: read
id-token: write
concurrency:
# never cancel a run mid-publish: a version half-published cannot be taken back. GitHub holds
# one pending run per group, so a newer push replaces an older queued one - no release is lost,
# as each run reads the version from its own commit.
group: ${{ github.workflow }}
cancel-in-progress: false
env:
LIB_DIR: ui/lib/ui
jobs:
publish:
name: Publish @ggml-org/llama-ui
if: github.repository == 'ggml-org/llama.cpp'
runs-on: ubuntu-26.04-arm
steps:
- name: Checkout code
uses: actions/checkout@v6
- name: Setup Node.js
uses: actions/setup-node@v6
with:
node-version: "24"
registry-url: 'https://registry.npmjs.org'
package-manager-cache: false
- name: Check npm can publish with an OIDC token
run: |
npm -v
node -e '
const v = process.argv[1];
const [major, minor] = v.split(".").map(Number);
if (major < 11 || (major === 11 && minor < 5)) {
console.error(`npm ${v} cannot publish with trusted publishing: 11.5.1 or newer is needed`);
process.exit(1);
}
' "$(npm -v)"
- name: Install dependencies
run: npm ci
working-directory: ${{ env.LIB_DIR }}
- name: Run type checking
run: npm run check
working-directory: ${{ env.LIB_DIR }}
- name: Build the package and lint it
run: npm run prepack
working-directory: ${{ env.LIB_DIR }}
- name: Compare the version with npm
id: version
# on a PR this is advisory and must not fail the job: a registry outage is not the
# contributor's mistake. On master the publish decision rests on the answer, so it fails.
continue-on-error: ${{ github.event_name == 'pull_request' }}
run: node scripts/npm-version-check.mjs
working-directory: ${{ env.LIB_DIR }}
- name: Note a library change that ships no new version
if: github.event_name == 'pull_request' && always() && steps.version.outcome == 'success' && steps.version.outputs.should_publish != 'true'
env:
NOTE: ${{ steps.version.outputs.note }}
run: echo "::warning::$NOTE"
- name: Publish
# master only. npm's trusted publisher matches on the repository and the workflow file,
# not the branch, so this condition is the only thing stopping a manual run from a PR
# branch - one that anyone with write access can start - from publishing code that no
# one reviewed. A dispatch on master remains the way to re-run a release.
if: github.ref == 'refs/heads/master' && steps.version.outputs.should_publish == 'true' && inputs.dry_run != true
# --ignore-scripts publishes the tarball prepack built and publint passed rather than
# rebuilding it here; --provenance is implied by trusted publishing and named to be
# explicit. A package published with nothing in it cannot be taken back, so refuse first.
env:
TAG: ${{ steps.version.outputs.tag }}
run: |
test -d dist || { echo "dist/ is missing: nothing was built, so nothing is published"; exit 1; }
npm publish --access public --provenance --ignore-scripts --tag "$TAG"
working-directory: ${{ env.LIB_DIR }}
- name: Rename the build to the unscoped alias
# after the scoped publish, so nothing above sees the changed name. Not on a PR: the
# advisory check there is about the version, which the alias shares.
if: github.event_name != 'pull_request'
run: node scripts/alias-package.mjs
working-directory: ${{ env.LIB_DIR }}
- name: Compare the alias with npm
id: alias
if: github.event_name != 'pull_request'
run: node scripts/npm-version-check.mjs
working-directory: ${{ env.LIB_DIR }}
- name: Publish the alias
# its own verdict rather than the scoped one's, so an alias that fell behind catches up
# on the next run instead of waiting for a bump
if: github.ref == 'refs/heads/master' && steps.alias.outputs.should_publish == 'true' && inputs.dry_run != true
env:
TAG: ${{ steps.alias.outputs.tag }}
run: |
test -d dist || { echo "dist/ is missing: nothing was built, so nothing is published"; exit 1; }
npm publish --access public --provenance --ignore-scripts --tag "$TAG"
working-directory: ${{ env.LIB_DIR }}
- name: Report
if: always()
# every value reaches the shell through env, not through the script, so a package name
# from a PR's package.json cannot inject into this step
env:
NAME: ${{ steps.version.outputs.name || '@ggml-org/llama-ui' }}
VERSION: ${{ steps.version.outputs.version || 'unknown' }}
LATEST: ${{ steps.version.outputs.latest }}
VERDICT: ${{ steps.version.outputs.verdict || 'not reached' }}
NOTE: ${{ steps.version.outputs.note }}
SHOULD: ${{ steps.version.outputs.should_publish }}
DRY: ${{ inputs.dry_run }}
ALIAS_LATEST: ${{ steps.alias.outputs.latest }}
ALIAS_VERDICT: ${{ steps.alias.outputs.verdict || 'not reached' }}
ALIAS_NOTE: ${{ steps.alias.outputs.note }}
ALIAS_SHOULD: ${{ steps.alias.outputs.should_publish }}
EVENT: ${{ github.event_name }}
run: |
if [ "$DRY" = "true" ]; then published="no (dry run)"
elif [ "$SHOULD" = "true" ]; then published=yes
else published=no
fi
{
echo "### $NAME $VERSION"
echo
echo "| | |"
echo "| --- | --- |"
echo "| npm latest was | ${LATEST:-nothing published yet} |"
echo "| verdict | \`$VERDICT\` |"
echo "| published | $published |"
echo
echo "> $NOTE"
} >> "$GITHUB_STEP_SUMMARY"
[ "$EVENT" = "pull_request" ] && exit 0
if [ "$DRY" = "true" ]; then published="no (dry run)"
elif [ "$ALIAS_SHOULD" = "true" ]; then published=yes
else published=no
fi
{
echo
echo "### llama-ui (alias) $VERSION"
echo
echo "| | |"
echo "| --- | --- |"
echo "| npm latest was | ${ALIAS_LATEST:-nothing published yet} |"
echo "| verdict | \`$ALIAS_VERDICT\` |"
echo "| published | $published |"
echo
echo "> $ALIAS_NOTE"
} >> "$GITHUB_STEP_SUMMARY"
- name: Mind the verdict
if: always() && github.event_name != 'pull_request' && steps.version.outcome == 'success'
# a version going backwards is a mistake; a version not bumped is usually one too, but it
# is also what a stories-only commit looks like, so that only fails the run when the
# repository variable UI_LIB_PUBLISH_FAIL_ON_UNBUMPED is set to true
env:
NOTE: ${{ steps.version.outputs.note }}
VERDICT: ${{ steps.version.outputs.verdict }}
FAIL_ON_UNBUMPED: ${{ vars.UI_LIB_PUBLISH_FAIL_ON_UNBUMPED }}
run: |
case "$VERDICT" in
stale)
echo "::error::$NOTE"
exit 1 ;;
published)
if [ "$FAIL_ON_UNBUMPED" = "true" ]; then
echo "::error::$NOTE"
exit 1
fi
echo "::warning::$NOTE" ;;
esac
+5 -5
View File
@@ -33,15 +33,15 @@ jobs:
uses: actions/download-artifact@v8
with:
name: llama-ui.zip
path: tools/ui/dist/
path: ui/apps/chat/dist/
run-id: ${{ inputs.run_id }}
github-token: ${{ github.token }}
- name: Create distribution archive
run: |
tar -czf dist.tar.gz -C tools/ui/dist .
tar -czf dist.tar.gz -C ui/apps/chat/dist .
sha256sum dist.tar.gz > dist.tar.gz.sha256
mv dist.tar.gz dist.tar.gz.sha256 tools/ui/dist/
mv dist.tar.gz dist.tar.gz.sha256 ui/apps/chat/dist/
- name: Install Hugging Face Hub CLI
run: pip install -U huggingface_hub
@@ -52,12 +52,12 @@ jobs:
- name: Sync built files to Hugging Face bucket (version tag)
run: |
# Upload the built files to the Hugging Face bucket under the release version
hf buckets sync tools/ui/dist hf://buckets/ggml-org/${{ env.HF_BUCKET_NAME }}/${{ inputs.version_tag }} --delete --quiet
hf buckets sync ui/apps/chat/dist hf://buckets/ggml-org/${{ env.HF_BUCKET_NAME }}/${{ inputs.version_tag }} --delete --quiet
- name: Sync built files to Hugging Face bucket (latest)
run: |
# Also upload to the 'latest' directory for fallback downloads
hf buckets sync tools/ui/dist hf://buckets/ggml-org/${{ env.HF_BUCKET_NAME }}/latest --delete --quiet
hf buckets sync ui/apps/chat/dist hf://buckets/ggml-org/${{ env.HF_BUCKET_NAME }}/latest --delete --quiet
- name: Verify upload
run: |
+17 -13
View File
@@ -17,7 +17,9 @@ on:
paths: [
'.github/workflows/ui-self-hosted.yml',
'.github/workflows/ui-build-self-hosted.yml',
'tools/ui/**.*',
'ui/apps/chat/**.*',
'ui/lib/ui/**.*',
'ui/lib/voice/**.*',
'tools/server/tests/**.*'
]
pull_request:
@@ -25,7 +27,9 @@ on:
paths: [
'.github/workflows/ui-self-hosted.yml',
'.github/workflows/ui-build-self-hosted.yml',
'tools/ui/**.*',
'ui/apps/chat/**.*',
'ui/lib/ui/**.*',
'ui/lib/voice/**.*',
'tools/server/tests/**.*'
]
@@ -67,33 +71,33 @@ jobs:
- name: Install dependencies
id: setup
run: npm ci
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Download built UI artifacts
uses: actions/download-artifact@v6
with:
name: llama-ui.zip
path: tools/ui/dist/
path: ui/apps/chat/dist/
- name: Run type checking
if: ${{ always() && steps.setup.conclusion == 'success' }}
run: npm run check
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Run linting
if: ${{ always() && steps.setup.conclusion == 'success' }}
run: npm run lint
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Run Client tests
if: ${{ always() && steps.setup.conclusion == 'success' }}
run: npm run test:client
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Run Unit tests
if: ${{ always() && steps.setup.conclusion == 'success' }}
run: npm run test:unit
working-directory: tools/ui
working-directory: ui/apps/chat
e2e-tests:
name: E2E Tests
@@ -109,25 +113,25 @@ jobs:
- name: Install dependencies
id: setup
run: npm ci
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Download built UI artifacts
uses: actions/download-artifact@v6
with:
name: llama-ui.zip
path: tools/ui/dist/
path: ui/apps/chat/dist/
- name: Build Storybook
if: ${{ always() && steps.setup.conclusion == 'success' }}
run: npm run build-storybook
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Run UI tests
if: ${{ always() && steps.setup.conclusion == 'success' }}
run: npm run test:ui -- --testTimeout=60000
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Run E2E tests
if: ${{ always() && steps.setup.conclusion == 'success' }}
run: npm run test:e2e
working-directory: tools/ui
working-directory: ui/apps/chat
+84
View File
@@ -0,0 +1,84 @@
name: UI Storybook
# Builds Llama UI and Llama Chat's Storybook (ui/apps/chat) and publishes it to its Hugging Face Space
# whenever the UI changes on master, so the Space always shows the latest build.
on:
workflow_dispatch:
push:
branches:
- master
paths: [
'.github/workflows/ui-storybook.yml',
'ui/apps/chat/**',
'ui/lib/ui/**'
]
cache-mode: none
permissions:
contents: read
# only the newest build matters: a newer push replaces a deploy still in progress
concurrency:
group: ${{ github.workflow }}
cancel-in-progress: true
jobs:
deploy:
name: Deploy to Hugging Face Spaces
if: github.repository == 'ggml-org/llama.cpp'
runs-on: hf-jobs-cpu-upgrade:ubuntu26_04
env:
HF_SPACE: ${{ vars.HF_SPACE_UI_STORYBOOK || 'ggml-org/llama-ui-storybook' }}
steps:
- name: Checkout code
uses: actions/checkout@v6
- name: Setup Node.js
uses: actions/setup-node@v6
with:
node-version: "24"
package-manager-cache: false
- name: Install dependencies
run: npm ci
working-directory: ui/apps/chat
- name: Build Storybook
run: npm run build-storybook
working-directory: ui/apps/chat
- name: Write the Space card
run: |
cat > ui/apps/chat/storybook-static/README.md << EOF
---
title: Llama UI Storybook
emoji: 🦙
colorFrom: gray
colorTo: gray
sdk: static
pinned: false
---
The Storybook of Llama UI and Llama Chat, built from [ggml-org/llama.cpp](https://github.com/ggml-org/llama.cpp) (\`ui/apps/chat\`) at ${GITHUB_SHA}.
EOF
- name: Install Hugging Face Hub CLI
run: |
sudo apt-get update
sudo apt-get install -y python3 python3-venv
python3 -m venv .venv-hf
.venv-hf/bin/pip install -U huggingface_hub==1.28.0
- name: Upload to the Space
env:
# a token with write access to the Space; HF_TOKEN_UI_STATIC_OUTPUT if that one has it
HF_TOKEN: ${{ secrets.HF_TOKEN_UI_STORYBOOK || secrets.HF_TOKEN_UI_STATIC_OUTPUT }}
run: |
# --delete '*' removes files the new build no longer has; the Space's .gitattributes is always kept
.venv-hf/bin/hf upload "$HF_SPACE" ui/apps/chat/storybook-static . \
--repo-type space \
--delete '*' \
--commit-message "Storybook from ggml-org/llama.cpp@${GITHUB_SHA::10}"
+91 -17
View File
@@ -13,7 +13,8 @@ on:
paths: [
'.github/workflows/ui.yml',
'.github/workflows/ui-build.yml',
'tools/ui/**.*',
'ui/apps/chat/**.*',
'ui/lib/**.*',
'tools/server/tests/**.*'
]
pull_request:
@@ -21,7 +22,8 @@ on:
paths: [
'.github/workflows/ui.yml',
'.github/workflows/ui-build.yml',
'tools/ui/**.*',
'ui/apps/chat/**.*',
'ui/lib/**.*',
'tools/server/tests/**.*'
]
@@ -66,46 +68,118 @@ jobs:
with:
node-version: "24"
# cache: "npm"
# cache-dependency-path: "tools/ui/package-lock.json"
# cache-dependency-path: "ui/apps/chat/package-lock.json"
package-manager-cache: false
- name: Download built UI artifacts
uses: actions/download-artifact@v6
with:
name: llama-ui.zip
path: tools/ui/dist/
path: ui/apps/chat/dist/
- name: Install dependencies
id: setup
if: ${{ steps.node.conclusion == 'success' }}
run: npm ci
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Run type checking
if: ${{ always() && steps.setup.conclusion == 'success' }}
run: npm run check
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Run linting
if: ${{ always() && steps.setup.conclusion == 'success' }}
run: npm run lint
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Check DESIGN.md is current with the tokens
if: ${{ always() && steps.setup.conclusion == 'success' }}
run: npm run check:design
working-directory: ui/apps/chat
- name: Install Playwright browsers
id: playwright
if: ${{ always() && steps.setup.conclusion == 'success' }}
run: npx playwright install --with-deps
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Run Client tests
if: ${{ always() && steps.playwright.conclusion == 'success' }}
run: npm run test:client
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Run Unit tests (uses pre-built dist/ from ui-build)
if: ${{ always() && steps.playwright.conclusion == 'success' }}
run: npm run test:unit
working-directory: tools/ui
working-directory: ui/apps/chat
lib-checks:
name: Library checks
runs-on: ubuntu-24.04
steps:
- name: Checkout code
uses: actions/checkout@v6
with:
ref: ${{ github.event.inputs.sha || github.event.pull_request.head.sha || github.sha || github.head_ref || github.ref_name }}
- name: Setup Node.js
uses: actions/setup-node@v6
with:
node-version: "24"
package-manager-cache: false
- name: Install dependencies
run: npm ci
working-directory: ui/lib/ui
- name: Run type checking
run: npm run check
working-directory: ui/lib/ui
- name: Install Playwright browsers
run: npx playwright install --with-deps chromium
working-directory: ui/lib/ui
# the unit tests and the Storybook stories (ui/lib/ui/tests/stories), which run as tests in Chromium
- name: Run tests
run: npm test
working-directory: ui/lib/ui
- name: Build the package
run: npm run prepack
working-directory: ui/lib/ui
voice-checks:
name: Voice checks
runs-on: ubuntu-24.04
steps:
- name: Checkout code
uses: actions/checkout@v6
with:
ref: ${{ github.event.inputs.sha || github.event.pull_request.head.sha || github.sha || github.head_ref || github.ref_name }}
- name: Setup Node.js
uses: actions/setup-node@v6
with:
node-version: "24"
package-manager-cache: false
- name: Install dependencies
run: npm ci
working-directory: ui/lib/voice
- name: Run type checking
run: npm run check
working-directory: ui/lib/voice
- name: Run tests
run: npm test
working-directory: ui/lib/voice
- name: Build the package
run: npm run prepack
working-directory: ui/lib/voice
e2e-tests:
name: E2E Tests
@@ -124,38 +198,38 @@ jobs:
with:
node-version: "24"
# cache: "npm"
# cache-dependency-path: "tools/ui/package-lock.json"
# cache-dependency-path: "ui/apps/chat/package-lock.json"
package-manager-cache: false
- name: Install dependencies
id: setup
if: ${{ steps.node.conclusion == 'success' }}
run: npm ci
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Download built UI artifacts (reuses ui-build)
uses: actions/download-artifact@v6
with:
name: llama-ui.zip
path: tools/ui/dist/
path: ui/apps/chat/dist/
- name: Install Playwright browsers
id: playwright
if: ${{ always() && steps.setup.conclusion == 'success' }}
run: npx playwright install --with-deps
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Build Storybook
if: ${{ always() && steps.playwright.conclusion == 'success' }}
run: npm run build-storybook
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Run UI tests
if: ${{ always() && steps.playwright.conclusion == 'success' }}
run: npm run test:ui -- --testTimeout=60000
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Run E2E tests (uses pre-built dist/ from ui-build)
if: ${{ always() && steps.playwright.conclusion == 'success' }}
run: npm run test:e2e
working-directory: tools/ui
working-directory: ui/apps/chat
+1 -1
View File
@@ -12,7 +12,7 @@ permissions:
jobs:
update:
name: Update Winget Package
runs-on: ubuntu-24.04 # previously ubuntu-latest
runs-on: ubuntu-26.04
if: github.repository_owner == 'ggml-org'
steps:
+6
View File
@@ -0,0 +1,6 @@
Project-specific instructions:
## Coding
- In `test-backend-ops.cpp`, do not mention specific backends (e.g. Metal, CUDA) in comments
-33
View File
@@ -1,33 +0,0 @@
You are a coding agent. Here are some very important rules that you must follow:
General:
- Be very precise and concise when writing code, comments, explanations, etc.
- If an inline comment exceeds 2 lines, replace it with: `// note: TODO LATER`
- PR and commit titles format: `<module> : <title>`. Lookup recents for examples
- Don't try to build or run the code unless you are explicitly asked to do so
- Use the `gh` CLI tool when querying PRs, issues, or other GitHub resources
- When [MODEL] is needed, first try to get it from the `PI_MODEL_NAME` env var before asking the user
- Never read the `AGENTS.md` file
Coding:
- When in doubt, always refer to the CONTRIBUTING.md file of the project
- In `test-backend-ops.cpp`, do not mention specific backends (e.g. Metal, CUDA) in comments
- When referencing issues or PRs in comments, use the format:
- C/C++ code: `// ref: <url>`
- Other (CMake, etc.): `# ref: <url>`
Pull requests (PRs):
- New branch names are prefixed with "gg/"
- Before opening a pull request, ask the user to confirm the description
- Don't explicitly wrap lines in the PR description (each paragraph and bullet is a single line)
- When creating a pull request, look for the repository's PR template and follow it
- For the AI usage disclosure section, write "YES. pi:llama.cpp/[MODEL]"
- If `PI_MODEL_NAME` env var is not set, ask the user to tell you what model was used and write it in place of [MODEL]
- Always create the pull requests in draft mode
- Never reply to review comments or post comments on issues/PRs without explicit permission from the user
Commits:
- On every commit that you make, include a "Assisted-by: pi:llama.cpp/[MODEL]" tag
- Do not explicitly set the git author in commits - rely on the default git config
- Always use `--no-gpg-sign` when committing
- Never `git push` without explicit confirmation from the user
+3
View File
@@ -246,3 +246,6 @@ Chat template and parser:
- [PEG parser](docs/development/parsing.md) - alternative to regex that llama.cpp uses to parse model's output
- [Auto parser](docs/autoparser.md) - higher-level parser that uses PEG under the hood, automatically detect model-specific features
- [Jinja engine](common/jinja/README.md)
Testing:
- [Slice model](skills/slice-model/SKILL.md) - build a small slice (few layers/experts) of a big HF model without downloading the full weights, to test conversion and inference
+285 -1
View File
File diff suppressed because it is too large. Load diff
+8 -4
View File
@@ -3,8 +3,8 @@ project("llama.cpp" C CXX)
include(CheckIncludeFileCXX)
### llama.cpp version
set(LLAMA_VERSION_MAJOR 0)
set(LLAMA_VERSION_MINOR 6)
set(LLAMA_VERSION_MAJOR 1)
set(LLAMA_VERSION_MINOR 0)
set(LLAMA_VERSION_PATCH 0)
set(LLAMA_VERSION_BASE "${LLAMA_VERSION_MAJOR}.${LLAMA_VERSION_MINOR}.${LLAMA_VERSION_PATCH}")
@@ -103,7 +103,7 @@ endif()
# subprocess spawning isn't a supported/sandbox-friendly operation on mobile OSes or in WASM
if (CMAKE_SYSTEM_NAME STREQUAL "iOS" OR CMAKE_SYSTEM_NAME STREQUAL "Android" OR ANDROID
OR CMAKE_SYSTEM_NAME STREQUAL "Emscripten" OR EMSCRIPTEN)
OR CMAKE_SYSTEM_NAME STREQUAL "Emscripten" OR EMSCRIPTEN OR CMAKE_SYSTEM_NAME STREQUAL "WASI")
set(LLAMA_SUBPROCESS_DEFAULT OFF)
else()
set(LLAMA_SUBPROCESS_DEFAULT ON)
@@ -134,8 +134,9 @@ option(LLAMA_BUILD_TOOLS "llama: build tools"
option(LLAMA_BUILD_EXAMPLES "llama: build examples" ${LLAMA_STANDALONE})
option(LLAMA_BUILD_SERVER "llama: build server example" ${LLAMA_STANDALONE})
option(LLAMA_BUILD_APP "llama: build the unified binary" ${LLAMA_STANDALONE})
option(LLAMA_BUILD_UI "llama: build the embedded Web UI for server" OFF)
option(LLAMA_BUILD_UI "llama: build Llama Chat, the embedded chat app, for server" OFF)
option(LLAMA_USE_PREBUILT_UI "llama: use prebuilt UI from HF Bucket when available" ON)
option(LLAMA_CONNECT "llama: include the llama-connect tool (downloads a prebuilt binary)" OFF)
option(LLAMA_TOOLS_INSTALL "llama: install tools" ${LLAMA_TOOLS_INSTALL_DEFAULT})
option(LLAMA_TESTS_INSTALL "llama: install tests" ON)
@@ -149,6 +150,7 @@ option(LLAMA_LLGUIDANCE "llama-common: include LLGuidance library for structured
# Required for relocatable CMake package
include(${CMAKE_CURRENT_SOURCE_DIR}/cmake/build-info.cmake)
include(${CMAKE_CURRENT_SOURCE_DIR}/cmake/common.cmake)
include(${CMAKE_CURRENT_SOURCE_DIR}/cmake/win-app.cmake)
if (NOT DEFINED LLAMA_BUILD_NUMBER)
set(LLAMA_BUILD_NUMBER ${BUILD_NUMBER})
@@ -268,6 +270,8 @@ if (LLAMA_BUILD_MTMD AND NOT (LLAMA_BUILD_COMMON AND LLAMA_BUILD_TOOLS))
add_subdirectory(tools/mtmd)
endif()
llama_apply_app_manifests()
#
# install
#
+25 -30
View File
@@ -2,20 +2,29 @@
# multiple collaborators per item can be specified
#
# ggml-org/ci : CISC, danbev, ggerganov, netrunnereve, ngxson, taronaeo
# ggml-org/ggml-base : aendk, fairydreaming, ggerganov
# ggml-org/ggml-cann : hipudding
# ggml-org/ggml-cuda : JohannesGaessler, am17an, IMbackK, ORippler
# ggml-org/ggml-cpu : angt, chaxu01, ggerganov, max-krasnyansky, taronaeo
# ggml-org/ggml-cuda : am17an, gaugarg-nv, IMbackK, JohannesGaessler, ORippler, thelittlefireman, ynankani
# ggml-org/ggml-et : marty1885
# ggml-org/ggml-hexagon : lhez, max-krasnyansky
# ggml-org/ggml-metal : ggerganov
# ggml-org/ggml-hip : IMbackK, thelittlefireman
# ggml-org/ggml-metal : forforever73, ggerganov, jhen0409, masterFoad
# ggml-org/ggml-opencl : lhez, max-krasnyansky
# ggml-org/ggml-rpc : rgerganov
# ggml-org/ggml-sycl : arthw
# ggml-org/ggml-openvino : ravi9, wine99
# ggml-org/ggml-riscv : xctan
# ggml-org/ggml-rpc : am17an, rgerganov
# ggml-org/ggml-sycl : arthw, Titaniumtown
# ggml-org/ggml-vulkan : 0cc4m, jeffbolznv
# ggml-org/ggml-webgpu : reeselevine, yomaytk
# ggml-org/ggml-zdnn : taronaeo
# ggml-org/llama-common : ggerganov, aldehir, angt, danbev, ngxson, pwilkin
# ggml-org/ggml-zdnn : AlekseiNikiforovIBM, Andreas-Krebbel, taronaeo
# ggml-org/llama-base : am17an, CISC, ggerganov, ngxson, sanmai, ServeurpersoCom
# ggml-org/llama-common : aldehir, angt, bartowski1182, danbev, ggerganov, ngxson, nikwen, pwilkin, ruixiang63, sanmai
# ggml-org/llama-examples : ggerganov
# ggml-org/llama-models : CISC, danbev, ggerganov, ngxson
# ggml-org/llama-mtmd : ngxson
# ggml-org/llama-server : ggerganov, ngxson, allozaur, angt, ServeurpersoCom
# ggml-org/llama-ui : allozaur
# ggml-org/llama-server : allozaur, angt, ggerganov, ngxson, ServeurpersoCom
# ggml-org/llama-ui : allozaur
/.devops/*.Dockerfile @ngxson
/.github/actions/ @ggml-org/ci
@@ -29,28 +38,12 @@
/conversion/ @CISC
/convert_*.py @CISC
/docs/backend/snapdragon/ @ggml-org/ggml-hexagon
/examples/batched.swift/ @ggerganov
/examples/batched/ @ggerganov
/examples/convert-llama2c-to-ggml/ @ggerganov
/examples/ @ggml-org/llama-examples
/examples/debug/ @danbev @pwilkin
/examples/deprecation-warning/ @ggerganov
/examples/diffusion/ @am17an
/examples/embedding/ @ggerganov
/examples/eval-callback/ @ggerganov
/examples/export-docs/ @ggerganov
/examples/gen-docs/ @ggerganov
/examples/gguf/ @ggerganov
/examples/llama.android/ @ggerganov @hanyin-arm @naco-siren
/examples/llama.swiftui/ @ggerganov
/examples/llama.vim @ggerganov
/examples/lookahead/ @ggerganov
/examples/lookup/ @JohannesGaessler
/examples/model-conversion/ @danbev
/examples/parallel/ @ggerganov
/examples/passkey/ @ggerganov
/examples/retrieval/ @ggerganov
/examples/speculative-simple/ @ggerganov
/examples/speculative/ @ggerganov
/ggml/cmake/ @ggerganov
/ggml/include/ @ggerganov
/ggml/src/ggml-backend-meta.cpp @JohannesGaessler
@@ -62,7 +55,7 @@
/ggml/src/ggml-cuda/ @ggml-org/ggml-cuda
/ggml/src/ggml-cuda/vendors/hip.h @IMbackK
/ggml/src/ggml-hexagon/ @ggml-org/ggml-hexagon
/ggml/src/ggml-hip/ @IMbackK
/ggml/src/ggml-hip/ @ggml-org/ggml-hip
/ggml/src/ggml-et/ @marty1885
/ggml/src/ggml-impl.h @ggerganov
/ggml/src/ggml-metal/ @ggml-org/ggml-metal
@@ -76,7 +69,7 @@
/ggml/src/ggml-virtgpu/ @kpouget
/ggml/src/ggml-vulkan/ @ggml-org/ggml-vulkan
/ggml/src/ggml-webgpu/ @ggml-org/ggml-webgpu
/ggml/src/ggml-zdnn/ @ggml-org/ggml-zdnn @Andreas-Krebbel @AlekseiNikiforovIBM
/ggml/src/ggml-zdnn/ @ggml-org/ggml-zdnn
/ggml/src/ggml-zendnn/ @avinashcpandey @Jiten1parmar
/ggml/src/ggml.c @ggerganov
/ggml/src/ggml.cpp @ggerganov
@@ -87,16 +80,18 @@
/scripts/get* @ggerganov
/scripts/sync* @ggerganov
/scripts/snapdragon/ @ggml-org/ggml-hexagon
/src/ @ggerganov
/src/ @ggml-org/llama-base
/src/llama-adapter.* @CISC
/src/llama-arch.* @CISC
/src/llama-chat.* @ngxson
/src/llama-graph.* @CISC
/src/llama-model.* @CISC
/src/llama-vocab.* @CISC
/src/models/ @CISC
/src/llama-moe-cache.* @am17an
/src/models/ @ggml-org/llama-models
/tests/ @ggerganov
/tests/test-chat.* @pwilkin
/tests/test-server-component.cpp @ggml-org/llama-server
/tools/batched-bench/ @ggerganov
/tools/cli/ @ngxson
/tools/completion/ @ggerganov
@@ -107,7 +102,7 @@
/tools/rpc/ @ggml-org/ggml-rpc
/tools/server/* @ggml-org/llama-server # no subdir
/tools/server/tests/ @ggml-org/llama-server
/tools/ui/ @ggml-org/llama-ui
/ui/ @ggml-org/llama-ui
/tools/tokenize/ @ggerganov
/tools/tts/ @ggerganov
/vendor/ @ggerganov
+12
View File
@@ -16,6 +16,8 @@ The project differentiates between 3 levels of contributors:
>
> Detailed information regarding permissible and restricted uses of AI can be found in the [AGENTS.md](AGENTS.md) file.
### Requirements
If AI is used to generate any portion of the code, contributors must adhere to the following requirements:
1. Explicitly disclose the manner in which AI was employed.
@@ -24,6 +26,16 @@ If AI is used to generate any portion of the code, contributors must adhere to t
4. Be prepared to explain every line of code you submit when asked about it by a maintainer.
5. It is strictly prohibited to use AI to write your posts for you (bug reports, feature requests, pull request descriptions, Github discussions, responding to humans, ...).
### Checklist
> [!IMPORTANT]
>
> **If you use AI, do all of the following:**
>
> 1. **Make sure [AGENTS.md](AGENTS.md) is loaded into your agent's context.** Skipping this step is one of the most common reasons PRs get rejected.
> 2. **Search for related discussions, PRs and issues** to get the broader context before writing any code.
> 3. **Run a local review pass before pushing anything**, using the [`code-review`](skills/code-review/SKILL.md) skill, once the change is implemented. Fix what it finds.
For more info, please refer to the [AGENTS.md](AGENTS.md) file.
# Pull requests (for contributors & collaborators)
+1738
View File
File diff suppressed because it is too large. Load diff
+30 -114
View File
@@ -1,100 +1,53 @@
# llama.cpp
![llama](https://raw.githubusercontent.com/ggml-org/llama.brand/refs/heads/master/cover/llama-cpp/cover-llama-cpp-dark.svg)
![llama.cpp](https://raw.githubusercontent.com/ggml-org/llama-brand/refs/heads/master/logo/svg/cover/llama-cpp/cover-llama-cpp-dark.svg)
<div align="center">
<b>LLM inference in C/C++</b>
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](https://opensource.org/licenses/MIT)
[![Release](https://img.shields.io/github/v/release/ggml-org/llama.cpp?filter=v*&color=brightgreen)](https://github.com/ggml-org/llama.cpp/releases?q=tag:v0)
[![Release](https://img.shields.io/github/v/release/ggml-org/llama.cpp?color=brightgreen)](https://github.com/ggml-org/llama.cpp/releases?q=tag:v1)
[![Nightly](https://img.shields.io/github/v/release/ggml-org/llama.cpp?label=nightly&filter=b*&color=orange)](https://github.com/ggml-org/llama.cpp/releases?q=b)
[![Server](https://img.shields.io/github/actions/workflow/status/ggml-org/llama.cpp/server.yml?label=Server)](https://github.com/ggml-org/llama.cpp/actions/workflows/server.yml)
[![Docker](https://img.shields.io/github/actions/workflow/status/ggml-org/llama.cpp/docker.yml?label=Docker)](https://github.com/ggml-org/llama.cpp/actions/workflows/docker.yml)
[![Winget](https://img.shields.io/github/actions/workflow/status/ggml-org/llama.cpp/winget.yml?label=Winget)](https://github.com/ggml-org/llama.cpp/actions/workflows/winget.yml)
[ggml](https://github.com/ggml-org/ggml) / [ops](https://github.com/ggml-org/llama.cpp/blob/master/docs/ops.md) / [maintainer PRs](https://github.com/ggml-org/llama.cpp/issues?q=is%3Apr%20is%3Aopen%20draft%3AFalse%20(author%3Argerganov%20OR%20author%3AKitaitiMakoto%20OR%20author%3Adanbev%20OR%20author%3Aaldehir%20OR%20author%3Amax-krasnyansky%20OR%20author%3ACISC%20OR%20author%3Aggerganov%20OR%20author%3Aam17an%20OR%20author%3Ajhen0409%20OR%20author%3Abartowski1182%20OR%20author%3Anikwen%20OR%20author%3Ahipudding%20OR%20author%3Aravi9%20OR%20author%3AServeurpersoCom%20OR%20author%3Apwilkin%20OR%20author%3Areeselevine%20OR%20author%3Angxson%20OR%20author%3Ajeffbolznv%20OR%20author%3Amarty1885%20OR%20author%3A0cc4m%20OR%20author%3ATitaniumtown%20OR%20author%3Aangt%20OR%20author%3AIMbackK%20OR%20author%3Aarthw%20OR%20author%3AJohannesGaessler%20OR%20author%3AORippler%20OR%20author%3Aruixiang63%20OR%20author%3Axctan%20OR%20author%3Aallozaur%20OR%20author%3Ayomaytk%20OR%20author%3Aaendk%20OR%20author%3Awine99%20OR%20author%3Agaugarg-nv%20OR%20author%3Ataronaeo%20OR%20author%3Aforforever73%20OR%20author%3Alhez%20OR%20author%3Anetrunnereve%20OR%20author%3Afairydreaming)%20sort%3Aupdated-desc) / [dev stats](https://github.com/ggml-org/llama.cpp-dev) / [lib llama API](https://github.com/ggml-org/llama.cpp/issues/9289) / [llama-server REST API](https://github.com/ggml-org/llama.cpp/issues/9291)
[v1.0.0](https://huggingface.co/blog/ggml-org/llama-cpp-v1) / [ggml](https://github.com/ggml-org/ggml) / [ops](https://github.com/ggml-org/llama.cpp/blob/master/docs/ops.md) / [maintainer PRs](https://github.com/ggml-org/llama.cpp/pulls?q=is%3Apr%20is%3Aopen%20draft%3AFalse%20(author%3Asanmai%20OR%20author%3Argerganov%20OR%20author%3AKitaitiMakoto%20OR%20author%3Adanbev%20OR%20author%3Aaldehir%20OR%20author%3Amax-krasnyansky%20OR%20author%3ACISC%20OR%20author%3Aggerganov%20OR%20author%3Aam17an%20OR%20author%3Ajhen0409%20OR%20author%3Abartowski1182%20OR%20author%3Anikwen%20OR%20author%3Ahipudding%20OR%20author%3Aravi9%20OR%20author%3AServeurpersoCom%20OR%20author%3Athelittlefireman%20OR%20author%3Apwilkin%20OR%20author%3Areeselevine%20OR%20author%3Angxson%20OR%20author%3Ajeffbolznv%20OR%20author%3Amarty1885%20OR%20author%3A0cc4m%20OR%20author%3ATitaniumtown%20OR%20author%3Aangt%20OR%20author%3AIMbackK%20OR%20author%3Aarthw%20OR%20author%3AJohannesGaessler%20OR%20author%3AORippler%20OR%20author%3Aruixiang63%20OR%20author%3Axctan%20OR%20author%3AmasterFoad%20OR%20author%3Aallozaur%20OR%20author%3AAndreas-Krebbel%20OR%20author%3Ayomaytk%20OR%20author%3Aaendk%20OR%20author%3Awine99%20OR%20author%3Agaugarg-nv%20OR%20author%3Ataronaeo%20OR%20author%3Aforforever73%20OR%20author%3Achaxu01%20OR%20author%3Alhez%20OR%20author%3AAlekseiNikiforovIBM%20OR%20author%3Anetrunnereve%20OR%20author%3Afairydreaming%20OR%20author%3Aynankani)%20sort%3Aupdated-desc) / [dev stats](https://github.com/ggml-org/llama.cpp-dev) / [lib llama API](https://github.com/ggml-org/llama.cpp/issues/9289) / [llama-server REST API](https://github.com/ggml-org/llama.cpp/issues/9291)
</div>
## This fork - Polaris / GCN tuning for large MoE models
## Description
Changes and measurements for running large MoE models with their experts offloaded to system RAM
(`--n-cpu-moe`) on an old GCN card. The two code changes below are auto-on, need no flag, and are
**token-identical** to mainline. Everything else here is tuning guidance.
The main goal of `llama.cpp` is to enable LLM (and MLLM) inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud.
Findings on an **RX 580 8GB** (Polaris / GCN, PCIe 3.0 x16, no fp16, no matrix cores) with
**Qwen3.5-35B-A3B Q4_K_M**, `-b 2048 -ub 2048`:
The `llama.cpp` project is built on top of the [ggml](https://github.com/ggml-org/ggml) library.
- **Flash-attention `mask_opt` is enabled for GCN large head sizes (this fork's own change).**
Upstream disables it on GCN; it is a **lossless** win in high-context prefill - it skips
fully-masked causal blocks and the per-block mask add on fully-visible ones, which is real work on
a card whose attention is compute-bound (no matrix cores). Auto-on, no flag. On Qwen3.5-35B
(head_dim 256): pp2048 **+8% @ 16k, +12% @ 32k**, growing with depth; perplexity bit-identical.
- **`-b 2048 -ub 2048` is the biggest prefill lever** (the default `-ub 512` roughly halves pp).
- **Tune `--n-cpu-moe` to context length.** Keep some expert layers resident in spare VRAM for short
prompts (e.g. `ncmoe 28` on the 35B, ~+5% over all-host); at long context the KV cache needs that
VRAM, so raise it (`ncmoe 40`, all experts on host). Keep flash attention on (`-fa 1`).
## This fork - RX 580 (Polaris / GCN) serving of large MoE models
**Recommended RX 580 / Polaris serving command** (per model):
A private fork maintained for one machine: an AMD RX 580 8 GB (RADV Vulkan) serving Qwen3.6-35B-A3B,
gemma-4-26B-A4B and smaller models under llama-swap, with the MoE experts in host RAM
(`--n-cpu-moe`). **[FORK.md](FORK.md) is the source of truth**: every fork change with its
invariants, rebase hazards and validation, the measured dead ends, the production configuration and
the measurement traps. The rebase and deploy procedure is
[docs/fork-rebase-runbook.md](docs/fork-rebase-runbook.md); `scripts/fork-validate.sh` validates a
build against a reference; `scripts/rx580-bench/` is the benchmark harness.
```bash
llama-server -hf <repo>:<quant> -ngl 99 --n-cpu-moe 40 -b 2048 -ub 2048 -fa 1
```
In short, the fork adds:
Lower `--n-cpu-moe` (e.g. 28) if the model plus your context budget leave spare VRAM; keep it high
for long-context / agentic use. At long context the bottleneck is attention compute (GPU-bound), so
`mask_opt` (above) is where the remaining prefill wins come from, not the MoE-transfer path.
- three Polaris prefill wins, on without a flag and token-identical to mainline: the MoE routing-ids
readback skip in `sched_copy_experts`, flash-attention `mask_opt` for GCN at head size >= 256, and
a tiled-transpose fast path for the delta-net conv-state concat;
- opt-in VRAM time-sharing between several llama-server processes (`LLAMA_SLEEP_VRAM_ONLY`,
`LLAMA_SLEEP_EVICT_KV`, `LLAMA_MMPROJ_ONDEMAND`, `LLAMA_VRAM_ARENA`): a model releases everything it
holds in VRAM and gets it back without a restart, so its RAM prompt cache survives;
- fixes to the server's sleep, wake and shutdown paths and to cold teardown that the sharing exposed;
- validation-only env knobs (`GGML_VK_CONCAT_FASTPATH`, `LLAMA_MOE_IDS_SKIP`), unset in production.
### Very large MoE (experts bigger than the GTT limit)
Measured on the same RX 580 with **Laguna-S-2.1 118B IQ2_M** (48 layers, 256 experts, 10 used,
experts 30.7 GiB of a 34.7 GiB file, all host-resident at `--n-cpu-moe 48`). The advice above changes
in this regime:
- **`-lm none` (formerly `--no-mmap`) stops being an option.** Its pinned host buffer is charged
against the amdgpu GTT limit (~31.4 GiB here, about half of system RAM). At 30.7 GiB of experts
the model no longer loads, and the allocation spike can OOM the box. Use mmap and accept the
staging copy.
- **The routing-ids readback is pure overhead at prefill batch sizes** and this fork now skips it.
To decide which experts to upload, the scheduler's expert-copy callback
(`llama_context::sched_copy_experts`) reads the ids back from the device that has just produced
them, forcing a full pipeline flush once per MoE layer per eval. With 2048 tokens x 10 experts over
256 experts every expert is used anyway. Skipping it is exact - `mul_mat_id` only reads the rows
the ids point at, and with `--moe-cache-mib` the cached experts are still copied device-to-device.
Auto-on at `4 * n_expert` ids and above; decode keeps the old path.
**pp2048 +4.7% @ 32k depth, +0.9% @ 16k, tg +3%.**
- **Where the time actually goes** (`GGML_VK_PERF_LOGGER=1`, depth 0, 22.65 s per 2048-token eval,
17.18 s of it GPU-busy so ~24% is H2D stall): expert `MUL_MAT_ID` **52%**, attention projections
**24%**, `FLASH_ATTN_EXT` **16%**, everything else 8%. The expert matmuls run at 1686-2139 GFLOP/s
while dense `MUL_MAT` q5_K/q6_K in the same graph reaches 3096-3691 - **the single largest
remaining opportunity on this hardware is closing that gap**, not the transfer path.
- **Interleaved SWA keeps its own small KV cache**, so a sliding-window layer costs the same at any
depth (`n_kv` pinned at `n_swa * n_seq_max + n_ubatch`). On this model 36 of 48 layers are O(1) in
depth and the entire high-context slowdown comes from the 12 full-attention layers.
- **`--parallel 1`** is worth setting for a solo large model: the server otherwise auto-selects 4
slots, and the SWA cache is sized `n_swa * n_seq_max + n_ubatch`, so 4 slots cost 4096 cells
instead of 2560. Measured **223 MiB of VRAM freed** at 64k context.
Dead ends measured on this hardware, recorded so they are not retried:
| Change | Result |
| --- | ---: |
| flash-attn `shmem_staging` enabled for GCN | **-6.7% @ 16k, -7.4% @ 32k** |
| `-b 4096 -ub 4096` (to amortize the fixed per-eval expert upload) | flat (-1%) |
| `--n-cpu-moe` 44 instead of 48 | +1.5%, but does not fit at 64k ctx |
| `mask_opt` gate relaxed below head_dim 256 | -18.5% @ 16k |
`shmem_staging` looks like a certain win (without it each rowgroup re-reads the whole K/V block
through a 16 KiB L1) but the `kvsh` stride of `D/4+1` dwords is 4 mod 32, which costs an 8-way LDS
bank conflict on wave64 - that `+1` padding is tuned for warp32. `-ub 4096` fails because halving the
number of expert uploads is exactly cancelled by intra-ubatch attention growing quadratically.
**Serving note that outweighs all of the above.** With a model this large, anything that restarts the
process is far more expensive than any kernel win: the server's prompt cache is RAM-only with no disk
backing, so a restart forces a full re-prefill of the conversation. If a model swapper can evict this
model to run a small helper model (chat-title generation and the like), fix that first - keeping the
process alive across a swap took a repeat turn from a 21,960 ms prefill down to 225 ms.
Production settings for the 35B are in FORK.md section 5 (`-lm none -t 6 -ngl 99 --n-cpu-moe 40
-b 2048 -ub 2048 -fa 1 --parallel 1`, ctx 131072). Keep all experts on the host (`--n-cpu-moe 40`)
for long contexts: lower values are faster only at short context and run out of VRAM as the KV grows
(FORK.md section 4).
## Quick start
@@ -130,50 +83,12 @@ llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF
<i>VLM session with <b>llama cli</b></i>
</td>
<td align="center">
<img width="1392" height="958" alt="Built-in web UI against `llama serve` running Qwen 3.6" src="https://github.com/user-attachments/assets/b402f972-2e32-4def-8771-8d849f08cf2e" />
<i>Built-in web UI against <b>llama serve</b></i>
<img width="1392" height="958" alt="Llama Chat, the built-in chat app, against `llama serve` running Qwen 3.6" src="https://github.com/user-attachments/assets/b402f972-2e32-4def-8771-8d849f08cf2e" />
<i>Llama Chat, the built-in chat app, against <b>llama serve</b></i>
</td>
</tr>
<table>
## Description
The main goal of `llama.cpp` is to enable LLM (and VLM) inference with minimal setup and state-of-the-art performance on
a wide range of hardware - locally and in the cloud.
- Plain C/C++ implementation without any dependencies
- Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks
- AVX, AVX2, AVX512 and AMX support for x86 architectures
- RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE support for RISC-V architectures
- 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use
- Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA)
- Vulkan and SYCL backend support
- CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity
The `llama.cpp` project is build on top of the [ggml](https://github.com/ggml-org/ggml) library.
## Supported backends
| Backend | Target devices |
| --- | --- |
| [BLAS](docs/build.md#blas-build) | All |
| [BLIS](docs/backend/BLIS.md) | All |
| [CANN](docs/build.md#cann) | Ascend NPU |
| [CUDA](docs/build.md#cuda) | Nvidia GPU |
| [HIP](docs/build.md#hip) | AMD GPU |
| [Hexagon](docs/backend/snapdragon/README.md) | Snapdragon |
| [IBM zDNN](docs/backend/zDNN.md) | IBM Z & LinuxONE |
| [MUSA](docs/build.md#musa) | Moore Threads GPU |
| [Metal](docs/build.md#metal-build) | Apple Silicon |
| [OpenCL](docs/backend/OPENCL.md) | Adreno GPU |
| [OpenVINO [In Progress]](docs/backend/OPENVINO.md) | Intel CPUs, GPUs, and NPUs |
| [RPC](https://github.com/ggml-org/llama.cpp/tree/master/tools/rpc) | All |
| [SYCL](docs/backend/SYCL.md) | Intel GPU |
| [VirtGPU](docs/backend/VirtGPU.md) | VirtGPU APIR |
| [Vulkan](docs/build.md#vulkan) | GPU |
| [WebGPU](docs/build.md#webgpu) | All |
| [ZenDNN](docs/build.md#zendnn) | AMD CPU |
## Documentation
#### Tools
@@ -181,6 +96,7 @@ The `llama.cpp` project is build on top of the [ggml](https://github.com/ggml-or
- [cli](tools/cli/README.md)
- [completion](tools/completion/README.md)
- [server](tools/server/README.md)
- [Text-to-speech (TTS) models](docs/multimodal/tts.md)
- [GBNF grammars](grammars/README.md)
#### Development
+1 -1
View File
@@ -46,7 +46,7 @@ Only vulnerabilities that fall within these parts of the project are considered
- `ggml/**/*`
- `gguf-py/**/*`
- `tools/server/*`, **excluding** the following topics:
- Web UI
- Llama Chat, the built-in chat app (`ui/apps/chat`)
- Features marked as experimental
- Features not recommended for use in untrusted environments (e.g., router, MCP)
- Bugs that can lead to Denial-of-Service attack
+88 -109
View File
@@ -122,6 +122,8 @@ if [ ! -z ${GG_BUILD_VULKAN} ]; then
MACOS_RUNNER_CUSTOM_SPIRV_HEADERS_LOCATION="${MACOS_RUNNER_CUSTOM_VULKAN_CMAKE_LOCATION}/SPIRV-Headers/SPIRV-HeadersConfig.cmake"
if [[ -f "${MACOS_RUNNER_CUSTOM_SPIRV_HEADERS_LOCATION}" || -h "${MACOS_RUNNER_CUSTOM_SPIRV_HEADERS_LOCATION}" ]]; then
CMAKE_EXTRA="${CMAKE_EXTRA} -DSPIRV-Headers_DIR=${MACOS_RUNNER_CUSTOM_VULKAN_CMAKE_LOCATION}/SPIRV-Headers"
# SPIRV-Headers walks up three directories and appends /include, yielding /usr/local/lib/include; the extra vulkan/ level likely breaks its prefix calculation.
CMAKE_EXTRA="${CMAKE_EXTRA} -DGGML_VULKAN_SPIRV_INCLUDE_DIR=/usr/local/include"
fi
fi
@@ -180,23 +182,33 @@ if [ ! -z ${GG_BUILD_OPENVINO} ]; then
CMAKE_EXTRA="${CMAKE_EXTRA} -DGGML_OPENVINO=ON"
# TODO: fix failing tests on OpenVINO backend
CTEST_EXTRA="-E test-llama-archs|^test-recurrent-state-|test-save-load-state"
CTEST_EXTRA="-E test-llama-archs|test-llama-context|test-fp8-kv"
fi
## helpers
# download a file if it does not exist or if it is outdated
# download a file unless already cached in $MNT
function gg_wget {
local out=$1
local url=$2
local cwd=`pwd`
local name=`basename $url`
local tmp=.gg-wget.$$.tmp
mkdir -p $out
cd $out
# should not re-download if file is the same
wget -nv -c -N $url
# cache first: HF artifacts are static, HF sends no Last-Modified for -N
if [ ! -s $name ]; then
# publish with mv: concurrent runners never read partial files
rm -rf $tmp && mkdir $tmp
(cd $tmp && wget -nv $url)
if [ -s $tmp/$name ]; then
mv -f $tmp/$name $name
fi
rm -rf $tmp
fi
cd $cwd
}
@@ -254,7 +266,7 @@ function gg_run_ctest_release {
(time cmake --build . --config Release -j$(nproc)) 2>&1 | tee -a $OUT/${ci}-make.log
if [ -z ${GG_BUILD_LOW_PERF} ]; then
(time ctest -C Release --output-on-failure -L 'main|python' ${CTEST_EXTRA}) 2>&1 | tee -a $OUT/${ci}-ctest.log
(time ctest -C Release --output-on-failure -L main ${CTEST_EXTRA}) 2>&1 | tee -a $OUT/${ci}-ctest.log
else
(time ctest -C Release --output-on-failure -L main -E test-opt ${CTEST_EXTRA}) 2>&1 | tee -a $OUT/${ci}-ctest.log
fi
@@ -286,25 +298,6 @@ function gg_run_test_llama_archs_tensor_split {
set +e
}
# test_llama_archs_models
function gg_run_test_llama_archs_models {
cd ${SRC}
set -e
# TODO: fix and re-enable `test-llama-archs` on OpenVINO
# TODO: the `test-llama-archs` currently does not build on Windows, so we check if the binary exists
if [ -z ${GG_BUILD_OPENVINO} ] && [ -f ./build-ci-release/bin/test-llama-archs ]; then
rm -rf build-ci-models && mkdir -p build-ci-models
# generate the dummy models used by the model-dependent tests
./build-ci-release/bin/test-llama-archs -o build-ci-models 2>&1
fi
set +e
}
# test_scripts
function gg_run_test_scripts {
@@ -312,8 +305,8 @@ function gg_run_test_scripts {
set -e
(cd ./tools/gguf-split && time bash tests.sh "$SRC/build-ci-release/bin" "$MNT/models") 2>&1 | tee -a $OUT/${ci}-scripts.log
(cd ./tools/quantize && time bash tests.sh "$SRC/build-ci-release/bin" "$MNT/models") 2>&1 | tee -a $OUT/${ci}-scripts.log
(cd ./tools/gguf-split && time bash tests.sh "$SRC/build-ci-release/bin" "$OUT/scratch") 2>&1 | tee -a $OUT/${ci}-scripts.log
(cd ./tools/quantize && time bash tests.sh "$SRC/build-ci-release/bin" "$OUT/scratch") 2>&1 | tee -a $OUT/${ci}-scripts.log
set +e
}
@@ -374,10 +367,14 @@ function gg_run_qwen3_0_6b {
gg_wget models-mnt/wikitext/ https://huggingface.co/datasets/ggml-org/ci/resolve/main/wikitext-2-raw-v1.zip
unzip -o models-mnt/wikitext/wikitext-2-raw-v1.zip -d models-mnt/wikitext/
rm -rf $OUT/scratch/qwen3-0.6B $OUT/scratch/wikitext
mkdir -p $OUT/scratch/qwen3-0.6B
unzip -o models-mnt/wikitext/wikitext-2-raw-v1.zip -d $OUT/scratch/wikitext/
path_models="../models-mnt/qwen3/0.6B"
path_wiki="../models-mnt/wikitext/wikitext-2-raw"
path_local="$OUT/scratch/qwen3-0.6B"
path_wiki="$OUT/scratch/wikitext/wikitext-2-raw"
rm -rf build-ci-release && mkdir build-ci-release && cd build-ci-release
@@ -386,72 +383,15 @@ function gg_run_qwen3_0_6b {
(cmake -G "${CMAKE_GENERATOR}" -DCMAKE_BUILD_TYPE=Release ${CMAKE_EXTRA} .. ) 2>&1 | tee -a $OUT/${ci}-cmake.log
(time cmake --build . --config Release -j$(nproc)) 2>&1 | tee -a $OUT/${ci}-make.log
python3 ../convert_hf_to_gguf.py ${path_models} --outfile ${path_models}/ggml-model-f16.gguf --outtype f16
python3 ../convert_hf_to_gguf.py ${path_models} --outfile ${path_models}/ggml-model-bf16.gguf --outtype bf16
python3 ../convert_hf_to_gguf.py ${path_models} --outfile ${path_local}/ggml-model-f16.gguf --outtype f16
python3 ../convert_hf_to_gguf.py ${path_models} --outfile ${path_local}/ggml-model-bf16.gguf --outtype bf16
model_f16="${path_models}/ggml-model-f16.gguf"
model_bf16="${path_models}/ggml-model-bf16.gguf"
model_q8_0="${path_models}/ggml-model-q8_0.gguf"
model_q4_0="${path_models}/ggml-model-q4_0.gguf"
model_q4_1="${path_models}/ggml-model-q4_1.gguf"
model_q5_0="${path_models}/ggml-model-q5_0.gguf"
model_q5_1="${path_models}/ggml-model-q5_1.gguf"
model_q2_k="${path_models}/ggml-model-q2_k.gguf"
model_q3_k="${path_models}/ggml-model-q3_k.gguf"
model_q4_k="${path_models}/ggml-model-q4_k.gguf"
model_q5_k="${path_models}/ggml-model-q5_k.gguf"
model_q6_k="${path_models}/ggml-model-q6_k.gguf"
model_f16="${path_local}/ggml-model-f16.gguf"
model_bf16="${path_local}/ggml-model-bf16.gguf"
model_q4_0="${path_local}/ggml-model-q4_0.gguf"
wiki_test="${path_wiki}/wiki.test.raw"
./bin/llama-quantize ${model_bf16} ${model_q8_0} q8_0 $(nproc)
./bin/llama-quantize ${model_bf16} ${model_q4_0} q4_0 $(nproc)
./bin/llama-quantize ${model_bf16} ${model_q4_1} q4_1 $(nproc)
./bin/llama-quantize ${model_bf16} ${model_q5_0} q5_0 $(nproc)
./bin/llama-quantize ${model_bf16} ${model_q5_1} q5_1 $(nproc)
./bin/llama-quantize ${model_bf16} ${model_q2_k} q2_k $(nproc)
./bin/llama-quantize ${model_bf16} ${model_q3_k} q3_k $(nproc)
./bin/llama-quantize ${model_bf16} ${model_q4_k} q4_k $(nproc)
./bin/llama-quantize ${model_bf16} ${model_q5_k} q5_k $(nproc)
./bin/llama-quantize ${model_bf16} ${model_q6_k} q6_k $(nproc)
(time ./bin/llama-fit-params --model ${model_f16} 2>&1 | tee -a $OUT/${ci}-fp-f16.log)
(time ./bin/llama-completion -no-cnv --model ${model_f16} -ngl 99 -c 1024 -s 1234 -n 64 --ignore-eos -p "I believe the meaning of life is" ) 2>&1 | tee -a $OUT/${ci}-tg-f16.log
(time ./bin/llama-completion -no-cnv --model ${model_bf16} -ngl 99 -c 1024 -s 1234 -n 64 --ignore-eos -p "I believe the meaning of life is" ) 2>&1 | tee -a $OUT/${ci}-tg-bf16.log
(time ./bin/llama-completion -no-cnv --model ${model_q8_0} -ngl 99 -c 1024 -s 1234 -n 64 --ignore-eos -p "I believe the meaning of life is" ) 2>&1 | tee -a $OUT/${ci}-tg-q8_0.log
(time ./bin/llama-completion -no-cnv --model ${model_q4_0} -ngl 99 -c 1024 -s 1234 -n 64 --ignore-eos -p "I believe the meaning of life is" ) 2>&1 | tee -a $OUT/${ci}-tg-q4_0.log
(time ./bin/llama-completion -no-cnv --model ${model_q4_1} -ngl 99 -c 1024 -s 1234 -n 64 --ignore-eos -p "I believe the meaning of life is" ) 2>&1 | tee -a $OUT/${ci}-tg-q4_1.log
(time ./bin/llama-completion -no-cnv --model ${model_q5_0} -ngl 99 -c 1024 -s 1234 -n 64 --ignore-eos -p "I believe the meaning of life is" ) 2>&1 | tee -a $OUT/${ci}-tg-q5_0.log
(time ./bin/llama-completion -no-cnv --model ${model_q5_1} -ngl 99 -c 1024 -s 1234 -n 64 --ignore-eos -p "I believe the meaning of life is" ) 2>&1 | tee -a $OUT/${ci}-tg-q5_1.log
(time ./bin/llama-completion -no-cnv --model ${model_q2_k} -ngl 99 -c 1024 -s 1234 -n 64 --ignore-eos -p "I believe the meaning of life is" ) 2>&1 | tee -a $OUT/${ci}-tg-q2_k.log
(time ./bin/llama-completion -no-cnv --model ${model_q3_k} -ngl 99 -c 1024 -s 1234 -n 64 --ignore-eos -p "I believe the meaning of life is" ) 2>&1 | tee -a $OUT/${ci}-tg-q3_k.log
(time ./bin/llama-completion -no-cnv --model ${model_q4_k} -ngl 99 -c 1024 -s 1234 -n 64 --ignore-eos -p "I believe the meaning of life is" ) 2>&1 | tee -a $OUT/${ci}-tg-q4_k.log
(time ./bin/llama-completion -no-cnv --model ${model_q5_k} -ngl 99 -c 1024 -s 1234 -n 64 --ignore-eos -p "I believe the meaning of life is" ) 2>&1 | tee -a $OUT/${ci}-tg-q5_k.log
(time ./bin/llama-completion -no-cnv --model ${model_q6_k} -ngl 99 -c 1024 -s 1234 -n 64 --ignore-eos -p "I believe the meaning of life is" ) 2>&1 | tee -a $OUT/${ci}-tg-q6_k.log
(time ./bin/llama-perplexity --model ${model_f16} -f ${wiki_test} -ngl 99 -c 1024 -b 512 --chunks 2 ) 2>&1 | tee -a $OUT/${ci}-tg-f16.log
if [ -z ${GG_BUILD_NO_BF16} ]; then
(time ./bin/llama-perplexity --model ${model_bf16} -f ${wiki_test} -ngl 99 -c 1024 -b 512 --chunks 2 ) 2>&1 | tee -a $OUT/${ci}-tg-bf16.log
fi
(time ./bin/llama-perplexity --model ${model_q8_0} -f ${wiki_test} -ngl 99 -c 1024 -b 512 --chunks 2 ) 2>&1 | tee -a $OUT/${ci}-tg-q8_0.log
(time ./bin/llama-perplexity --model ${model_q4_0} -f ${wiki_test} -ngl 99 -c 1024 -b 512 --chunks 2 ) 2>&1 | tee -a $OUT/${ci}-tg-q4_0.log
(time ./bin/llama-perplexity --model ${model_q4_1} -f ${wiki_test} -ngl 99 -c 1024 -b 512 --chunks 2 ) 2>&1 | tee -a $OUT/${ci}-tg-q4_1.log
(time ./bin/llama-perplexity --model ${model_q5_0} -f ${wiki_test} -ngl 99 -c 1024 -b 512 --chunks 2 ) 2>&1 | tee -a $OUT/${ci}-tg-q5_0.log
(time ./bin/llama-perplexity --model ${model_q5_1} -f ${wiki_test} -ngl 99 -c 1024 -b 512 --chunks 2 ) 2>&1 | tee -a $OUT/${ci}-tg-q5_1.log
(time ./bin/llama-perplexity --model ${model_q2_k} -f ${wiki_test} -ngl 99 -c 1024 -b 512 --chunks 2 ) 2>&1 | tee -a $OUT/${ci}-tg-q2_k.log
(time ./bin/llama-perplexity --model ${model_q3_k} -f ${wiki_test} -ngl 99 -c 1024 -b 512 --chunks 2 ) 2>&1 | tee -a $OUT/${ci}-tg-q3_k.log
(time ./bin/llama-perplexity --model ${model_q4_k} -f ${wiki_test} -ngl 99 -c 1024 -b 512 --chunks 2 ) 2>&1 | tee -a $OUT/${ci}-tg-q4_k.log
(time ./bin/llama-perplexity --model ${model_q5_k} -f ${wiki_test} -ngl 99 -c 1024 -b 512 --chunks 2 ) 2>&1 | tee -a $OUT/${ci}-tg-q5_k.log
(time ./bin/llama-perplexity --model ${model_q6_k} -f ${wiki_test} -ngl 99 -c 1024 -b 512 --chunks 2 ) 2>&1 | tee -a $OUT/${ci}-tg-q6_k.log
(time ./bin/llama-imatrix --model ${model_f16} -f ${wiki_test} -ngl 99 -c 1024 -b 512 --chunks 2 ) 2>&1 | tee -a $OUT/${ci}-imatrix.log
(time ./bin/test-save-load-state --model ${model_q4_0} -ngl 10 -c 1024 -fa off --no-op-offload) 2>&1 | tee -a $OUT/${ci}-save-load-state.log
(time ./bin/test-save-load-state --model ${model_q4_0} -ngl 10 -c 1024 -fa on --no-op-offload) 2>&1 | tee -a $OUT/${ci}-save-load-state.log
(time ./bin/test-save-load-state --model ${model_q4_0} -ngl 99 -c 1024 -fa off ) 2>&1 | tee -a $OUT/${ci}-save-load-state.log
(time ./bin/test-save-load-state --model ${model_q4_0} -ngl 99 -c 1024 -fa on ) 2>&1 | tee -a $OUT/${ci}-save-load-state.log
function check_ppl {
qnt="$1"
ppl=$(echo "$2" | grep -oE "[0-9]+\.[0-9]+" | tail -n 1)
@@ -465,20 +405,52 @@ function gg_run_qwen3_0_6b {
return 0
}
for qnt in q8_0 q4_0 q4_1 q5_0 q5_1 q2_k q3_k q4_k q5_k q6_k; do
model="${path_local}/ggml-model-${qnt}.gguf"
./bin/llama-quantize ${model_bf16} ${model} ${qnt} $(nproc)
(time ./bin/llama-completion -no-cnv --model ${model} -ngl 99 -c 1024 -s 1234 -n 64 --ignore-eos -p "I believe the meaning of life is" ) 2>&1 | tee -a $OUT/${ci}-tg-${qnt}.log
(time ./bin/llama-perplexity --model ${model} -f ${wiki_test} -ngl 99 -c 1024 -b 512 --chunks 2 ) 2>&1 | tee -a $OUT/${ci}-tg-${qnt}.log
# note: ppl > 20.0 for q2_k with this model
if [ "${qnt}" != "q2_k" ]; then
check_ppl "${qnt}" "$(cat $OUT/${ci}-tg-${qnt}.log | grep "^\[1\]")"
fi
if [ "${qnt}" = "q4_0" ]; then
# q4_0 is kept for other tests
q4_0="${path_models}/ggml-model-q4_0.gguf"
ln ${model} "$q4_0" 2>/dev/null || { cp ${model} $q4_0.tmp.$$ && mv -f $q4_0.tmp.$$ $q4_0; }
else
rm -f ${model}
fi
done
(time ./bin/llama-fit-params --model ${model_f16} 2>&1 | tee -a $OUT/${ci}-fp-f16.log)
(time ./bin/llama-completion -no-cnv --model ${model_f16} -ngl 99 -c 1024 -s 1234 -n 64 --ignore-eos -p "I believe the meaning of life is" ) 2>&1 | tee -a $OUT/${ci}-tg-f16.log
(time ./bin/llama-completion -no-cnv --model ${model_bf16} -ngl 99 -c 1024 -s 1234 -n 64 --ignore-eos -p "I believe the meaning of life is" ) 2>&1 | tee -a $OUT/${ci}-tg-bf16.log
(time ./bin/llama-perplexity --model ${model_f16} -f ${wiki_test} -ngl 99 -c 1024 -b 512 --chunks 2 ) 2>&1 | tee -a $OUT/${ci}-tg-f16.log
if [ -z ${GG_BUILD_NO_BF16} ]; then
(time ./bin/llama-perplexity --model ${model_bf16} -f ${wiki_test} -ngl 99 -c 1024 -b 512 --chunks 2 ) 2>&1 | tee -a $OUT/${ci}-tg-bf16.log
fi
(time ./bin/llama-imatrix --model ${model_f16} -f ${wiki_test} -ngl 99 -c 1024 -b 512 --chunks 2 ) 2>&1 | tee -a $OUT/${ci}-imatrix.log
(time ./bin/test-llama-context --model ${model_q4_0} -ngl 10 -c 1024 -fa off --no-op-offload) 2>&1 | tee -a $OUT/${ci}-test-llama-context.log
(time ./bin/test-llama-context --model ${model_q4_0} -ngl 99 -c 1024 -fa off ) 2>&1 | tee -a $OUT/${ci}-test-llama-context.log
(time ./bin/test-llama-context --model ${model_q4_0} -ngl 10 -c 1024 -fa on --no-op-offload) 2>&1 | tee -a $OUT/${ci}-test-llama-context.log
(time ./bin/test-llama-context --model ${model_q4_0} -ngl 99 -c 1024 -fa on ) 2>&1 | tee -a $OUT/${ci}-test-llama-context.log
check_ppl "f16" "$(cat $OUT/${ci}-tg-f16.log | grep "^\[1\]")"
if [ -z ${GG_BUILD_NO_BF16} ]; then
check_ppl "bf16" "$(cat $OUT/${ci}-tg-bf16.log | grep "^\[1\]")"
fi
check_ppl "q8_0" "$(cat $OUT/${ci}-tg-q8_0.log | grep "^\[1\]")"
check_ppl "q4_0" "$(cat $OUT/${ci}-tg-q4_0.log | grep "^\[1\]")"
check_ppl "q4_1" "$(cat $OUT/${ci}-tg-q4_1.log | grep "^\[1\]")"
check_ppl "q5_0" "$(cat $OUT/${ci}-tg-q5_0.log | grep "^\[1\]")"
check_ppl "q5_1" "$(cat $OUT/${ci}-tg-q5_1.log | grep "^\[1\]")"
#check_ppl "q2_k" "$(cat $OUT/${ci}-tg-q2_k.log | grep "^\[1\]")" # note: ppl > 20.0 for this quant and model
check_ppl "q3_k" "$(cat $OUT/${ci}-tg-q3_k.log | grep "^\[1\]")"
check_ppl "q4_k" "$(cat $OUT/${ci}-tg-q4_k.log | grep "^\[1\]")"
check_ppl "q5_k" "$(cat $OUT/${ci}-tg-q5_k.log | grep "^\[1\]")"
check_ppl "q6_k" "$(cat $OUT/${ci}-tg-q6_k.log | grep "^\[1\]")"
# scratch is rebuilt on every run; q4_0 is already published to $MNT
rm -rf $OUT/scratch/qwen3-0.6B $OUT/scratch/wikitext
set +e
}
@@ -501,6 +473,8 @@ function gg_run_embd_bge_small {
gg_wget models-mnt/bge-small/1_Pooling https://huggingface.co/BAAI/bge-small-en-v1.5/raw/main/1_Pooling/config.json
path_models="../models-mnt/bge-small"
path_local="$OUT/scratch/bge-small"
rm -rf $path_local && mkdir -p $path_local
rm -rf build-ci-release && mkdir build-ci-release && cd build-ci-release
@@ -509,10 +483,10 @@ function gg_run_embd_bge_small {
(cmake -G "${CMAKE_GENERATOR}" -DCMAKE_BUILD_TYPE=Release ${CMAKE_EXTRA} .. ) 2>&1 | tee -a $OUT/${ci}-cmake.log
(time cmake --build . --config Release -j$(nproc)) 2>&1 | tee -a $OUT/${ci}-make.log
python3 ../convert_hf_to_gguf.py ${path_models} --outfile ${path_models}/ggml-model-f16.gguf
python3 ../convert_hf_to_gguf.py ${path_models} --outfile ${path_local}/ggml-model-f16.gguf
model_f16="${path_models}/ggml-model-f16.gguf"
model_q8_0="${path_models}/ggml-model-q8_0.gguf"
model_f16="${path_local}/ggml-model-f16.gguf"
model_q8_0="${path_local}/ggml-model-q8_0.gguf"
./bin/llama-quantize ${model_f16} ${model_q8_0} q8_0
@@ -521,6 +495,8 @@ function gg_run_embd_bge_small {
(time ./bin/llama-embedding --model ${model_f16} -p "I believe the meaning of life is" -ngl 99 -c 0 --no-op-offload) 2>&1 | tee -a $OUT/${ci}-tg-f16.log
(time ./bin/llama-embedding --model ${model_q8_0} -p "I believe the meaning of life is" -ngl 99 -c 0 --no-op-offload) 2>&1 | tee -a $OUT/${ci}-tg-q8_0.log
rm -rf $path_local
set +e
}
@@ -537,6 +513,8 @@ function gg_run_rerank_tiny {
gg_wget models-mnt/rerank-tiny/ https://huggingface.co/jinaai/jina-reranker-v1-tiny-en/raw/main/vocab.json
path_models="../models-mnt/rerank-tiny"
path_local="$OUT/scratch/rerank-tiny"
rm -rf $path_local && mkdir -p $path_local
rm -rf build-ci-release && mkdir build-ci-release && cd build-ci-release
@@ -545,9 +523,9 @@ function gg_run_rerank_tiny {
(cmake -G "${CMAKE_GENERATOR}" -DCMAKE_BUILD_TYPE=Release ${CMAKE_EXTRA} .. ) 2>&1 | tee -a $OUT/${ci}-cmake.log
(time cmake --build . --config Release -j$(nproc)) 2>&1 | tee -a $OUT/${ci}-make.log
python3 ../convert_hf_to_gguf.py ${path_models} --outfile ${path_models}/ggml-model-f16.gguf
python3 ../convert_hf_to_gguf.py ${path_models} --outfile ${path_local}/ggml-model-f16.gguf
model_f16="${path_models}/ggml-model-f16.gguf"
model_f16="${path_local}/ggml-model-f16.gguf"
(time ./bin/llama-fit-params --model ${model_f16} 2>&1 | tee -a $OUT/${ci}-fp-f16.log)
@@ -577,6 +555,8 @@ function gg_run_rerank_tiny {
check_score "rerank score 1" "$(cat $OUT/${ci}-rk-f16.log | grep "rerank score 1")" "0.00" "0.05" | tee -a $OUT/${ci}-rk-f16.log
check_score "rerank score 2" "$(cat $OUT/${ci}-rk-f16.log | grep "rerank score 2")" "0.10" "0.30" | tee -a $OUT/${ci}-rk-f16.log
rm -rf $path_local
set +e
}
@@ -683,11 +663,11 @@ if [ -z ${GG_BUILD_LOW_PERF} ]; then
ln -sfn ${mnt_models} ${SRC}/models-mnt
# Create a fresh python3 venv and enter it
if ! python3 -m venv "$MNT/venv"; then
echo "Error: Failed to create Python virtual environment at $MNT/venv."
if ! python3 -m venv "$OUT/venv"; then
echo "Error: Failed to create Python virtual environment at $OUT/venv."
exit 1
fi
source "$MNT/venv/bin/activate"
source "$OUT/venv/bin/activate"
pip install -r ${SRC}/requirements.txt --disable-pip-version-check
pip install --editable gguf-py --disable-pip-version-check
@@ -700,7 +680,6 @@ test $ret -eq 0 && gg_run ctest_release
test $ret -eq 0 && gg_run test_backend_ops
test $ret -eq 0 && gg_run test_llama_archs_models
test $ret -eq 0 && gg_run test_llama_archs_tensor_split
if [ -z ${GG_BUILD_LOW_PERF} ]; then
+1 -1
View File
@@ -2,7 +2,7 @@ include("ggml/cmake/common.cmake")
function(llama_add_compile_flags)
if (LLAMA_FATAL_WARNINGS)
if (CMAKE_CXX_COMPILER_ID MATCHES "GNU" OR CMAKE_CXX_COMPILER_ID MATCHES "Clang")
if (CMAKE_CXX_COMPILER_ID MATCHES "GNU" OR CMAKE_CXX_COMPILER_ID MATCHES "Clang" OR CMAKE_CXX_COMPILER_ID MATCHES "IntelLLVM")
list(APPEND C_FLAGS -Werror)
list(APPEND CXX_FLAGS -Werror)
elseif (CMAKE_CXX_COMPILER_ID STREQUAL "MSVC")
+2 -2
View File
@@ -2,11 +2,11 @@ get_filename_component(DEST_DIR "${DEST}" DIRECTORY)
file(MAKE_DIRECTORY "${DEST_DIR}")
if(NOT EXISTS "${DEST}")
message(STATUS "Downloading ${NAME} from ggml-org/models...")
message(STATUS "Downloading ${NAME} from ${REPO}...")
endif()
file(DOWNLOAD
"https://huggingface.co/ggml-org/models/resolve/main/${NAME}?download=true"
"https://huggingface.co/${REPO}/resolve/main/${NAME}?download=true"
"${DEST}"
TLS_VERIFY ON
EXPECTED_HASH ${HASH}
+66
View File
@@ -0,0 +1,66 @@
set(LLAMA_WIN_APP_MANIFEST "${CMAKE_CURRENT_LIST_DIR}/../common/win-app.manifest")
if(WIN32)
if(MSVC OR CMAKE_CXX_SIMULATE_ID STREQUAL "MSVC")
set(LLAMA_WIN_APP_MSVC_ABI TRUE)
else()
set(LLAMA_WIN_APP_MSVC_ABI FALSE)
endif()
if(NOT LLAMA_WIN_APP_MSVC_ABI)
if(CMAKE_VERSION VERSION_LESS 3.21)
message(FATAL_ERROR "CMake 3.21 or newer is required to build for Windows with the GNU ABI")
endif()
enable_language(RC)
set(LLAMA_WIN_APP_MANIFEST_RC "${CMAKE_CURRENT_BINARY_DIR}/win-app-manifest.rc")
file(CONFIGURE OUTPUT "${LLAMA_WIN_APP_MANIFEST_RC}" CONTENT "1 24 \"@LLAMA_WIN_APP_MANIFEST@\"\n" @ONLY)
set_property(DIRECTORY APPEND PROPERTY CMAKE_CONFIGURE_DEPENDS "${LLAMA_WIN_APP_MANIFEST}")
endif()
endif()
function(llama_add_app_manifest target)
if(NOT WIN32)
return()
endif()
get_target_property(_aliased ${target} ALIASED_TARGET)
if(_aliased)
return()
endif()
get_target_property(_type ${target} TYPE)
if(NOT _type STREQUAL "EXECUTABLE")
return()
endif()
if(LLAMA_WIN_APP_MSVC_ABI)
target_sources(${target} PRIVATE "${LLAMA_WIN_APP_MANIFEST}")
else()
if(NOT TARGET llama-win-app-manifest-res)
add_library(llama-win-app-manifest-res OBJECT ${LLAMA_WIN_APP_MANIFEST_RC})
set_source_files_properties(${LLAMA_WIN_APP_MANIFEST_RC} PROPERTIES OBJECT_DEPENDS "${LLAMA_WIN_APP_MANIFEST}")
endif()
target_link_libraries(${target} PRIVATE "$<TARGET_OBJECTS:llama-win-app-manifest-res>")
endif()
endfunction()
function(llama_apply_app_manifests_dir dir)
get_property(_targets DIRECTORY "${dir}" PROPERTY BUILDSYSTEM_TARGETS)
foreach(_target IN LISTS _targets)
llama_add_app_manifest(${_target})
endforeach()
get_property(_subdirs DIRECTORY "${dir}" PROPERTY SUBDIRECTORIES)
foreach(_subdir IN LISTS _subdirs)
llama_apply_app_manifests_dir("${_subdir}")
endforeach()
endfunction()
function(llama_apply_app_manifests)
if(NOT WIN32)
return()
endif()
get_property(_subdirs DIRECTORY "${CMAKE_CURRENT_SOURCE_DIR}" PROPERTY SUBDIRECTORIES)
foreach(_dir IN LISTS _subdirs)
get_filename_component(_name "${_dir}" NAME)
if(_name STREQUAL "app" OR _name STREQUAL "examples" OR _name STREQUAL "pocs" OR _name STREQUAL "tests" OR _name STREQUAL "tools")
llama_apply_app_manifests_dir("${_dir}")
endif()
endforeach()
endfunction()
+9
View File
@@ -66,6 +66,8 @@ add_library(${TARGET}
chat-diff-analyzer.cpp
chat-peg-parser.cpp
chat-peg-parser.h
chat-tts-norm.cpp
chat-tts-norm.h
chat.cpp
chat.h
common.cpp
@@ -78,6 +80,8 @@ add_library(${TARGET}
download.h
fit.cpp
fit.h
file-lock.cpp
file-lock.h
hf-cache.cpp
hf-cache.h
http.h
@@ -184,3 +188,8 @@ if (LLAMA_LLGUIDANCE)
endif()
target_link_libraries(${TARGET} PUBLIC llama Threads::Threads)
if (CMAKE_SYSTEM_NAME MATCHES "AIX")
# AIX's flock() function comes from libbsd.a
target_link_libraries(${TARGET} PRIVATE -lbsd)
endif()
+197 -111
View File
@@ -4,6 +4,7 @@
#include "chat.h"
#include "common.h"
#include "download.h"
#include "http.h"
#include "json-schema-to-grammar.h"
#include "json.h"
#include "llama.h"
@@ -12,16 +13,6 @@
#include "speculative.h"
#include "preset.h"
// fix problem with std::min and std::max
#if defined(_WIN32)
#define WIN32_LEAN_AND_MEAN
#ifndef NOMINMAX
# define NOMINMAX
#endif
#include <windows.h>
#include <shellapi.h>
#endif
#include <algorithm>
#include <cinttypes>
#include <climits>
@@ -38,20 +29,6 @@
#include <thread> // for hardware_concurrency
#include <vector>
#ifndef __EMSCRIPTEN__
#ifdef __linux__
#include <linux/limits.h>
#elif defined(_WIN32)
# if !defined(PATH_MAX)
# define PATH_MAX MAX_PATH
# endif
#elif defined(_AIX)
#include <sys/limits.h>
#else
#include <sys/syslimits.h>
#endif
#endif
#define LLAMA_MAX_URL_LENGTH 2084 // Maximum URL Length in Chrome: 2083
using json = common_json;
@@ -305,6 +282,7 @@ const std::vector<ggml_type> kv_cache_types = {
GGML_TYPE_F32,
GGML_TYPE_F16,
GGML_TYPE_BF16,
GGML_TYPE_F8_E4M3,
GGML_TYPE_Q8_0,
GGML_TYPE_Q4_0,
GGML_TYPE_Q4_1,
@@ -351,7 +329,7 @@ static bool parse_bool_value(const std::string & value) {
static std::string get_default_local_path(const std::string & url) {
auto f = string_split<std::string>(url, '#').front();
f = string_split<std::string>(f, '?').front();
return fs_path_to_utf8(fs_get_cache_file(string_split<std::string>(f, '/').back()));
return fs_get_cache_file(string_split<std::string>(f, '/').back()).string();
}
static bool spec_types_is_default(const common_params & params) {
@@ -371,6 +349,14 @@ common_models_handler common_models_handler_init(const common_params & params, l
params.speculative.types.end(),
COMMON_SPECULATIVE_TYPE_DRAFT_DFLASH) != params.speculative.types.end();
const bool spec_type_draft_dflash2 = std::find(params.speculative.types.begin(),
params.speculative.types.end(),
COMMON_SPECULATIVE_TYPE_DRAFT_DFLASH2) != params.speculative.types.end();
const bool spec_type_draft_lilicorr = std::find(params.speculative.types.begin(),
params.speculative.types.end(),
COMMON_SPECULATIVE_TYPE_DRAFT_LILICORR) != params.speculative.types.end();
const bool spec_type_draft_eagle3 = std::find(params.speculative.types.begin(),
params.speculative.types.end(),
COMMON_SPECULATIVE_TYPE_DRAFT_EAGLE3) != params.speculative.types.end();
@@ -395,7 +381,7 @@ common_models_handler common_models_handler_init(const common_params & params, l
opts.offline = params.offline;
opts.download_mtp = spec_type_draft_mtp;
opts.download_eagle3 = spec_type_draft_eagle3;
opts.download_dflash = spec_type_draft_dflash;
opts.download_dflash = spec_type_draft_dflash || spec_type_draft_dflash2 || spec_type_draft_lilicorr;
opts.download_dspark = spec_type_draft_dspark;
opts.download_mmproj = use_mmproj && !params.no_mmproj
&& params.mmproj.path.empty() && params.mmproj.url.empty();
@@ -423,7 +409,26 @@ bool common_models_handler_is_preset_repo(const common_models_handler & handler)
return !handler.plan.preset.url.empty();
}
static common_download_opts download_opts_for_url(const common_download_opts & opts, const std::string & url) {
common_download_opts result = opts;
try {
const auto hf = common_http_parse_url(common_get_model_endpoint());
const auto parts = common_http_parse_url(url);
if (parts.scheme == hf.scheme &&
parts.host == hf.host &&
parts.port == hf.port &&
string_starts_with(result.bearer_token, "hf_")) {
return result;
}
} catch (const std::exception &) {
}
result.bearer_token.clear();
return result;
}
static std::vector<common_download_task> build_url_tasks(const common_params_model & model, common_download_opts opts) {
opts = download_opts_for_url(opts, model.url);
auto parts = common_download_get_all_parts(model.url);
std::vector<common_download_task> tasks;
@@ -470,6 +475,14 @@ void common_models_handler_apply(common_models_handler & handler, common_params
auto opts = handler.opts; // copy
opts.callback = callback;
// another process may hold the cache lock during finalization:
// let the download callback cancel that wait too
auto finalize = [callback](const hf_cache::hf_file & f) {
return hf_cache::finalize_file(f, [callback]() {
return !(callback && callback->is_cancelled());
});
};
// handle plain "url" if needed
auto handle_url = [&](common_params_model & model) {
if (!model.url.empty()) {
@@ -504,7 +517,7 @@ void common_models_handler_apply(common_models_handler & handler, common_params
common_download_task task;
task.url = params.mmproj.url;
task.local_path = params.mmproj.path;
task.opts = opts;
task.opts = download_opts_for_url(opts, task.url);
tasks.push_back(task);
}
bool had_spec_url = false;
@@ -512,13 +525,13 @@ void common_models_handler_apply(common_models_handler & handler, common_params
common_download_task task;
task.url = params.speculative.draft.mparams.url;
task.local_path = params.speculative.draft.mparams.path;
task.opts = opts;
task.opts = download_opts_for_url(opts, task.url);
tasks.push_back(task);
had_spec_url = true;
}
// handle hf_plan tasks
auto add_tasks = [&opts, &tasks](const hf_cache::hf_files & model_files,
auto add_tasks = [&opts, &tasks, &finalize](const hf_cache::hf_files & model_files,
const hf_cache::hf_file & primary,
common_params_model & model) {
for (size_t i = 0; i < model_files.size(); ++i) {
@@ -527,9 +540,9 @@ void common_models_handler_apply(common_models_handler & handler, common_params
tasks.emplace_back(model_file, opts, [&, is_primary]() {
if (is_primary) {
// the primary file is the first split (00001-of), use it as model path
model.path = hf_cache::finalize_file(model_file);
model.path = finalize(model_file);
} else {
hf_cache::finalize_file(model_file);
finalize(model_file);
}
});
}
@@ -581,9 +594,9 @@ void common_models_handler_apply(common_models_handler & handler, common_params
tasks.emplace_back(plan_spec.mtp, opts, [&]() {
// only use the discovered MTP head when no draft path is set yet
if (params.speculative.draft.mparams.path.empty()) {
params.speculative.draft.mparams.path = hf_cache::finalize_file(plan_spec.mtp);
params.speculative.draft.mparams.path = finalize(plan_spec.mtp);
} else {
hf_cache::finalize_file(plan_spec.mtp);
finalize(plan_spec.mtp);
}
});
}
@@ -591,9 +604,9 @@ void common_models_handler_apply(common_models_handler & handler, common_params
tasks.emplace_back(plan_spec.dflash, opts, [&]() {
// only use the discovered DFlash sidecar when no draft path is set yet
if (params.speculative.draft.mparams.path.empty()) {
params.speculative.draft.mparams.path = hf_cache::finalize_file(plan_spec.dflash);
params.speculative.draft.mparams.path = finalize(plan_spec.dflash);
} else {
hf_cache::finalize_file(plan_spec.dflash);
finalize(plan_spec.dflash);
}
});
}
@@ -601,9 +614,9 @@ void common_models_handler_apply(common_models_handler & handler, common_params
tasks.emplace_back(plan_spec.eagle3, opts, [&]() {
// only use the discovered Eagle3 sidecar when no draft path is set yet
if (params.speculative.draft.mparams.path.empty()) {
params.speculative.draft.mparams.path = hf_cache::finalize_file(plan_spec.eagle3);
params.speculative.draft.mparams.path = finalize(plan_spec.eagle3);
} else {
hf_cache::finalize_file(plan_spec.eagle3);
finalize(plan_spec.eagle3);
}
});
}
@@ -611,9 +624,9 @@ void common_models_handler_apply(common_models_handler & handler, common_params
tasks.emplace_back(plan_spec.dspark, opts, [&]() {
// only use the discovered DSpark sidecar when no draft path is set yet
if (params.speculative.draft.mparams.path.empty()) {
params.speculative.draft.mparams.path = hf_cache::finalize_file(plan_spec.dspark);
params.speculative.draft.mparams.path = finalize(plan_spec.dspark);
} else {
hf_cache::finalize_file(plan_spec.dspark);
finalize(plan_spec.dspark);
}
});
}
@@ -634,16 +647,16 @@ void common_models_handler_apply(common_models_handler & handler, common_params
}
if (!plan.mmproj.local_path.empty()) {
tasks.emplace_back(plan.mmproj, opts, [&]() {
params.mmproj.path = hf_cache::finalize_file(plan.mmproj);
params.mmproj.path = finalize(plan.mmproj);
});
}
if (!plan.mtp.local_path.empty() && !had_spec_url) {
tasks.emplace_back(plan.mtp, opts, [&]() {
// only fall back to the discovered MTP head when no draft was explicitly provided
if (params.speculative.draft.mparams.empty()) {
params.speculative.draft.mparams.path = hf_cache::finalize_file(plan.mtp);
params.speculative.draft.mparams.path = finalize(plan.mtp);
} else {
hf_cache::finalize_file(plan.mtp);
finalize(plan.mtp);
}
});
}
@@ -651,9 +664,9 @@ void common_models_handler_apply(common_models_handler & handler, common_params
tasks.emplace_back(plan.dflash, opts, [&]() {
// only fall back to the discovered DFlash sidecar when no draft was explicitly provided
if (params.speculative.draft.mparams.empty()) {
params.speculative.draft.mparams.path = hf_cache::finalize_file(plan.dflash);
params.speculative.draft.mparams.path = finalize(plan.dflash);
} else {
hf_cache::finalize_file(plan.dflash);
finalize(plan.dflash);
}
});
}
@@ -661,9 +674,9 @@ void common_models_handler_apply(common_models_handler & handler, common_params
tasks.emplace_back(plan.eagle3, opts, [&]() {
// only fall back to the discovered Eagle3 sidecar when no draft was explicitly provided
if (params.speculative.draft.mparams.empty()) {
params.speculative.draft.mparams.path = hf_cache::finalize_file(plan.eagle3);
params.speculative.draft.mparams.path = finalize(plan.eagle3);
} else {
hf_cache::finalize_file(plan.eagle3);
finalize(plan.eagle3);
}
});
}
@@ -671,9 +684,9 @@ void common_models_handler_apply(common_models_handler & handler, common_params
tasks.emplace_back(plan.dspark, opts, [&]() {
// only fall back to the discovered DSpark sidecar when no draft was explicitly provided
if (params.speculative.draft.mparams.empty()) {
params.speculative.draft.mparams.path = hf_cache::finalize_file(plan.dspark);
params.speculative.draft.mparams.path = finalize(plan.dspark);
} else {
hf_cache::finalize_file(plan.dspark);
finalize(plan.dspark);
}
});
}
@@ -681,7 +694,7 @@ void common_models_handler_apply(common_models_handler & handler, common_params
tasks.emplace_back(plan.preset, opts, [&]() {
// if HF repo is a preset repo, we simply run server in router mode with the preset.ini file
params.models_preset_hf = params.model.hf_repo; // only for showing a warning
params.models_preset = hf_cache::finalize_file(plan.preset);
params.models_preset = finalize(plan.preset);
// clear the model so the server starts in router mode
params.model.path.clear();
params.model.hf_repo.clear();
@@ -726,7 +739,7 @@ static void common_params_apply_system_config(common_params & params, llama_exam
std::vector<std::filesystem::path> paths;
#if defined(_WIN32)
const std::filesystem::path program_data = common_get_path_from_env("PROGRAMDATA");
const std::filesystem::path program_data = common_get_env("PROGRAMDATA");
if (!program_data.empty()) {
paths.push_back(program_data / "llama.cpp" / "config.ini");
}
@@ -754,7 +767,7 @@ static void common_params_apply_system_config(common_params & params, llama_exam
common_preset_context ctx(ex);
ctx.ignore_unknown_keys = true; // the same config file is shared by all programs
for (const auto & path : found) {
LOG_INF("using config file: %s\n", fs_path_to_utf8(path).c_str());
LOG_INF("using config file: %s\n", path.string().c_str());
common_preset global;
common_presets presets = ctx.load_from_ini(path, global);
global.apply_to_params(params);
@@ -899,6 +912,10 @@ static bool common_params_parse_ex(int argc, char ** argv, common_params_context
throw std::invalid_argument("error: --prompt-cache-all not supported in interactive mode yet\n");
}
if (params.sampling.ignore_eos && !common_grammar_value(params.sampling.grammar).empty()) {
throw std::invalid_argument("error: --ignore-eos is not supported with --grammar or --json-schema\n");
}
const bool skip_model_download =
// server will call common_params_handle_models() later, so we skip it here
ctx_arg.ex == LLAMA_EXAMPLE_SERVER ||
@@ -916,7 +933,7 @@ static bool common_params_parse_ex(int argc, char ** argv, common_params_context
// TODO @ngxson : maybe show a list of available models in CLI in this case
bool can_skip_model = params.usage || params.completion || !params.server_base.empty();
if (!can_skip_model && params.model.path.empty()) {
throw std::invalid_argument("error: --model is required\n");
throw std::invalid_argument("error: --model or -hf is required\n");
}
}
@@ -938,8 +955,8 @@ static bool common_params_parse_ex(int argc, char ** argv, common_params_context
}
const bool mcp_enabled = !params.mcp_servers_config.empty() || !params.mcp_servers_json.empty();
if ((!params.server_tools.empty() || mcp_enabled) && !params.cors_origins_explicit) {
LOG_WRN("server tools or MCP servers are enabled, using localhost as default CORS origin (change via --cors-origins)\n");
if ((!params.server_tools.empty() || mcp_enabled || params.ui_mcp_proxy) && !params.cors_origins_explicit) {
LOG_WRN("server tools, MCP servers or MCP proxy are enabled, using localhost as default CORS origin (change via --cors-origins)\n");
params.cors_origins = "localhost";
}
@@ -1251,44 +1268,7 @@ bool common_params_to_map(int argc, char ** argv, llama_example ex, std::map<com
return true;
}
#ifdef _WIN32
struct utf8_argv {
std::vector<std::string> buf;
std::vector<char*> ptrs;
};
static utf8_argv make_utf8_argv() {
utf8_argv out;
int wargc = 0;
LPWSTR* wargv = CommandLineToArgvW(GetCommandLineW(), &wargc);
if (!wargv) return out;
out.buf.reserve(wargc);
for (int i = 0; i < wargc; ++i) {
int n = WideCharToMultiByte(CP_UTF8, WC_ERR_INVALID_CHARS, wargv[i], -1, nullptr, 0, nullptr, nullptr);
if (n <= 0) { out.buf.emplace_back(); continue; }
auto& s = out.buf.emplace_back();
s.resize(static_cast<size_t>(n - 1));
(void)WideCharToMultiByte(CP_UTF8, 0, wargv[i], -1, s.data(), n, nullptr, nullptr);
}
LocalFree(wargv);
out.ptrs.reserve(out.buf.size() + 1);
for (auto& s : out.buf) out.ptrs.push_back(s.data());
out.ptrs.push_back(nullptr);
return out;
}
#endif
bool common_params_parse(int argc, char ** argv, common_params & params, llama_example ex, void(*print_usage)(int, char **)) {
#ifdef _WIN32
auto utf8 = make_utf8_argv();
// repair argv only when it matches the process command line
if (static_cast<int>(utf8.buf.size()) == argc) {
argv = utf8.ptrs.data();
}
#endif
auto ctx_arg = common_params_parser_init(params, ex, print_usage);
const common_params params_org = ctx_arg.params; // the example can modify the default params
@@ -1479,7 +1459,7 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
));
add_opt(common_arg(
{"--server-base"}, "URL",
string_format("connect to this server instead of starting a new one, example: 'http://localhost:8080' (default: none)"),
string_format("connect to this server instead of starting a new one, example: 'http://localhost:9931' (default: none)"),
[](common_params & params, const std::string & value) {
params.server_base = value;
}
@@ -1717,12 +1697,30 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
).set_env("LLAMA_ARG_CHECKPOINT_MIN_SPACING_NT").set_examples({LLAMA_EXAMPLE_SERVER}));
add_opt(common_arg(
{"-cram", "--cache-ram"}, "N",
string_format("set the maximum cache size in MiB (default: %d, -1 - no limit, 0 - disable)"
string_format("set the maximum cache size in MiB (default: %d, 0 for decision models, -1 - no limit, 0 - disable)"
"[(more info)](https://github.com/ggml-org/llama.cpp/pull/16391)", params.cache_ram_mib),
[](common_params & params, int value) {
params.cache_ram_mib = value;
params.cache_ram_explicit = true;
}
).set_env("LLAMA_ARG_CACHE_RAM").set_examples({LLAMA_EXAMPLE_SERVER, LLAMA_EXAMPLE_CLI}));
add_opt(common_arg(
{"--cache-disk-path"}, "PATH",
"save the state of each slot with a prompt to this directory on shutdown, and restore it for a prompt that starts with it after a restart (default: disabled)",
[](common_params & params, const std::string & value) {
params.cache_disk_path = value;
}
).set_env("LLAMA_ARG_CACHE_DISK_PATH").set_examples({LLAMA_EXAMPLE_SERVER}));
add_opt(common_arg(
{"--cache-disk-size"}, "N",
string_format("size limit of the --cache-disk-path directory in MiB, the least recently used files are removed (default: %d, 0 - no limit)", params.cache_disk_mib),
[](common_params & params, int value) {
if (value < 0) {
throw std::invalid_argument("cache-disk-size must be non-negative");
}
params.cache_disk_mib = value;
}
).set_env("LLAMA_ARG_CACHE_DISK_SIZE").set_examples({LLAMA_EXAMPLE_SERVER}));
add_opt(common_arg(
{"-kvu", "--kv-unified"},
{"-no-kvu", "--no-kv-unified"},
@@ -1730,7 +1728,7 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
[](common_params & params, bool value) {
params.kv_unified = value;
}
).set_env("LLAMA_ARG_KV_UNIFIED").set_examples({LLAMA_EXAMPLE_SERVER, LLAMA_EXAMPLE_PERPLEXITY, LLAMA_EXAMPLE_BATCHED, LLAMA_EXAMPLE_BENCH, LLAMA_EXAMPLE_PARALLEL}));
).set_env("LLAMA_ARG_KV_UNIFIED").set_examples({LLAMA_EXAMPLE_SERVER, LLAMA_EXAMPLE_PERPLEXITY, LLAMA_EXAMPLE_BATCHED, LLAMA_EXAMPLE_BENCH, LLAMA_EXAMPLE_PARALLEL, LLAMA_EXAMPLE_SPECULATIVE}));
add_opt(common_arg(
{"--cache-idle-slots"},
{"--no-cache-idle-slots"},
@@ -2532,15 +2530,6 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
params.ppl_output_type = value;
}
).set_examples({LLAMA_EXAMPLE_PERPLEXITY}));
add_opt(common_arg(
{"-dt", "--defrag-thold"}, "N",
string_format("KV cache defragmentation threshold (DEPRECATED)"),
[](common_params & params, const std::string & value) {
GGML_UNUSED(params);
GGML_UNUSED(value);
LOG_WRN("DEPRECATED: --defrag-thold is deprecated and no longer necessary to specify\n");
}
).set_env("LLAMA_ARG_DEFRAG_THOLD"));
if (ex == LLAMA_EXAMPLE_SERVER) {
// this is to make sure this option appears in the server-specific section of the help message
add_opt(common_arg(
@@ -2786,6 +2775,17 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
params.moe_cache_size = (size_t) value*1024*1024;
}
).set_env("LLAMA_ARG_MOE_CACHE_MIB"));
add_opt(common_arg(
{"--moe-cache-cpu-share"}, "F",
string_format("share of the MoE cache misses that the CPU computes from the host experts instead of uploading them to the GPU once the cache is full, from 0 to 1 (default: %.2f)", (double) params.moe_cache_cpu_share),
[](common_params & params, const std::string & value) {
const float share = std::stof(value);
if (share < 0.0f || share > 1.0f) {
throw std::invalid_argument("invalid value");
}
params.moe_cache_cpu_share = share;
}
).set_env("LLAMA_ARG_MOE_CACHE_CPU_SHARE"));
add_opt(common_arg(
{"-ncffn", "--n-cpu-ffn"}, "N",
"keep the dense FFN weights of the first N layers in the CPU\n"
@@ -2850,7 +2850,7 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
const std::regex regex{ R"([,/]+)" };
std::sregex_token_iterator it{ arg_next.begin(), arg_next.end(), regex, -1 };
std::vector<std::string> split_arg{ it, {} };
if (split_arg.size() >= llama_max_devices()) {
if (split_arg.size() > llama_max_devices()) {
throw std::invalid_argument(
string_format("got %zu input configs, but system only has %zu devices", split_arg.size(), llama_max_devices())
);
@@ -2916,7 +2916,7 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
const std::regex regex{ R"([,/]+)" };
std::sregex_token_iterator it{ arg_next.begin(), arg_next.end(), regex, -1 };
std::vector<std::string> split_arg{ it, {} };
if (split_arg.size() >= llama_max_devices()) {
if (split_arg.size() > llama_max_devices()) {
throw std::invalid_argument(
string_format("got %zu input configs, but system only has %zu devices", split_arg.size(), llama_max_devices())
);
@@ -3053,6 +3053,16 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
params.model.path = value;
}
).set_examples({LLAMA_EXAMPLE_COMMON, LLAMA_EXAMPLE_EXPORT_LORA, LLAMA_EXAMPLE_DOWNLOAD, LLAMA_EXAMPLE_TOKENIZE}).set_env("LLAMA_ARG_MODEL"));
add_opt(common_arg(
{"--ple", "--shard"}, "FNAME,...",
"extra GGUF shards for the model, comma separated; may be given more than once\n"
"paths may live anywhere, including on different drives; --ple is an alias for --shard",
[](common_params & params, const std::string & value) {
for (const auto & item : parse_csv_row(value)) {
params.model.shards.push_back(item);
}
}
).set_examples({LLAMA_EXAMPLE_COMMON}).set_env("LLAMA_ARG_SHARD"));
add_opt(common_arg(
{"-mu", "--model-url"}, "MODEL_URL",
"model download url (default: unused)",
@@ -3371,7 +3381,7 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
{"--cors-origins"}, "ORIGINS",
string_format(
"comma-separated list of allowed origins for CORS (default: %s)\n"
"if set to special value 'localhost', reflect the Origin header only if it is localhost",
"if set to special value 'localhost', only allow same-origin requests (e.g. the built-in UI)",
params.cors_origins.c_str()),
[](common_params & params, const std::string & value) {
params.cors_origins = value;
@@ -3427,11 +3437,20 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
add_opt(common_arg(
{"--ui-mcp-proxy", "--webui-mcp-proxy"},
{"--no-ui-mcp-proxy", "--no-webui-mcp-proxy"},
"experimental: whether to enable MCP CORS proxy - do not enable in untrusted environments (default: disabled)",
"experimental: whether to enable MCP CORS proxy, for Llama Chat to reach public MCP servers without CORS headers; local and private addresses are refused unless --ui-mcp-proxy-allow-private-urls is set - do not enable in untrusted environments (default: disabled)\n"
"note: for security reasons, this will limit --cors-origins to localhost by default",
[](common_params & params, bool value) {
params.ui_mcp_proxy = value;
}
).set_examples({LLAMA_EXAMPLE_SERVER}).set_env("LLAMA_ARG_UI_MCP_PROXY"));
add_opt(common_arg(
{"--ui-mcp-proxy-allow-private-urls"},
"allow the MCP CORS proxy to reach local or private addresses (e.g. MCP servers on localhost or LAN) - anyone who can reach the proxy can then reach your local network (default: disabled, only public addresses)\n"
"note: only has effect with --ui-mcp-proxy",
[](common_params & params) {
params.ui_mcp_proxy_allow_private_urls = true;
}
).set_examples({LLAMA_EXAMPLE_SERVER}).set_env("LLAMA_ARG_UI_MCP_PROXY_ALLOW_PRIVATE_URLS"));
add_opt(common_arg(
{"--tools"}, "TOOL1,TOOL2,...",
"experimental: whether to enable built-in tools for AI agents - do not enable in untrusted environments (default: no tools)\n"
@@ -3453,6 +3472,13 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
params.server_tools_runtime = value;
}
).set_examples({LLAMA_EXAMPLE_SERVER}).set_env("LLAMA_ARG_TOOLS_RUNTIME"));
add_opt(common_arg(
{"--allow-unauthenticated-tools"},
"allow server tools, MCP servers or MCP proxy on a non-loopback address without an API key - only for trusted networks (default: disabled)",
[](common_params & params) {
params.server_tools_allow_unauth = true;
}
).set_examples({LLAMA_EXAMPLE_SERVER}).set_env("LLAMA_ARG_ALLOW_UNAUTHENTICATED_TOOLS"));
add_opt(common_arg(
{"--mcp-servers-config"}, "PATH",
"experimental: path to JSON file with MCP server definitions (Cursor-compatible format) - do not enable in untrusted environments (default: none)\n"
@@ -3488,11 +3514,31 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
add_opt(common_arg(
{"--ui", "--webui"},
{"--no-ui", "--no-webui"},
string_format("whether to enable the Web UI (default: %s)", params.ui ? "enabled" : "disabled"),
string_format("whether to enable Llama Chat, the built-in chat app (default: %s)", params.ui ? "enabled" : "disabled"),
[](common_params & params, bool value) {
params.ui = value;
}
).set_examples({LLAMA_EXAMPLE_SERVER}).set_env("LLAMA_ARG_UI"));
add_opt(common_arg(
{"--connect"},
string_format("(experimental) open a peer-to-peer tunnel for server using llama-connect (default: %s)", params.server_connect ? "enabled" : "disabled"),
[](common_params & params) {
params.server_connect = true;
}
).set_examples({LLAMA_EXAMPLE_SERVER}).set_env("LLAMA_ARG_CONNECT"));
add_opt(common_arg(
{"--connect-code"}, "CODE",
"manually specify a 40-character code for --connect (default: generate a new one for each run)",
[](common_params & params, const std::string & value) {
std::string val = value;
string_replace_all(val, "-", "");
string_replace_all(val, " ", "");
if (val.size() != 40) {
throw std::invalid_argument(string_format("error: invalid connect code '%s', must be 40 characters\n", value.c_str()));
}
params.server_connect_code = val;
}
).set_examples({LLAMA_EXAMPLE_SERVER}).set_env("LLAMA_ARG_CONNECT_CODE"));
add_opt(common_arg(
{"--embedding", "--embeddings"},
string_format("restrict to only support embedding use case; use only with dedicated embedding models (default: %s)", params.embedding ? "enabled" : "disabled"),
@@ -3643,7 +3689,7 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
params.slot_save_path += DIRECTORY_SEPARATOR;
}
}
).set_examples({LLAMA_EXAMPLE_SERVER}));
).set_examples({LLAMA_EXAMPLE_SERVER}).set_env("LLAMA_ARG_SLOT_SAVE_PATH"));
add_opt(common_arg(
{"--media-path"}, "PATH",
"directory for loading local media files; files can be accessed via file:// URLs using relative paths (default: disabled)",
@@ -3658,6 +3704,13 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
}
}
).set_examples({LLAMA_EXAMPLE_SERVER}));
add_opt(common_arg(
{"--media-allow-private-urls"},
"allow http(s) media URLs (e.g. image_url) that point to local or private addresses (default: disabled, only public addresses)",
[](common_params & params) {
params.media_allow_private_urls = true;
}
).set_examples({LLAMA_EXAMPLE_SERVER}).set_env("LLAMA_ARG_MEDIA_ALLOW_PRIVATE_URLS"));
add_opt(common_arg(
{"--models-dir"}, "PATH",
"directory containing models for the router server (default: disabled)",
@@ -3679,6 +3732,13 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
params.models_max = value;
}
).set_examples({LLAMA_EXAMPLE_SERVER}).set_env("LLAMA_ARG_MODELS_MAX"));
add_opt(common_arg(
{"--models-memory-margin"}, "N",
string_format("for router server, MiB of device memory to keep free per device, unload LRU models until a new one fits (default: %d, 0 = disabled)", params.models_memory_margin),
[](common_params & params, int value) {
params.models_memory_margin = value;
}
).set_examples({LLAMA_EXAMPLE_SERVER}).set_env("LLAMA_ARG_MODELS_MEMORY_MARGIN"));
add_opt(common_arg(
{"--models-autoload"},
{"--no-models-autoload"},
@@ -4173,6 +4233,16 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
params.speculative.draft.n_min = value;
}
).set_spec().set_examples({LLAMA_EXAMPLE_SPECULATIVE, LLAMA_EXAMPLE_LOOKUP, LLAMA_EXAMPLE_SERVER, LLAMA_EXAMPLE_CLI}).set_env("LLAMA_ARG_SPEC_DRAFT_N_MIN"));
add_opt(common_arg(
{"--spec-draft-n-ubatch"}, "N",
string_format("physical maximum batch size of the draft context, use a smaller value to spare VRAM (default: %d - same as --n-ubatch)", params.speculative.draft.n_ubatch),
[](common_params & params, int value) {
if (value < 0) {
throw std::invalid_argument("invalid value");
}
params.speculative.draft.n_ubatch = value;
}
).set_spec().set_examples({LLAMA_EXAMPLE_SPECULATIVE, LLAMA_EXAMPLE_SERVER, LLAMA_EXAMPLE_CLI}).set_env("LLAMA_ARG_SPEC_DRAFT_N_UBATCH"));
add_opt(common_arg(
{"--spec-synth-len"}, "L",
"target mean synthetic acceptance length, including the target token (benchmarking only)",
@@ -4461,11 +4531,19 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
add_opt(common_arg(
{"--tts-lang"}, "FNAME",
"language (ISO 639-1) for audio generation\n"
"see tts/README.md for per-model usage notes",
"see docs/multimodal/tts.md for per-model usage notes",
[](common_params & params, const std::string & value) {
params.tts_lang = value;
}
).set_examples({LLAMA_EXAMPLE_TTS}));
).set_examples({LLAMA_EXAMPLE_TTS, LLAMA_EXAMPLE_SERVER}));
add_opt(common_arg(
{"--tts-voice"}, "VOICE",
"voice for audio generation (model-dependent)\n"
"see docs/multimodal/tts.md for per-model usage notes",
[](common_params & params, const std::string & value) {
params.tts_voice = value;
}
).set_examples({LLAMA_EXAMPLE_TTS, LLAMA_EXAMPLE_SERVER}));
add_opt(common_arg(
{"--tts-speaker-file"}, "FNAME",
"speaker file path for audio generation",
@@ -4473,6 +4551,14 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
params.tts_speaker_file = value;
}
).set_examples({LLAMA_EXAMPLE_TTS}));
add_opt(common_arg(
{"--tts-speaker-text"}, "TEXT",
"transcript of --tts-speaker-file, for models that need it to clone the voice\n"
"see docs/multimodal/tts.md for per-model usage notes",
[](common_params & params, const std::string & value) {
params.tts_speaker_text = value;
}
).set_examples({LLAMA_EXAMPLE_TTS}));
//
// diffusion params
+3 -4
View File
@@ -58,11 +58,10 @@ common_chat_params peg_generator::generate_parser(const common_chat_template &
data.generation_prompt += msg.render_content();
}
data.prompt += data.generation_prompt;
data.prompt.append(data.generation_prompt);
}
auto parser = autoparser.build_parser(inputs, parser_generation_prompt);
data.parser = parser.save();
data.parser = autoparser.build_parser(inputs, parser_generation_prompt);
// Build grammar if tools are present
bool has_tools =
@@ -78,7 +77,7 @@ common_chat_params peg_generator::generate_parser(const common_chat_template &
if (include_grammar) {
data.grammar_lazy = !has_response_format && inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_AUTO;
data.grammar = build_grammar([&](const common_grammar_builder & builder) {
parser.build_grammar(builder, data.grammar_lazy);
data.parser.build_grammar(builder, data.grammar_lazy);
});
// Set grammar triggers based on tool section markers (fall back to per-call markers)
+1 -1
View File
@@ -322,7 +322,7 @@ std::string apply_template(const common_chat_template & tmpl, const template_par
tmpl_params.extra_context["enable_thinking"] = params.enable_thinking;
try {
return common_chat_template_direct_apply(tmpl, tmpl_params);
return common_chat_template_direct_apply(tmpl, tmpl_params).str();
} catch (const std::exception & e) {
LOG_DBG("Template application failed: %s\n", e.what());
return ERR_TMPL;
+67 -49
View File
@@ -131,6 +131,12 @@ static std::string normalize_quotes_to_json(const std::string & input) {
continue;
}
// Models can emit raw control characters inside string literals; escape them to keep the JSON valid.
if ((in_single_quoted || in_double_quoted) && static_cast<unsigned char>(c) < 0x20) {
result += escape_json_string_inner(std::string(1, c));
continue;
}
// Handle quote characters
if (c == '"') {
if (in_single_quoted) {
@@ -348,7 +354,8 @@ void common_chat_peg_mapper::map(const common_peg_ast_node & node) {
}
if (is_tool_id && current_tool) {
auto text = trim_trailing_space(node.text);
auto sanitized = node.sanitized_text();
auto text = trim_trailing_space(sanitized);
if (text.size() >= 2 && text.front() == '"' && text.back() == '"') {
text = text.substr(1, text.size() - 2);
}
@@ -401,7 +408,7 @@ void common_chat_peg_mapper::map(const common_peg_ast_node & node) {
}
if ((is_arg_value || is_arg_string_value) && current_tool) {
std::string value_content = std::string(node.text);
std::string value_content = node.sanitized_text();
std::string value_to_add;
if (value_content.empty() && is_arg_string_value) {
@@ -555,7 +562,8 @@ common_peg_parser common_chat_peg_builder::python_or_json_value() {
common_peg_parser common_chat_peg_builder::python_style_tool_calls(
const ordered_json & tools,
bool parallel_tool_calls,
bool allow_json_literals) {
bool allow_json_literals,
bool allow_undeclared) {
if (!tools.is_array() || tools.empty()) {
return eps();
}
@@ -613,6 +621,18 @@ common_peg_parser common_chat_peg_builder::python_style_tool_calls(
tool_choices |= rule("tool-" + std::to_string(i), tool_parser);
}
// last, so declared tools win when parsing
if (allow_undeclared) {
auto ident = rule("tool-undeclared-ident", chars("[a-zA-Z_]", 1, 1) + chars("[a-zA-Z0-9_.]", 0, -1));
auto value = tool_arg_value(allow_json_literals ? python_or_json_value() : python_value());
auto arg = tool_arg(tool_arg_open(tool_arg_name(ident) + literal("=")) + value + tool_arg_close(eps()));
auto args = optional(arg + zero_or_more("," + space() + arg));
tool_choices |= rule("tool-undeclared", tool(tool_open(tool_name(ident) + literal("(")) +
space() + tool_args(args) + space() + tool_close(literal(")"))
));
}
if (parallel_tool_calls) {
return "[" + space() + tool_choices + zero_or_more("," + space() + tool_choices) + space() + "]";
}
@@ -966,6 +986,21 @@ void common_chat_peg_gemma4_mapper::from_ast(const common_peg_ast_arena & arena,
}
}
using gemma4 = common_chat_peg_gemma4_mapper;
// Collects the nearest descendants carrying tag, looking through the rules the parser
// wraps around the tagged nodes.
static void gemma4_collect(const common_peg_ast_arena & arena, const common_peg_ast_node & node, const std::string & tag, std::vector<common_peg_ast_id> & out) {
for (auto child_id : node.children) {
const auto & child = arena.get(child_id);
if (child.tag == tag) {
out.push_back(child_id);
} else {
gemma4_collect(arena, child, tag, out);
}
}
}
static std::string gemma4_to_json(const common_peg_ast_arena & arena, common_peg_ast_id id) {
const auto & node = arena.get(id);
@@ -973,51 +1008,45 @@ static std::string gemma4_to_json(const common_peg_ast_arena & arena, common_peg
return "";
}
if (node.rule == "gemma4-number" || node.rule == "gemma4-bool" || node.rule == "gemma4-null") {
if (node.tag == gemma4::NUMBER || node.tag == gemma4::BOOL || node.tag == gemma4::NULL_) {
return std::string(node.text);
}
if (node.rule == "gemma4-string-content") {
return escape_json_string_inner(std::string(node.text));
if (node.tag == gemma4::STRING_CONTENT) {
return escape_json_string_inner(node.sanitized_text());
}
if (node.rule == "gemma4-string") {
if (node.tag == gemma4::STRING) {
std::string result = "\"";
if (!node.children.empty()) {
result += gemma4_to_json(arena, node.children[0]);
if (!node.is_partial) {
result += "\"";
}
for (auto child_id : node.children) {
result += gemma4_to_json(arena, child_id);
}
if (!node.is_partial) {
result += "\"";
}
return result;
}
if (node.rule == "gemma4-array") {
if (node.tag == gemma4::ARRAY) {
std::string result = "[";
bool add_comma = false;
for (auto child_id : node.children) {
if (add_comma) {
for (size_t i = 0; i < node.children.size(); i++) {
if (i > 0) {
result += ',';
}
add_comma = true;
result += gemma4_to_json(arena, child_id);
result += gemma4_to_json(arena, node.children[i]);
}
if (!node.is_partial) {
result += ']';
}
return result;
}
if (node.rule == "gemma4-dict-key-name") {
return std::string(node.text);
}
if (node.tag == gemma4::DICT_KEY) {
auto name_id = arena.find_by_tag(node, gemma4::DICT_KEY_NAME);
if (node.rule == "gemma4-dict-key") {
std::string result = "\"";
if (!node.children.empty()) {
result += escape_json_string_inner(gemma4_to_json(arena, node.children[0]));
if (name_id != COMMON_PEG_INVALID_AST_ID) {
result += escape_json_string_inner(std::string(arena.get(name_id).text));
}
if (!node.is_partial) {
result += "\":";
@@ -1025,40 +1054,29 @@ static std::string gemma4_to_json(const common_peg_ast_arena & arena, common_peg
return result;
}
if (node.rule == "gemma4-dict-kv") {
std::string result;
for (auto child_id : node.children) {
result += gemma4_to_json(arena, child_id);
}
return result;
}
if (node.tag == gemma4::DICT) {
std::vector<common_peg_ast_id> members;
gemma4_collect(arena, node, gemma4::DICT_KV, members);
if (node.rule == "gemma4-dict") {
std::string result = "{";
bool add_comma = false;
for (auto child_id : node.children) {
if (add_comma) {
for (size_t i = 0; i < members.size(); i++) {
if (i > 0) {
result += ',';
}
add_comma = true;
result += gemma4_to_json(arena, child_id);
result += gemma4_to_json(arena, members[i]);
}
if (!node.is_partial) {
result += '}';
}
return result;
}
if (node.rule == "gemma4-value") {
if (!node.children.empty()) {
return gemma4_to_json(arena, node.children[0]);
}
return "";
// members, rules and other wrappers
std::string result;
for (auto child_id : node.children) {
result += gemma4_to_json(arena, child_id);
}
return "";
return result;
}
void common_chat_peg_gemma4_mapper::visit(const common_peg_ast_arena & arena, common_peg_ast_id id) {
@@ -1195,7 +1213,7 @@ static std::string minimax_m3_value_to_json(const common_peg_ast_arena & arena,
}
if (node.tag == common_chat_peg_builder::TOOL_ARG_STRING_VALUE) {
return "\"" + escape_json_string_inner(std::string(node.text)) + (closed ? "\"" : "");
return "\"" + escape_json_string_inner(node.sanitized_text()) + (closed ? "\"" : "");
}
// Numbers and booleans are written verbatim by the template
+15 -1
View File
@@ -34,6 +34,18 @@ class common_chat_peg_mapper {
class common_chat_peg_gemma4_mapper : public common_chat_peg_mapper {
public:
// Tags of the Gemma 4 argument syntax. The parser may wrap them in any rules.
static constexpr const char * STRING = "gemma4-string";
static constexpr const char * STRING_CONTENT = "gemma4-string-content";
static constexpr const char * NUMBER = "gemma4-number";
static constexpr const char * BOOL = "gemma4-bool";
static constexpr const char * NULL_ = "gemma4-null";
static constexpr const char * ARRAY = "gemma4-array";
static constexpr const char * DICT = "gemma4-dict";
static constexpr const char * DICT_KV = "gemma4-dict-kv";
static constexpr const char * DICT_KEY = "gemma4-dict-key";
static constexpr const char * DICT_KEY_NAME = "gemma4-dict-key-name";
common_chat_peg_gemma4_mapper(common_chat_msg & msg) : common_chat_peg_mapper(msg) {}
virtual void from_ast(const common_peg_ast_arena & arena, const common_peg_parse_result & result);
private:
@@ -149,9 +161,11 @@ class common_chat_peg_builder : public common_peg_parser_builder {
// Helper for Python-style function call format: name(arg1="value1", arg2=123)
// Used by LFM2 and similar templates
// allow_undeclared: also accept calls to names not in tools (any identifier, any arguments)
common_peg_parser python_style_tool_calls(const common_json & tools,
bool parallel_tool_calls,
bool allow_json_literals);
bool allow_json_literals,
bool allow_undeclared = false);
private:
// Python values plus JSON true/false/null.
+902
View File
@@ -0,0 +1,902 @@
#include "chat-tts-norm.h"
#include "jinja/runtime.h"
#include "jinja/value.h"
#include "json.h"
#include <algorithm>
#include <climits>
#include <stdexcept>
#include <string>
#include <vector>
// TTS normalization patterns, see docs/tts-norm-patterns.md for the specification
// keep this file in sync with the python reference: scripts/chat_tts_norm.py
using namespace jinja;
//
// character classes
// they are defined with explicit codepoint ranges, independent of any unicode database version,
// so that the C++ and python implementations agree on every codepoint
//
struct tts_norm_range {
uint32_t lo;
uint32_t hi;
};
static const tts_norm_range TTS_NORM_SPACE[] = {
{0x0009, 0x000D}, {0x0020, 0x0020}, {0x0085, 0x0085}, {0x00A0, 0x00A0}, {0x1680, 0x1680},
{0x2000, 0x200A}, {0x2028, 0x2029}, {0x202F, 0x202F}, {0x205F, 0x205F}, {0x3000, 0x3000},
};
static const tts_norm_range TTS_NORM_PUNCT[] = {
{0x0021, 0x002F}, {0x003A, 0x0040}, {0x005B, 0x0060}, {0x007B, 0x007E},
{0x00A1, 0x00A9}, {0x00AB, 0x00B4}, {0x00B6, 0x00B9}, {0x00BB, 0x00BF}, {0x00D7, 0x00D7}, {0x00F7, 0x00F7},
{0x2010, 0x2027}, {0x2030, 0x205E}, {0x20A0, 0x20CF}, {0x2100, 0x218F}, {0x2190, 0x2BFF},
{0x3001, 0x3003}, {0x3008, 0x3020}, {0x3030, 0x3030}, {0x303D, 0x303D}, {0x30FB, 0x30FB},
{0xFE10, 0xFE19}, {0xFE30, 0xFE4F}, {0xFE50, 0xFE6B},
{0xFF01, 0xFF0F}, {0xFF1A, 0xFF20}, {0xFF3B, 0xFF40}, {0xFF5B, 0xFF65},
{0x1F000, 0x1FAFF},
};
static const tts_norm_range TTS_NORM_CONTROL[] = {
{0x0000, 0x001F}, {0x007F, 0x009F}, {0x200B, 0x200F}, {0x2060, 0x206F}, {0xFEFF, 0xFEFF},
};
static const tts_norm_range TTS_NORM_HAN[] = {
{0x2E80, 0x2FDF}, {0x3005, 0x3005}, {0x3007, 0x3007}, {0x3021, 0x3029}, {0x3038, 0x303B},
{0x3400, 0x4DBF}, {0x4E00, 0x9FFF}, {0xF900, 0xFAFF}, {0x20000, 0x323AF},
};
template<size_t N>
static bool tts_norm_in(const tts_norm_range (&ranges)[N], uint32_t c) {
for (const auto & r : ranges) {
if (c >= r.lo && c <= r.hi) {
return true;
}
}
return false;
}
static bool tts_norm_is_alpha(uint32_t c) {
if ((c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z')) {
return true;
}
if (c == 0xAA || c == 0xB5 || c == 0xBA) {
return true;
}
return c >= 0xC0 && !tts_norm_in(TTS_NORM_SPACE, c) && !tts_norm_in(TTS_NORM_PUNCT, c) && !tts_norm_in(TTS_NORM_CONTROL, c);
}
// kind is one of "dspawulhc"
static bool tts_norm_class_match(char kind, uint32_t c) {
switch (kind) {
case 'd': return c >= '0' && c <= '9';
case 's': return tts_norm_in(TTS_NORM_SPACE, c);
case 'p': return tts_norm_in(TTS_NORM_PUNCT, c);
case 'a': return tts_norm_is_alpha(c);
case 'w': return tts_norm_is_alpha(c) || (c >= '0' && c <= '9');
case 'u': return c >= 'A' && c <= 'Z';
case 'l': return c >= 'a' && c <= 'z';
case 'h': return tts_norm_in(TTS_NORM_HAN, c);
case 'c': return tts_norm_in(TTS_NORM_CONTROL, c);
}
return false;
}
static bool tts_norm_is_class_letter(uint32_t c) {
if (c >= 'A' && c <= 'Z') {
c += 'a' - 'A';
}
return c < 0x80 && std::string("dspawulhc").find(static_cast<char>(c)) != std::string::npos;
}
static uint32_t tts_norm_ascii_lower(uint32_t c) {
return (c >= 'A' && c <= 'Z') ? c + ('a' - 'A') : c;
}
static uint32_t tts_norm_ascii_swapcase(uint32_t c) {
if (c >= 'A' && c <= 'Z') return c + ('a' - 'A');
if (c >= 'a' && c <= 'z') return c - ('a' - 'A');
return c;
}
static bool tts_norm_is_ascii_alnum(uint32_t c) {
return (c >= '0' && c <= '9') || (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z');
}
// invalid bytes are decoded as U+FFFD, one per byte; offs[i] is the byte offset of codepoint i, offs[n] = text size
static void tts_norm_decode_utf8(const std::string & s, std::vector<uint32_t> & cpts, std::vector<size_t> & offs) {
cpts.clear();
offs.clear();
size_t i = 0;
const size_t n = s.size();
while (i < n) {
offs.push_back(i);
const unsigned char c = static_cast<unsigned char>(s[i]);
uint32_t cp = 0;
size_t len = 0;
if (c < 0x80) {
cp = c; len = 1;
} else if ((c & 0xE0) == 0xC0) {
cp = c & 0x1F; len = 2;
} else if ((c & 0xF0) == 0xE0) {
cp = c & 0x0F; len = 3;
} else if ((c & 0xF8) == 0xF0) {
cp = c & 0x07; len = 4;
}
bool ok = len > 0 && i + len <= n;
for (size_t k = 1; ok && k < len; k++) {
const unsigned char cc = static_cast<unsigned char>(s[i + k]);
if ((cc & 0xC0) != 0x80) {
ok = false;
} else {
cp = (cp << 6) | (cc & 0x3F);
}
}
if (!ok) {
cpts.push_back(0xFFFD);
i += 1;
} else {
cpts.push_back(cp);
i += len;
}
}
offs.push_back(n);
}
//
// AST
//
enum class tts_norm_op {
LITERAL, // one codepoint
SET, // one codepoint from a set; class escapes like %d are sets with a single class
ANY, // any codepoint
START, // ^
END, // $
BOUNDARY, // %b
GROUP, // (...), (?:...), (?<name>...)
LOOKAHEAD, // (?=...), (?!...)
LOOKBEHIND, // (?<=c), (?<!c) - exactly one codepoint
};
struct tts_norm_class {
char kind;
bool neg;
};
struct tts_norm_set {
bool neg = false;
std::vector<std::pair<uint32_t, uint32_t>> ranges;
std::vector<tts_norm_class> classes;
};
struct tts_norm_item;
using tts_norm_seq = std::vector<tts_norm_item>;
struct tts_norm_pattern_node {
tts_norm_op op = tts_norm_op::GROUP;
uint32_t cp = 0; // LITERAL
tts_norm_set set; // SET
std::vector<tts_norm_seq> alts; // GROUP, LOOKAHEAD; LOOKBEHIND holds a single item
int capture = -1; // GROUP: capture index, -1 for non-capturing
bool neg = false; // LOOKAHEAD, LOOKBEHIND
};
static constexpr int TTS_NORM_INF = INT_MAX;
static constexpr int TTS_NORM_MAX_REPEAT = 1000;
static constexpr size_t TTS_NORM_MAX_PATTERN_LEN = 4096;
struct tts_norm_item {
tts_norm_pattern_node atom;
int min = 1;
int max = 1;
};
static bool tts_norm_is_zero_width(tts_norm_op op) {
return op == tts_norm_op::START || op == tts_norm_op::END || op == tts_norm_op::BOUNDARY
|| op == tts_norm_op::LOOKAHEAD || op == tts_norm_op::LOOKBEHIND;
}
//
// parser
//
struct tts_norm_parser {
std::vector<uint32_t> s;
size_t p = 0;
size_t & n_caps;
tts_norm_parser(std::vector<uint32_t> src, size_t & n_caps) : s(std::move(src)), n_caps(n_caps) {}
[[noreturn]] void fail(const std::string & msg) const {
throw std::runtime_error("invalid TTS norm pattern: " + msg + " at position " + std::to_string(p));
}
bool eof() const { return p >= s.size(); }
uint32_t peek() const { return s[p]; }
bool peek_is(uint32_t c) const { return !eof() && s[p] == c; }
void expect(uint32_t c, const char * what) {
if (!peek_is(c)) {
fail(std::string("expected ") + what);
}
p++;
}
std::vector<tts_norm_seq> parse_alts() {
std::vector<tts_norm_seq> alts;
alts.push_back(parse_seq());
while (peek_is('|')) {
p++;
alts.push_back(parse_seq());
}
return alts;
}
tts_norm_seq parse_seq() {
tts_norm_seq seq;
while (!eof() && peek() != '|' && peek() != ')') {
tts_norm_item item;
item.atom = parse_atom();
parse_quantifier(item);
seq.push_back(std::move(item));
}
return seq;
}
int parse_int() {
if (eof() || peek() < '0' || peek() > '9') {
fail("expected number in quantifier");
}
int v = 0;
while (!eof() && peek() >= '0' && peek() <= '9') {
v = v * 10 + static_cast<int>(peek() - '0');
if (v > TTS_NORM_MAX_REPEAT) {
fail("repetition count exceeds " + std::to_string(TTS_NORM_MAX_REPEAT));
}
p++;
}
return v;
}
void parse_quantifier(tts_norm_item & item) {
if (eof()) {
return;
}
const uint32_t c = peek();
if (c != '?' && c != '*' && c != '+' && c != '{') {
return;
}
if (tts_norm_is_zero_width(item.atom.op)) {
fail("zero-width assertion cannot be repeated");
}
p++;
if (c == '?') {
item.min = 0; item.max = 1;
} else if (c == '*') {
item.min = 0; item.max = TTS_NORM_INF;
} else if (c == '+') {
item.min = 1; item.max = TTS_NORM_INF;
} else {
item.min = parse_int();
item.max = item.min;
if (peek_is(',')) {
p++;
item.max = peek_is('}') ? TTS_NORM_INF : parse_int();
}
expect('}', "'}' to close quantifier");
if (item.max < item.min) {
fail("quantifier max is smaller than min");
}
}
if (!eof() && (peek() == '?' || peek() == '*' || peek() == '+' || peek() == '{')) {
fail("multiple quantifiers");
}
}
tts_norm_pattern_node parse_group() {
// '(' already consumed
tts_norm_pattern_node node;
node.op = tts_norm_op::GROUP;
if (peek_is('?')) {
p++;
if (eof()) {
fail("unterminated group");
}
const uint32_t c = peek();
if (c == ':') {
p++;
} else if (c == '=' || c == '!') {
p++;
node.op = tts_norm_op::LOOKAHEAD;
node.neg = c == '!';
} else if (c == '<' && p + 1 < s.size() && (s[p + 1] == '=' || s[p + 1] == '!')) {
node.op = tts_norm_op::LOOKBEHIND;
node.neg = s[p + 1] == '!';
p += 2;
tts_norm_item item;
if (eof() || peek() == ')') {
fail("lookbehind must contain exactly one codepoint matcher");
}
item.atom = parse_atom();
if (item.atom.op != tts_norm_op::LITERAL && item.atom.op != tts_norm_op::SET && item.atom.op != tts_norm_op::ANY) {
fail("lookbehind must contain exactly one codepoint matcher");
}
if (!peek_is(')')) {
fail("lookbehind must contain exactly one codepoint matcher");
}
p++;
node.alts.push_back(tts_norm_seq{});
node.alts[0].push_back(std::move(item));
return node;
} else if (c == 'i') {
fail("(?i) is only allowed at the start of the pattern");
} else {
fail("unknown group type");
}
} else {
node.capture = static_cast<int>(n_caps++);
}
node.alts = parse_alts();
expect(')', "')' to close group");
return node;
}
// after '%' inside or outside of a set
// returns true if it is a class (stored in cls), false if it is a literal (stored in lit)
bool parse_escape(bool in_set, tts_norm_class & cls, uint32_t & lit, bool & boundary) {
boundary = false;
if (eof()) {
fail("dangling '%'");
}
const uint32_t e = s[p++];
if (tts_norm_is_class_letter(e)) {
cls.kind = static_cast<char>(tts_norm_ascii_lower(e));
cls.neg = e >= 'A' && e <= 'Z';
return true;
}
if (e == 'b' && !in_set) {
boundary = true;
return false;
}
if (tts_norm_is_ascii_alnum(e)) {
p--;
fail(std::string("unknown escape '%") + static_cast<char>(e) + "'");
}
lit = e;
return false;
}
tts_norm_pattern_node parse_set() {
// '[' already consumed
tts_norm_pattern_node node;
node.op = tts_norm_op::SET;
if (peek_is('^')) {
node.set.neg = true;
p++;
}
bool empty = true;
while (true) {
if (eof()) {
fail("unterminated set");
}
if (peek() == ']') {
p++;
break;
}
empty = false;
uint32_t lo = 0;
if (peek() == '%') {
p++;
tts_norm_class cls{};
bool boundary = false;
if (parse_escape(true, cls, lo, boundary)) {
node.set.classes.push_back(cls);
continue;
}
} else {
lo = s[p++];
}
uint32_t hi = lo;
if (peek_is('-') && p + 1 < s.size() && s[p + 1] != ']') {
p++;
if (peek() == '%') {
p++;
tts_norm_class cls{};
bool boundary = false;
if (parse_escape(true, cls, hi, boundary)) {
fail("character class cannot be used in a range");
}
} else {
hi = s[p++];
}
if (hi < lo) {
fail("invalid range in set");
}
}
node.set.ranges.emplace_back(lo, hi);
}
if (empty) {
fail("empty set");
}
return node;
}
tts_norm_pattern_node parse_atom() {
const uint32_t c = s[p++];
tts_norm_pattern_node node;
switch (c) {
case '(':
return parse_group();
case '[':
return parse_set();
case '.':
node.op = tts_norm_op::ANY;
return node;
case '^':
node.op = tts_norm_op::START;
return node;
case '$':
node.op = tts_norm_op::END;
return node;
case '%': {
tts_norm_class cls{};
uint32_t lit = 0;
bool boundary = false;
if (parse_escape(false, cls, lit, boundary)) {
node.op = tts_norm_op::SET;
node.set.classes.push_back(cls);
} else if (boundary) {
node.op = tts_norm_op::BOUNDARY;
} else {
node.op = tts_norm_op::LITERAL;
node.cp = lit;
}
return node;
}
case '?':
case '*':
case '+':
case '{':
p--;
fail("nothing to repeat");
case ']':
case '}':
p--;
fail(std::string("unescaped '") + static_cast<char>(c) + "', use '%" + static_cast<char>(c) + "'");
default:
node.op = tts_norm_op::LITERAL;
node.cp = c;
return node;
}
}
};
//
// matcher
// PEG-style: alternatives are ordered (the first one that matches wins) and quantifiers are greedy and possessive
// (they never give back what they consumed), so there is no backtracking
//
struct tts_norm_matcher {
const std::vector<uint32_t> & t;
bool icase;
std::vector<std::pair<int64_t, int64_t>> caps;
bool is_w(size_t i) const {
return i < t.size() && tts_norm_class_match('w', t[i]);
}
bool set_match(const tts_norm_set & st, uint32_t c) const {
auto test = [&](uint32_t x) {
for (const auto & r : st.ranges) {
if (x >= r.first && x <= r.second) {
return true;
}
}
for (const auto & k : st.classes) {
if (tts_norm_class_match(k.kind, x) != k.neg) {
return true;
}
}
return false;
};
bool m = test(c);
if (!m && icase) {
const uint32_t o = tts_norm_ascii_swapcase(c);
if (o != c) {
m = test(o);
}
}
return m != st.neg;
}
bool match_alts(const std::vector<tts_norm_seq> & alts, size_t i, size_t & end) {
for (const auto & seq : alts) {
auto saved = caps;
if (match_seq(seq, i, end)) {
return true;
}
caps = std::move(saved);
}
return false;
}
bool match_seq(const tts_norm_seq & seq, size_t i, size_t & end) {
size_t pos = i;
for (const auto & it : seq) {
int count = 0;
while (count < it.max) {
const bool is_group = it.atom.op == tts_norm_op::GROUP;
std::vector<std::pair<int64_t, int64_t>> saved;
if (is_group) {
saved = caps;
}
size_t e = pos;
if (!match_once(it.atom, pos, e)) {
if (is_group) {
caps = std::move(saved);
}
break;
}
if (e == pos) {
// an empty iteration would repeat forever, it satisfies any remaining minimum
count = std::max(count + 1, it.min);
break;
}
pos = e;
count++;
}
if (count < it.min) {
return false;
}
}
end = pos;
return true;
}
bool match_once(const tts_norm_pattern_node & node, size_t i, size_t & end) {
const size_t n = t.size();
switch (node.op) {
case tts_norm_op::LITERAL:
if (i < n && (t[i] == node.cp || (icase && tts_norm_ascii_lower(t[i]) == tts_norm_ascii_lower(node.cp)))) {
end = i + 1;
return true;
}
return false;
case tts_norm_op::SET:
if (i < n && set_match(node.set, t[i])) {
end = i + 1;
return true;
}
return false;
case tts_norm_op::ANY:
if (i < n) {
end = i + 1;
return true;
}
return false;
case tts_norm_op::START:
end = i;
return i == 0;
case tts_norm_op::END:
end = i;
return i == n;
case tts_norm_op::BOUNDARY:
end = i;
return (i > 0 && is_w(i - 1)) != is_w(i);
case tts_norm_op::GROUP: {
size_t e = i;
if (!match_alts(node.alts, i, e)) {
return false;
}
if (node.capture >= 0) {
caps[node.capture] = {static_cast<int64_t>(i), static_cast<int64_t>(e)};
}
end = e;
return true;
}
case tts_norm_op::LOOKAHEAD: {
// captures made inside a lookahead are discarded
auto saved = caps;
size_t e = i;
const bool ok = match_alts(node.alts, i, e);
caps = std::move(saved);
end = i;
return ok != node.neg;
}
case tts_norm_op::LOOKBEHIND: {
size_t e = 0;
const bool ok = i > 0 && match_once(node.alts[0][0].atom, i - 1, e) && e == i;
end = i;
return ok != node.neg;
}
}
return false;
}
};
//
// common_tts_norm_pattern
//
common_tts_norm_pattern::common_tts_norm_pattern(const std::string & src) {
if (src.size() > TTS_NORM_MAX_PATTERN_LEN) {
throw std::runtime_error("invalid TTS norm pattern: pattern is longer than " + std::to_string(TTS_NORM_MAX_PATTERN_LEN) + " bytes");
}
std::vector<uint32_t> cpts;
std::vector<size_t> offs;
tts_norm_decode_utf8(src, cpts, offs);
tts_norm_parser parser(std::move(cpts), n_caps);
static const std::vector<uint32_t> icase_prefix = {'(', '?', 'i', ')'};
if (parser.s.size() >= 4 && std::equal(icase_prefix.begin(), icase_prefix.end(), parser.s.begin())) {
icase = true;
parser.p = 4;
}
root = std::make_unique<tts_norm_pattern_node>();
root->op = tts_norm_op::GROUP;
root->alts = parser.parse_alts();
if (!parser.eof()) {
parser.fail("unbalanced ')'");
}
}
common_tts_norm_pattern::~common_tts_norm_pattern() = default;
common_tts_norm_pattern::common_tts_norm_pattern(common_tts_norm_pattern &&) noexcept = default;
common_tts_norm_pattern & common_tts_norm_pattern::operator=(common_tts_norm_pattern &&) noexcept = default;
bool common_tts_norm_pattern::match_at(const std::vector<uint32_t> & cpts, size_t pos, size_t & end, std::vector<std::pair<int64_t, int64_t>> & groups) const {
tts_norm_matcher m{cpts, icase, std::vector<std::pair<int64_t, int64_t>>(n_caps, {-1, -1})};
size_t e = pos;
if (!m.match_alts(root->alts, pos, e)) {
return false;
}
end = e;
groups = std::move(m.caps);
return true;
}
size_t common_tts_norm_pattern::n_groups() const {
return n_caps;
}
//
// scan / sub
//
std::vector<common_tts_norm_segment> common_tts_norm_scan(const std::string & text, const std::vector<const common_tts_norm_pattern *> & rules) {
std::vector<uint32_t> cpts;
std::vector<size_t> offs;
tts_norm_decode_utf8(text, cpts, offs);
std::vector<common_tts_norm_segment> out;
const size_t n = cpts.size();
size_t i = 0;
size_t pending = 0; // start of the current unmatched run
std::vector<std::pair<int64_t, int64_t>> groups;
while (i < n) {
bool matched = false;
for (size_t r = 0; r < rules.size(); r++) {
size_t end = i;
if (rules[r]->match_at(cpts, i, end, groups) && end > i) {
if (pending < i) {
out.push_back({-1, offs[pending], offs[i], {}});
}
common_tts_norm_segment seg;
seg.rule = static_cast<int>(r);
seg.start = offs[i];
seg.end = offs[end];
for (const auto & g : groups) {
if (g.first < 0) {
seg.groups.emplace_back(std::string::npos, std::string::npos);
} else {
seg.groups.emplace_back(offs[g.first], offs[g.second]);
}
}
out.push_back(std::move(seg));
i = end;
pending = i;
matched = true;
break;
}
}
if (!matched) {
i++;
}
}
if (pending < n) {
out.push_back({-1, offs[pending], offs[n], {}});
}
return out;
}
struct tts_norm_repl_piece {
std::string literal;
int group = -1; // 0 = whole match, 1.. = groups
};
static std::vector<tts_norm_repl_piece> tts_norm_parse_repl(const std::string & repl, const common_tts_norm_pattern & pattern) {
std::vector<tts_norm_repl_piece> pieces;
std::string lit;
auto flush = [&]() {
if (!lit.empty()) {
pieces.push_back({lit, -1});
lit.clear();
}
};
for (size_t i = 0; i < repl.size(); i++) {
if (repl[i] != '%') {
lit += repl[i];
continue;
}
if (i + 1 >= repl.size()) {
throw std::runtime_error("invalid TTS norm replacement: dangling '%'");
}
const char c = repl[++i];
if (c == '%') {
lit += '%';
} else if (c >= '0' && c <= '9') {
const int g = c - '0';
if (static_cast<size_t>(g) > pattern.n_groups()) {
throw std::runtime_error("invalid TTS norm replacement: group %" + std::string(1, c) + " does not exist");
}
flush();
pieces.push_back({"", g});
} else {
throw std::runtime_error("invalid TTS norm replacement: unknown escape '%" + std::string(1, c) + "'");
}
}
flush();
return pieces;
}
std::string common_tts_norm_sub(const std::string & text, const common_tts_norm_pattern & pattern, const std::string & repl) {
const auto pieces = tts_norm_parse_repl(repl, pattern);
std::string out;
for (const auto & seg : common_tts_norm_scan(text, {&pattern})) {
if (seg.rule < 0) {
out.append(text, seg.start, seg.end - seg.start);
continue;
}
for (const auto & piece : pieces) {
if (piece.group < 0) {
out += piece.literal;
} else if (piece.group == 0) {
out.append(text, seg.start, seg.end - seg.start);
} else {
const auto & g = seg.groups[piece.group - 1];
if (g.first != std::string::npos) {
out.append(text, g.first, g.second - g.first);
}
}
}
}
return out;
}
//
// template builtins
//
// rules can be: a pattern string (rule name "match"), a dict {name: pattern}, or a list of [name, pattern] pairs
static std::vector<std::pair<std::string, std::string>> tts_norm_rules_from_value(const value & v) {
std::vector<std::pair<std::string, std::string>> rules;
if (is_val<value_string>(v)) {
rules.emplace_back("match", v->as_string().str());
} else if (is_val<value_object>(v)) {
for (const auto & kv : v->as_ordered_object()) {
if (!is_val<value_string>(kv.second)) {
throw std::runtime_error("tts_norm_scan: rule pattern must be a string");
}
rules.emplace_back(kv.first->as_string().str(), kv.second->as_string().str());
}
} else if (is_val<value_array>(v)) {
for (const auto & item : v->as_array()) {
if (!is_val<value_array>(item) || item->as_array().size() != 2
|| !is_val<value_string>(item->as_array()[0]) || !is_val<value_string>(item->as_array()[1])) {
throw std::runtime_error("tts_norm_scan: each rule must be a [name, pattern] pair of strings");
}
rules.emplace_back(item->as_array()[0]->as_string().str(), item->as_array()[1]->as_string().str());
}
} else {
throw std::runtime_error("tts_norm_scan: rules must be a string, a dict or a list of [name, pattern] pairs, got " + v->type());
}
return rules;
}
static value tts_norm_builtin_scan(const func_args & args) {
args.ensure_count(2, 2);
args.ensure_val<value_string>(args.get_pos(0));
const string text_val = args.get_pos(0)->as_string();
const std::string text = text_val.str();
const bool is_input = text_val.all_parts_are_input();
const auto rule_srcs = tts_norm_rules_from_value(args.get_pos(1));
std::vector<common_tts_norm_pattern> patterns;
patterns.reserve(rule_srcs.size());
std::vector<const common_tts_norm_pattern *> ptrs;
for (const auto & r : rule_srcs) {
patterns.emplace_back(r.second);
}
for (const auto & p : patterns) {
ptrs.push_back(&p);
}
auto mk_str = [&](size_t start, size_t end) -> value {
return mk_val<value_string>(string(text.substr(start, end - start), is_input));
};
auto out = mk_val<value_array>();
for (const auto & seg : common_tts_norm_scan(text, ptrs)) {
auto groups = mk_val<value_array>();
for (const auto & g : seg.groups) {
// groups that did not participate are empty strings, so they render as nothing and are falsy
groups->push_back(g.first == std::string::npos ? mk_str(0, 0) : mk_str(g.first, g.second));
}
auto obj = mk_val<value_object>();
obj->insert("rule", seg.rule < 0 ? value(mk_val<value_none>()) : value(mk_val<value_string>(rule_srcs[seg.rule].first)));
obj->insert("text", mk_str(seg.start, seg.end));
obj->insert("groups", groups);
out->push_back(obj);
}
return out;
}
static value tts_norm_builtin_sub(const func_args & args) {
args.ensure_count(3, 3);
args.ensure_vals<value_string, value_string, value_string>();
const string text_val = args.get_pos(0)->as_string();
const common_tts_norm_pattern pattern(args.get_pos(1)->as_string().str());
const std::string res = common_tts_norm_sub(text_val.str(), pattern, args.get_pos(2)->as_string().str());
return mk_val<value_string>(string(res, text_val.all_parts_are_input()));
}
bool common_tts_dict_find(const common_tts_dict & dict, const std::string & key, std::string & value) {
const auto it = std::lower_bound(dict.keys.begin(), dict.keys.end(), key);
if (it == dict.keys.end() || *it != key) {
return false;
}
const size_t idx = it - dict.keys.begin();
if (idx >= dict.values.size()) {
throw std::runtime_error("tts_dict: malformed dictionary, a key has no value");
}
value = dict.values[idx];
return true;
}
void common_chat_tts_norm_add_builtins(context & ctx, std::shared_ptr<const common_tts_dicts> dicts, common_tts_g2p g2p) {
ctx.set_val("tts_norm_scan", mk_val<value_func>("tts_norm_scan", tts_norm_builtin_scan));
ctx.set_val("tts_norm_sub", mk_val<value_func>("tts_norm_sub", tts_norm_builtin_sub));
ctx.set_val("tts_dict", mk_val<value_func>("tts_dict", [dicts](const func_args & args) -> value {
args.ensure_count(2, 2);
args.ensure_vals<value_string, value_string>();
const std::string name = args.get_pos(0)->as_string().str();
const auto it = dicts ? dicts->find(name) : common_tts_dicts::const_iterator();
if (!dicts || it == dicts->end()) {
throw std::runtime_error("tts_dict: unknown dictionary '" + name + "'");
}
std::string doc;
if (!common_tts_dict_find(it->second, args.get_pos(1)->as_string().str(), doc)) {
return mk_val<value_none>();
}
return value_from_json(common_json::parse(doc), false);
}));
// none when the caller gives no letter-to-sound model, or the model has no pronunciation
ctx.set_val("tts_g2p", mk_val<value_func>("tts_g2p", [g2p](const func_args & args) -> value {
args.ensure_count(1, 1);
args.ensure_vals<value_string>();
const std::string out = g2p ? g2p(args.get_pos(0)->as_string().str()) : std::string();
if (out.empty()) {
return mk_val<value_none>();
}
return mk_val<value_string>(out);
}));
}
+81
View File
@@ -0,0 +1,81 @@
#pragma once
#include <cstdint>
#include <functional>
#include <map>
#include <memory>
#include <string>
#include <utility>
#include <vector>
// TTS normalization patterns: a small, fully specified matching language for TTS text normalization templates
//
// it is NOT a regex engine: the semantics are defined by llama.cpp (see docs/tts-norm-patterns.md), so that this
// implementation and the python reference (scripts/chat_tts_norm.py) give the same result for every input
//
// functions exposed to task templates (e.g. "tts") via common_chat_tts_norm_add_builtins():
// tts_norm_scan(text, rules) -> list of segments {rule, text, groups}
// tts_norm_sub(text, pattern, repl) -> string
// tts_dict(name, key) -> value of key in a dictionary of the model, or none
namespace jinja {
struct context;
}
struct tts_norm_pattern_node;
class common_tts_norm_pattern {
public:
// throws std::runtime_error on invalid pattern
explicit common_tts_norm_pattern(const std::string & src);
~common_tts_norm_pattern();
common_tts_norm_pattern(common_tts_norm_pattern &&) noexcept;
common_tts_norm_pattern & operator=(common_tts_norm_pattern &&) noexcept;
// match anchored at codepoint index pos; on success, end is the codepoint index after the match
// groups[i] holds [start, end) codepoint indices of capture i+1, or {-1, -1} if the group did not participate
bool match_at(const std::vector<uint32_t> & cpts, size_t pos, size_t & end, std::vector<std::pair<int64_t, int64_t>> & groups) const;
size_t n_groups() const;
private:
std::unique_ptr<tts_norm_pattern_node> root;
size_t n_caps = 0;
bool icase = false;
};
// segment produced by common_tts_norm_scan(); byte offsets into the input text
struct common_tts_norm_segment {
int rule = -1; // index of the matched rule, -1 for unmatched text
size_t start = 0;
size_t end = 0;
std::vector<std::pair<size_t, size_t>> groups; // byte offsets; {npos, npos} for groups that did not participate
};
// split text into matched and unmatched segments
// at each position, rules are tried in order and the first non-empty match wins; otherwise one codepoint is skipped
std::vector<common_tts_norm_segment> common_tts_norm_scan(const std::string & text, const std::vector<const common_tts_norm_pattern *> & rules);
// replace every non-overlapping match of pattern with repl
// repl references: %0 (whole match), %1..%9 (groups), %% (literal percent)
std::string common_tts_norm_sub(const std::string & text, const common_tts_norm_pattern & pattern, const std::string & repl);
// a dictionary of the model, e.g. a pronunciation lexicon
struct common_tts_dict {
std::vector<std::string> keys; // sorted by their bytes
std::vector<std::string> values; // the JSON document of each key
};
using common_tts_dicts = std::map<std::string, common_tts_dict>;
// binary search of key in a dictionary; on success, value is set to the JSON document
bool common_tts_dict_find(const common_tts_dict & dict, const std::string & key, std::string & value);
// pronunciation of a word by a letter-to-sound model, as phonemes separated by spaces (e.g. CMUdict notation)
// empty when there is none; the notation is the one of the model, its "tts" template knows it
using common_tts_g2p = std::function<std::string(const std::string & word)>;
// add tts_norm_scan(), tts_norm_sub(), tts_dict() and tts_g2p() to the context of a task template
// they are not jinja global builtins on purpose: chat templates are not affected
void common_chat_tts_norm_add_builtins(jinja::context & ctx, std::shared_ptr<const common_tts_dicts> dicts = nullptr,
common_tts_g2p g2p = nullptr);
+528 -129
View File
@@ -3,12 +3,14 @@
#include "chat-auto-parser-helpers.h"
#include "chat-auto-parser.h"
#include "chat-peg-parser.h"
#include "chat-tts-norm.h"
#include "common.h"
#include "ggml.h"
#include "json-schema-to-grammar.h"
#include "json.h"
#include "log.h"
#include "parsers/parsers.h"
#include "sampling.h"
#include "jinja/value.h"
#include "jinja/runtime.h"
@@ -112,38 +114,6 @@ const char * common_chat_role_to_string(common_chat_role role) {
return "";
}
json common_chat_msg_delimiters::to_json() const {
json result = json::array();
for (const auto & d : delimiters) {
result.push_back({
{ "role", common_chat_role_to_string(d.role) },
{ "delimiter", d.delimiter },
});
}
return result;
}
common_chat_msg_delimiters common_chat_msg_delimiters_parse(const json & delimiters) {
common_chat_msg_delimiters result;
if (!delimiters.is_array()) {
return result;
}
result.delimiters.reserve(delimiters.size());
for (const auto & d : delimiters) {
if (!d.is_object()) {
continue;
}
result.delimiters.push_back({
common_chat_role_from_string(d.value("role", std::string())),
d.value("delimiter", std::string()),
});
}
return result;
}
void common_chat_msg_delimiters::tokenize(const llama_vocab * vocab) {
for (auto & d : delimiters) {
d.tokens = common_tokenize(vocab, d.delimiter, false, true);
@@ -221,10 +191,17 @@ json common_chat_msg::to_json_oaicompat(bool concat_typed_text) const {
} else {
auto & parts = jmsg["content"] = json::array();
for (const auto & part : content_parts) {
parts.push_back({
{"type", part.type},
{"text", part.text},
});
if (part.type == "tool_reference") {
parts.push_back({
{"type", part.type},
{"name", part.name},
});
} else {
parts.push_back({
{"type", part.type},
{"text", part.text},
});
}
}
}
} else {
@@ -340,6 +317,8 @@ struct common_chat_templates {
bool has_explicit_template; // Model had builtin template or template overridden was specified.
std::unique_ptr<common_chat_template> template_default; // always set (defaults to chatml)
std::unique_ptr<common_chat_template> template_tool_use;
std::unique_ptr<common_chat_template> template_tts; // the prompt of a text to speak
std::shared_ptr<common_tts_dicts> tts_dicts; // dictionaries of the model for tts_dict(), by name
};
common_chat_tool_choice common_chat_tool_choice_parse_oaicompat(const std::string & tool_choice) {
@@ -401,12 +380,16 @@ std::vector<common_chat_msg> common_chat_msgs_parse_oaicompat(const json & messa
throw std::invalid_argument("Missing content part type: " + part.dump());
}
const auto & type = part.at("type");
if (type != "text" && type != "media_marker") {
throw std::invalid_argument("Unsupported content part type: " + type.dump());
}
common_chat_msg_content_part msg_part;
msg_part.type = type;
msg_part.text = part.at("text");
if (type == "text" || type == "media_marker") {
msg_part.text = part.at("text");
} else if (type == "tool_reference") {
// names a request tool, the template renders its definition
msg_part.name = part.at("name");
} else {
throw std::invalid_argument("Unsupported content part type: " + type.dump());
}
msg.content_parts.push_back(msg_part);
}
} else if (!content.is_null()) {
@@ -459,6 +442,41 @@ std::vector<common_chat_msg> common_chat_msgs_parse_oaicompat(const json & messa
if (message.contains("tool_call_id")) {
msg.tool_call_id = message.at("tool_call_id");
}
{
json extra = json::object();
for (const auto & [k, v] : message.items()) {
if (k != "role" && k != "content" && k != "tool_calls" && k != "reasoning_content" &&
k != "name" && k != "tool_call_id") {
extra[k] = v;
}
}
if (has_tool_calls) {
// per tool call: keys besides type/function/id, and function keys besides name/arguments
json tc_extra = json::array();
bool any = false;
for (const auto & tool_call : message.at("tool_calls")) {
json e = json::object();
for (const auto & [k, v] : tool_call.items()) {
if (k != "type" && k != "function" && k != "id") {
e[k] = v;
}
}
for (const auto & [k, v] : tool_call.at("function").items()) {
if (k != "name" && k != "arguments") {
e["function"][k] = v;
}
}
any = any || !e.empty();
tc_extra.push_back(e);
}
if (any) {
extra["__tool_call_extras"] = tc_extra;
}
}
if (!extra.empty()) {
msg.extra_json = extra.dump();
}
}
msgs.push_back(msg);
}
@@ -547,6 +565,50 @@ static json render_message_to_json(const std::vector<common_chat_msg> & msgs, co
return messages_inp_normalizer(c).normalize(messages);
}
// DeepSeek V4.1 reads non-OpenAI message keys and joins content blocks with "\n\n" around images,
// so keep media markers as separate text parts and restore the extra keys
static json render_message_to_json_deepseek_v4_1(const std::vector<common_chat_msg> & msgs, const jinja::caps & c) {
json messages = json::array();
for (const auto & msg : msgs) {
json jmsg;
if (msg.contains_media() && msg.content.empty()) {
common_chat_msg typed = msg;
for (auto & part : typed.content_parts) {
part.type = "text";
}
jmsg = typed.to_json_oaicompat(/* concat_typed_text= */ false);
} else {
jmsg = msg.to_json_oaicompat(/* concat_typed_text= */ false);
}
if (!msg.extra_json.empty()) {
const json extra = json::parse(msg.extra_json);
for (const auto & [k, v] : extra.items()) {
if (k == "__tool_call_extras") {
if (!jmsg.contains("tool_calls")) {
continue;
}
auto & tcs = jmsg.at("tool_calls");
for (size_t i = 0; i < tcs.size() && i < v.size(); i++) {
for (const auto & [tk, tv] : v[i].items()) {
if (tk == "function") {
for (const auto & [fk, fv] : tv.items()) {
tcs[i]["function"][fk] = fv;
}
} else {
tcs[i][tk] = tv;
}
}
}
} else if (!jmsg.contains(k)) {
jmsg[k] = v;
}
}
}
messages.push_back(std::move(jmsg));
}
return messages_inp_normalizer(c).normalize(messages);
}
// DEPRECATED: only used in tests
json common_chat_msgs_to_json_oaicompat(const std::vector<common_chat_msg> & msgs, bool concat_typed_text) {
jinja::caps c;
@@ -562,13 +624,17 @@ json common_chat_tools_to_json_oaicompat(const std::vector<common_chat_tool> & t
auto result = json::array();
for (const auto & tool : tools) {
json function {
{ "name", tool.name },
{ "description", tool.description },
{ "parameters", json::parse(tool.parameters) },
};
if (tool.defer_loading) {
function["defer_loading"] = true;
}
result.push_back({
{ "type", "function" },
{ "function", {
{ "name", tool.name },
{ "description", tool.description },
{ "parameters", json::parse(tool.parameters) },
}},
{ "function", function },
});
}
return result;
@@ -605,10 +671,19 @@ std::vector<common_chat_tool> common_chat_tools_parse_oaicompat(const json & too
}
const auto & function = tool.at("function");
// "defer_loading" sits on the function, or on the tool next to "type"; the function wins
const json & defer = function.contains("defer_loading") ? function.at("defer_loading")
: tool.value("defer_loading", json(false));
if (!defer.is_boolean()) {
throw std::invalid_argument("Expected 'defer_loading' to be a boolean, got " + defer.dump());
}
result.push_back({
/* .name = */ function.at("name"),
/* .description = */ function.value("description", ""),
/* .parameters = */ function.value("parameters", json::object()).dump(),
/* .defer_loading = */ defer.get<bool>(),
});
}
}
@@ -620,8 +695,11 @@ std::vector<common_chat_tool> common_chat_tools_parse_oaicompat(const json & too
}
common_chat_continuation common_chat_continuation_parse(const common_json & value) {
if (value.is_boolean() && value.get<bool>()) {
return COMMON_CHAT_CONTINUATION_AUTO;
if (value.is_null()) {
return COMMON_CHAT_CONTINUATION_NONE;
}
if (value.is_boolean()) {
return value.get<bool>() ? COMMON_CHAT_CONTINUATION_AUTO : COMMON_CHAT_CONTINUATION_NONE;
}
if (value.is_string()) {
auto value_str = value.get<std::string>();
@@ -632,7 +710,7 @@ common_chat_continuation common_chat_continuation_parse(const common_json & valu
return COMMON_CHAT_CONTINUATION_CONTENT;
}
}
return COMMON_CHAT_CONTINUATION_NONE;
throw std::invalid_argument("Invalid continue_final_message: expected a boolean, \"content\" or \"reasoning_content\"");
}
bool common_chat_verify_template(const std::string & tmpl, bool use_jinja) {
@@ -675,7 +753,7 @@ std::string common_chat_format_single(const struct common_chat_templates * tmpls
if (!past_msg.empty()) {
inputs.messages = past_msg;
inputs.add_generation_prompt = false;
fmt_past_msg = common_chat_templates_apply(tmpls, inputs).prompt;
fmt_past_msg = common_chat_templates_apply(tmpls, inputs).prompt.str();
}
std::ostringstream ss;
// if the past_msg ends with a newline, we must preserve it in the formatted version
@@ -685,7 +763,7 @@ std::string common_chat_format_single(const struct common_chat_templates * tmpls
// format chat with new_msg
inputs.messages.push_back(new_msg);
inputs.add_generation_prompt = add_ass;
auto fmt_new_msg = common_chat_templates_apply(tmpls, inputs).prompt;
auto fmt_new_msg = common_chat_templates_apply(tmpls, inputs).prompt.str();
// get the diff part
ss << fmt_new_msg.substr(fmt_past_msg.size(), fmt_new_msg.size() - fmt_past_msg.size());
return ss.str();
@@ -709,7 +787,7 @@ std::string common_chat_format_example(const struct common_chat_templates *
add_simple_msg("user", "Hello");
add_simple_msg("assistant", "Hi there");
add_simple_msg("user", "How are you?");
return common_chat_templates_apply(tmpls, inputs).prompt;
return common_chat_templates_apply(tmpls, inputs).prompt.str();
}
#define CHATML_TEMPLATE_SRC \
@@ -728,19 +806,119 @@ bool common_chat_templates_was_explicit(const struct common_chat_templates * tmp
return tmpls->has_explicit_template;
}
common_chat_prompt_preset common_chat_get_asr_prompt(const common_chat_templates * chat_templates) {
// renders a task template (e.g. "tts") with the given variables
static std::string common_chat_template_render(const common_chat_template & tmpl, const json & vars, const std::shared_ptr<common_tts_dicts> & tts_dicts,
const common_tts_g2p & g2p = nullptr) {
jinja::context ctx(tmpl.source());
// text normalization helpers: tts_norm_scan(), tts_norm_sub(), tts_dict() and tts_g2p(), see docs/tts-norm-patterns.md
common_chat_tts_norm_add_builtins(ctx, tts_dicts, g2p);
jinja::global_from_json(ctx, vars, false);
ctx.set_val("bos_token", jinja::mk_val<jinja::value_string>(tmpl.bos_token()));
ctx.set_val("eos_token", jinja::mk_val<jinja::value_string>(tmpl.eos_token()));
jinja::runtime runtime(ctx);
return jinja::runtime::gather_string_parts(runtime.execute(tmpl.prog))->as_string().str();
}
// LFM3 format detection: template renders deferred tools inside <additional_tools>...</additional_tools>
static bool is_lfm3_template(const std::string & src) {
return src.find("<additional_tools>") != std::string::npos;
}
static std::string string_lower(std::string s) {
std::transform(s.begin(), s.end(), s.begin(), [](unsigned char c) { return (char) std::tolower(c); });
return s;
}
// VibeVoice-ASR: default system prompt of the template written by the converter
static bool is_vibevoice_asr_template(const std::string & src) {
return src.find("transcribes audio input into text output in JSON format") != std::string::npos;
}
// LFM3-Audio: language code -> name
static std::string lfm3_language_name(const std::string & language) {
static const std::map<std::string, std::string> names = {
{ "ar", "Arabic" }, { "da", "Danish" }, { "de", "German" }, { "el", "Greek" }, { "en", "English" },
{ "es", "Spanish" }, { "fi", "Finnish" }, { "fr", "French" }, { "he", "Hebrew" }, { "it", "Italian" },
{ "ja", "Japanese" }, { "ko", "Korean" }, { "nl", "Dutch" }, { "pl", "Polish" }, { "pt", "Portuguese" },
{ "sv", "Swedish" }, { "tr", "Turkish" }, { "zh", "Chinese" },
};
const auto it = names.find(string_lower(language));
return it != names.end() ? it->second : language;
}
// LFM3-Audio: no language = auto detect
static std::string lfm3_asr_system_prompt(const std::string & language) {
if (language.empty()) {
return "Perform ASR.";
}
return "Perform ASR in " + lfm3_language_name(language) + ".";
}
common_chat_prompt_preset common_chat_get_asr_prompt(const common_chat_templates * chat_templates, const std::string & language,
double audio_length, const std::string & context) {
common_chat_prompt_preset asr_preset;
asr_preset.system = "";
asr_preset.user = "Transcribe audio to text";
if (chat_templates && chat_templates->template_default && is_lfm2_template(chat_templates->template_default->source())) {
asr_preset.system = "Perform ASR.";
asr_preset.user = "";
if (chat_templates && chat_templates->template_default) {
const std::string src = chat_templates->template_default->source();
if (is_lfm2_template(src)) {
asr_preset.system = "Perform ASR.";
asr_preset.user = "";
} else if (is_lfm3_template(src)) {
asr_preset.system = lfm3_asr_system_prompt(language);
asr_preset.user = "";
asr_preset.has_language = true;
} else if (is_vibevoice_asr_template(src)) {
// without the audio length the model does not know where the audio ends and loops
const char * keys = "transcribe it with these keys: Start time, End time, Speaker ID, Content";
if (context.empty()) {
asr_preset.user = string_format("\nThis is a %.2f seconds audio, please %s", audio_length, keys);
} else {
asr_preset.user = string_format("\nThis is a %.2f seconds audio, with extra info: %s\n\nPlease %s", audio_length, context.c_str(), keys);
asr_preset.has_context = true;
}
asr_preset.media_first = true;
asr_preset.needs_audio_length = true;
}
}
return asr_preset;
}
std::string common_chat_get_tts_prompt(const common_chat_templates * chat_templates, const std::string & text,
const std::string & language, const std::string & voice, const common_tts_g2p & g2p) {
if (!chat_templates || !chat_templates->template_tts) {
return "";
}
return common_chat_template_render(*chat_templates->template_tts, {
{"messages", json::array({{{"role", "user"}, {"content", text}}})},
{"language", language},
{"voice", voice},
}, chat_templates->tts_dicts, g2p);
}
std::string common_chat_get_speech_system_prompt(const common_chat_templates * chat_templates,
const std::string & language, const std::string & voice) {
if (!chat_templates || !chat_templates->template_default || !is_lfm3_template(chat_templates->template_default->source())) {
return "";
}
// same voices as the "tts" template
const std::string v = string_lower(voice);
const bool has_voice = v == "female" || v == "male";
if (language.empty() && !has_voice) {
return "";
}
std::string out = "Speak";
if (!language.empty()) {
out += " in " + lfm3_language_name(language);
}
if (has_voice) {
out += " using a " + v + " voice";
}
return out + ".";
}
std::string common_chat_templates_source(const struct common_chat_templates * tmpls, const std::string & variant) {
if (!variant.empty()) {
if (variant == "tool_use") {
@@ -796,18 +974,6 @@ common_chat_templates_ptr common_chat_templates_init(const struct llama_model *
"{%- if false %}");
}
// TODO @aldehir : this is a temporary fix, pending Minja changes
// Ref: https://github.com/ggml-org/llama.cpp/pull/17713#issuecomment-3631342664
if (default_template_src.find("[TOOL_CALLS]") != std::string::npos
// search for the error message and patch it
&& default_template_src.find("if (message['content'] is none or") != std::string::npos) {
string_replace_all(default_template_src,
"{%- if (message['content'] is none or message['content'] == '' or "
"message['content']|length == 0) and (message['tool_calls'] is not defined or "
"message['tool_calls'] is none or message['tool_calls']|length == 0) %}",
"{%- if false %}");
}
std::string token_bos = bos_token_override;
std::string token_eos = eos_token_override;
bool add_bos = false;
@@ -851,6 +1017,35 @@ common_chat_templates_ptr common_chat_templates_init(const struct llama_model *
LOG_ERR("%s: failed to parse tool use chat template (ignoring it): %s\n", __func__, e.what());
}
}
// task templates of the model, kept with a chat template override
if (model) {
if (const char * src = llama_model_chat_template(model, "tts")) {
try {
tmpls->template_tts = std::make_unique<common_chat_template>(src, token_bos, token_eos);
} catch (const std::exception & e) {
LOG_ERR("%s: failed to parse the tts chat template (ignoring it): %s\n", __func__, e.what());
}
}
// dictionaries for tts_dict(), listed by name in tokenizer.tts_dicts
tmpls->tts_dicts = std::make_shared<common_tts_dicts>();
for (int32_t i = 0; i < llama_model_meta_arr_n(model, "tokenizer.tts_dicts"); i++) {
const std::string name = llama_model_meta_arr_str(model, "tokenizer.tts_dicts", i);
const std::string key = "tokenizer.tts_dict." + name + ".keys";
const std::string val_key = "tokenizer.tts_dict." + name + ".values";
const int32_t n = llama_model_meta_arr_n(model, key.c_str());
if (n < 0 || llama_model_meta_arr_n(model, val_key.c_str()) != n) {
LOG_ERR("%s: the tts dictionary '%s' has no value for each of its keys (ignoring it)\n", __func__, name.c_str());
continue;
}
auto & dict = (*tmpls->tts_dicts)[name];
dict.keys.reserve(n);
dict.values.reserve(n);
for (int32_t j = 0; j < n; j++) {
dict.keys.emplace_back(llama_model_meta_arr_str(model, key.c_str(), j));
dict.values.emplace_back(llama_model_meta_arr_str(model, val_key.c_str(), j));
}
}
}
return tmpls;
}
@@ -902,7 +1097,7 @@ common_reasoning_format common_reasoning_format_from_name(const std::string & fo
throw std::runtime_error("Unknown reasoning format: " + format);
}
std::string common_chat_template_direct_apply_impl(
common_chat_prompt common_chat_template_direct_apply_impl(
const common_chat_template & tmpl,
const autoparser::generation_params & inputs,
const std::optional<json> & messages_override,
@@ -948,24 +1143,37 @@ std::string common_chat_template_direct_apply_impl(
jinja::global_from_json(ctx, inp, inputs.mark_input);
// bos/eos come from the model, not from the user
ctx.set_val("bos_token", jinja::mk_val<jinja::value_string>(tmpl.bos_token()));
ctx.set_val("eos_token", jinja::mk_val<jinja::value_string>(tmpl.eos_token()));
// render
jinja::runtime runtime(ctx);
const jinja::value results = runtime.execute(tmpl.prog);
auto parts = jinja::runtime::gather_string_parts(results);
std::string result = parts->as_string().str();
common_chat_prompt result;
for (auto & part : parts->val_str.parts) {
result.parts.push_back({std::move(part.val), part.is_input});
}
// TODO: improve this later
if (inputs.add_bos && string_starts_with(result, tmpl.bos_token())) {
result = result.substr(tmpl.bos_token().size());
if (inputs.add_bos && !result.parts.empty()) {
auto & first = result.parts.front();
if (!first.is_input && string_starts_with(first.text, tmpl.bos_token())) {
first.text.erase(0, tmpl.bos_token().size());
}
}
if (inputs.add_eos && string_ends_with(result, tmpl.eos_token())) {
result = result.substr(0, result.size() - tmpl.eos_token().size());
if (inputs.add_eos && !result.parts.empty()) {
auto & last = result.parts.back();
if (!last.is_input && string_ends_with(last.text, tmpl.eos_token())) {
last.text.resize(last.text.size() - tmpl.eos_token().size());
}
}
return result;
}
std::string common_chat_template_direct_apply(
common_chat_prompt common_chat_template_direct_apply(
const common_chat_template & tmpl,
const autoparser::generation_params & inputs) {
return common_chat_template_direct_apply_impl(tmpl, inputs, std::nullopt, std::nullopt, std::nullopt);
@@ -981,9 +1189,9 @@ std::string common_chat_template_generation_prompt_impl(
autoparser::generation_params params = inputs;
params.add_generation_prompt = false;
params.continue_final_message = COMMON_CHAT_CONTINUATION_NONE;
std::string no_gen_prompt = common_chat_template_direct_apply_impl(tmpl, params, messages_override, tools_override, additional_context);
std::string no_gen_prompt = common_chat_template_direct_apply_impl(tmpl, params, messages_override, tools_override, additional_context).str();
params.add_generation_prompt = true;
std::string gen_prompt = common_chat_template_direct_apply_impl(tmpl, params, messages_override, tools_override, additional_context);
std::string gen_prompt = common_chat_template_direct_apply_impl(tmpl, params, messages_override, tools_override, additional_context).str();
size_t prefix_len = 0;
size_t min_size = std::min(no_gen_prompt.size(), gen_prompt.size());
@@ -1087,41 +1295,61 @@ static json common_chat_extra_context() {
return ctx;
}
std::optional<common_chat_params> common_chat_try_specialized_template(
const common_chat_template & tmpl,
const std::string & src,
autoparser::generation_params & params) {
static common_chat_params common_chat_params_init_lfm2_tokens(const common_chat_template & tmpl, const autoparser::generation_params & inputs) {
return common_chat_params_init_lfm2(tmpl, inputs, /* tool_list_tokens = */ true);
}
static common_chat_params common_chat_params_init_lfm2_5(const common_chat_template & tmpl, const autoparser::generation_params & inputs) {
return common_chat_params_init_lfm2(tmpl, inputs, /* tool_list_tokens = */ false);
}
// Pick the dedicated handler for a template from its source, or null for the autoparser.
// Order matters: the first match wins, and later checks assume the earlier ones did not match.
static common_chat_params_init_fn common_chat_template_detect_params_init(const std::string & src) {
// Ministral/Mistral Large 3 - uses special reasoning structure fixes, can't use autoparser
// Note: Mistral Small 3.2 uses [CALL_ID] which Ministral doesn't have, so we can distinguish them
if (src.find("[SYSTEM_PROMPT]") != std::string::npos && src.find("[TOOL_CALLS]") != std::string::npos &&
src.find("[ARGS]") != std::string::npos && src.find("[CALL_ID]") == std::string::npos) {
LOG_DBG("Using specialized template: Ministral/Magistral Large 3\n");
return common_chat_params_init_ministral_3(tmpl, params);
return common_chat_params_init_ministral_3;
}
// LLM-jp-4.1 - GPT-OSS dialect (spaces after special tokens, <|end|>-separated parallel calls)
if (src.find("chat_format=llm-jp-harmony-v1") != std::string::npos) {
LOG_DBG("Using specialized template: LLM-jp Harmony v1\n");
return common_chat_params_init_llm_jp_harmony(tmpl, params);
return common_chat_params_init_llm_jp_harmony;
}
// Harmony XML tool calls, checked before the GPT-OSS JSON calls
if (src.find("<|channel|>") != std::string::npos && src.find("<|constrain|>xml") != std::string::npos &&
src.find("rfl:parameter") != std::string::npos) {
LOG_DBG("Using specialized template: Harmony XML\n");
return common_chat_params_init_harmony_xml;
}
// GPT-OSS - has unique channel-based structure that needs dedicated handler
if (src.find("<|channel|>") != std::string::npos) {
LOG_DBG("Using specialized template: GPT-OSS\n");
return common_chat_params_init_gpt_oss(tmpl, params);
return common_chat_params_init_gpt_oss;
}
// Muse Spark shares the Muse recipient protocol and adds a commentary recipient.
if (src.find("to=commentary") != std::string::npos && src.find("<|eom|>") != std::string::npos) {
LOG_DBG("Using specialized template: Muse Spark\n");
return common_chat_params_init_muse_spark;
}
// Muse Glimmer format using " to=<recipient>" recipients and <|eom|>/<|eot|> message terminators.
if (src.find("<atem:function_calls>") != std::string::npos && src.find("<|eom|>") != std::string::npos) {
LOG_DBG("Using specialized template: Muse Glimmer\n");
return common_chat_params_init_muse_glimmer(tmpl, params);
return common_chat_params_init_muse_glimmer;
}
// Functionary v3.2 - uses recipient-based format with >>>recipient\n{content}
// Detection: template has ">>>all" for content and ">>>" prefix for tool calls
if (src.find(">>>all") != std::string::npos && src.find(">>>${recipient}") != std::string::npos) {
LOG_DBG("Using specialized template: Functionary v3.2\n");
return common_chat_params_init_functionary_v3_2(tmpl, params);
return common_chat_params_init_functionary_v3_2;
}
// Kimi K2 Thinking - uses unique tool call ID format: functions.<name>:<index>
@@ -1129,14 +1357,14 @@ std::optional<common_chat_params> common_chat_try_specialized_template(
if (src.find("<|tool_calls_section_begin|>") != std::string::npos &&
src.find("<|tool_call_begin|>") != std::string::npos) {
LOG_DBG("Using specialized template: Kimi K2 Thinking\n");
return common_chat_params_init_kimi_k2(tmpl, params);
return common_chat_params_init_kimi_k2;
}
// Kimi K3 - the <|open|>/<|close|>/<|end_of_msg|> markers are unique to it
if (src.find("<|open|>") != std::string::npos && src.find("<|close|>") != std::string::npos &&
src.find("<|end_of_msg|>") != std::string::npos) {
LOG_DBG("Using specialized template: Kimi K3\n");
return common_chat_params_init_kimi_k3(tmpl, params);
return common_chat_params_init_kimi_k3;
}
// K2 Horizon - <|ifm|im_start|> turns, <ifm|think*> reasoning picked by reasoning_effort and
@@ -1144,7 +1372,7 @@ std::optional<common_chat_params> common_chat_try_specialized_template(
if (src.find("<|ifm|im_start|>") != std::string::npos &&
src.find("<ifm|tool_calls>") != std::string::npos) {
LOG_DBG("Using specialized template: K2 Horizon\n");
return common_chat_params_init_k2_horizon(tmpl, params);
return common_chat_params_init_k2_horizon;
}
// Ling 3.0 / Bailing V3 - <role>X</role> sections with <arg_key>/<arg_value> tagged
@@ -1152,7 +1380,7 @@ std::optional<common_chat_params> common_chat_try_specialized_template(
if (src.find("<role>ASSISTANT</role>") != std::string::npos &&
src.find("<arg_key>") != std::string::npos) {
LOG_DBG("Using specialized template: Ling 3.0 (Bailing V3)\n");
return common_chat_params_init_ling3(tmpl, params);
return common_chat_params_init_ling3;
}
// Cohere2 MoE / North Code - marker-wrapped format with <|START_TEXT|> content and
@@ -1161,19 +1389,19 @@ std::optional<common_chat_params> common_chat_try_specialized_template(
if (src.find("<|START_TEXT|>") != std::string::npos &&
src.find("<|START_ACTION|>") != std::string::npos) {
LOG_DBG("Using specialized template: Cohere2 MoE\n");
return common_chat_params_init_cohere2moe(tmpl, params);
return common_chat_params_init_cohere2moe;
}
if (is_lfm2_template(src)) {
LOG_DBG("Using specialized template: LFM2\n");
return common_chat_params_init_lfm2(tmpl, params, /* tool_list_tokens = */ true);
return common_chat_params_init_lfm2_tokens;
}
// LFM2.5 format detection: template uses plain "List of tools: [...]" with no special tokens
if (src.find("List of tools: [") != std::string::npos &&
src.find("<|tool_list_start|>") == std::string::npos) {
LOG_DBG("Using specialized template: LFM2.5\n");
return common_chat_params_init_lfm2(tmpl, params, /* tool_list_tokens = */ false);
return common_chat_params_init_lfm2_5;
}
// GigaChatV3 format detection
@@ -1181,7 +1409,7 @@ std::optional<common_chat_params> common_chat_try_specialized_template(
src.find("<|message_sep|>") != std::string::npos &&
src.find("<|function_call|>") == std::string::npos) {
LOG_DBG("Using specialized template: GigaChatV3\n");
return common_chat_params_init_gigachat_v3(tmpl, params);
return common_chat_params_init_gigachat_v3;
}
// MiniMax-M3: the namespace token "]<]minimax[>[" collides with the autoparser's
@@ -1190,29 +1418,26 @@ std::optional<common_chat_params> common_chat_try_specialized_template(
src.find("<tool_call>") != std::string::npos &&
src.find("<invoke name=") != std::string::npos) {
LOG_DBG("Using specialized template: MiniMax-M3\n");
return common_chat_params_init_minimax_m3(tmpl, params);
return common_chat_params_init_minimax_m3;
}
// DeepSeek V3.2/V4 format detection: template defines dsml_token and uses it for tool calls.
// The template source contains the token as a variable assignment, not as a literal in markup.
// V3.2 names the tool call block "function_calls", V4 names it "tool_calls".
if (src.find("dsml_token") != std::string::npos &&
src.find("DSML") != std::string::npos &&
(src.find("function_calls") != std::string::npos ||
src.find("tool_calls") != std::string::npos)) {
LOG_DBG("Using specialized template: DeepSeek V3.2/V4\n");
return common_chat_params_init_deepseek_v3_2(tmpl, params);
// V4.1 tag names have a leading space and its template does not name the dsml_token variable.
if ((src.find("dsml_token") != std::string::npos &&
src.find("DSML") != std::string::npos &&
(src.find("function_calls") != std::string::npos ||
src.find("tool_calls") != std::string::npos)) ||
src.find("|DSML| calls>") != std::string::npos) {
LOG_DBG("Using specialized template: DeepSeek V3.2/V4/V4.1\n");
return common_chat_params_init_deepseek_v3_2;
}
// Gemma4 format detection
if (src.find("'<|tool_call>call:'") != std::string::npos) {
if (src.find("{#- OpenAI Chat Completions:") == std::string::npos) {
// apply workarounds if using the older gemma4 templates
LOG_WRN("%s: detected an outdated gemma4 chat template, applying compatibility workarounds. "
"Consider updating to the official template.\n", __func__);
workaround::convert_tool_responses_gemma4(params.messages);
}
return common_chat_params_init_gemma4(tmpl, params);
LOG_DBG("Using specialized template: Gemma4\n");
return common_chat_params_init_gemma4;
}
// MiniCPM5 - XML tool calls with <function name="..."><param name="...">...</param></function>
@@ -1220,14 +1445,14 @@ std::optional<common_chat_params> common_chat_try_specialized_template(
src.find("<function name=\"") != std::string::npos &&
src.find("<param name=\"") != std::string::npos) {
LOG_DBG("Using specialized template: MiniCPM5\n");
return common_chat_params_init_minicpm5(tmpl, params);
return common_chat_params_init_minicpm5;
}
// TranslateGemma - user content must follow a custom schema with language codes
if (src.find("[source_lang_code]") != std::string::npos &&
src.find("[target_lang_code]") != std::string::npos) {
LOG_DBG("Using specialized template: TranslateGemma\n");
return common_chat_params_init_translate_gemma(tmpl, params);
return common_chat_params_init_translate_gemma;
}
// Qwen3-Coder XML tool calls, also used by Nemotron Nano 3, Qwen3.5 and StepFun-3.5-Flash
@@ -1237,10 +1462,57 @@ std::optional<common_chat_params> common_chat_try_specialized_template(
// Exclude models that don't use \n between tags
src.find("'<tool_call><function=' ~ tool_call.name ~ '>'") == std::string::npos) {
LOG_DBG("Using specialized template: Qwen3-Coder\n");
return common_chat_params_init_qwen3_coder(tmpl, params);
return common_chat_params_init_qwen3_coder;
}
return std::nullopt;
// Inkling 1 and 2
if (is_inkling_template(src)) {
LOG_DBG("Using specialized template: Inkling\n");
return common_chat_params_init_inkling;
}
return nullptr;
}
common_chat_template::common_chat_template(const std::string & src, const std::string & bos_token, const std::string & eos_token) {
jinja::lexer lexer;
auto lexer_res = lexer.tokenize(src);
this->prog = jinja::parse_from_tokens(lexer_res);
this->src = lexer_res.source;
this->bos_tok = bos_token;
this->eos_tok = eos_token;
this->caps = jinja::caps_get(prog);
// LOG_INF("%s: caps:\n%s\n", __func__, this->caps.to_string().c_str());
this->params_init = common_chat_template_detect_params_init(this->src);
if (this->params_init) {
return;
}
// The analysis depends only on the template, so run it once here instead of on every apply.
// A failure is kept for apply to report, so a bad template still loads like it did before.
try {
analysis = std::make_unique<autoparser::autoparser>();
analysis->analyze_template(*this);
} catch (const std::exception & e) {
analysis.reset();
analysis_error = e.what();
}
}
common_chat_template::~common_chat_template() = default;
common_chat_template::common_chat_template(common_chat_template &&) = default;
common_chat_template & common_chat_template::operator=(common_chat_template &&) = default;
std::optional<common_chat_params> common_chat_try_specialized_template(
const common_chat_template & tmpl,
const autoparser::generation_params & params) {
if (!tmpl.params_init) {
return std::nullopt;
}
return tmpl.params_init(tmpl, params);
}
static common_chat_params common_chat_templates_apply_jinja(const struct common_chat_templates * tmpls,
@@ -1260,7 +1532,9 @@ static common_chat_params common_chat_templates_apply_jinja(const struct common_
workaround::trim_all_content(trimmed_messages);
messages_to_render = &trimmed_messages;
}
params.messages = render_message_to_json(*messages_to_render, tmpl.original_caps());
params.messages = src.find("|DSML| calls>") != std::string::npos ?
render_message_to_json_deepseek_v4_1(*messages_to_render, tmpl.original_caps()) :
render_message_to_json(*messages_to_render, tmpl.original_caps());
params.tool_choice = inputs.tool_choice;
params.reasoning_format = inputs.reasoning_format;
params.enable_thinking = inputs.enable_thinking;
@@ -1342,21 +1616,23 @@ static common_chat_params common_chat_templates_apply_jinja(const struct common_
data.prompt = common_chat_template_direct_apply_impl(tmpl, params_copy);
data.generation_prompt = common_chat_template_generation_prompt_impl(tmpl, params);
data.format = COMMON_CHAT_FORMAT_PEG_NATIVE;
auto parser = build_chat_peg_parser([&data](common_chat_peg_builder &p) {
data.parser = build_chat_peg_parser([&data](common_chat_peg_builder &p) {
return p.literal(data.generation_prompt) << p.content(p.rest());
});
data.parser = parser.save();
return data;
}
if (auto result = common_chat_try_specialized_template(tmpl, src, params)) {
if (auto result = common_chat_try_specialized_template(tmpl, params)) {
return *result;
}
if (!tmpl.analysis) {
throw std::invalid_argument("Unable to generate parser for this template. Automatic parser generation failed: " + tmpl.analysis_error);
}
try {
LOG_DBG("%s: using differential autoparser\n", __func__);
struct autoparser::autoparser autoparser;
autoparser.analyze_template(tmpl);
const auto & autoparser = *tmpl.analysis;
auto auto_params = autoparser::peg_generator::generate_parser(tmpl, params, autoparser);
common_chat_msg_delimiters delimiters;
@@ -1377,8 +1653,7 @@ static common_chat_params common_chat_templates_apply_jinja(const struct common_
auto_params.thinking_end_tags = {std::move(end_tag)};
}
}
common_peg_arena arena;
arena.load(auto_params.parser);
const auto & arena = auto_params.parser;
LOG_DBG("%s: generated parser:\n%s\n\nparser generation prompt: %s\n", __func__, arena.dump(arena.root()).c_str(), auto_params.generation_prompt.c_str());
return auto_params;
} catch (const std::exception & e) {
@@ -1443,7 +1718,7 @@ static common_chat_params common_chat_templates_apply_legacy(const struct common
}
common_chat_params params;
params.prompt = std::string(buf.data(), res);
params.prompt = common_chat_prompt(std::string(buf.data(), res));
if (!inputs.json_schema.empty()) {
params.grammar = json_schema_to_grammar(json::parse(inputs.json_schema));
} else {
@@ -1459,6 +1734,46 @@ common_chat_params common_chat_templates_apply(const struct common_chat_template
common_chat_templates_apply_legacy(tmpls, inputs);
}
std::string common_chat_prompt::str() const {
if (parts.size() == 1) {
return parts[0].text;
}
size_t len = 0;
for (const auto & part : parts) {
len += part.text.size();
}
std::string out;
out.reserve(len);
for (const auto & part : parts) {
out += part.text;
}
return out;
}
void common_chat_prompt::append(const std::string & text, bool is_input) {
if (text.empty()) {
return;
}
if (!parts.empty() && parts.back().is_input == is_input) {
parts.back().text += text;
} else {
parts.push_back({text, is_input});
}
}
void common_chat_prompt::replace_last(const std::string & from, const std::string & to) {
for (auto it = parts.rbegin(); it != parts.rend(); ++it) {
if (it->is_input) {
continue;
}
size_t pos = it->text.rfind(from);
if (pos != std::string::npos) {
it->text.replace(pos, from.size(), to);
return;
}
}
}
void common_chat_input::append(const std::string & piece, llama_token token) {
if (piece.empty()) {
return;
@@ -1525,9 +1840,10 @@ common_chat_msg common_chat_peg_parse(const common_peg_arena & src_pars
const common_chat_input & input,
bool is_partial,
const common_chat_parser_params & params) {
const common_peg_arena & parser = src_parser.empty() ?
build_chat_peg_parser([](common_chat_peg_builder & p) { return p.content(p.rest()) + p.end(); }) :
src_parser;
// both branches must be lvalues, a temporary here would copy the arena on every call
static const common_peg_arena content_only =
build_chat_peg_parser([](common_chat_peg_builder & p) { return p.content(p.rest()) + p.end(); });
const common_peg_arena & parser = src_parser.empty() ? content_only : src_parser;
if (src_parser.empty()) {
LOG_DBG("No parser definition detected, assuming pure content parser.");
@@ -1539,6 +1855,9 @@ common_chat_msg common_chat_peg_parse(const common_peg_arena & src_pars
//LOG_DBG("Parsing PEG input with format %s: %s\n", common_chat_format_name(params.format), effective_input.text.c_str());
common_peg_parse_flags flags = COMMON_PEG_PARSE_FLAG_LENIENT;
if (!is_partial) {
flags |= COMMON_PEG_PARSE_FLAG_COMPLETE;
}
if (params.debug) {
flags |= COMMON_PEG_PARSE_FLAG_DEBUG;
}
@@ -1598,12 +1917,92 @@ common_chat_msg common_chat_peg_parse(const common_peg_arena & src_pars
return msg;
}
std::map<std::string, bool> common_chat_templates_get_caps(const common_chat_templates * chat_templates) {
common_chat_session::common_chat_session(const common_chat_templates * tmpls,
const llama_vocab * vocab,
const common_chat_templates_inputs & inputs,
const common_chat_session_params & params) {
auto applied = common_chat_templates_apply(tmpls, inputs);
templated = true;
prompt_parts = std::move(applied.prompt);
result.role = "assistant";
grammar_text = std::move(applied.grammar);
grammar_lazy = applied.grammar_lazy;
stops = std::move(applied.additional_stops);
generation_prompt_text = applied.generation_prompt;
thinking_start = std::move(applied.thinking_start_tag);
thinking_ends = std::move(applied.thinking_end_tags);
parser_params.format = applied.format;
parser_params.generation_prompt = vocab ? common_chat_input_tokenize(vocab, applied.generation_prompt)
: common_chat_input(applied.generation_prompt);
parser_params.debug = params.debug;
parser_params.parser = std::move(applied.parser);
delimiters = std::move(applied.message_delimiters);
if (vocab) {
common_params_sampling resolved;
resolved.grammar_lazy = applied.grammar_lazy;
common_sampling_add_preserved_tokens(resolved, vocab, applied.preserved_tokens);
common_sampling_add_grammar_triggers(resolved, vocab, std::move(applied.grammar_triggers));
preserved_tokens = std::move(resolved.preserved_tokens);
grammar_triggers = std::move(resolved.grammar_triggers);
delimiters.tokenize(vocab);
} else {
grammar_triggers = std::move(applied.grammar_triggers);
}
if (inputs.continue_final_message != COMMON_CHAT_CONTINUATION_NONE && !params.echo) {
// start from the prefill so it is not emitted as part of the first delta
result = common_chat_parse(input, true, parser_params);
}
}
void common_chat_session::apply_sampling(common_params_sampling & sampling) const {
if (!templated) {
return;
}
if (!grammar_text.empty()) {
sampling.grammar = {COMMON_GRAMMAR_TYPE_TOOL_CALLS, grammar_text};
}
sampling.grammar_lazy = grammar_lazy;
sampling.generation_prompt = generation_prompt_text;
sampling.preserved_tokens.insert(preserved_tokens.begin(), preserved_tokens.end());
sampling.grammar_triggers.insert(sampling.grammar_triggers.end(), grammar_triggers.begin(), grammar_triggers.end());
}
const common_chat_msg & common_chat_session::feed(const common_chat_input & chunk) {
GGML_ASSERT(!finished && "feed() after finish()");
input.append(chunk);
auto msg = common_chat_parse(input, true, parser_params);
if (!msg.empty()) {
result = std::move(msg);
}
return result;
}
const common_chat_msg & common_chat_session::finish(const common_chat_input & chunk) {
GGML_ASSERT(!finished && "finish() called twice");
finished = true;
input.append(chunk);
auto msg = common_chat_parse(input, false, parser_params);
if (!msg.empty()) {
result = std::move(msg);
}
return result;
}
common_json common_chat_templates_get_caps(const common_chat_templates * chat_templates) {
GGML_ASSERT(chat_templates != nullptr);
GGML_ASSERT(chat_templates->template_default != nullptr);
if (chat_templates->template_tool_use != nullptr) {
// take the more expressive template when available
return chat_templates->template_tool_use->caps.to_map();
}
return chat_templates->template_default->caps.to_map();
// take the more expressive template when available
const auto & caps = chat_templates->template_tool_use != nullptr ? chat_templates->template_tool_use->caps
: chat_templates->template_default->caps;
common_json result = caps.to_map();
result["reasoning_effort_levels"] = caps.reasoning_effort_levels;
result["reasoning_effort_rejected"] = caps.reasoning_effort_rejected;
return result;
}
+129 -31
View File
@@ -22,8 +22,16 @@ struct common_chat_templates;
namespace autoparser {
struct generation_params;
struct autoparser;
} // namespace autoparser
struct common_chat_params;
struct common_chat_template;
// Builds the prompt and parser for a template that has a dedicated handler (see common/parsers)
using common_chat_params_init_fn = common_chat_params (*)(const common_chat_template & tmpl,
const autoparser::generation_params & inputs);
struct common_chat_tool_call {
std::string name;
std::string arguments;
@@ -37,6 +45,7 @@ struct common_chat_tool_call {
struct common_chat_msg_content_part {
std::string type;
std::string text;
std::string name; // tool_reference: name of a tool from the request tools
// TODO @ngxson : no known chat templates support reasoning_content in content parts yet
// this can be useful for models with interleaved thinking (like Kimi-K2)
@@ -44,7 +53,7 @@ struct common_chat_msg_content_part {
// std::string reasoning_content;
bool operator==(const common_chat_msg_content_part & other) const {
return type == other.type && text == other.text;
return type == other.type && text == other.text && name == other.name;
}
};
@@ -54,19 +63,20 @@ struct common_chat_template {
std::string eos_tok;
std::string src;
chat_template_caps caps;
// Dedicated handler picked once from the source, null when the differential autoparser is used
common_chat_params_init_fn params_init = nullptr;
common_chat_template(const std::string & src, const std::string & bos_token, const std::string & eos_token) {
jinja::lexer lexer;
auto lexer_res = lexer.tokenize(src);
this->prog = jinja::parse_from_tokens(lexer_res);
// Differential analysis, run once here when there is no dedicated handler. Null when there
// is one, or when the analysis failed, in which case analysis_error says why.
std::unique_ptr<autoparser::autoparser> analysis;
std::string analysis_error;
this->src = lexer_res.source;
this->bos_tok = bos_token;
this->eos_tok = eos_token;
common_chat_template(const std::string & src, const std::string & bos_token, const std::string & eos_token);
this->caps = jinja::caps_get(prog);
// LOG_INF("%s: caps:\n%s\n", __func__, this->caps.to_string().c_str());
}
// autoparser is incomplete here, so these are defined where it is complete
~common_chat_template();
common_chat_template(common_chat_template &&);
common_chat_template & operator=(common_chat_template &&);
const std::string & source() const { return src; }
const std::string & bos_token() const { return bos_tok; }
@@ -85,6 +95,8 @@ struct common_chat_msg {
std::string reasoning_content;
std::string tool_name;
std::string tool_call_id;
// non-OpenAI message keys as a JSON object string, only for templates that read them (DeepSeek V4.1)
std::string extra_json;
common_json to_json_oaicompat(bool concat_typed_text = false) const;
@@ -209,14 +221,15 @@ struct common_chat_msg_delimiters {
// split tokens into message spans. skips maps a start index to a length of a region to jump over without matching
common_chat_msg_spans split(const llama_tokens & tokens, const std::map<size_t, size_t> & skips = {}) const;
common_json to_json() const;
};
struct common_chat_tool {
std::string name;
std::string description;
std::string parameters;
// withheld from the system prompt, a tool_reference reveals it later in the conversation
bool defer_loading = false;
};
enum common_chat_tool_choice {
@@ -266,9 +279,31 @@ struct common_chat_templates_inputs {
bool force_pure_content = false;
};
struct common_chat_prompt_part {
std::string text;
bool is_input = false; // from user input, special tokens inside must not be parsed
};
// rendered chat template, split into template parts and user input parts
struct common_chat_prompt {
std::vector<common_chat_prompt_part> parts;
common_chat_prompt() = default;
// single part of template text
explicit common_chat_prompt(std::string text) { parts.push_back({std::move(text), false}); }
std::string str() const;
void append(const std::string & text, bool is_input = false);
// replace the last occurrence of `from`, user input parts are skipped
void replace_last(const std::string & from, const std::string & to);
};
struct common_chat_params {
common_chat_format format = COMMON_CHAT_FORMAT_CONTENT_ONLY;
std::string prompt;
common_chat_prompt prompt;
std::string grammar;
bool grammar_lazy = false;
std::string generation_prompt;
@@ -278,7 +313,7 @@ struct common_chat_params {
std::vector<common_grammar_trigger> grammar_triggers;
std::vector<std::string> preserved_tokens;
std::vector<std::string> additional_stops;
std::string parser;
common_peg_arena parser;
common_chat_msg_delimiters message_delimiters;
};
@@ -310,16 +345,10 @@ common_chat_input common_chat_input_tokenize(const llama_vocab * vocab, const st
// per-message parsing syntax
// should be derived from common_chat_params
struct common_chat_parser_params {
common_chat_format format = COMMON_CHAT_FORMAT_CONTENT_ONLY;
common_reasoning_format reasoning_format = COMMON_REASONING_FORMAT_NONE; // TODO: refactor this to "bool parse_reasoning"
// Whether reasoning_content should be inlined in the content (e.g. for reasoning_format=deepseek in stream mode)
bool reasoning_in_content = false;
common_chat_input generation_prompt;
bool parse_tool_calls = true;
bool is_continuation = false;
bool echo = false; // Include assistant prefilled msg in output
bool debug = false; // Enable debug output for PEG parser
common_peg_arena parser = {};
common_chat_format format = COMMON_CHAT_FORMAT_CONTENT_ONLY;
common_chat_input generation_prompt;
bool debug = false; // Enable debug output for PEG parser
common_peg_arena parser = {};
common_chat_parser_params() = default;
common_chat_parser_params(const common_chat_params & chat_params) {
format = chat_params.format;
@@ -365,6 +394,60 @@ const char * common_chat_format_name(common_chat_format format);
common_chat_msg common_chat_parse(const common_chat_input & input, bool is_partial, const common_chat_parser_params & params);
common_chat_msg common_chat_peg_parse(const common_peg_arena & src_parser, const common_chat_input & input, bool is_partial, const common_chat_parser_params & params);
struct common_chat_session_params {
bool echo = false; // include the assistant prefill in the output when continuing a message
bool debug = false; // enable debug output for the PEG parser
};
class common_chat_session {
public:
common_chat_session() { result.role = "assistant"; }
common_chat_session(const common_chat_templates * tmpls,
const llama_vocab * vocab,
const common_chat_templates_inputs & inputs,
const common_chat_session_params & params = {});
const common_chat_prompt & prompt() const { return prompt_parts; }
common_chat_format format() const { return parser_params.format; }
const common_chat_msg & msg() const { return result; }
const common_peg_arena & parser() const { return parser_params.parser; }
const std::string & grammar() const { return grammar_text; }
const std::string & generation_prompt() const { return generation_prompt_text; }
const std::string & thinking_start_tag() const { return thinking_start; }
const std::vector<std::string> & thinking_end_tags() const { return thinking_ends; }
const std::vector<std::string> & additional_stops() const { return stops; }
const common_chat_msg_delimiters & message_delimiters() const { return delimiters; }
void apply_sampling(common_params_sampling & sampling) const;
bool has_template() const { return templated; }
const common_chat_msg & feed(const common_chat_input & chunk);
const common_chat_msg & finish(const common_chat_input & chunk = {});
private:
common_chat_prompt prompt_parts;
std::string grammar_text;
bool grammar_lazy = false;
std::vector<common_grammar_trigger> grammar_triggers;
std::set<llama_token> preserved_tokens;
std::vector<std::string> stops;
std::string generation_prompt_text;
std::string thinking_start;
std::vector<std::string> thinking_ends;
common_chat_parser_params parser_params;
common_chat_msg_delimiters delimiters;
common_chat_input input;
common_chat_msg result;
bool templated = false;
bool finished = false;
};
// used by arg and server
const char * common_reasoning_format_name(common_reasoning_format format);
common_reasoning_format common_reasoning_format_from_name(const std::string & format);
@@ -389,9 +472,9 @@ common_json common_chat_tools_to_json_oaicompat(const std::vector<common_chat_to
common_json common_chat_tool_parameters(const common_json & function);
// get template caps, useful for reporting to server /props endpoint
std::map<std::string, bool> common_chat_templates_get_caps(const common_chat_templates * chat_templates);
common_json common_chat_templates_get_caps(const common_chat_templates * chat_templates);
std::string common_chat_template_direct_apply(
common_chat_prompt common_chat_template_direct_apply(
const common_chat_template & tmpl,
const autoparser::generation_params & inputs);
@@ -401,16 +484,31 @@ std::string common_chat_template_generation_prompt(
std::optional<common_chat_params> common_chat_try_specialized_template(
const common_chat_template & tmpl,
const std::string & src,
autoparser::generation_params & params);
const autoparser::generation_params & params);
// specialized per-task preset
struct common_chat_prompt_preset {
std::string system;
std::string user;
bool has_language = false; // the preset already names the requested language
bool has_context = false; // the user text already holds the context
bool media_first = false; // the media goes before the user text
bool needs_audio_length = false; // the user text depends on the audio length, call again with it
};
common_chat_prompt_preset common_chat_get_asr_prompt(const common_chat_templates * chat_templates);
// audio_length: in seconds, 0 if unknown
// context: extra info from the user (e.g. names, hotwords), only used by some presets, see has_context
common_chat_prompt_preset common_chat_get_asr_prompt(const common_chat_templates * chat_templates, const std::string & language = "",
double audio_length = 0.0, const std::string & context = "");
common_chat_msg_delimiters common_chat_msg_delimiters_parse(const common_json & delimiters);
// prompt of the model's "tts" chat template for the text to speak, empty if the model has none
// language and voice are optional, the template defines their defaults
// g2p (optional): the letter-to-sound model of the template's tts_g2p(), e.g. mtmd_gen_audio_g2p()
std::string common_chat_get_tts_prompt(const common_chat_templates * chat_templates, const std::string & text,
const std::string & language = "", const std::string & voice = "",
const std::function<std::string(const std::string & word)> & g2p = nullptr);
// system prompt line for the speech language and voice (e.g. LFM3-Audio), empty if not used
std::string common_chat_get_speech_system_prompt(const common_chat_templates * chat_templates,
const std::string & language, const std::string & voice);
+245 -148
View File
@@ -26,6 +26,7 @@
#include <fstream>
#include <iostream>
#include <iterator>
#include <random>
#include <regex>
#include <sstream>
#include <string>
@@ -218,6 +219,23 @@ int32_t common_cpu_get_num_math() {
}
}
}
#elif defined(__riscv) && defined(__linux__)
std::ifstream cpuinfo("/proc/cpuinfo");
std::string line;
int32_t n_a100 = 0;
while (std::getline(cpuinfo, line)) {
std::istringstream fields(line);
std::string key;
std::string separator;
uint64_t marchid;
if (fields >> key >> separator >> std::hex >> marchid &&
key == "marchid" && separator == ":" && marchid == UINT64_C(0x8000000041000002)) {
++n_a100;
}
}
if (n_a100 > 0) {
return n_a100;
}
#elif defined(__powerpc64__) || defined(__powerpc__)
int32_t smt_factor = 1;
int phy_cpus = common_cpu_get_num_physical_cores();
@@ -741,50 +759,56 @@ static inline bool glob_class_match(const char c, const char * pattern, const ch
}
// simple glob: * matches non-/ chars, ** matches anything including /, [] matches character class
static inline bool glob_match(const char * pattern, const char * str) {
if (*pattern == '\0') {
return *str == '\0';
}
if (pattern[0] == '*' && pattern[1] == '*') {
const char * p = pattern + 2;
if (glob_match(p, str)) return true;
if (*str != '\0') return glob_match(pattern, str + 1);
// only wildcards recurse, too many wildcards: no match
static bool glob_match(const char * pattern, const char * str, int depth = 0) {
if (depth >= COMMON_RECURSIVE_LIMIT) {
return false;
}
if (*pattern == '*') {
const char * p = pattern + 1;
for (; *str != '\0' && *str != '/'; str++) {
if (glob_match(p, str)) return true;
}
return glob_match(p, str);
}
if (*pattern == '?' && *str != '\0' && *str != '/') {
return glob_match(pattern + 1, str + 1);
}
if (*pattern == '[') {
const char * class_end = pattern + 1;
// If first character after '[' is ']' or '-', treat it as literal
if (*class_end == ']' || *class_end == '-') {
class_end++;
}
while (*class_end != '\0' && *class_end != ']') {
class_end++;
}
if (*class_end == ']') {
if (*str == '\0') return false;
bool matched = glob_class_match(*str, pattern + 1, class_end);
return matched && glob_match(class_end + 1, str + 1);
} else {
if (*str == '[') {
return glob_match(pattern + 1, str + 1);
while (*pattern != '\0') {
if (pattern[0] == '*' && pattern[1] == '*') {
const char * p = pattern + 2;
for (;; str++) {
if (glob_match(p, str, depth + 1)) return true;
if (*str == '\0') return false;
}
return false;
}
if (*pattern == '*') {
const char * p = pattern + 1;
for (; *str != '\0' && *str != '/'; str++) {
if (glob_match(p, str, depth + 1)) return true;
}
return glob_match(p, str, depth + 1);
}
if (*pattern == '?' && *str != '\0' && *str != '/') {
pattern++;
str++;
continue;
}
if (*pattern == '[') {
const char * class_end = pattern + 1;
// If first character after '[' is ']' or '-', treat it as literal
if (*class_end == ']' || *class_end == '-') {
class_end++;
}
while (*class_end != '\0' && *class_end != ']') {
class_end++;
}
if (*class_end == ']') {
if (*str == '\0' || !glob_class_match(*str, pattern + 1, class_end)) return false;
pattern = class_end + 1;
str++;
continue;
}
if (*str != '[') return false;
pattern++;
str++;
continue;
}
if (*pattern != *str) return false;
pattern++;
str++;
}
if (*pattern == *str) {
return glob_match(pattern + 1, str + 1);
}
return false;
return *str == '\0';
}
bool glob_match(const std::string & pattern, const std::string & str) {
@@ -871,27 +895,7 @@ bool fs_validate_filename(const std::string & filename, bool allow_subdirs) {
return true;
}
#include <iostream>
#ifdef _WIN32
std::wstring utf8_to_wstring(const std::string & str) {
if (str.empty()) {
return std::wstring();
}
int size = MultiByteToWideChar(CP_UTF8, 0, str.c_str(), (int)str.size(), NULL, 0);
if (size <= 0) {
return std::wstring();
}
std::wstring wstr(size, 0);
MultiByteToWideChar(CP_UTF8, 0, str.c_str(), (int)str.size(), &wstr[0], size);
return wstr;
}
std::string wstring_to_utf8(const std::wstring & str) {
if (str.empty()) {
return std::string();
@@ -910,16 +914,14 @@ std::string wstring_to_utf8(const std::wstring & str) {
}
#endif
// returns the path as a UTF-8 string, preserving its separators
std::string fs_path_to_utf8(const std::filesystem::path & path) {
const auto value = path.u8string();
return std::string(value.begin(), value.end());
}
void fs_write_atomic(const std::filesystem::path & path, const std::string & data) {
// a random suffix gives every writer its own temporary file, across processes and containers
std::random_device rd;
const uint64_t id = ((uint64_t) rd() << 32) | rd();
std::error_code ec;
std::filesystem::path path_tmp = path;
path_tmp += ".tmp";
path_tmp += "." + std::to_string(id) + ".tmp";
if (path.has_parent_path()) {
std::filesystem::create_directories(path.parent_path(), ec);
@@ -935,7 +937,7 @@ void fs_write_atomic(const std::filesystem::path & path, const std::string & dat
if (file.fail() || ec) {
std::filesystem::remove(path_tmp, ec);
throw std::runtime_error("failed to write file: " + fs_path_to_utf8(path));
throw std::runtime_error("failed to write file: " + path.string());
}
}
@@ -961,20 +963,9 @@ void common_set_env(const std::string & name, const std::string & value) {
#endif
}
std::filesystem::path common_get_path_from_env(const std::string & name) {
#if defined(_WIN32)
const std::wstring wname = utf8_to_wstring(name);
const wchar_t * wvalue = _wgetenv(wname.c_str());
return wvalue ? std::filesystem::path(wvalue) : std::filesystem::path();
#else
const char * value = std::getenv(name.c_str());
return value ? std::filesystem::path(value) : std::filesystem::path();
#endif
}
#if !defined(_WIN32)
static std::filesystem::path get_home_directory() {
std::filesystem::path home = common_get_path_from_env("HOME");
std::filesystem::path home = common_get_env("HOME");
if (!home.empty()) {
return home;
}
@@ -987,19 +978,19 @@ static std::filesystem::path get_home_directory() {
#endif
std::filesystem::path fs_get_cache_directory() {
std::filesystem::path cache_directory = common_get_path_from_env("LLAMA_CACHE");
std::filesystem::path cache_directory = common_get_env("LLAMA_CACHE");
if (!cache_directory.empty()) {
return cache_directory;
}
#if defined(_WIN32)
cache_directory = common_get_path_from_env("LOCALAPPDATA");
cache_directory = common_get_env("LOCALAPPDATA");
if (cache_directory.empty()) {
throw std::runtime_error("Failed to find %LOCALAPPDATA% directory");
}
#elif defined(__APPLE__)
cache_directory = get_home_directory() / "Library/Caches";
#else
cache_directory = common_get_path_from_env("XDG_CACHE_HOME");
cache_directory = common_get_env("XDG_CACHE_HOME");
if (cache_directory.empty()) {
cache_directory = get_home_directory() / ".cache";
}
@@ -1010,12 +1001,12 @@ std::filesystem::path fs_get_cache_directory() {
std::filesystem::path fs_get_config_directory() {
std::filesystem::path config_directory;
#if defined(_WIN32)
config_directory = common_get_path_from_env("APPDATA");
config_directory = common_get_env("APPDATA");
if (config_directory.empty()) {
throw std::runtime_error("Failed to find %APPDATA% directory");
}
#else
config_directory = common_get_path_from_env("XDG_CONFIG_HOME");
config_directory = common_get_env("XDG_CONFIG_HOME");
if (config_directory.empty()) {
config_directory = get_home_directory() / ".config";
}
@@ -1029,9 +1020,9 @@ std::filesystem::path fs_get_cache_file(const std::string & filename) {
std::error_code ec;
common_create_directories(cache_directory, ec);
if (ec) {
throw std::runtime_error("failed to create cache directory: " + fs_path_to_utf8(cache_directory));
throw std::runtime_error("failed to create cache directory: " + cache_directory.string());
}
return cache_directory / std::filesystem::u8path(filename);
return cache_directory / filename;
}
//
@@ -1170,8 +1161,16 @@ static const std::map<common_decision_type, std::string> COMMON_DECISION_TYPE_NA
{ COMMON_DECISION_TYPE_LAYA, "laya" },
{ COMMON_DECISION_TYPE_CLEF, "clef" },
{ COMMON_DECISION_TYPE_PPLX_DECIDER, "pplx-decider" },
{ COMMON_DECISION_TYPE_VLLM_DECISION2, "vllm-decision2" },
{ COMMON_DECISION_TYPE_VLLM_D3, "vllm-d3" },
{ COMMON_DECISION_TYPE_LFM2_D1, "lfm2-d1" },
{ COMMON_DECISION_TYPE_LFM2_D1_OMNI, "lfm2-d1-omni" },
{ COMMON_DECISION_TYPE_GRANITE_SWITCH, "granite-switch"},
{ COMMON_DECISION_TYPE_H2O_LIGHTNING, "h2o-lightning" },
{ COMMON_DECISION_TYPE_MINICPM_OCR_JEV, "openbmb-minicpm-ocr-jev" },
{ COMMON_DECISION_TYPE_LFM3_D1, "lfm3-d1" },
{ COMMON_DECISION_TYPE_LFM3_D1_OMNI, "lfm3-d1-omni" },
// for future models, ALWAYS add org name prefix to avoid name collisions, e.g. "orgname-modelname"
};
static common_decision_type common_decision_type_from_string(const std::string & str) {
@@ -1195,7 +1194,9 @@ common_decision_type common_get_decision_type(const struct llama_model * model)
return common_decision_type_from_string(buf);
}
common_decision_type common_get_decision_type(const std::string & fname) {
common_gguf_info common_get_gguf_info(const std::string & fname) {
common_gguf_info info;
struct gguf_init_params gguf_params = {
/* .no_alloc = */ true,
/* .ctx = */ nullptr,
@@ -1203,31 +1204,73 @@ common_decision_type common_get_decision_type(const std::string & fname) {
gguf_context_ptr gguf_ctx(gguf_init_from_file(fname.c_str(), gguf_params));
if (!gguf_ctx) {
return COMMON_DECISION_TYPE_UNKNOWN; // missing or unreadable file
return info; // missing or unreadable file
}
std::string arch;
const int64_t arch_id = gguf_find_key(gguf_ctx.get(), "general.architecture");
if (arch_id < 0) {
return COMMON_DECISION_TYPE_UNKNOWN; // no architecture in the metadata
if (arch_id < 0 || gguf_get_kv_type(gguf_ctx.get(), arch_id) != GGUF_TYPE_STRING) {
return info; // no architecture in the metadata
}
if (gguf_get_kv_type(gguf_ctx.get(), arch_id) != GGUF_TYPE_STRING) {
return COMMON_DECISION_TYPE_UNKNOWN; // malformed metadata
}
arch = gguf_get_val_str(gguf_ctx.get(), arch_id);
const std::string arch = gguf_get_val_str(gguf_ctx.get(), arch_id);
if (arch.empty()) {
return COMMON_DECISION_TYPE_UNKNOWN;
return info;
}
const std::string key = arch + ".decision.type";
const int64_t type_id = gguf_find_key(gguf_ctx.get(), key.c_str());
const int64_t type_id = gguf_find_key(gguf_ctx.get(), (arch + ".decision.type").c_str());
if (type_id < 0) {
return COMMON_DECISION_TYPE_NONE;
info.decision_type = COMMON_DECISION_TYPE_NONE;
} else if (gguf_get_kv_type(gguf_ctx.get(), type_id) == GGUF_TYPE_STRING) {
info.decision_type = common_decision_type_from_string(gguf_get_val_str(gguf_ctx.get(), type_id));
}
if (gguf_get_kv_type(gguf_ctx.get(), type_id) != GGUF_TYPE_STRING) {
return COMMON_DECISION_TYPE_UNKNOWN; // malformed metadata
// same key and type as the model loader
const int64_t ctx_id = gguf_find_key(gguf_ctx.get(), (arch + ".context_length").c_str());
if (ctx_id >= 0 && gguf_get_kv_type(gguf_ctx.get(), ctx_id) == GGUF_TYPE_UINT32) {
info.n_ctx_train = gguf_get_val_u32(gguf_ctx.get(), ctx_id);
}
return common_decision_type_from_string(gguf_get_val_str(gguf_ctx.get(), type_id));
// same key as llama_model_chat_template(model, nullptr)
info.has_chat_tmpl = gguf_find_key(gguf_ctx.get(), "tokenizer.chat_template") >= 0;
return info;
}
void common_fit_apply(common_params & params, llama_model_params & mparams, llama_context_params & cparams) {
if (!params.fit_params) {
return;
}
COM_TRC("%s", "fitting params to device memory ...\n");
COM_TRC("%s", "(for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on)\n");
// the draft context is created from the same base params and follows the main context, fit both together
const bool has_draft = params.speculative.has_dft();
const bool spec_mtp = std::find(params.speculative.types.begin(), params.speculative.types.end(),
COMMON_SPECULATIVE_TYPE_DRAFT_MTP) != params.speculative.types.end();
common_params params_dft = common_base_params_to_speculative(params);
auto mparams_dft = common_model_params_to_llama(params_dft);
auto cparams_dft = common_context_params_to_llama(params_dft);
if (spec_mtp) {
cparams_dft.ctx_type = LLAMA_CONTEXT_TYPE_MTP;
}
cparams_dft.n_rs_seq = 0;
const common_fit_extra_model extra = {
/*.paths_model =*/ params_dft.model.get_paths(),
/*.mparams =*/ &mparams_dft,
/*.cparams =*/ &cparams_dft,
/*.shares_model =*/ !has_draft, // an MTP context runs on the weights of the main model
};
common_fit_params(params.model.get_paths(), &mparams, &cparams,
params.tensor_split,
params.tensor_buft_overrides.data(),
params.fit_params_target.data(),
params.fit_params_min_ctx,
has_draft || spec_mtp ? &extra : nullptr,
params.verbosity >= LOG_LEVEL_DEBUG ? GGML_LOG_LEVEL_DEBUG : GGML_LOG_LEVEL_ERROR);
}
common_init_result::common_init_result(common_params & params, bool model_only) :
@@ -1235,41 +1278,9 @@ common_init_result::common_init_result(common_params & params, bool model_only)
auto mparams = common_model_params_to_llama(params);
auto cparams = common_context_params_to_llama(params);
if (params.fit_params) {
COM_TRC("%s", "fitting params to device memory ...\n");
COM_TRC("%s", "(for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on)\n");
common_fit_apply(params, mparams, cparams);
// the draft context is created from the same base params and follows the main context, fit both together
const bool has_draft = params.speculative.has_dft();
const bool spec_mtp = std::find(params.speculative.types.begin(), params.speculative.types.end(),
COMMON_SPECULATIVE_TYPE_DRAFT_MTP) != params.speculative.types.end();
common_params params_dft = common_base_params_to_speculative(params);
auto mparams_dft = common_model_params_to_llama(params_dft);
auto cparams_dft = common_context_params_to_llama(params_dft);
if (spec_mtp) {
cparams_dft.ctx_type = LLAMA_CONTEXT_TYPE_MTP;
}
cparams_dft.n_rs_seq = 0;
const common_fit_extra_model extra = {
/*.path_model =*/ params_dft.model.path.c_str(),
/*.mparams =*/ &mparams_dft,
/*.cparams =*/ &cparams_dft,
/*.shares_model =*/ !has_draft, // an MTP context runs on the weights of the main model
};
common_fit_params(params.model.path.c_str(), &mparams, &cparams,
params.tensor_split,
params.tensor_buft_overrides.data(),
params.fit_params_target.data(),
params.fit_params_min_ctx,
has_draft || spec_mtp ? &extra : nullptr,
params.verbosity >= LOG_LEVEL_DEBUG ? GGML_LOG_LEVEL_DEBUG : GGML_LOG_LEVEL_ERROR);
}
llama_model * model = llama_model_load_from_file(params.model.path.c_str(), mparams);
llama_model * model = common_model_load(params.model.get_paths(), mparams);
if (model == NULL) {
return;
}
@@ -1285,19 +1296,34 @@ common_init_result::common_init_result(common_params & params, bool model_only)
// these decision models return a score for each token via the embeddings output
// TODO: maybe improve this in the future
const auto decision_type = common_get_decision_type(model);
// minicpm-ocr-jev reads its class scores from the pooled output, the pooling type is the one of the GGUF
if (decision_type == COMMON_DECISION_TYPE_LAYA || decision_type == COMMON_DECISION_TYPE_KEV || decision_type == COMMON_DECISION_TYPE_CLEF ||
decision_type == COMMON_DECISION_TYPE_LFM2_D1_OMNI) {
decision_type == COMMON_DECISION_TYPE_LFM2_D1_OMNI || decision_type == COMMON_DECISION_TYPE_VLLM_DECISION2 ||
decision_type == COMMON_DECISION_TYPE_MINICPM_OCR_JEV) {
const auto pooling_type = decision_type == COMMON_DECISION_TYPE_MINICPM_OCR_JEV ? LLAMA_POOLING_TYPE_UNSPECIFIED : LLAMA_POOLING_TYPE_NONE;
params.embedding = true;
params.pooling_type = LLAMA_POOLING_TYPE_NONE;
params.pooling_type = pooling_type;
cparams.embeddings = true;
cparams.pooling_type = LLAMA_POOLING_TYPE_NONE;
cparams.pooling_type = pooling_type;
cparams.n_outputs_max = cparams.n_batch;
cparams.n_outputs_max_per_seq = 1;
LOG_INF("%s", "decision model reads the embeddings output, enabling embedding mode\n");
}
// 0 is n_batch
if ((decision_type == COMMON_DECISION_TYPE_LFM3_D1 || decision_type == COMMON_DECISION_TYPE_LFM3_D1_OMNI) && cparams.n_outputs_max != 0) {
cparams.n_outputs_max = std::max(cparams.n_outputs_max, (uint32_t) COMMON_DECISION_LFM3_D1_SET_SIZE);
}
if (llama_model_classifier_free_guidance(model)) {
params.kv_unified = true;
cparams.kv_unified = true;
cparams.n_seq_max *= 2;
}
// embeddings need the whole batch in one ubatch, so n_batch must not be larger than n_ubatch
// (server.cpp does this check for --embedding, but before the model is loaded)
if (cparams.embeddings && cparams.n_batch > cparams.n_ubatch) {
@@ -1400,6 +1426,14 @@ std::vector<llama_adapter_lora_ptr> & common_init_result::lora() {
return pimpl->lora;
}
// only for warmup and probe decodes, fill zeros as dummy input
static void common_batch_set_zero_state(common_batch & batch, const llama_model * model, std::vector<float> & zeros) {
zeros.assign(llama_model_n_embd_out(model), 0.0f);
for (int32_t i = 0; i < batch.size(); ++i) {
batch.set_embd_state(i, { zeros.data(), 1, zeros.size() });
}
}
common_init_result_ptr common_init_from_params(common_params & params, bool model_only) {
common_init_result_ptr res(new common_init_result(params, model_only));
@@ -1430,7 +1464,7 @@ common_init_result_ptr common_init_from_params(common_params & params, bool mode
if (params.control_vector_layer_start <= 0) params.control_vector_layer_start = 1;
if (params.control_vector_layer_end <= 0) params.control_vector_layer_end = llama_model_n_layer(model);
const auto cvec = common_control_vector_load(params.control_vectors);
const auto cvec = common_control_vector_load(params.control_vectors, llama_model_n_layer(model));
if (cvec.n_embd == -1) {
return res;
}
@@ -1506,6 +1540,8 @@ common_init_result_ptr common_init_from_params(common_params & params, bool mode
if (llama_model_has_decoder(model)) {
tmp.resize(std::min(tmp.size(), (size_t) params.n_batch));
common_batch batch = common_batch_get_one(lctx, tmp);
std::vector<float> zeros;
common_batch_set_zero_state(batch, model, zeros);
llama_process(lctx, LLAMA_PROCESS_TYPE_DECODE, batch.get());
}
llama_memory_clear(llama_get_memory(lctx), true);
@@ -1573,6 +1609,8 @@ common_context_seq_rm_type common_context_can_seq_rm(llama_context * ctx) {
int ret;
{
common_batch batch = common_batch_get_one(ctx, tmp);
std::vector<float> zeros;
common_batch_set_zero_state(batch, llama_get_model(ctx), zeros);
ret = llama_process(ctx, LLAMA_PROCESS_TYPE_DECODE, batch.get());
}
if (ret != 0) {
@@ -1689,6 +1727,19 @@ struct llama_model_params common_model_params_to_llama(common_params & params) {
return mparams;
}
struct llama_model * common_model_load(const std::vector<std::string> & paths, const struct llama_model_params & mparams) {
if (paths.size() == 1) {
return llama_model_load_from_file(paths[0].c_str(), mparams);
}
std::vector<const char *> c_paths;
c_paths.reserve(paths.size());
for (const auto & path : paths) {
c_paths.push_back(path.c_str());
}
return llama_model_load_from_splits(c_paths.data(), c_paths.size(), mparams);
}
struct llama_context_params common_context_params_to_llama(const common_params & params) {
auto cparams = llama_context_default_params();
@@ -1725,7 +1776,8 @@ struct llama_context_params common_context_params_to_llama(const common_params &
cparams.type_k = params.cache_type_k;
cparams.type_v = params.cache_type_v;
cparams.moe_cache_size = params.moe_cache_size;
cparams.moe_cache_size = params.moe_cache_size;
cparams.moe_cache_cpu_share = params.moe_cache_cpu_share;
return cparams;
}
@@ -1948,7 +2000,7 @@ float common_embd_similarity_cos(const float * embd1, const float * embd2, int n
// Control vector utils
//
static common_control_vector_data common_control_vector_load_one(const common_control_vector_load_info & load_info) {
static common_control_vector_data common_control_vector_load_one(const common_control_vector_load_info & load_info, int32_t n_layer) {
common_control_vector_data result = { -1, {} };
ggml_context * ctx = nullptr;
@@ -1989,6 +2041,10 @@ static common_control_vector_data common_control_vector_load_one(const common_co
COM_ERR("invalid (zero) direction tensor layer index in %s\n", load_info.fname.c_str());
result.n_embd = -1;
break;
} else if (layer_idx > n_layer) {
COM_ERR("direction tensor layer index %d exceeds the model layer count %d in %s\n", layer_idx, n_layer, load_info.fname.c_str());
result.n_embd = -1;
break;
}
struct ggml_tensor * tensor = ggml_get_tensor(ctx, name.c_str());
@@ -2012,10 +2068,10 @@ static common_control_vector_data common_control_vector_load_one(const common_co
}
// extend if necessary - do not store data for layer 0 (it's not used)
result.data.resize(std::max(result.data.size(), static_cast<size_t>(result.n_embd * layer_idx)), 0.0f);
result.data.resize(std::max(result.data.size(), (size_t) result.n_embd * layer_idx), 0.0f);
const float * src = (const float *) tensor->data;
float * dst = result.data.data() + result.n_embd * (layer_idx - 1); // layer 1 at [0]
float * dst = result.data.data() + (size_t) result.n_embd * (layer_idx - 1); // layer 1 at [0]
for (int j = 0; j < result.n_embd; j++) {
dst[j] += src[j] * load_info.strength; // allows multiple directions for same layer in same file
}
@@ -2033,11 +2089,11 @@ static common_control_vector_data common_control_vector_load_one(const common_co
return result;
}
common_control_vector_data common_control_vector_load(const std::vector<common_control_vector_load_info> & load_infos) {
common_control_vector_data common_control_vector_load(const std::vector<common_control_vector_load_info> & load_infos, int32_t n_layer) {
common_control_vector_data result = { -1, {} };
for (const auto & info : load_infos) {
auto cur = common_control_vector_load_one(info);
auto cur = common_control_vector_load_one(info, n_layer);
if (cur.n_embd == -1) {
result.n_embd = -1;
@@ -2158,7 +2214,7 @@ void common_batch::clear() {
}
int32_t common_batch::add(llama_token id, llama_pos pos, llama_seq_id seq_id, bool output) {
tokens.push_back({ id, { pos, 0, 0, 0 }, seq_id, output, { nullptr, 0, 0 }, {} });
tokens.push_back({ id, { pos, 0, 0, 0 }, seq_id, output, { nullptr, 0, 0 }, { nullptr, 0, 0 }, {} });
return size() - 1;
}
@@ -2196,8 +2252,34 @@ bool common_batch::set_embd(int32_t idx, llama_embd embd) {
return true;
}
bool common_batch::set_embd_state(int32_t idx, llama_embd state) {
if (idx < 0 || idx >= size() || tokens[idx].state.data != nullptr) {
return false;
}
tokens[idx].state = state;
return true;
}
bool common_batch::set_embd_state_row(int32_t idx, int32_t row) {
if (idx < 0 || idx >= size() || row < 0 || tokens[idx].state.data != nullptr || tokens[idx].state_row >= 0) {
return false;
}
tokens[idx].state_row = row;
return true;
}
int32_t common_batch::add_embd(llama_embd embd, const llama_pos * pos, llama_seq_id seq_id, bool output) {
token t = { LLAMA_TOKEN_NULL, { 0, 0, 0, 0 }, seq_id, output, embd, {} };
token t = { LLAMA_TOKEN_NULL, { 0, 0, 0, 0 }, seq_id, output, embd, { nullptr, 0, 0 }, {} };
for (int32_t j = 0; j < n_pos; ++j) {
t.pos[j] = pos[j];
}
tokens.push_back(t);
return size() - 1;
}
int32_t common_batch::add_embd_row(int32_t row, const llama_pos * pos, llama_seq_id seq_id, bool output) {
token t = { LLAMA_TOKEN_NULL, { 0, 0, 0, 0 }, seq_id, output, { nullptr, 0, 0 }, { nullptr, 0, 0 }, {} };
t.embd_row = row;
for (int32_t j = 0; j < n_pos; ++j) {
t.pos[j] = pos[j];
}
@@ -2225,6 +2307,12 @@ llama_batch_ext * common_batch::get_sub_batch(int32_t off, int32_t n) {
if (t.embd.data && !llama_batch_ext_set_embd_token(res, idx, t.embd)) {
GGML_ABORT("%s: failed to set the embedding of token %d at index %d\n", __func__, t.id, i);
}
} else if (t.embd_row >= 0) {
idx = llama_batch_ext_add_embd_row(res, t.seq_id, t.embd_row);
if (idx < 0) {
GGML_ABORT("%s: failed to add embd row %d at index %d (error %d, n = %d)\n", __func__, t.embd_row, i, idx, n);
}
llama_batch_ext_set_pos(res, idx, t.pos.data());
} else {
idx = llama_batch_ext_add_embd(res, t.seq_id, t.embd);
if (idx < 0) {
@@ -2242,9 +2330,18 @@ llama_batch_ext * common_batch::get_sub_batch(int32_t off, int32_t n) {
if (t.output) {
llama_batch_ext_set_output_logits(res, idx, true);
}
if (t.state.data) {
llama_batch_ext_set_embd_state(res, idx, t.state); // contexts without a state input ignore it
}
if (t.state_row >= 0 && !llama_batch_ext_set_embd_state_row(res, idx, t.state_row)) {
GGML_ABORT("%s: failed to set the state row of the entry at index %d\n", __func__, i);
}
if (t.decision_order != 0) {
llama_batch_ext_set_decision_order(res, idx, (llama_decision_order) t.decision_order);
}
if (t.decision_group != 0 && !llama_batch_ext_set_decision_group(res, idx, t.decision_group)) {
GGML_ABORT("%s: failed to set the decision group of the entry at index %d\n", __func__, i);
}
}
return res;
+107 -28
View File
@@ -31,6 +31,8 @@
#define DIRECTORY_SEPARATOR '/'
#endif // _WIN32
#define COMMON_RECURSIVE_LIMIT 128
#define COM_DBG(fmt, ...) LOG_DBG("cmn %12.*s: " fmt, 12, __func__, __VA_ARGS__)
#define COM_TRC(fmt, ...) LOG_TRC("cmn %12.*s: " fmt, 12, __func__, __VA_ARGS__)
#define COM_INF(fmt, ...) LOG_INF("cmn %12.*s: " fmt, 12, __func__, __VA_ARGS__)
@@ -172,18 +174,20 @@ enum common_params_sampling_config : uint64_t {
};
enum common_speculative_type {
COMMON_SPECULATIVE_TYPE_NONE, // no speculative decoding
COMMON_SPECULATIVE_TYPE_DRAFT_SIMPLE, // standalone draft model speculative decoding
COMMON_SPECULATIVE_TYPE_DRAFT_EAGLE3, // Eagle3 speculative decoding
COMMON_SPECULATIVE_TYPE_DRAFT_MTP, // Multi-token prediction
COMMON_SPECULATIVE_TYPE_DRAFT_DFLASH, // DFlash speculative decoding
COMMON_SPECULATIVE_TYPE_DRAFT_DSPARK, // DSpark speculative decoding (DFlash + Markov head)
COMMON_SPECULATIVE_TYPE_NGRAM_SIMPLE, // simple self-speculative decoding based on n-grams
COMMON_SPECULATIVE_TYPE_NGRAM_MAP_K, // self-speculative decoding with n-gram keys only
COMMON_SPECULATIVE_TYPE_NGRAM_MAP_K4V, // self-speculative decoding with n-gram keys and 4 m-gram values
COMMON_SPECULATIVE_TYPE_NONE, // no speculative decoding
COMMON_SPECULATIVE_TYPE_DRAFT_SIMPLE, // standalone draft model speculative decoding
COMMON_SPECULATIVE_TYPE_DRAFT_EAGLE3, // Eagle3 speculative decoding
COMMON_SPECULATIVE_TYPE_DRAFT_MTP, // Multi-token prediction
COMMON_SPECULATIVE_TYPE_DRAFT_DFLASH, // DFlash speculative decoding
COMMON_SPECULATIVE_TYPE_DRAFT_DSPARK, // DSpark speculative decoding (DFlash + Markov head)
COMMON_SPECULATIVE_TYPE_DRAFT_DFLASH2, // DFlash2 speculative decoding (DFlash + selector)
COMMON_SPECULATIVE_TYPE_DRAFT_LILICORR, // LiliCorr speculative decoding (DFlash + scorer)
COMMON_SPECULATIVE_TYPE_NGRAM_SIMPLE, // simple self-speculative decoding based on n-grams
COMMON_SPECULATIVE_TYPE_NGRAM_MAP_K, // self-speculative decoding with n-gram keys only
COMMON_SPECULATIVE_TYPE_NGRAM_MAP_K4V, // self-speculative decoding with n-gram keys and 4 m-gram values
COMMON_SPECULATIVE_TYPE_NGRAM_MOD,
COMMON_SPECULATIVE_TYPE_NGRAM_CACHE, // self-speculative decoding with 3-level n-gram cache
COMMON_SPECULATIVE_TYPE_COUNT // number of types, unknown type
COMMON_SPECULATIVE_TYPE_NGRAM_CACHE, // self-speculative decoding with 3-level n-gram cache
COMMON_SPECULATIVE_TYPE_COUNT // number of types, unknown type
};
// Grammar type enumeration
@@ -309,6 +313,18 @@ struct common_params_model {
std::string hf_file = ""; // HF file
std::string docker_repo = ""; // Docker repo
std::vector<std::string> shards;
std::vector<std::string> get_paths() const {
std::vector<std::string> paths = { path };
for (const auto & shard : shards) {
if (std::find(paths.begin(), paths.end(), shard) == paths.end()) {
paths.push_back(shard);
}
}
return paths;
}
std::string get_name() const {
if (!hf_repo.empty()) {
return hf_repo;
@@ -343,6 +359,8 @@ struct common_params_speculative_draft {
int32_t n_gpu_layers = -1; // number of layers to store in VRAM for the draft model (-1 - use default)
int32_t n_ubatch = 0; // physical batch size of the draft context (0 - use the target's n_ubatch)
ggml_type cache_type_k = GGML_TYPE_F16; // KV cache data type for the K
ggml_type cache_type_v = GGML_TYPE_F16; // KV cache data type for the V
@@ -398,7 +416,12 @@ struct common_params_speculative {
uint32_t need_n_rs_seq() const {
bool needs_rs_seq = std::any_of(types.begin(), types.end(), [&](auto t) {
return t == COMMON_SPECULATIVE_TYPE_DRAFT_MTP || t == COMMON_SPECULATIVE_TYPE_DRAFT_EAGLE3 || t == COMMON_SPECULATIVE_TYPE_DRAFT_DFLASH || t == COMMON_SPECULATIVE_TYPE_DRAFT_DSPARK;
return t == COMMON_SPECULATIVE_TYPE_DRAFT_MTP ||
t == COMMON_SPECULATIVE_TYPE_DRAFT_EAGLE3 ||
t == COMMON_SPECULATIVE_TYPE_DRAFT_DFLASH ||
t == COMMON_SPECULATIVE_TYPE_DRAFT_DSPARK ||
t == COMMON_SPECULATIVE_TYPE_DRAFT_DFLASH2 ||
t == COMMON_SPECULATIVE_TYPE_DRAFT_LILICORR;
});
return needs_rs_seq ? draft.n_max : 0u;
@@ -593,7 +616,8 @@ struct common_params {
ggml_type cache_type_k = GGML_TYPE_F16; // KV cache data type for the K
ggml_type cache_type_v = GGML_TYPE_F16; // KV cache data type for the V
size_t moe_cache_size = 0; // GPU cache size in bytes for the MoE experts kept in the CPU, split among the GPUs like the layers
size_t moe_cache_size = 0; // GPU cache size in bytes for the MoE experts kept in the CPU, split among the GPUs like the layers
float moe_cache_cpu_share = 0.0f; // share of the MoE cache misses computed on the CPU instead of uploaded
common_conversation_mode conversation_mode = COMMON_CONVERSATION_MODE_AUTO;
@@ -625,7 +649,7 @@ struct common_params {
std::string cls_sep = "\t"; // separator of classification sequences
// server params
int32_t port = 8080; // server listens on this network port
int32_t port = 9931; // server listens on this network port
bool reuse_port = false; // allow multiple sockets to bind to the same port
int32_t timeout_read = 3600; // http read timeout in seconds
int32_t timeout_write = timeout_read; // http write timeout in seconds
@@ -638,6 +662,13 @@ struct common_params {
int32_t kv_unified_per_slot = 0; // max context per parallel slot; 0 = unset
int32_t checkpoint_min_step = 8192; // minimum spacing between context checkpoints
int32_t cache_ram_mib = 8192; // -1 = no limit, 0 - disable, 1 = 1 MiB, etc.
bool cache_ram_explicit = false; // set by --cache-ram, decision models disable the cache otherwise
std::string cache_disk_path; // directory of the disk prompt cache, empty = disabled
int32_t cache_disk_mib = 4096; // size limit of cache_disk_path in MiB, 0 = no limit
// llama-connect params
bool server_connect = false;
std::string server_connect_code = "";
std::string public_path = ""; // NOLINT
std::string api_prefix = ""; // NOLINT
@@ -673,6 +704,7 @@ struct common_params {
// UI configs
bool ui = true;
bool ui_mcp_proxy = false;
bool ui_mcp_proxy_allow_private_urls = false; // allow the MCP CORS proxy to reach local or private addresses
std::string ui_config_json;
// "advanced" endpoints are disabled by default for better security
@@ -683,6 +715,7 @@ struct common_params {
// enable built-in tools
std::vector<std::string> server_tools;
std::string server_tools_runtime;
bool server_tools_allow_unauth = false; // allow tools/MCP on a non-loopback address without API key
// MCP server configs (Cursor-compatible JSON)
std::string mcp_servers_config; // path to JSON file with MCP server definitions
@@ -692,6 +725,7 @@ struct common_params {
std::string models_dir = ""; // directory containing models for the router server
std::string models_preset = ""; // directory containing model presets for the router server
int models_max = 4; // maximum number of models to load simultaneously
int models_memory_margin = 0; // MiB of device memory to keep free when loading models (0 = no memory accounting)
bool models_autoload = true; // automatically load models when requested via the router server
std::string models_preset_hf = ""; // show a warning about remote presets on router loaded (if not empty)
@@ -699,6 +733,7 @@ struct common_params {
std::string slot_save_path;
std::string media_path; // path to directory for loading media files
bool media_allow_private_urls = false; // allow http(s) media URLs on local or private addresses
float slot_prompt_similarity = 0.1f;
@@ -762,7 +797,9 @@ struct common_params {
// TTS params
std::string tts_lang = "";
std::string tts_voice = "";
std::string tts_speaker_file = "";
std::string tts_speaker_text = "";
bool is_gen_docs = false; // whether we are running inside llama-gen-docs
};
@@ -895,13 +932,9 @@ bool glob_match(const std::string & pattern, const std::string & str);
//
#ifdef _WIN32
std::wstring utf8_to_wstring(const std::string & str);
std::string wstring_to_utf8(const std::wstring & str);
std::string wstring_to_utf8(const std::wstring & str);
#endif
// returns the path as a UTF-8 string, preserving its separators
std::string fs_path_to_utf8(const std::filesystem::path & path);
//
// Environment utils
//
@@ -911,9 +944,6 @@ std::string fs_path_to_utf8(const std::filesystem::path & path);
std::string common_get_env(const std::string & name);
void common_set_env(const std::string & name, const std::string & value);
// reads a path from the environment, an unset variable gives an empty path
std::filesystem::path common_get_path_from_env(const std::string & name);
//
// Filesystem utils
//
@@ -963,16 +993,31 @@ enum common_decision_type {
COMMON_DECISION_TYPE_LAYA, // score of one marker token per option, read from the embeddings output
COMMON_DECISION_TYPE_CLEF, // all questions in one prompt, score of option i read from the embeddings output at row i
COMMON_DECISION_TYPE_PPLX_DECIDER, // same as openjev, label codes of 1 or 2 letters
COMMON_DECISION_TYPE_VLLM_DECISION2, // score of each option read from the embeddings output, the options of a question are marked in the batch
COMMON_DECISION_TYPE_VLLM_D3, // same as pplx-decider, non-causal attention
COMMON_DECISION_TYPE_LFM2_D1, // same as openjev, the labels depend on the question type
COMMON_DECISION_TYPE_LFM2_D1_OMNI, // same as laya, other prompt layout
COMMON_DECISION_TYPE_GRANITE_SWITCH, // same as openjev, run by the decision aLoRA of granite switch
COMMON_DECISION_TYPE_H2O_LIGHTNING, // same as openjev, the labels are read after an answer prefill
COMMON_DECISION_TYPE_MINICPM_OCR_JEV, // one score per class, the pooled embeddings output of the last token
COMMON_DECISION_TYPE_LFM3_D1, // same as lfm2-d1, non-causal, label codes A..Z then AA, AB..
COMMON_DECISION_TYPE_LFM3_D1_OMNI, // same as lfm3-d1
COMMON_DECISION_TYPE_UNKNOWN, // a decision model of a type that is not supported
};
// lfm3-d1: max questions packed in one prompt
#define COMMON_DECISION_LFM3_D1_SET_SIZE 16
common_decision_type common_get_decision_type(const struct llama_model * model);
// same as above, but reads a GGUF file; it does not load the model
// returns COMMON_DECISION_TYPE_UNKNOWN if the file is missing, unreadable, or invalid
common_decision_type common_get_decision_type(const std::string & fname);
// metadata of a GGUF file, read without loading the model
struct common_gguf_info {
common_decision_type decision_type = COMMON_DECISION_TYPE_UNKNOWN; // UNKNOWN if the file is missing, unreadable, or invalid
uint32_t n_ctx_train = 0; // 0 if unknown
bool has_chat_tmpl = false; // has the default tokenizer.chat_template
};
common_gguf_info common_get_gguf_info(const std::string & fname);
// note: defines the model, context, samplers, ets. lifetimes
struct common_init_result {
@@ -999,6 +1044,11 @@ common_init_result_ptr common_init_from_params(common_params & params, bool mode
struct llama_model_params common_model_params_to_llama ( common_params & params);
struct llama_context_params common_context_params_to_llama(const common_params & params);
// apply --fit to mparams and cparams, no-op if params.fit_params is false
void common_fit_apply(common_params & params, struct llama_model_params & mparams, struct llama_context_params & cparams);
struct llama_model * common_model_load(const std::vector<std::string> & paths, const struct llama_model_params & mparams);
// clear LoRA adapters from context, then apply new list of adapters
void common_set_adapter_lora(struct llama_context * ctx, std::vector<common_adapter_lora_info> & lora);
@@ -1070,8 +1120,12 @@ struct common_batch {
llama_seq_id seq_id; // the first sequence id, see add_seq()
bool output;
llama_embd embd; // non-owning view of the data passed to add_embd()/set_embd(), data == NULL if none
llama_embd state; // non-owning view of the data passed to set_embd_state(), data == NULL if none
std::vector<llama_seq_id> seq_ids_extra; // see add_seq()
int32_t decision_order = 0; // see llama_batch_ext_set_decision_order()
int32_t decision_group = 0; // see llama_batch_ext_set_decision_group()
int32_t embd_row = -1; // see add_embd_row(), -1 if none
int32_t state_row = -1; // see set_embd_state_row(), -1 if none
};
std::vector<token> tokens; // mirror of the entries, tokens[i] describes batch index i
@@ -1107,10 +1161,20 @@ struct common_batch {
// attach a token embedding to the entry at idx, can only be set once per entry
bool set_embd(int32_t idx, llama_embd embd);
// attach a state embedding (e.g. the target hidden state for MTP) to the entry at idx, can only be set once per entry
bool set_embd_state(int32_t idx, llama_embd state);
// the state of entry idx is row `row` of the h_nextn device buffer of the state source context, see llama_set_ctx_state_src()
bool set_embd_state_row(int32_t idx, int32_t row);
// add an embedding-only entry (no token id)
// pos points to n_pos positions
int32_t add_embd(llama_embd embd, const llama_pos * pos, llama_seq_id seq_id, bool output);
// add an entry whose embedding is row `row` of the layer input device buffer of the context's ctx_other
// pos points to n_pos positions
int32_t add_embd_row(int32_t row, const llama_pos * pos, llama_seq_id seq_id, bool output);
int32_t size() const { return (int32_t) tokens.size(); }
};
@@ -1207,8 +1271,9 @@ struct common_control_vector_load_info {
};
// Load control vectors, scale each by strength, and add them together.
// Direction layer indices must be in [1, n_layer].
// On error, returns {-1, empty}
common_control_vector_data common_control_vector_load(const std::vector<common_control_vector_load_info> & load_infos);
common_control_vector_data common_control_vector_load(const std::vector<common_control_vector_load_info> & load_infos, int32_t n_layer);
//
// Split utils
@@ -1260,6 +1325,20 @@ enum ggml_opt_optimizer_type common_opt_get_optimizer(const char *);
// prompt utils
//
// leave default-inserted elements uninitialized; callers must fill them before use
template <typename T>
struct common_no_init_allocator : std::allocator<T> {
template <typename U> struct rebind { using other = common_no_init_allocator<U>; };
common_no_init_allocator() = default;
template <typename U> common_no_init_allocator(const common_no_init_allocator<U> &) {}
template <typename U> void construct(U * p) { ::new ((void *) p) U; }
};
// sequence state data, filled by llama_state_seq_get_data_ext()
using common_state_data = std::vector<uint8_t, common_no_init_allocator<uint8_t>>;
struct common_prompt_checkpoint {
int64_t n_tokens;
@@ -1269,8 +1348,8 @@ struct common_prompt_checkpoint {
llama_pos pos_min;
llama_pos pos_max;
std::vector<uint8_t> data_tgt;
std::vector<uint8_t> data_dft;
common_state_data data_tgt;
common_state_data data_dft;
// (optional) speculative-decoding implementation state stashed with the checkpoint
// (e.g. eagle3's deferred-boundary g_embd row)
+302 -118
View File
@@ -4,6 +4,7 @@
#include "common.h"
#include "log.h"
#include "download.h"
#include "file-lock.h"
#include "hf-cache.h"
#include "json.h"
@@ -39,9 +40,12 @@
// downloader
//
// the suffix of the file a download writes until it completes
static const std::string PARTIAL_SUFFIX = ".downloadInProgress";
static void write_etag(const std::string & path, const std::string & etag) {
const std::string etag_path = path + ".etag";
fs_write_atomic(std::filesystem::u8path(etag_path), etag);
fs_write_atomic(etag_path, etag);
LOG_DBG("%s: file etag saved: %s\n", __func__, etag_path.c_str());
}
@@ -74,6 +78,18 @@ std::pair<std::string, std::string> common_download_split_repo_tag(const std::st
return {hf_repo, tag};
}
// file name from url, without the query string
static std::string url_get_filename(const std::string & url) {
std::string filename = url;
if (auto pos = filename.rfind('/'); pos != std::string::npos) {
filename = filename.substr(pos + 1);
}
if (auto pos = filename.find('?'); pos != std::string::npos) {
filename = filename.substr(0, pos);
}
return filename;
}
class ProgressBar : public common_download_callback {
static inline std::mutex mutex;
static inline std::map<const ProgressBar *, int> lines;
@@ -97,14 +113,8 @@ public:
ProgressBar() = default;
void on_start(const common_download_progress & p) override {
filename = p.url;
filename = url_get_filename(p.url);
if (auto pos = filename.rfind('/'); pos != std::string::npos) {
filename = filename.substr(pos + 1);
}
if (auto pos = filename.find('?'); pos != std::string::npos) {
filename = filename.substr(0, pos);
}
for (size_t i = 0; i < filename.size(); ++i) {
if ((filename[i] & 0xC0) != 0x80) {
if (len++ == 39) {
@@ -186,6 +196,13 @@ static bool common_pull_file(httplib::Client & cli,
[&](const httplib::Response &response) {
if (p.downloaded > 0 && response.status != 206) {
LOG_WRN("%s: server did not respond with 206 Partial Content for a resume request. Status: %d\n", func, response.status);
if (response.status == 416 || response.status == 200) {
// the kept partial cannot be resumed: 416 for a complete or stale one,
// 200 from a server that ignores the range; drop it, so the next
// attempt starts over instead of wedging on the same answer
ofs.close();
remove(path_tmp.c_str());
}
return false;
}
if (p.downloaded == 0 && response.status != 200) {
@@ -214,7 +231,7 @@ static bool common_pull_file(httplib::Client & cli,
if (progress_step >= p.total / 1000 || p.downloaded == p.total) {
if (callback) {
callback->on_update(p);
if (callback->is_cancelled()) {
if (callback->is_cancelled() && (p.total == 0 || p.downloaded != p.total)) {
return false;
}
}
@@ -247,15 +264,61 @@ static bool common_pull_file(httplib::Client & cli,
static int common_download_file_single_online(const std::string & url,
const std::string & path,
const common_download_opts & opts,
bool skip_etag) {
bool skip_etag,
const std::string & lock_path,
const std::string & alt_path) {
static const int max_attempts = 3;
static const int retry_delay_seconds = 2;
const bool file_exists = std::filesystem::exists(std::filesystem::u8path(path));
const std::string path_temporary = path + PARTIAL_SUFFIX;
const std::string path_lock = lock_path.empty() ? path + ".lock" : lock_path;
common_download_progress p;
p.url = url;
if (opts.callback) {
opts.callback->on_start(p);
}
// the completion callback may itself take the same lock (the HF cache
// finalizes blobs under it), so every exit goes through finish()
common_file_lock lock(path_lock);
auto finish = [&](int status) {
lock.close();
if (opts.callback) {
opts.callback->on_done(p, is_http_status_ok(status));
}
return status;
};
// alt_path is the HF snapshot: the blob may have been moved there when symlinks are unavailable
auto cache_hit = [&] {
return std::filesystem::exists(path) || (!alt_path.empty() && std::filesystem::exists(alt_path));
};
if (skip_etag && cache_hit()) {
LOG_DBG("%s: using cached file: %s\n", __func__, path.c_str());
return finish(304); // 304 Not Modified - fake cached response
}
// take the lock before touching the cache, so two processes cannot race on the same file
bool waited = false;
if (!lock.acquire([&] {
if (!waited) {
LOG_INF("%s: file '%s' is being downloaded by another process, waiting...\n", __func__, url_get_filename(url).c_str());
waited = true;
}
return !(opts.callback && opts.callback->is_cancelled());
})) {
return finish(-1);
}
// under the lock: another process may have finished while we waited
bool file_exists = cache_hit();
if (file_exists && skip_etag) {
LOG_DBG("%s: using cached file: %s\n", __func__, path.c_str());
return 304; // 304 Not Modified - fake cached response
return finish(304); // 304 Not Modified - fake cached response
}
auto [cli, parts] = common_http_client(url);
@@ -284,9 +347,9 @@ static int common_download_file_single_online(const std::string & url,
LOG_TRC("%s: HEAD failed, status: %d\n", __func__, head ? head->status : -1);
if (file_exists) {
LOG_TRC("%s: using cached file (HEAD failed): %s\n", __func__, path.c_str());
return 304; // 304 Not Modified - fake cached response
return finish(304); // 304 Not Modified - fake cached response
}
return head ? head->status : -1;
return finish(head ? head->status : -1);
}
std::string etag;
@@ -294,8 +357,6 @@ static int common_download_file_single_online(const std::string & url,
etag = head->get_header_value("ETag");
}
common_download_progress p;
p.url = url;
if (head->has_header("Content-Length")) {
try {
p.total = std::stoull(head->get_header_value("Content-Length"));
@@ -312,16 +373,16 @@ static int common_download_file_single_online(const std::string & url,
if (file_exists) {
if (etag.empty()) {
LOG_DBG("%s: using cached file (no server etag): %s\n", __func__, path.c_str());
return 304; // 304 Not Modified - fake cached response
return finish(304); // 304 Not Modified - fake cached response
}
if (!last_etag.empty() && last_etag == etag) {
LOG_DBG("%s: using cached file (same etag): %s\n", __func__, path.c_str());
return 304; // 304 Not Modified - fake cached response
return finish(304); // 304 Not Modified - fake cached response
}
// pass this point, the file exists but is different from the server version, so we need to redownload it
if (remove(path.c_str()) != 0) {
LOG_ERR("%s: unable to delete file: %s\n", __func__, path.c_str());
return -1;
return finish(-1);
}
}
@@ -331,11 +392,24 @@ static int common_download_file_single_online(const std::string & url,
}
bool success = false;
const std::string path_temporary = path + ".downloadInProgress";
int delay = retry_delay_seconds;
if (!skip_etag && !etag.empty()) {
if (std::filesystem::exists(path_temporary) && read_etag(path_temporary) != etag) {
LOG_WRN("%s: partial file does not match the server etag, discarding it: %s\n", __func__, path_temporary.c_str());
if (remove(path_temporary.c_str()) != 0) {
LOG_ERR("%s: unable to delete file: %s\n", __func__, path_temporary.c_str());
return finish(-1);
}
}
write_etag(path_temporary, etag);
}
if (opts.callback) {
opts.callback->on_start(p);
// report the resume offset, so a resumed file does not start at 0
p.downloaded = supports_ranges && std::filesystem::exists(path_temporary) ?
std::filesystem::file_size(path_temporary) : 0;
opts.callback->on_update(p);
}
for (int i = 0; i < max_attempts; ++i) {
@@ -372,77 +446,88 @@ static int common_download_file_single_online(const std::string & url,
}
if (!etag.empty() && !skip_etag) {
write_etag(path, etag);
remove((path_temporary + ".etag").c_str());
}
success = true;
break;
}
}
if (opts.callback) {
opts.callback->on_done(p, success);
}
if (opts.callback && opts.callback->is_cancelled() &&
std::filesystem::exists(path_temporary)) {
if (remove(path_temporary.c_str()) != 0) {
LOG_ERR("%s: unable to delete temporary file: %s\n", __func__, path_temporary.c_str());
}
}
// a stopped download keeps path_temporary, so the next attempt resumes from it
if (!success) {
LOG_ERR("%s: download failed after %d attempts\n", __func__, max_attempts);
return -1; // max attempts reached
}
return head->status;
return finish(success ? head->status : -1);
}
std::pair<long, std::vector<char>> common_remote_get_content(const std::string & url,
std::pair<long, std::vector<char>> common_remote_get_content(const std::string & url_in,
const common_remote_params & params) {
auto [cli, parts] = common_http_client(url);
static constexpr int max_redirects = 5;
std::string url = url_in;
for (int n_redirects = 0; ; n_redirects++) {
auto [cli, parts] = common_http_client(url);
httplib::Headers headers;
for (const auto & h : params.headers) {
headers.emplace(h.first, h.second);
httplib::Headers headers;
for (const auto & h : params.headers) {
headers.emplace(h.first, h.second);
}
if (headers.find("User-Agent") == headers.end()) {
headers.emplace("User-Agent", "llama-cpp/" + std::string(llama_build_info()));
}
if (params.timeout > 0) {
cli.set_read_timeout(params.timeout, 0);
cli.set_write_timeout(params.timeout, 0);
}
if (params.resolve_host) {
// pin checked address, follow redirects manually to check each hop
cli.set_direct_hostname_addr_map({{parts.host, params.resolve_host(parts.host)}});
cli.set_follow_location(false);
}
std::vector<char> buf;
auto res = cli.Get(parts.path, headers,
[&](const char *data, size_t len) {
buf.insert(buf.end(), data, data + len);
return params.max_size == 0 ||
buf.size() <= static_cast<size_t>(params.max_size);
},
nullptr
);
if (!res) {
throw std::runtime_error("error: cannot make GET request");
}
if (params.resolve_host && res->status >= 300 && res->status < 400 && res->has_header("Location")) {
if (n_redirects >= max_redirects) {
throw std::runtime_error("error: too many redirects");
}
url = common_http_resolve_location(url, res->get_header_value("Location"));
continue;
}
return { res->status, std::move(buf) };
}
if (headers.find("User-Agent") == headers.end()) {
headers.emplace("User-Agent", "llama-cpp/" + std::string(llama_build_info()));
}
if (params.timeout > 0) {
cli.set_read_timeout(params.timeout, 0);
cli.set_write_timeout(params.timeout, 0);
}
std::vector<char> buf;
auto res = cli.Get(parts.path, headers,
[&](const char *data, size_t len) {
buf.insert(buf.end(), data, data + len);
return params.max_size == 0 ||
buf.size() <= static_cast<size_t>(params.max_size);
},
nullptr
);
if (!res) {
throw std::runtime_error("error: cannot make GET request");
}
return { res->status, std::move(buf) };
}
int common_download_file_single(const std::string & url,
const std::string & path,
const common_download_opts & opts,
bool skip_etag) {
bool skip_etag,
const std::string & lock_path,
const std::string & alt_path) {
if (!opts.offline) {
ProgressBar tty_cb;
common_download_opts online_opts = opts;
if (!online_opts.callback) {
online_opts.callback = &tty_cb;
}
return common_download_file_single_online(url, path, online_opts, skip_etag);
return common_download_file_single_online(url, path, online_opts, skip_etag, lock_path, alt_path);
}
if (!std::filesystem::exists(std::filesystem::u8path(path))) {
if (!std::filesystem::exists(path)) {
LOG_ERR("%s: required file is not available in cache (offline mode): %s\n", __func__, path.c_str());
return -1;
}
@@ -594,33 +679,65 @@ static hf_cache::hf_file find_best_sibling(const hf_cache::hf_files & files,
return best;
}
static hf_cache::hf_file find_best_mmproj(const hf_cache::hf_files & files,
const std::string & model) {
return find_best_sibling(files, model, "mmproj");
static const std::vector<common_download_sidecar> sidecars = {
{ "mmproj", "mmproj", false, &common_download_opts::download_mmproj, &common_download_hf_plan::mmproj },
{ "mtp", "mtp-", true, &common_download_opts::download_mtp, &common_download_hf_plan::mtp },
{ "dflash", "dflash-", true, &common_download_opts::download_dflash, &common_download_hf_plan::dflash },
{ "eagle3", "eagle3-", true, &common_download_opts::download_eagle3, &common_download_hf_plan::eagle3 },
{ "dspark", "dspark-", true, &common_download_opts::download_dspark, &common_download_hf_plan::dspark },
};
const std::vector<common_download_sidecar> & common_download_sidecars() {
return sidecars;
}
static hf_cache::hf_file find_best_mtp(const hf_cache::hf_files & files,
const std::string & model,
const std::string & tag = "") {
return find_best_sibling(files, model, "mtp-", tag);
bool common_download_opts_add_sidecar(common_download_opts & opts, const std::string & kind) {
for (const auto & s : sidecars) {
if (kind == s.name) {
opts.*s.opt = true;
return true;
}
}
return false;
}
static hf_cache::hf_file find_best_eagle3(const hf_cache::hf_files & files,
const std::string & model,
const std::string & tag = "") {
return find_best_sibling(files, model, "eagle3-", tag);
std::vector<std::string> common_download_plan_sidecars(const common_download_hf_plan & plan) {
std::vector<std::string> out;
for (const auto & s : sidecars) {
if (!(plan.*s.file).path.empty()) {
out.emplace_back(s.name);
}
}
return out;
}
static hf_cache::hf_file find_best_dflash(const hf_cache::hf_files & files,
const std::string & model,
const std::string & tag = "") {
return find_best_sibling(files, model, "dflash-", tag);
std::vector<common_download_partial> common_download_plan_partials(const common_download_hf_plan & plan) {
std::vector<common_download_partial> out;
// a file in the cache resolves to its snapshot entry, a file still to download to its blob
auto add = [&](const std::string & kind, const hf_cache::hf_file & f) {
if (!f.local_path.empty() && f.local_path != f.final_path) {
out.push_back({ kind, f.local_path + PARTIAL_SUFFIX });
}
};
for (const auto & f : plan.model_files) {
add("", f);
}
for (const auto & s : sidecars) {
add(s.name, plan.*s.file);
}
return out;
}
static hf_cache::hf_file find_best_dspark(const hf_cache::hf_files & files,
const std::string & model,
const std::string & tag = "") {
return find_best_sibling(files, model, "dspark-", tag);
// true when `name` carries the keyword of a sidecar kind
static bool name_is_sidecar(const std::string & name) {
for (const auto & s : sidecars) {
if (name.find(s.keyword) != std::string::npos) {
return true;
}
}
return false;
}
static bool gguf_filename_is_model(const std::string & filepath) {
@@ -633,12 +750,7 @@ static bool gguf_filename_is_model(const std::string & filepath) {
filename = filename.substr(pos + 1);
}
return filename.find("mmproj") == std::string::npos &&
filename.find("imatrix") == std::string::npos &&
filename.find("mtp-") == std::string::npos &&
filename.find("eagle3-") == std::string::npos &&
filename.find("dflash-") == std::string::npos &&
filename.find("dspark-") == std::string::npos;
return filename.find("imatrix") == std::string::npos && !name_is_sidecar(filename);
}
static hf_cache::hf_file find_best_model(const hf_cache::hf_files & files,
@@ -652,7 +764,8 @@ static hf_cache::hf_file find_best_model(const hf_cache::hf_files & files,
}
for (const auto & t : tags) {
std::regex pattern(t + "[.-]", std::regex::icase);
// match the tag as a full token, so ":F16" does not match a BF16 file
std::regex pattern("(^|[-./])" + t + "[.-]", std::regex::icase);
for (const auto & f : files) {
if (gguf_filename_is_model(f.path) &&
std::regex_search(f.path, pattern)) {
@@ -730,8 +843,11 @@ common_download_hf_plan common_download_get_hf_plan(const common_params_model &
}
} else {
primary = find_best_model(all, tag);
// a requested sidecar can resolve on its own, without a full model of the same tag
if (primary.path.empty() && !opts.download_mtp && !opts.download_dflash && !opts.download_eagle3 && !opts.download_dspark) {
// a requested draft can resolve on its own, without a full model of the same tag
const bool want_draft = std::any_of(sidecars.begin(), sidecars.end(), [&](const common_download_sidecar & s) {
return s.draft && opts.*s.opt;
});
if (primary.path.empty() && !want_draft) {
LOG_ERR("%s: no GGUF files found in repository %s\n", __func__, repo.c_str());
list_available_gguf_files(all);
return plan;
@@ -743,24 +859,22 @@ common_download_hf_plan common_download_get_hf_plan(const common_params_model &
plan.model_files = get_split_files(all, primary);
}
if (opts.download_mmproj && !primary.path.empty()) {
plan.mmproj = find_best_mmproj(all, primary.path);
}
if (opts.download_mtp) {
plan.mtp = find_best_mtp(all, primary.path, tag);
}
if (opts.download_dflash) {
plan.dflash = find_best_dflash(all, primary.path, tag);
}
if (opts.download_eagle3) {
plan.eagle3 = find_best_eagle3(all, primary.path, tag);
}
if (opts.download_dspark) {
plan.dspark = find_best_dspark(all, primary.path, tag);
// a draft resolves at the tag, the mmproj pairs with the model
for (const auto & s : sidecars) {
if (!(opts.*s.opt)) {
continue;
}
if (s.draft) {
plan.*s.file = find_best_sibling(all, primary.path, s.keyword, tag);
} else if (!primary.path.empty()) {
plan.*s.file = find_best_sibling(all, primary.path, s.keyword);
}
}
if (primary.path.empty() &&
plan.mtp.local_path.empty() && plan.dflash.local_path.empty() && plan.eagle3.local_path.empty() && plan.dspark.local_path.empty()) {
const bool has_draft = std::any_of(sidecars.begin(), sidecars.end(), [&](const common_download_sidecar & s) {
return s.draft && !(plan.*s.file).local_path.empty();
});
if (primary.path.empty() && !has_draft) {
LOG_ERR("%s: no GGUF files found in repository %s\n", __func__, repo.c_str());
list_available_gguf_files(all);
}
@@ -773,7 +887,7 @@ void common_download_run_tasks(const std::vector<common_download_task> & tasks)
for (const auto & task : tasks) {
futures.push_back(std::async(std::launch::async,
[&task]() {
return common_download_file_single(task.url, task.local_path, task.opts, task.is_hf);
return common_download_file_single(task.url, task.local_path, task.opts, task.is_hf, task.lock_path, task.alt_path);
}
));
}
@@ -908,7 +1022,7 @@ std::string common_docker_resolve_model(const std::string & docker) {
std::string model_filename = repo;
std::replace(model_filename.begin(), model_filename.end(), '/', '_');
model_filename += "_" + tag + ".gguf";
std::string local_path = fs_path_to_utf8(fs_get_cache_file(model_filename));
std::string local_path = fs_get_cache_file(model_filename).string();
const std::string blob_url = url_prefix + "/blobs/" + gguf_digest;
common_download_opts opts;
@@ -926,6 +1040,11 @@ std::string common_docker_resolve_model(const std::string & docker) {
}
}
// a cached file that lists a model under its tag: the first split of a model, not a sidecar
static bool is_listed_model(const gguf_split_info & split) {
return split.index == 1 && !split.tag.empty() && !name_is_sidecar(split.prefix);
}
std::vector<common_cached_model_info> common_list_cached_models() {
std::unordered_set<std::string> seen;
std::vector<common_cached_model_info> result;
@@ -934,12 +1053,7 @@ std::vector<common_cached_model_info> common_list_cached_models() {
for (const auto & f : files) {
auto split = get_gguf_split_info(f.path);
if (split.index != 1 || split.tag.empty() ||
split.prefix.find("mmproj") != std::string::npos ||
split.prefix.find("mtp-") != std::string::npos ||
split.prefix.find("eagle3-") != std::string::npos ||
split.prefix.find("dflash-") != std::string::npos ||
split.prefix.find("dspark-") != std::string::npos) {
if (!is_listed_model(split)) {
continue;
}
if (seen.insert(f.repo_id + ":" + split.tag).second) {
@@ -970,6 +1084,44 @@ std::string common_download_resolve_path(const std::string & hf_repo_with_tag, c
return find_best_model(files, tag).local_path;
}
// the files -hf loads from the cache, the model with every sidecar, for each cached
// tag of `repo_id` other than `tag`
static std::unordered_set<std::string> files_in_use(const std::string & repo_id, const std::string & tag) {
common_download_opts opts;
opts.offline = true;
for (const auto & s : sidecars) {
opts.*s.opt = true;
}
std::unordered_set<std::string> tags;
std::unordered_set<std::string> used;
for (const auto & f : hf_cache::get_cached_files(repo_id)) {
auto split = get_gguf_split_info(f.path);
if (!is_listed_model(split) || split.tag == tag || !tags.insert(split.tag).second) {
continue;
}
common_params_model model;
model.hf_repo = repo_id + ":" + split.tag;
auto plan = common_download_get_hf_plan(model, opts);
for (const auto & m : plan.model_files) {
used.insert(m.local_path);
}
for (const auto & s : sidecars) {
const auto & sf = plan.*s.file;
if (!sf.local_path.empty()) {
used.insert(sf.local_path);
}
}
}
return used;
}
static bool remove_snapshot_entries(const std::string & repo_id, const std::vector<std::filesystem::path> & to_remove);
bool common_download_remove(const std::string & hf_repo_with_tag) {
namespace fs = std::filesystem;
@@ -989,11 +1141,13 @@ bool common_download_remove(const std::string & hf_repo_with_tag) {
return false;
}
// collect snapshot entries whose tag matches
// collect snapshot entries whose tag matches, but a file another tag loads stays
const auto used = files_in_use(repo_id, tag_upper);
std::vector<fs::path> to_remove;
for (const auto & f : files) {
auto split = get_gguf_split_info(f.path);
if (split.tag == tag_upper) {
if (split.tag == tag_upper && used.find(f.local_path) == used.end()) {
to_remove.emplace_back(f.local_path);
}
}
@@ -1002,6 +1156,13 @@ bool common_download_remove(const std::string & hf_repo_with_tag) {
return false;
}
return remove_snapshot_entries(repo_id, to_remove);
}
// remove snapshot entries of `repo_id`, then the blobs no remaining entry references
static bool remove_snapshot_entries(const std::string & repo_id, const std::vector<std::filesystem::path> & to_remove) {
namespace fs = std::filesystem;
// resolve blob paths from symlinks before deleting snapshot entries
std::vector<fs::path> blobs_to_check;
for (const auto & p : to_remove) {
@@ -1053,3 +1214,26 @@ bool common_download_remove(const std::string & hf_repo_with_tag) {
return true;
}
bool common_download_remove_sidecars(const std::string & hf_repo_with_tag, const common_download_opts & opts) {
common_params_model model;
model.hf_repo = hf_repo_with_tag;
common_download_opts cached = opts;
cached.offline = true;
auto plan = common_download_get_hf_plan(model, cached);
std::vector<std::filesystem::path> to_remove;
for (const auto & s : sidecars) {
const auto & f = plan.*s.file;
if (!f.local_path.empty()) {
to_remove.emplace_back(f.local_path);
}
}
if (to_remove.empty()) {
return false;
}
return remove_snapshot_entries(common_download_split_repo_tag(hf_repo_with_tag).first, to_remove);
}
+49 -3
View File
@@ -31,6 +31,8 @@ struct common_remote_params {
common_header_list headers;
long timeout = 0; // in seconds, 0 means no timeout
long max_size = 0; // unlimited if 0
// if set: called per hop (redirects too), returns the address to connect to, throws to refuse
std::function<std::string(const std::string & host)> resolve_host;
};
// get remote file content, returns <http_code, raw_response_body>
@@ -69,12 +71,15 @@ struct common_download_task {
std::string local_path;
std::function<void()> on_done;
bool is_hf = false;
std::string lock_path;
std::string alt_path; // HF snapshot path, also treated as cached (the blob may be moved there)
common_download_task() = default;
common_download_task(hf_cache::hf_file f,
const common_download_opts & opts,
std::function<void()> on_done = nullptr)
: opts(opts), url(f.url), local_path(f.local_path), on_done(on_done), is_hf(true) {}
: opts(opts), url(f.url), local_path(f.local_path), on_done(on_done), is_hf(true),
lock_path(hf_cache::get_lock_path(f.repo_id, f.oid)), alt_path(f.final_path) {}
};
void common_download_run_tasks(const std::vector<common_download_task> & tasks);
@@ -92,10 +97,14 @@ std::string common_download_resolve_path(const std::string & hf_repo_with_tag, c
// download single file from url to local path
// returns status code or -1 on error
// skip_etag: if true, don't read/write .etag files (for HF cache where filename is the hash)
// lock_path: lock file used to serialize downloads of the same file; when empty, "<path>.lock" is used
// alt_path: additional path treated as present in cache (see cache_hit in download.cpp)
int common_download_file_single(const std::string & url,
const std::string & path,
const common_download_opts & opts = {},
bool skip_etag = false);
bool skip_etag = false,
const std::string & lock_path = "",
const std::string & alt_path = "");
// resolve and download model from Docker registry
// return local path to downloaded model file
@@ -104,10 +113,17 @@ std::string common_docker_resolve_model(const std::string & docker);
// Remove a cached model from disk
// input format: "user/model" or "user/model:tag"
// - if tag is omitted, removes the entire repo cache directory
// - if tag is present, removes only files matching that tag (and orphaned blobs)
// - if tag is present, removes only files matching that tag (and orphaned blobs),
// except the files another cached tag of the repo still loads
// returns true if anything was removed
bool common_download_remove(const std::string & hf_repo_with_tag);
// remove the cached sidecar files `opts` selects for "user/model:tag", as the cached
// plan resolves them, and leave the model files in place; a sidecar file another tag
// shares goes for every tag that resolves it
// returns true if anything was removed
bool common_download_remove_sidecars(const std::string & hf_repo_with_tag, const common_download_opts & opts);
struct common_download_hf_plan {
hf_cache::hf_file primary;
hf_cache::hf_files model_files;
@@ -119,3 +135,33 @@ struct common_download_hf_plan {
hf_cache::hf_file preset; // if set, only this file is downloaded
};
common_download_hf_plan common_download_get_hf_plan(const common_params_model & model, const common_download_opts & opts);
// a sidecar kind a download can pull next to its model: mmproj, and the draft heads
// of --spec-type draft-<name>
struct common_download_sidecar {
const char * name; // the kind, as the API names it
const char * keyword; // the part of a file name that marks a file of this kind
bool draft; // a draft head, resolved at the tag even without a model
bool common_download_opts::* opt; // the option that requests it
hf_cache::hf_file common_download_hf_plan::* file; // the plan file it resolves to
};
// every sidecar kind, the single list the resolution, the listing and the removal follow
const std::vector<common_download_sidecar> & common_download_sidecars();
// select the sidecar `kind` in `opts`; returns false for an unknown kind
bool common_download_opts_add_sidecar(common_download_opts & opts, const std::string & kind);
// the sidecar kinds `plan` resolves a file for
std::vector<std::string> common_download_plan_sidecars(const common_download_hf_plan & plan);
// a file `plan` still has to download, at the path its partial takes while it is
// incomplete; kind is the sidecar it belongs to, empty for the model files
struct common_download_partial {
std::string kind;
std::string path;
};
// the partials the download of `plan` writes, for the files not in the cache yet
std::vector<common_download_partial> common_download_plan_partials(const common_download_hf_plan & plan);
+179
View File
@@ -0,0 +1,179 @@
#include "file-lock.h"
#include "log.h"
#include <chrono>
#include <filesystem>
#include <thread>
#include <fcntl.h>
#include <sys/stat.h>
#if defined(_WIN32)
#define WIN32_LEAN_AND_MEAN
#ifndef NOMINMAX
#define NOMINMAX
#endif
#include <windows.h>
#include <io.h>
#else
#include <sys/file.h>
#include <unistd.h>
#include <cerrno>
#include <cstring>
#endif
common_file_lock::common_file_lock(const std::string & lock_path) {
this->lock_path = lock_path;
}
#if defined(_WIN32)
// the \\?\ prefix (for paths longer than 255 chars) is only accepted on absolute paths
// with backslash separators: normalize first, then check the length of the absolute form
static std::wstring as_extended_path(const std::string & path) {
// convert from the narrow encoding std::filesystem uses on this system
int wlen = MultiByteToWideChar(CP_ACP, 0, path.c_str(), (int) path.size(), nullptr, 0);
if (wlen <= 0) {
return std::wstring(path.begin(), path.end()); // best effort for exotic encodings
}
std::wstring wide(wlen, L'\0');
MultiByteToWideChar(CP_ACP, 0, path.c_str(), (int) path.size(), wide.data(), wlen);
if (wide.rfind(L"\\\\?\\", 0) == 0) {
return wide;
}
// GetFullPathNameW resolves relative paths and normalizes separators
DWORD size = GetFullPathNameW(wide.c_str(), 0, nullptr, nullptr);
if (size == 0) {
return wide;
}
std::wstring full(size, L'\0');
DWORD written = GetFullPathNameW(wide.c_str(), size, full.data(), nullptr);
if (written == 0 || written >= size) {
return wide;
}
full.resize(written);
if (full.size() <= 255) {
return wide; // short enough: open the path exactly as given
}
if (full.rfind(L"\\\\", 0) == 0) {
return L"\\\\?\\UNC\\" + full.substr(2); // \\server\share -> \\?\UNC\server\share
}
return L"\\\\?\\" + full;
}
#endif
common_file_lock::status common_file_lock::try_lock() {
if (lock_path.empty()) {
return status::acquired; // no lock path, nothing to lock
}
#if defined(_WIN32)
if (fd < 0) {
// opened with a wide path: extended paths cannot go through the narrow CRT
std::wstring lock_path_w = as_extended_path(lock_path);
fd = _wopen(lock_path_w.c_str(), _O_RDWR | _O_CREAT | _O_BINARY | _O_NOINHERIT, _S_IREAD | _S_IWRITE);
if (fd < 0) {
LOG_ERR("%s: cannot open lock file: %s\n", __func__, lock_path.c_str());
return status::error;
}
}
// lock the first byte, the file itself stays empty
OVERLAPPED ov = {};
locked = LockFileEx((HANDLE) _get_osfhandle(fd),
LOCKFILE_EXCLUSIVE_LOCK | LOCKFILE_FAIL_IMMEDIATELY, 0, 1, 0, &ov);
if (locked) {
return status::acquired;
}
if (GetLastError() == ERROR_LOCK_VIOLATION) {
return status::busy;
}
LOG_ERR("%s: file locking failed (%d)\n", __func__, (int) GetLastError());
return status::error;
#else
if (fd < 0) {
int open_flags = O_RDWR | O_CREAT | O_CLOEXEC;
#ifdef O_NOFOLLOW
open_flags |= O_NOFOLLOW;
#endif
// group-writable, so users sharing a cache can wait on the lock
fd = open(lock_path.c_str(), open_flags, 0664);
if (fd < 0) {
LOG_ERR("%s: cannot open lock file: %s\n", __func__, lock_path.c_str());
return status::error;
}
// open() filters the mode through the umask
(void) fchmod(fd, 0664);
}
if (flock(fd, LOCK_EX | LOCK_NB) == 0) {
locked = true;
return status::acquired;
}
// EACCES is reported as contention on some filesystems
if (errno == EWOULDBLOCK || errno == EAGAIN || errno == EACCES) {
return status::busy;
}
// flock is not implemented (some NFS/FUSE mounts): report it instead of racing
LOG_ERR("%s: file locking failed (%s)\n", __func__, strerror(errno));
return status::error;
#endif
}
bool common_file_lock::acquire(const std::function<bool()> & keep_waiting) {
// the lock file is opened by try_lock, its parent must exist by then
if (!lock_path.empty()) {
std::error_code ec;
auto lock_dir = std::filesystem::path(lock_path).parent_path();
if (!lock_dir.empty()) {
std::filesystem::create_directories(lock_dir, ec);
if (ec) {
LOG_ERR("%s: unable to create lock directory '%s': %s\n", __func__, lock_dir.string().c_str(), ec.message().c_str());
return false;
}
}
}
while (true) {
switch (try_lock()) {
case status::acquired:
return true;
case status::busy:
if (keep_waiting && !keep_waiting()) {
return false;
}
std::this_thread::sleep_for(std::chrono::milliseconds(100));
continue;
case status::error:
return false;
}
}
}
// the lock file is never deleted: another process could then create a fresh
// file at the same path and lock that
void common_file_lock::close() {
#if defined(_WIN32)
if (fd >= 0) {
if (locked) {
OVERLAPPED ov = {};
UnlockFileEx((HANDLE) _get_osfhandle(fd), 0, 1, 0, &ov);
}
_close(fd);
}
#else
if (fd >= 0) {
if (locked) {
flock(fd, LOCK_UN);
}
::close(fd);
}
#endif
fd = -1;
locked = false;
}
common_file_lock::~common_file_lock() {
close();
}
+36
View File
@@ -0,0 +1,36 @@
#pragma once
#include <functional>
#include <string>
// Cross-process exclusive lock (flock on POSIX, LockFileEx on Windows).
// Lazy: acquire() opens the lock file, so a read-only cache works if the lock is never acquired.
class common_file_lock {
public:
common_file_lock(const std::string & lock_path);
~common_file_lock();
// take the lock, waiting while another process holds it;
// keep_waiting is polled before each retry, false cancels the wait.
// Also returns false when locking is unavailable.
bool acquire(const std::function<bool()> & keep_waiting = {});
void close();
common_file_lock(const common_file_lock &) = delete;
common_file_lock & operator=(const common_file_lock &) = delete;
private:
enum class status {
acquired,
busy, // another process holds the lock
error, // file locking is unavailable: no exclusion is possible
};
status try_lock();
std::string lock_path;
int fd = -1;
bool locked = false;
};
+233 -118
View File
@@ -1,6 +1,8 @@
#include "fit.h"
#include "common.h"
#include "json.h"
#include "llama-cpp.h"
#include "log.h"
#include "../src/llama-ext.h"
@@ -27,46 +29,56 @@ class common_params_fit_exception : public std::runtime_error {
using std::runtime_error::runtime_error;
};
// RAII: send llama logs below min_level to the debug log
struct common_fit_log_scope {
common_fit_log_scope(ggml_log_level min_level) : min_level(min_level) {
llama_log_get(&original_logger.callback, &original_logger.user_data);
llama_log_set([](ggml_log_level level, const char * text, void * user_data) {
const common_fit_log_scope * self = (const common_fit_log_scope *) user_data;
const ggml_log_level level_eff = level >= self->min_level ? level : GGML_LOG_LEVEL_DEBUG;
self->original_logger.callback(level_eff, text, self->original_logger.user_data);
}, this);
}
~common_fit_log_scope() {
llama_log_set(original_logger.callback, original_logger.user_data);
}
common_fit_log_scope(const common_fit_log_scope &) = delete;
common_fit_log_scope & operator=(const common_fit_log_scope &) = delete;
private:
struct {
ggml_log_callback callback;
void * user_data;
} original_logger;
ggml_log_level min_level;
};
static std::vector<llama_device_memory_data> common_get_device_memory_data_impl(
const char * path_model,
const std::vector<std::string> & paths_model,
const llama_model_params * mparams,
const llama_context_params * cparams,
std::vector<ggml_backend_dev_t> & devs,
uint32_t & hp_ngl,
uint32_t & hp_ngl_skip,
uint32_t & hp_n_ctx_train,
uint32_t & hp_n_expert,
ggml_log_level log_level) {
struct user_data_t {
struct {
ggml_log_callback callback;
void * user_data;
} original_logger;
ggml_log_level min_level; // prints below this log level go to debug log
};
user_data_t ud;
llama_log_get(&ud.original_logger.callback, &ud.original_logger.user_data);
ud.min_level = log_level;
llama_log_set([](ggml_log_level level, const char * text, void * user_data) {
const user_data_t * ud = (const user_data_t *) user_data;
const ggml_log_level level_eff = level >= ud->min_level ? level : GGML_LOG_LEVEL_DEBUG;
ud->original_logger.callback(level_eff, text, ud->original_logger.user_data);
}, &ud);
common_fit_log_scope log_scope(log_level);
llama_model_params mparams_copy = *mparams;
mparams_copy.no_alloc = true;
mparams_copy.load_mode = LLAMA_LOAD_MODE_NONE;
llama_model * model = llama_model_load_from_file(path_model, mparams_copy);
llama_model * model = common_model_load(paths_model, mparams_copy);
if (model == nullptr) {
llama_log_set(ud.original_logger.callback, ud.original_logger.user_data);
throw std::runtime_error("failed to load model");
}
llama_context * ctx = llama_init_from_model(model, *cparams);
if (ctx == nullptr) {
llama_model_free(model);
llama_log_set(ud.original_logger.callback, ud.original_logger.user_data);
throw std::runtime_error("failed to create llama_context from model");
}
@@ -141,6 +153,8 @@ static std::vector<llama_device_memory_data> common_get_device_memory_data_impl(
if (mparams->load_mtp) {
hp_ngl += llama_model_n_layer_nextn(model);
}
// without MTP the NextN layers hold no weights but the model loader still counts them for n_gpu_layers
hp_ngl_skip = mparams->load_mtp ? 0 : llama_model_n_layer_nextn(model);
hp_n_ctx_train = llama_model_n_ctx_train(model);
hp_n_expert = llama_model_n_expert(model);
@@ -148,22 +162,93 @@ static std::vector<llama_device_memory_data> common_get_device_memory_data_impl(
llama_free(ctx);
llama_model_free(model);
llama_log_set(ud.original_logger.callback, ud.original_logger.user_data);
return ret;
}
std::vector<common_memory_breakdown_row> common_fit_measure(
const std::vector<std::string> & paths_model,
const llama_model_params * mparams,
const llama_context_params * cparams,
uint32_t * n_ctx_train,
uint32_t * n_ctx,
ggml_log_level log_level) {
common_fit_log_scope log_scope(log_level);
llama_model_params mparams_copy = *mparams;
mparams_copy.no_alloc = true;
mparams_copy.load_mode = LLAMA_LOAD_MODE_NONE;
llama_model_ptr model(common_model_load(paths_model, mparams_copy));
if (!model) {
throw std::runtime_error("failed to load model");
}
llama_context_ptr ctx(llama_init_from_model(model.get(), *cparams));
if (!ctx) {
throw std::runtime_error("failed to create llama_context from model");
}
if (n_ctx_train) {
*n_ctx_train = llama_model_n_ctx_train(model.get());
}
if (n_ctx) {
*n_ctx = llama_n_ctx(ctx.get());
}
return common_memory_breakdown_rows(ctx.get());
}
common_memory_reports common_measure_memory(common_params & params) {
const ggml_log_level log_level = params.verbosity >= LOG_LEVEL_DEBUG ? GGML_LOG_LEVEL_DEBUG : GGML_LOG_LEVEL_ERROR;
const std::vector<std::string> paths_model = params.model.get_paths();
common_memory_reports res;
auto mparams = common_model_params_to_llama(params);
auto cparams = common_context_params_to_llama(params);
// max: what --fit picks now, no-op with -fit off
{
auto mparams_fit = mparams;
auto cparams_fit = cparams;
common_fit_apply(params, mparams_fit, cparams_fit);
res.info_max = common_fit_measure(paths_model, &mparams_fit, &cparams_fit, &res.n_ctx_train, &res.n_ctx_max, log_level);
}
// min: the smallest context --fit may pick (--fit only changes n_ctx when it is 0)
uint32_t n_ctx_min = cparams.n_ctx;
if (n_ctx_min == 0 && params.fit_params) {
const uint32_t n_streams = cparams.kv_unified ? 1 : std::max<uint32_t>(1, cparams.n_seq_max);
// -c 0 sets fit_params_min_ctx to UINT32_MAX, cap at n_ctx_train like --fit
n_ctx_min = (uint32_t) std::min<uint64_t>(
uint64_t(params.fit_params_min_ctx) * n_streams,
uint64_t(res.n_ctx_train) * n_streams);
}
if (n_ctx_min == 0 || n_ctx_min >= res.n_ctx_max) {
// nothing to reduce, or -fit off with the trained context
res.n_ctx_min = res.n_ctx_max;
res.info_min = res.info_max;
} else {
cparams.n_ctx = n_ctx_min;
res.info_min = common_fit_measure(paths_model, &mparams, &cparams, nullptr, &res.n_ctx_min, log_level);
}
return res;
}
common_device_memory_data_vec common_get_device_memory_data(
const char * path_model,
const std::vector<std::string> & paths_model,
const llama_model_params * mparams,
const llama_context_params * cparams,
std::vector<ggml_backend_dev_t> & devs,
uint32_t & hp_ngl,
uint32_t & hp_ngl_skip,
uint32_t & hp_n_ctx_train,
uint32_t & hp_n_expert,
ggml_log_level log_level) {
std::vector<llama_device_memory_data> impl = common_get_device_memory_data_impl(
path_model, mparams, cparams, devs, hp_ngl, hp_n_ctx_train, hp_n_expert, log_level);
paths_model, mparams, cparams, devs, hp_ngl, hp_ngl_skip, hp_n_ctx_train, hp_n_expert, log_level);
common_device_memory_data_vec ret(impl.size());
for (size_t i = 0; i < impl.size(); i++) {
@@ -177,7 +262,7 @@ common_device_memory_data_vec common_get_device_memory_data(
}
static void common_params_fit_impl(
const char * path_model, struct llama_model_params * mparams, struct llama_context_params * cparams,
const std::vector<std::string> & paths_model, struct llama_model_params * mparams, struct llama_context_params * cparams,
float * tensor_split, struct llama_model_tensor_buft_override * tensor_buft_overrides,
size_t * margins_s, uint32_t n_ctx_min, const common_fit_extra_model * extra, enum ggml_log_level log_level) {
if (mparams->split_mode == LLAMA_SPLIT_MODE_TENSOR) {
@@ -189,6 +274,7 @@ static void common_params_fit_impl(
std::vector<ggml_backend_dev_t> devs;
uint32_t hp_ngl = 0; // hparams.n_gpu_layers
uint32_t hp_nsk = 0; // number of skipped NextN layers, counted by the model loader but not by hp_ngl
uint32_t hp_nct = 0; // hparams.n_ctx_train
uint32_t hp_nex = 0; // hparams.n_expert
@@ -210,6 +296,7 @@ static void common_params_fit_impl(
if (dmds_extra.empty() || n_ctx_extra != cparams->n_ctx) {
std::vector<ggml_backend_dev_t> devs_extra;
uint32_t ngl_extra = 0;
uint32_t nsk_extra = 0;
uint32_t nct_extra = 0;
uint32_t nex_extra = 0;
@@ -221,7 +308,7 @@ static void common_params_fit_impl(
dmds_t measured;
try {
measured = common_get_device_memory_data_impl(
extra->path_model, extra->mparams, extra->cparams, devs_extra, ngl_extra, nct_extra, nex_extra, log_level);
extra->paths_model, extra->mparams, extra->cparams, devs_extra, ngl_extra, nsk_extra, nct_extra, nex_extra, log_level);
} catch (const std::runtime_error & e) {
// the extra model is optional, fit the main model alone rather than giving up
LOG_WRN("%s: failed to measure the memory of the extra model, fitting without it: %s\n", __func__, e.what());
@@ -261,7 +348,7 @@ static void common_params_fit_impl(
// step 1: get data for default parameters and check whether any changes are necessary in the first place
LOG_TRC("%s: getting device memory data for initial parameters:\n", __func__);
dmds_t dmds_full = common_get_device_memory_data_impl(path_model, mparams, cparams, devs, hp_ngl, hp_nct, hp_nex, log_level);
dmds_t dmds_full = common_get_device_memory_data_impl(paths_model, mparams, cparams, devs, hp_ngl, hp_nsk, hp_nct, hp_nex, log_level);
// saturate instead of overflowing, this also preserves the UINT32_MAX sentinel of n_ctx_min:
const uint32_t n_ctx_max = (uint32_t) std::min<uint64_t>(uint64_t(hp_nct) * n_streams, UINT32_MAX);
@@ -273,7 +360,7 @@ static void common_params_fit_impl(
if (n_streams > 1) {
LOG_TRC("%s: context size unset and KV cache not unified -> using %" PRIu32 " for %" PRIu32 " sequences:\n",
__func__, n_ctx_max, n_streams);
dmds_full = common_get_device_memory_data_impl(path_model, mparams, cparams, devs, hp_ngl, hp_nct, hp_nex, log_level);
dmds_full = common_get_device_memory_data_impl(paths_model, mparams, cparams, devs, hp_ngl, hp_nsk, hp_nct, hp_nex, log_level);
}
}
add_extra_memory(dmds_full);
@@ -412,7 +499,7 @@ static void common_params_fit_impl(
int64_t sum_projected_used_min_ctx = 0;
cparams->n_ctx = n_ctx_min_total;
dmds_t dmds_min_ctx = common_get_device_memory_data_impl(path_model, mparams, cparams, devs, hp_ngl, hp_nct, hp_nex, log_level);
dmds_t dmds_min_ctx = common_get_device_memory_data_impl(paths_model, mparams, cparams, devs, hp_ngl, hp_nsk, hp_nct, hp_nex, log_level);
add_extra_memory(dmds_min_ctx);
if (nd == 0) {
sum_projected_used_min_ctx = dmds_min_ctx.back().mb.total();
@@ -559,6 +646,20 @@ static void common_params_fit_impl(
assert(uint32_t(mparams.n_gpu_layers) <= hp_ngl + 1);
uint32_t il0 = hp_ngl + 1 - mparams.n_gpu_layers; // start index for tensor buft overrides
// the skipped NextN layers come right before the output layer, put them on the same device as the output layer
// so that the layers assigned by the model loader match the layers above
if (hp_nsk > 0 && mparams.n_gpu_layers > 0) {
mparams.n_gpu_layers += hp_nsk;
if (nd > 1) {
// the last devices have no layers if not even the output layer fits
size_t id_last = nd - 1;
while (ngl_per_device[id_last].n_layer == 0) {
id_last--;
}
tensor_split[id_last] += hp_nsk;
}
}
mparams.tensor_split = tensor_split;
size_t itbo = 0;
@@ -594,7 +695,7 @@ static void common_params_fit_impl(
set_ngl_tensor_split_tbo(ngl_per_device, overflow_bufts, mparams_copy);
dmds_t dmd_nl = common_get_device_memory_data_impl(
path_model, &mparams_copy, cparams, devs, hp_ngl, hp_nct, hp_nex, log_level);
paths_model, &mparams_copy, cparams, devs, hp_ngl, hp_nsk, hp_nct, hp_nex, log_level);
add_extra_memory(dmd_nl);
LOG_TRC("%s: memory for test allocation by device:\n", func_name);
@@ -623,7 +724,7 @@ static void common_params_fit_impl(
LOG_TRC("%s: getting device memory data with all MoE tensors moved to system memory:\n", __func__);
dmds_t dmds_cpu_moe = common_get_device_memory_data_impl(
path_model, mparams, cparams, devs, hp_ngl, hp_nct, hp_nex, log_level);
paths_model, mparams, cparams, devs, hp_ngl, hp_nsk, hp_nct, hp_nex, log_level);
add_extra_memory(dmds_cpu_moe);
for (size_t id = 0; id < nd; id++) {
@@ -877,7 +978,7 @@ static void common_params_fit_impl(
}
enum common_params_fit_status common_fit_params(
const char * path_model,
const std::vector<std::string> & paths_model,
llama_model_params * mparams,
llama_context_params * cparams,
float * tensor_split,
@@ -889,7 +990,7 @@ enum common_params_fit_status common_fit_params(
const int64_t t0_us = llama_time_us();
common_params_fit_status status = COMMON_PARAMS_FIT_STATUS_SUCCESS;
try {
common_params_fit_impl(path_model, mparams, cparams, tensor_split, tensor_buft_overrides, margins, n_ctx_min, extra, log_level);
common_params_fit_impl(paths_model, mparams, cparams, tensor_split, tensor_buft_overrides, margins, n_ctx_min, extra, log_level);
LOG_TRC("%s: successfully fit params to free device memory\n", __func__);
} catch (const common_params_fit_exception & e) {
LOG_WRN("%s: failed to fit params to free device memory: %s\n", __func__, e.what());
@@ -903,8 +1004,7 @@ enum common_params_fit_status common_fit_params(
return status;
}
void common_memory_breakdown_print(const struct llama_context * ctx) {
//const auto & devices = ctx->get_model().devices;
std::vector<common_memory_breakdown_row> common_memory_breakdown_rows(const struct llama_context * ctx) {
const auto * model = llama_get_model(ctx);
std::vector<ggml_backend_dev_t> devices;
@@ -914,18 +1014,6 @@ void common_memory_breakdown_print(const struct llama_context * ctx) {
llama_memory_breakdown memory_breakdown = llama_get_memory_breakdown(ctx);
std::vector<std::array<std::string, 9>> table_data;
table_data.reserve(devices.size());
// same data as the table below, for --log-jsonl consumers
common_json rows = common_json::array();
const std::string template_header = "%s: | %s | %s %s %s %s %s %s %s |\n";
const std::string template_gpu = "%s: | %s | %s = %s + (%s = %s + %s + %s) + %s |\n";
const std::string template_other = "%s: | %s | %s %s %s = %s + %s + %s %s |\n";
table_data.push_back({template_header, "memory breakdown [MiB]", "total", "free", "self", "model", "context", "compute", "unaccounted"});
constexpr size_t MiB = 1024 * 1024;
const std::vector<std::string> desc_prefixes_strip = {"NVIDIA ", "GeForce ", "Tesla ", "AMD ", "Radeon ", "Instinct "};
// track seen buffer types to avoid double counting:
@@ -964,104 +1052,130 @@ void common_memory_breakdown_print(const struct llama_context * ctx) {
}
}
// print memory breakdown for each device:
std::vector<common_memory_breakdown_row> rows;
rows.reserve(devices.size() + 1 + memory_breakdown.size());
for (size_t i = 0; i < devices.size(); i++) {
ggml_backend_dev_t dev = devices[i];
llama_memory_breakdown_data mb = mb_dev[i];
const std::string name = ggml_backend_dev_name(dev);
std::string desc = ggml_backend_dev_description(dev);
common_memory_breakdown_row row;
row.kind = common_memory_breakdown_row::DEVICE;
row.name = ggml_backend_dev_name(dev);
row.description = ggml_backend_dev_description(dev);
for (const std::string & prefix : desc_prefixes_strip) {
if (desc.length() >= prefix.length() && desc.substr(0, prefix.length()) == prefix) {
desc = desc.substr(prefix.length());
if (row.description.length() >= prefix.length() && row.description.substr(0, prefix.length()) == prefix) {
row.description = row.description.substr(prefix.length());
}
}
size_t free, total;
ggml_backend_dev_memory(dev, &free, &total);
const size_t self = mb.model + mb.context + mb.compute;
const int64_t unaccounted = static_cast<int64_t>(total) - static_cast<int64_t>(free) - static_cast<int64_t>(self);
table_data.push_back({
template_gpu,
" - " + name + " (" + desc + ")",
std::to_string(total / MiB),
std::to_string(free / MiB),
std::to_string(self / MiB),
std::to_string(mb.model / MiB),
std::to_string(mb.context / MiB),
std::to_string(mb.compute / MiB),
std::to_string(unaccounted / static_cast<int64_t>(MiB))});
rows.push_back({
{"kind", "device"},
{"name", name},
{"description", desc},
{"total", total / MiB},
{"free", free / MiB},
{"self", self / MiB},
{"model", mb.model / MiB},
{"context", mb.context / MiB},
{"compute", mb.compute / MiB},
{"unaccounted", unaccounted / static_cast<int64_t>(MiB)},
});
row.total = total;
row.free = free;
row.model = mb_dev[i].model;
row.context = mb_dev[i].context;
row.compute = mb_dev[i].compute;
rows.push_back(std::move(row));
}
// print memory breakdown for host:
{
const size_t self = mb_host.model + mb_host.context + mb_host.compute;
table_data.push_back({
template_other,
" - Host",
"", // total
"", // free
std::to_string(self / MiB),
std::to_string(mb_host.model / MiB),
std::to_string(mb_host.context / MiB),
std::to_string(mb_host.compute / MiB),
""}); // unaccounted
rows.push_back({
{"kind", "host"},
{"name", "Host"},
{"self", self / MiB},
{"model", mb_host.model / MiB},
{"context", mb_host.context / MiB},
{"compute", mb_host.compute / MiB},
});
common_memory_breakdown_row row;
row.kind = common_memory_breakdown_row::HOST;
row.name = "Host";
row.model = mb_host.model;
row.context = mb_host.context;
row.compute = mb_host.compute;
rows.push_back(std::move(row));
}
// print memory breakdown for all remaining buffer types:
// all remaining buffer types:
for (const auto & buft_mb : memory_breakdown) {
ggml_backend_buffer_type_t buft = buft_mb.first;
const llama_memory_breakdown_data & mb = buft_mb.second;
if (seen_buffer_types.count(buft) == 1) {
continue;
}
const std::string name = ggml_backend_buft_name(buft);
const size_t self = mb.model + mb.context + mb.compute;
common_memory_breakdown_row row;
row.kind = common_memory_breakdown_row::BUFFER_TYPE;
row.name = ggml_backend_buft_name(buft);
row.model = mb.model;
row.context = mb.context;
row.compute = mb.compute;
rows.push_back(std::move(row));
seen_buffer_types.insert(buft);
}
return rows;
}
void common_memory_breakdown_print(const struct llama_context * ctx) {
const std::vector<common_memory_breakdown_row> rows = common_memory_breakdown_rows(ctx);
std::vector<std::array<std::string, 9>> table_data;
table_data.reserve(rows.size() + 1);
// same data as the table below, for --log-jsonl consumers
common_json json_rows = common_json::array();
const std::string template_header = "%s: | %s | %s %s %s %s %s %s %s |\n";
const std::string template_gpu = "%s: | %s | %s = %s + (%s = %s + %s + %s) + %s |\n";
const std::string template_other = "%s: | %s | %s %s %s = %s + %s + %s %s |\n";
table_data.push_back({template_header, "memory breakdown [MiB]", "total", "free", "self", "model", "context", "compute", "unaccounted"});
constexpr size_t MiB = 1024 * 1024;
for (const auto & row : rows) {
const size_t self = row.model + row.context + row.compute;
if (row.kind == common_memory_breakdown_row::DEVICE) {
const int64_t unaccounted = row.total - row.free - static_cast<int64_t>(self);
table_data.push_back({
template_gpu,
" - " + row.name + " (" + row.description + ")",
std::to_string(row.total / MiB),
std::to_string(row.free / MiB),
std::to_string(self / MiB),
std::to_string(row.model / MiB),
std::to_string(row.context / MiB),
std::to_string(row.compute / MiB),
std::to_string(unaccounted / static_cast<int64_t>(MiB))});
json_rows.push_back({
{"kind", "device"},
{"name", row.name},
{"description", row.description},
{"total", row.total / MiB},
{"free", row.free / MiB},
{"self", self / MiB},
{"model", row.model / MiB},
{"context", row.context / MiB},
{"compute", row.compute / MiB},
{"unaccounted", unaccounted / static_cast<int64_t>(MiB)},
});
continue;
}
table_data.push_back({
template_other,
" - " + name,
" - " + row.name,
"", // total
"", // free
std::to_string(self / MiB),
std::to_string(mb.model / MiB),
std::to_string(mb.context / MiB),
std::to_string(mb.compute / MiB),
std::to_string(row.model / MiB),
std::to_string(row.context / MiB),
std::to_string(row.compute / MiB),
""}); // unaccounted
rows.push_back({
{"kind", "buffer_type"},
{"name", name},
json_rows.push_back({
{"kind", row.kind == common_memory_breakdown_row::HOST ? "host" : "buffer_type"},
{"name", row.name},
{"self", self / MiB},
{"model", mb.model / MiB},
{"context", mb.context / MiB},
{"compute", mb.compute / MiB},
{"model", row.model / MiB},
{"context", row.context / MiB},
{"compute", row.compute / MiB},
});
seen_buffer_types.insert(buft);
}
for (size_t j = 1; j < table_data[0].size(); j++) {
@@ -1081,20 +1195,21 @@ void common_memory_breakdown_print(const struct llama_context * ctx) {
LOG_JSON("fit_memory_breakdown", common_json({
{"unit", "MiB"},
{"rows", rows},
{"rows", json_rows},
}));
}
void common_fit_print(
const char * path_model,
const std::vector<std::string> & paths_model,
llama_model_params * mparams,
llama_context_params * cparams) {
std::vector<ggml_backend_dev_t> devs;
uint32_t hp_ngl = 0; // hparams.n_gpu_layers
uint32_t hp_nsk = 0; // number of skipped NextN layers, counted by the model loader but not by hp_ngl
uint32_t hp_nct = 0; // hparams.n_ctx_train
uint32_t hp_nex = 0; // hparams.n_expert
auto dmd = common_get_device_memory_data_impl(path_model, mparams, cparams, devs, hp_ngl, hp_nct, hp_nex, GGML_LOG_LEVEL_ERROR);
auto dmd = common_get_device_memory_data_impl(paths_model, mparams, cparams, devs, hp_ngl, hp_nsk, hp_nct, hp_nex, GGML_LOG_LEVEL_ERROR);
GGML_ASSERT(dmd.size() == devs.size() + 1);
for (size_t id = 0; id < devs.size(); id++) {
Loaded 100 of 3102 files, more files were not shown because too many files have changed in this diff. Show more