* models: use flash-linear-attention's l2norm for gated delta net q/k
The GDN q/k normalization is defined by flash-linear-attention as
l2norm(x) = x * rsqrt(sum(x*x) + eps)
with eps inside the root. Every GDN call site in the tree uses ggml_l2_norm
instead, which is x / max(sqrt(sum(x*x)), eps), i.e.
torch.nn.functional.normalize - its CUDA kernel cites that page.
The clamp never engages at these magnitudes, so in practice llama.cpp
normalizes with no epsilon at all where the reference has one inside the
root.
transformers made the same substitution when it first added Qwen3-Next and
corrected it three days later in huggingface/transformers#40842, 'Fix the
misalignment between the l2norm in GDN of Qwen3-Next and the implementation
in the FLA library'. vLLM and SGLang vendor FLA rather than reimplementing
it, so neither ever had the clamp.
eps keeps coming from the checkpoint, exactly as every call site already
passed it. The references hardcode 1e-6 for this norm; that is a separate
question and the two agree on every GDN checkpoint in the wild.
ggml_l2_norm itself is correct and unchanged, as is rwkv7-base, its original
caller, which passes normalize's own default eps of 1e-12.
No new ggml op: rms_norm already carries eps inside the root, so
rms_norm(x, eps/n) * (1/sqrt(n)) is exactly x * rsqrt(sum(x*x) + eps).
* Update src/models/models.h
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* ui : update active conversation fields in place
updateCurrentNode, applyConversationUpdate, updateConversationTimestamp
and the pin toggle replaced the whole activeConversation object, so its
identity changed on every send, tool result and rename. ChatMessages
tracks that identity to refresh sibling info, so each replacement
triggered a full refetch of every message in the conversation. Write the
changed fields instead, mirroring updateMessageAtIndex.
Assisted-by: pi:zai-org/GLM-5.3
* ui : reuse the conversation load read for sibling info
Opening a conversation read every message from the database twice: once
in loadConversation for the active path, once in ChatMessages for the
sibling map. Hand the freshly read array over once so the chat screen
builds sibling info from it, and set the conversation and its messages
in one sync block so effects never see the new conversation paired with
the previous one's messages.
Assisted-by: pi:zai-org/GLM-5.3
* ui : memoize leaf walks in sibling map build
buildSiblingInfoMap resolves each sibling's leaf by walking the last-child
chain, once per sibling per message, so the walk repeats along the same
chains for every message in the conversation ( O(messages^2) on long
chats ). Memoize leaf resolution per build with path compression so each
edge is walked once.
Assisted-by: pi:zai-org/GLM-5.3
* ui : skip sibling refetch for in-place message edits
refreshAllMessages refetches every message of the conversation just to
rebuild sibling info, but preserve-responses and non-branching assistant
edits never create branches, so the sibling map stays valid. Refresh only
after actions that branch (editWithBranching kept) or delete.
Assisted-by: pi:zai-org/GLM-5.3
* ui : drop unused currentResponse reactive writes
Nothing reads chatStore.currentResponse, but setChatStreaming reassigned
it on every streamed chunk, so each token paid a reactive write and string
assignment for nothing. Remove the field and the clearUIState wrapper
that only reset it.
Assisted-by: pi:zai-org/GLM-5.3
* ui : reuse completed agentic turn sections during streaming
deriveAgenticSections runs in a $derived invalidated per streamed chunk,
but re-derived every turn of the session each time, so per-chunk cost grew
with session length. Cache completed turns keyed by their assistant message
plus reference checks on every field that feeds derivation; only the
streaming turn recomputes. Cache hits return the same section objects, so
tool block props stay stable and skip their per-chunk re-derive.
Assisted-by: pi:zai-org/GLM-5.3
* ui : share markdown block infrastructure
Every markdown block duplicated shared work: a full copy of the hljs
theme CSS per instance, and the remark/rehype plugin chain rebuilt on
every processMarkdown call ( once per block at mount, again per coalesced
chunk while streaming ). Use the single theme style element already
maintained by SyntaxHighlightedCode, and build pipelines once - shared
process-wide for attachment-less blocks, cached by attachments identity
otherwise.
Assisted-by: pi:zai-org/GLM-5.3
* ui : measure assistant layout only for the last message
Every assistant message ran getComputedStyle, getBoundingClientRect and
a ResizeObserver over the previous user bubble at mount, even off-screen
ones, forcing a layout pass per message while a long conversation
renders. The measured vars only feed the :last-child min-height rule, so
gate the effect on isLastAssistantMessage; one measurement and one
observer remain, and the effect re-runs when the last message changes.
Assisted-by: pi:zai-org/GLM-5.3
* ui : trim whole-blob scans in tool block headers
Tool block headers parsed their entire blobs at mount, even collapsed,
and most tool results and args are large plain text or embedded file
content: skip JSON.parse unless the blob starts with a JSON container,
prefilter search-result extraction with a Title:/URL: substring check,
and match the end-anchored exit-code marker against only the tail of exec
outputs.
Assisted-by: pi:zai-org/GLM-5.3
* ui : parse write_file and edit_file titles without the content blob
Both block headers parsed the full args JSON at mount, even collapsed, and
write_file and edit_file args embed the whole file content or edit
strings, so every block paid a full-blob JSON parse just to read the path.
Split the meta into a title tier that extracts the path with a targeted
key match (full parse only as fallback) and a body tier that keeps the
full parse; Svelte deriveds are lazy, and the body snippet renders only
while the block is expanded, so collapsed blocks no longer parse args.
Assisted-by: pi:zai-org/GLM-5.3
* ui : mount chat messages lazily near the viewport
Every message row mounted its full component tree on load, so the cycle
collector, GC and layout invalidation kept walking every live object and
DOM node even for rows the user never scrolls to - which dominated the
profile of long conversations. Wrap each row in a placeholder with an
IntersectionObserver ( two viewport heights of runway ) that swaps in the
real ChatMessage when the row approaches the viewport; the row shell
keeps the content-visibility sizing, and rows stay mounted once
realized. Rows targeted by the pending-edit flow mount eagerly.
Assisted-by: pi:zai-org/GLM-5.3
* ui : smooth the chat navigation animations
Slide the centered new-chat form to the bottom edge with a transform
instead of a bottom offset - layout-property transitions need the main
thread every frame and stutter while a long conversation loads, while
transform transitions run on the compositor. Fade the message list in
with a CSS animation keyed to the conversation id, disabled under
prefers-reduced-motion.
Assisted-by: pi:zai-org/GLM-5.3
* ui : follow the svelte runes guidance in chat message code
Two effects detected changes with manual previous-value refs and reset
flags. The permission request carries object identity, so its dismissal
is now a derived comparing the dismissed request; the continue request
is a bare boolean, so its dismissal only shrinks to a reset while no
request is pending. Also drop a dead if (browser) guard in the markdown
theme loader - effects never run on the server.
Assisted-by: pi:zai-org/GLM-5.3
* test : pin the chat perf invariants in the unit suite
Cover the fixes whose silent regression would be stale or wrong UI rather
than a crash: the turn-section cache must reuse unchanged turns yet
recompute on every field it compares; the sibling map must resolve the
same leaves after the leaf-walk memoization; the active conversation must
keep its identity through field updates; and the blob gates ( exec tail
window, plain-text result gate, search prefilter ) must keep accepting
what they gate. Only the risky invariants are pinned - no coverage for
coverage's sake.
Assisted-by: pi:zai-org/GLM-5.3
* refactor : address review remarks
Name the tool-arg string-field pattern, move the file tools' path field
aliases and the JSON container gates into lib/constants, and export the
write_file / edit_file meta types from $lib/types instead of the parser
modules.
Assisted-by: pi:zai-org/GLM-5.3
Remove the build-time C++ helper and external gzip dependency,
simplifying cross-compilation. Keep the generated C++ in templates for
readability and preserve fully embedded UI assets.
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* Reapply "sycl : add Kronecker product FWHT support for sizes 384, 640, 768, 12…" (#28184)
This reverts commit c845263f8b.
* tests : fix unused variable M in test-backend-ops
* tests: fix trailing space error and isolate kronecker tests for sycl backend only
define two new environment variables to better understand how much
memory is being allocated, and when. This has been invaluable in
inproving the --fit algorithm, and is likely to be useful when debugging
other memory-related issues.
`-lv 4` will be required to enable the following:
GGML_SYCL_MEMTRACE=1 will show per-site memory usage, updated whenever
it increases by more than 64MiB.
GGML_SYCL_MEMTRACE=2 will show every allocation and deallocation.
To change the default 64MiB threshold for reporting memory usage increases, use
GGML_SYCL_MEMTRACE_STEP.
A sample log line:
[SYCL-MEMTRACE] device memory query (dev): total 59493 MiB, free 4494, in use 54998; allocated 0 (buffers 0 + scratch 0), peak 0 MiB
* ui : fix MCP image attachments not displayed in tool block (#25789)
Fixes regression from #25450 where ChatMessageAgenticContent passed
message.extra instead of section.toolResultExtras to tool blocks,
leaving tool images invisible. Also fixes TOOL_RESULT_JSON_OPEN_REGEX
which misclassified "[Attachment saved: ...]" as JSON.
Fixes#25789
Assisted-by: Muse Spark
* Addressed PR comments: 1.- Removed ·?? mesage?extra· as it has no case left to cover 2.- Added ·[\· to cover the case of ·[[1, 2], [3, 4]]· case suggested in the PR comment 3.- Added unit test for covering up this regex case
* ui : fix MCP image attachments not displayed in tool block (ggml-org#25789) - Addressed lint error on regex (redundant \)
* opencl: add extended elementwise unary ops (sgn, step, elu, hardswish, hardsigmoid, floor, ceil, round, trunc)
Adds nine GGML_UNARY_OP_* elementwise ops that were falling back to CPU on the
OpenCL backend, following the same variant shape as the existing ABS op: f32,
f32_4 (vec4), f16, f16_4 (vec4), and stride-addressed f32_nc / f16_nc for
non-contiguous inputs. New kernels/unary_ext.cl (macro-generated), a shared
ggml_cl_unary_ext dispatch helper mirroring ggml_cl_abs, the supports_op cases,
and the compute-forward cases.
Values are computed in float (the f16 variants read/write half and convert), so
the conditional ops (step, elu) match the CPU reference; the vec4 forms use
select() for the branch.
Validated with test-backend-ops on Adreno 840 and 850 (E17): all nine ops pass
every case including the vec4 and non-contiguous variants (8/8 or 14/14).
* opencl: dispatch a contiguous f32 copy over the whole device
kernel_cpy_f32_f32 maps one workgroup to each (i01,i02,i03) row and strides the
row across that workgroup's lanes, and the host launches ne01*MIN(64,ne00) work
items. A tensor with few long rows therefore runs on a single workgroup. The
mamba2 and gated-delta-net recurrent state cache is one row of 524288 floats,
copied once per layer per graph, and lands on 64 work items.
When both sides are contiguous the copy is a linear move, so dispatch it over
the whole device: one work item per float4. Gated on ggml_is_contiguous for both
tensors and equal element counts, so copies already spread over many rows keep
the existing path. The kernel is created optionally, so a driver that rejects it
falls back rather than aborting.
vload4/vstore4 rather than a float4 cast: they require only the scalar type's
alignment, and these buffers carry an arbitrary 4-byte view offset.
CPY, DUP and CONT are 217/217 on Adreno 840 and 740 with the path enabled and
disabled. GGML_OPENCL_CPY_FLAT=0 forces the old kernel.
* opencl: support all easy-copy types in CONCAT
CONCAT was F32-only. Extend it to every "easy-copy" type -- any non-quantized
type with a block size of 1 and an element size of 1, 2, 4 or 8 bytes, i.e.
f16/bf16/i8/i16/i32/i64 as well as f32.
The kernels are keyed by element SIZE rather than by type, which is what CUDA
already does for the same op: one kernel per byte width (b1/b2/b4/b8) plus the
packed b4 fast path, instead of one per ggml type. supports_op gates on the
same property, so a new type of a supported width is picked up with no further
work.
Validated with test-backend-ops on Adreno 840 / A8X and X2-90 / X2E.
* model: add Tencent Hy 4 (hy_v4) preview architecture support
Adds support for the Tencent Hy 4 model (Hugging Face architecture
HYV4ForCausalLM, GGUF arch hy_v4):
Add HF -> GGUF conversion script (conversion/hy_v4.py) and wire it into the conversion registry
Register hy_v4 GGUF constants, arch enum, and writer support
Implement the hy-v4 model graph, hparams, vocab and context changes
Register the new arch in llama-arch and models registry
Extend arch tests to cover hy_v4
Assisted by Claude Opus 5
* Update convert_hf_to_gguf_update.py
Co-authored-by: fairydreaming <166155368+fairydreaming@users.noreply.github.com>
* Update conversion/base.py
Co-authored-by: fairydreaming <166155368+fairydreaming@users.noreply.github.com>
* convert : move hy_v4 entry to the same place as in convert_hf_to_gguf_update.py
* model : apply changes related to n_ff_exp becoming per-layer in Hy4-preview
* n_layer_all
---------
Co-authored-by: fairydreaming <166155368+fairydreaming@users.noreply.github.com>
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
This commit adds a cmake version configuration file to replace the
current compile definition solution for the version.
The motivation for this change is that I made a mistake and did not take
into consideration that the compile definition means that this will
become a compiler flag for all sources in the target. This means that
when a version update happens that will recompile all sources in the
target even if they have not changed.
Refs: https://github.com/ggml-org/llama.cpp/pull/28278
Use std::error_code overloads of fs::current_path() and
fs::directory_iterator in ggml_backend_load_best() so an
inaccessible search path (WebDAV mount, removed CWD) is
skipped instead of terminating the process with an uncaught
filesystem_error.
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
Let llama_print_build_info write to a caller-provided FILE* instead of
hardcoding stderr. The parameter defaults to stderr so existing callers
keep their current behavior.
The version command in llama-app now passes stdout, so plain version
output goes to stdout where users expect it.
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* src : add n_expert_used_max function
With Commit c61b98b875 ("model: add
NVIDIA Nemotron-3-Puzzle-75B-A9B (NemotronHPuzzle) support (#25444)") it
is now possible for each layer to have a specific number of experts but
there are a few checks that need to be updated to handle this upon model
loading. For example:
```console
llama_model_load: error loading model: model has expert layers but no expert layers are used
```
And later:
```console
/llama.cpp/src/llama-model-loader.cpp:955: GGML_ASSERT(n_ids_used > 0) failed
```
This commit adds the n_expert_used_max function so that these checks
can use it.
Refs: https://github.com/ggml-org/llama.cpp/pull/25444#issuecomment-5524976031
* src : use hparams.n_expert_used_max in llama_model_base::load_hparams
* src : use 0 as initial value for n_expert_used_max
Fuse RMS_NORM+MUL+ADD and ADD+ADD under GGML_SYCL_ENABLE_FUSION.
ADD+ADD uses the same binbcast indexing and type matrix as standalone
add() (f32, f16, f16/f32, i32, i16, bf16, including broadcast and
non-contiguous). Unsupported combinations fall back to two add() launches.
* opencl: quant lm_head / decode GEMV and medium-batch GEMM optimizations
* opencl: guard q4_K/q6_K tiled_ns convert-kernel registration for non-Adreno build
* opencl: gate q4_K MUL_MAT+GLU fusion dispatch to Adreno
* opencl: require the noshuffle weight layout in the q4_K GLU fusion gate
* opencl: do not take the vectorized f16 mrow GEMV path on an unaligned row stride
* opencl: pass the new get_scale_min_k4 stride argument at the row-major call sites
* opencl: enable the q4_K split-K decode GEMV only where it is measured to win
* opencl: record the X1-85 split-K datapoint (neutral, exclusion confirmed)
* opencl: restrict the tiled lm_head/embed GEMV default to X2E/A8X
* opencl: fix q4_K variant kernels to read the transposed scales layout
* opencl: keep the flat-GEMV large-m escape opt-in
* opencl: guard the o4 GEMV store against the rounded-up dispatch tail
* opencl: restore the tiled q4_K/q6_K layout on tensor read-back
* opencl: split-K for the q8_0 decode GEMV at small M
* opencl: keep the q6_K noshuffle correctness escape ahead of the opt-in gate
* server : use pytest-xdist for server tests
This commit adds pytest-xdist to the server tests. This is pytest
plugin that distributes test execution across multiple CPU cores.
Assisted-by: pi:llama.cpp/qwen3.8-27B
Refs: https://github.com/ggml-org/llama.cpp/pull/26734#issuecomment-5220707042
* remove server_base_port and BASE_PORT
* use worksteal and pytest builting tmp_path
* metal : support n_kv_max sparse mask hint in flash attention vec kernel
- add kernel_flash_attn_ext_vec_idx: compacts finite mask entries into
a per-row index list (Hillis-Steele scan, one threadgroup per row)
- extend vec FA kernel with optional sparse index gathering (FC slot 5)
- add host-side gate: sparse path when n_kv_max > 0, mask present,
supported head sizes / KV types, n_kv_max <= 4096
- new buffer region extra_idx for the index list
- pipeline getter extended with has_sparse param
- add test cases: head sizes, quant types, nb>1, nr23 variants,
sinks, ALiBi, softcap, permute, v_view_of_k, no-mask fallback
Note: multi-row (nb*nr23[1] > 1) cases still failing - rid mapping
in the store phase needs revisiting for the sparse path.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* metal : fix sparse flash attention row addressing
- kernel_flash_attn_ext_vec_idx: mask param is half* but nb31 is a byte
stride, so the per-row mask offset was scaled by 2x; cast to char*
before applying the byte strides
- kernel_flash_attn_ext_vec: sparse pidx param is char* so the per-row
element offset was under-scaled by sizeof(int); scale it by sizeof(int)
to get the correct byte offset
- fixes the multi-row (nb*nr23[1] > 1) sparse flash attention failures
Assisted-by: pi:llama.cpp/DeepSeek-v4-0731
* cont : use sparse vec FA for prefill
* metal : single-pass flash attention sparse index compaction
The idx kernel previously read the mask row twice: once to count the finite
entries (for the prefix scan) and again to recover their positions. Since the
kernel is memory-bound, this doubled the mask traffic.
Keep the finite positions in a per-thread register array during the count
pass and write them out directly, avoiding the second mask read. A dense
mask with more than NLOCAL finite entries in a slice falls back to re-reading
the mask to write the remaining positions.
Assisted-by: pi:llama.cpp/DeepSeek-v4-0731
* tests : add perf cases for sparse flash attention prefill
Measure the sparse vec FA kernel across KV sizes, n_kv_max hints and batch
sizes. Run with:
./build/bin/test-backend-ops -b MTL0 -o FLASH_ATTN_EXT -p "n_kv_max=[1-9]" perf
Assisted-by: pi:llama.cpp/DeepSeek-v4-0731
* qwen4 : enable sparse attention
* cont : adjust nsg
* cont : sync test-backend-ops
* cont : disable Qwen4 for now
* cont : clean-up + tests
* mtmd : mark context as const in more methods
Mark `mtmd_context` as `const` in:
- mtmd_bitmap_init_lazy
- mtmd_tokenize
- mtmd_tokenize_from_parts
- mtmd_helper_support_video
- mtmd_helper_bitmap_init_from_file
- mtmd_helper_bitmap_init_from_buf
- mtmd_helper_video_init
- mtmd_helper_video_init_from_buf
- mtmd_helper_model_can_chat
The tokenization functions in particular are useful to have marked
`const`, as that allows more easily telling the compiler that we can
safely tokenize from multiple threads (`mtmd_tokenize` is already
documented as thread-safe, this just reifies that in the signature).
* mtmd : mark tokenization input pointer as const
Mark the `bitmaps` and `parts` pointers in `mtmd_tokenize` and
`mtmd_tokenize_from_parts` as `const`. This allows more easily calling
these with immutable arrays / vectors.
* mtmd : mark llama_context as const in mtmd_helper_model_can_chat
* CUDA: Allow CUDA optimization per split for multi-GPU.
Previous guard caused multi-GPU to skip the graph optimization. The
graph is already split per device and the optimization doesnt run
over the whole model but once per split, and thus should be allowed.
However, the CUDA event ggml_cuda_concurrent_event belongs to
whichever GPU was "current" when created. If the pass ran while
GPU 0 was current, it would stick and during event creation for the
second GPU it would land on GPU 0.
The fix: set the device explicitly ggml_cuda_set_device(cuda_ctx->device);
Default behaviour remains unchanged, only active for GGML_CUDA_GRAPH_OPT=1.
Explicit device setting pattern re-used from ggml_backend_cuda_graph_compute.
* Update ggml/src/ggml-cuda/ggml-cuda.cu
Co-authored-by: Aman Gupta <amangupta052@gmail.com>
---------
Co-authored-by: tannerbruhn <tannerbruhn@users.noreply.github.com>
Co-authored-by: Aman Gupta <amangupta052@gmail.com>
Skip the nb[3] check when ne[3] == 1, the shader never reads it for a
single stream. Cache views carry the full-buffer stride there, so the old
check reduced to n_kv == kv_size and the path only engaged with the
cache full.
