An embedding-only graph has no logits, and build_pooling() returns
early when embeddings are off, so since the models no longer expand
t_embd the graph ends up empty and its inputs are never allocated.
Expand t_embd in that case.
* server: save only restorable slot states to the prompt cache
The idle-slot save loop stores every idle slot whenever a task starts,
while only completion tasks restore from the prompt cache. Rerank,
embedding, decision and infill states fill the cache up to its limit
without ever being read back, and slots already cached are offered
again at every task start.
Each slot now tracks with cache_pending whether its state is restorable
and missing from the cache: a completion release or an explicit slot
restore sets it, prompt_save consumes it, prompt_clear resets it. Both
save paths share this rule, and the LRU path restores only for
completion tasks.
* server: test that decision slots skip the prompt cache
Decision tasks never mark their slot as cacheable, so starting new
tasks must not save their idle slots to the RAM prompt cache, with or
without a unified KV cache.
* server: restore any reusing task from the prompt cache
Embedding and rerank tasks never reuse their state, so their slots are
cleared on release like child slots and the cache only holds states
that a later task can restore. Infill and decision tasks now restore
from the cache too, which replaces the cache_pending flag.
Decision models disable the prompt cache unless --cache-ram is given,
since their states seldom repeat.
* spec : add lilicorr support
* convert : use convention for hparam assignment
* add todo for hparam use in llama-batch.cpp
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* typo fix
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
The clamp kernel indexed source and destination as one flat contiguous array, so
a strided view as input (the new CLAMP v=1 case) read the wrong elements. Add a
strided kernel_clamp_nc, as the other unary ops have, and take it whenever the
source is not contiguous.
* sycl: persistent IQ3_S/IQ3_XXS weight reorder for SYCL decode and prefill
Add persistent reordered weight layouts for IQ3_S and IQ3_XXS. Weights
are reordered once at load time; the reordered MMVQ dot products then
walk dense super-blocks with dp4a and SWAR sign expansion instead of
gathering bits from the interleaved standard layout. Reorder-aware
dequantization feeds the batched prefill GEMMs so prompt processing
sees the same benefit.
Also harden the oneDNN build check: a missing DNNL package or a
CPU-only DNNL build now fails at configure time instead of silently
producing a degraded backend.
Measured on Intel Arc Pro B70 with Qwen3.8-27B-GSQ-RCO-IQ3_S (MTP),
-ngl 999 -b 4096 -ub 1024 -fa on:
tg256 @ 0 ctx 12.92 -> 23.54 tok/s (+82%)
tg256 @ 9,216 12.33 -> 21.69 tok/s (+76%)
tg256 @ 64,000 9.76 -> 14.91 tok/s (+53%)
pp8192 833.9 -> 833.6 tok/s (flat)
Qwen3.5-9B Q4_K_M / Q3_K_M are unchanged within noise (largest delta
-0.5% pp, tg flat), so standard quants are not affected.
The MMVQ GLU fusion widening and rms_norm+scale fusion are
intentionally not part of this change; master already carries them
via #28931.
* removing wait as per review request
* sycl : fix fusion.cpp line ending
* ggml-cpu : keep the FP16->FP32 lookup table on x86 with F16C (#30342)
#30270 routed both scalar conversions through F16C. For FP16->FP32, Clang 20.1.8
lowers the intrinsic to a vpinsrw merging into the live accumulator register of
the quantized dot products, which serialises the loop (19 vs 4 cycles/iteration
estimated, 45-65% slower Q8_0/Q4_0 MUL_MAT_ID kernels measured). Keep F16C for
FP32->FP16 as intended and the table for FP16->FP32.
* ggml-cpu : comment out the F16C FP16->FP32 define instead of removing it (review)
* ggml-cpu : size set_rows conversion buffer for all threads
SET_ROWS is planned with n_tasks = 1, but every thread runs it and uses its own row of the F16 -> F32 conversion buffer, so threads > 0 wrote past the work buffer when the graph had another multi-threaded op.
* move set_rows to n_task = n_threads group
* cuda, hip: GDN chunked kernel
Ported-from: https://github.com/tetherto/qvac-fabric-llm.cpp/pull/257
Source-commit: 53f0027d8070715fe9ad2131f3201bb0001d4eed
Co-authored-by: Aman Gupta <aman.gupta@tether.io>
* fix older builds
* review fixes
* cuda: enable GDN MMA on tested Ampere and Ada GPUs
Enable SM86 and SM89 after RTX 3090 and RTX 4090 correctness and prefill benchmarks. Keep other untested architectures on the existing fallback.
Assisted-by: Codex
* cuda : use refactored swizzle API in GDN MMA
* ggml : GDN only guarantees the final state when n_tokens > K
* cuda : add short GDN MMA phase comments
---------
Co-authored-by: gianni-cor <gianfranco.cordella@tether.io>
Co-authored-by: Aman Gupta <aman.gupta@tether.io>
* skill : add tools/ui review checklist to code-review skill
Adds a WebUI (tools/ui) section distilled from a year of review
feedback on tools/ui PRs: constants/enums placement, layering
(stores/services/utils/hooks), DRY and dead code, Svelte 5 idioms,
comment hygiene, streaming perf, frontend security, and PR hygiene
(static build, demos, scope discipline). Scoped to tools/ui only.
* skill : expand tools/ui constants/enums ruleset
Replaces the short constants bullet list with a decision ruleset
distilled from review outcomes: when a literal becomes a string
enum vs an enum-keyed map vs a declarative config vs a grouped
as-const object vs a scalar constant, naming and suffix
conventions (_REGEX, _MS, _PX, semantic separators), placement
and composition rules, and what legitimately stays inline.
* skill : drop stale static-build-commit requirement for tools/ui
Static build output is no longer stored in the repo - the UI is
built by CI and bundled at release time. Keep the requirement that
npm run build succeeds and the UI build job is green.
* skill : address copilot review of tools/ui checklist
- scope the video-demo requirement to user-facing interactive/visual
changes instead of every tools/ui PR
- narrow the constants trigger to reused/domain-significant literals,
keeping the inline boundary as the escape hatch
- carve out independent literals for persisted/wire-contract test
assertions so they don't become tautological via shared constants
- allow simple presentation-local $derived in components; only shared
business logic, orchestration and parsing leave the component
- match the existing barrel documentation style instead of requiring
per-export JSDoc everywhere
- keep non-obvious why/rationale comments; move only examples and
edge-case enumerations into tests
- point at the actual interpolate-size/grid-template-rows transition
pattern in CollapsibleRegion.svelte instead of calc-size()
* skill : use spaces for nested list indentation
EditorConfig CI requires space indentation for all files; the
scalar-constants sub-bullets used tabs. Matches the 2-space nested
list convention used in docs/.
* ggml: fix conv_1d result layout for batched input
The GEMM behind conv_1d folds the batch into its rows, so the product
sits in memory as [OL, N, OC] while the reshape claims [OL, OC, N].
The shape is right and the contents are transposed, so nothing fires:
no assert, no abort, the graph just runs and the channels come out
interleaved with the batches. With N == 1 the two readings describe
the same buffer, which is why the three vocoders in the tree never
hit it, they all squeeze to 2d right after the call.
Reshape to the real layout and permute OC and N back. The N == 1 path
keeps its direct reshape, so the current callers build the same graph
and pay for no extra copy.
tests/test-conv-1d.cpp compares one batched convolution against N
single batch convolutions built from the same weight and the same
input slices. Five of the seven geometries fail before this change and
all seven pass after. N == 1 and OC == 1 pass either way, since
swapping two dimensions when one of them is 1 changes nothing.
The batched path matters to downstream audio projects where several
synthesis lanes share a single graph. Nothing in mtmd batches a
vocoder today, each one streams one flow and carries its left context
between calls, but the op is correct now if one of them grows a lane
dimension.
* ggml: build the same conv_1d graph for every batch size
Address review from ggerganov: drop the N == 1 branch of ggml_conv_1d.
The GEMM result is always reshaped to [OC, N, OL] and permuted back,
as ggml_conv_2d does, so every batch size builds the same graph. For
N == 1 this adds a cont on an already ordered tensor.
* common : add --spec-draft-n-ubatch
Allow configuring the physical batch size of the draft context. The
drafters feed it at most the drafted tokens per sequence and chunk larger
batches by its own n_ubatch, so it does not need the ubatch of the target.
A smaller value shrinks the draft compute buffer and leaves more VRAM for
the target KV cache (the draft is fitted together with the target).
The default 0 keeps the current behavior of following --n-ubatch.
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
* cont : remove comment
[no ci]
Added that the oneapi toolkit includes the sycl dependencies. This makes it clear for first time users, that the toolkit includes all necessary dependencies, so that the user is guided to install the toolkit if they do not already have the dependencies separately
When every device reports zero free memory the split points are
normalized by a zero sum, the NaN split sends layers to a device index
past the end and loading aborts with vector::_M_range_check.
- add rpc_log_format_size() and print byte counts in KiB/MiB/GiB across the
RPC client, server and transport traces
- add rpc_log_format_tensor() and include tensor name, op, type and shape in
the set/get/memset/copy/init tensor logs
- fix rpc_cmd_name() returning UNKNOWN for SET_TENSOR_2D, GET_TENSOR_2D,
COMM_INIT, COMM_ALLREDUCE and COMM_FREE
- move the log helpers to ggml-rpc/log.{h,cpp} under a common rpc_log_ prefix
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
tojson printed floats through a C++ stream (6 significant digits, no
".0"), so 48.858844 became 48.8588 and 1.0 became 1 in tool definitions,
tool-call arguments and other template output. Write the shortest digits
that read back as the same value, as json.dumps does. The string and safe
filters of a float use the same code.
These graphs only produce embeddings, and build_pooling() already
expands its output for every pooling type, so the expand at the end of
the model graph is redundant.
The RDNA3.5 config table selects a 64-row / 128-thread tile up to J=32 and a
128-row / 256-thread tile above that. On gfx115x the smaller tile leaves enough
LDS free for a second workgroup per CU, while the larger one does not.
Extend the 64-row tile to the (type, J) pairs where it measured faster on
gfx1151, and leave every other entry untouched. J=96 is deliberately excluded
because it did not benefit.
test-backend-ops perf on gfx1151, 100 MUL_MAT shapes over the affected types
(N in 48..512, M both padded and unpadded), 3 rounds: median +2.1%, best
+26.0%, no shape outside run-to-run noise. 5 MUL_MAT_ID shapes: median +8.0%.
Assisted-by: Claude Opus 5
* llama : build the k-pool selection in the causal mask layout
- the glm5-next indexer folded the causal mask in through a [n_kv, n_tokens]
view: one reshape of the mask ahead of the add, and one of the result ahead
of every layer reading the selection, including the shared-indexer layers
- build the selection directly in the mask layout, as qwen4exp already does,
and drop the now-degenerate reshape in its attention as well
- assert the selection width matches the input it is taken from
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
* cont : remove comment
* unicode : add opt-in NFC normalization for Unigram tokenizers
Note: used help from Codex, Sol 6.1 Medium
* Fixing initial bot comments + linter error (using existing out instead of print)
* removed test_nfc_tokenizer
The d1-omni and kev prompts escape "<|name|>" in the request text with
std::regex_replace. libstdc++ recurses for each char of the name, so a
state of "<|" and 30000 letters overflows the stack of the HTTP thread
and llama-server dies with SIGSEGV.
server_decision_escape_special_tokens() writes the same text with a loop.
test-server-component compares it with the regex on every string of up to
6 chars of "<|>a-".
* hexagon: add configurable HTP DCVS/HMX power mode
* hexagon: address PR review feedback (drop CLI flag, align QAIRT power modes, comment/log cleanup)
* hexagon: add --hex-power-mode flag to run.py
Hooks GGML_HEXAGON_POWER_MODE into the unified run.py runner, following
the existing --hex-nhvx pattern. The env var must be set before the
process touches the Hexagon backend registry (see comment in
ggml_hexagon_init), so this is exposed as a runner flag rather than a
common/arg.cpp CLI option.
* hexagon: keep HTP awake for DCVS power profiles
* Apply suggestion from @max-krasnyansky
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
The no-cache attention pads K, V and the mask to 256, but T5 also adds
a relative position bias of [n_tokens, n_tokens], so the add with KQ
fails unless n_tokens divides the padded width. Pad the bias with
zeros up to the KV dimension of the mask, like K and V.
* llama : do not map foreign ranges into a backend buffer
A buffer created from a host pointer pins everything that it wraps. When
several contexts share the same file (lazy tensors, MoE tensors kept on the
CPU with -cmoe), the single buffer that spans the first and the last tensor
of a context also wraps the ranges owned by the other contexts, pinning data
that the backend never reads.
For models with per-layer token embeddings this pins the whole lazy tensor
into the GPU buffer (27 GiB for Qwen3.8-Flash-Next IQ4_NL), defeating the
point of reading it lazily.
Instead, compute the file ranges of each context and subtract the ranges
owned by the other contexts, then create one buffer per range. Weights that
are mapped in more than one context (e.g. token_embd.weight) keep being part
of the range of every context that uses them.
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
* llama : enable prefetch on macOS
The prefetch hint of the tensors marked as prefetchable is a no-op on macOS,
so their pages are faulted in one by one while the graph runs. madvise
(MADV_WILLNEED) behaves as expected here, so add the platform to the guard.
With the lazy per-layer token embeddings of Qwen3.8-Flash-Next IQ4_NL read
from a cold file cache, prompt processing goes from 356 / 375 / 475 t/s to
735 / 749 / 788 t/s for pp512 / pp2048 / pp4096.
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
* llama : use a plain buffer list per context
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
* graph: skip the LM head for pooled embeddings
A context that extracts pooled embeddings outputs one vector per
sequence, but every model still expanded its LM head, so the graph
computed n_vocab logits per token and the context kept a host buffer
for them.
The models now only declare t_logits, and set_outputs() expands it
unless the context extracts embeddings with a pooling type other than
NONE. Since the toggle changes the graph shape for a pooled context,
set_embeddings() schedules a re-reserve in that case.
* graph: drop the redundant t_embd expand
Address review from @CISC: with embeddings on, build_pooling() already
expands its output for every pooling type, and a decoder graph without
embeddings does not read t_embd, so set_outputs() only expands
t_logits.
* graph: add build_logits() for the LM head output
Address review from @ggerganov: the LM head is expanded by a dedicated
build_logits(), called from llama_model::build_graph() like
build_pooling(), and set_outputs() only marks the outputs again.
* graph: call build_logits() before build_sampling()
Address review from @ggerganov: build_sampling() reads t_logits, so the
LM head is built first. A pooled context then skips the sampling layers
too, since it has no logits to sample from.
* nit
* tests : consolidate the context state tests into test-llama-context
test-save-load-state, test-recurrent-state-rollback and
test-state-restore-fragmented check the same object from three places:
the KV cache and the recurrent state of a llama_context, saved, loaded,
copied, rolled back. Each had its own main, its own loop over the
models, its own copy of the token and the comparison helpers, and a run
of all three loaded the same models several times.
They are one binary now, and the case is the unit of the test:
tests/llama-context/test-case.h the interface a case implements
tests/llama-context/cases-kv.cpp the cases of the KV cache
tests/llama-context/cases-rs.cpp the cases of the recurrent state
tests/llama-context/test-util.cpp tokens, generation, comparison
tests/llama-context/test-runner.cpp the models of a case, on threads
tests/llama-context/test-log.cpp the log file of a model
A run is a table: a row per model, a column per case, a cell holds the
verdict of that pair. The models load on several threads (-j, --threads,
min(cores, 4) by default), the table is printed when it is complete and
the log of a model that failed is printed after it, in the order of the
rows. A case returns a verdict and the reason for it, and the runner is
the only one who reports - the test does not use tests/testing.h, a run
over 130 models is a matrix of verdicts and not a tree of subtests.
Staging no longer travels through the environment: llama_model_set_attn_rot
replaces the LLAMA_ATTN_ROT_DISABLE variable, which a model loaded in one
thread used to change for the models the other threads were loading, and
llama_memory_get_buffers is added for the cases that copy the buffers of
a memory. common_log_set_stdo keeps the console quiet while the table is
being built.
ggml_aligned_malloc allocates with posix_memalign when the thread
sanitizer is on. On macOS it allocates with the Mach VM, which the
sanitizer does not intercept, so it never sees a page range change hands
and reports the first write into recycled memory as a race - 30 of them
in this test, and every one of them false.
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
* cont : run single-threaded by default
* cont : try to build the tests on all platforms
* cont : trim down some body fat
* cont : fix
* cont : fix the fix