Commit Graph
11643 Commits
Author SHA1 Message Date
Pascal 1ddc262558 graph: keep the graph non empty without embeddings (#30522)
An embedding-only graph has no logits, and build_pooling() returns
early when embeddings are off, so since the models no longer expand
t_embd the graph ends up empty and its inputs are never allocated.
Expand t_embd in that case.
2026-10-15 06:38:12 +02:00
Pascal 2639a3b4e0 server: save only restorable slot states to the prompt cache (#30498)
* server: save only restorable slot states to the prompt cache

The idle-slot save loop stores every idle slot whenever a task starts,
while only completion tasks restore from the prompt cache. Rerank,
embedding, decision and infill states fill the cache up to its limit
without ever being read back, and slots already cached are offered
again at every task start.

Each slot now tracks with cache_pending whether its state is restorable
and missing from the cache: a completion release or an explicit slot
restore sets it, prompt_save consumes it, prompt_clear resets it. Both
save paths share this rule, and the LRU path restores only for
completion tasks.

* server: test that decision slots skip the prompt cache

Decision tasks never mark their slot as cacheable, so starting new
tasks must not save their idle slots to the RAM prompt cache, with or
without a unified KV cache.

* server: restore any reusing task from the prompt cache

Embedding and rerank tasks never reuse their state, so their slots are
cleared on release like child slots and the cache only holds states
that a later task can restore. Infill and decision tasks now restore
from the cache too, which replaces the cache_pending flag.

Decision models disable the prompt cache unless --cache-ram is given,
since their states seldom repeat.
2026-10-15 06:30:17 +02:00
Kartik GuliaandGeorgi Gerganov 843e7dcc15 spec : add lilicorr support (#30506)
* spec : add lilicorr support

* convert : use convention for hparam assignment

* add todo for hparam use in llama-batch.cpp

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* typo fix

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-10-15 05:49:25 +02:00
Ruben Ortlam 3e390af18d vulkan: Use L2-friendly workgroup scheduling for cm2 mulmat (#30215) 2026-10-15 05:26:10 +02:00
Hongqiang Wang a0e6dc7384 opencl: support CLAMP on non-contiguous views (#30186)
The clamp kernel indexed source and destination as one flat contiguous array, so
a strided view as input (the new CLAMP v=1 case) read the wrong elements. Add a
strided kernel_clamp_nc, as the other unary ops have, and take it whenever the
source is not contiguous.
2026-10-14 13:02:27 -07:00
stormy 23deec0255 devops: incorrect COPY instruction (#30529) 2026-10-15 02:09:28 +08:00
Sigbjørn Skjæret 683bbc0dce gguf-py : detect nfc normalization (#30540) 2026-10-14 20:08:09 +02:00
Anant Shrivastava 529ac893e7 sycl: IQ3 code reorder (#29107)
* sycl: persistent IQ3_S/IQ3_XXS weight reorder for SYCL decode and prefill

Add persistent reordered weight layouts for IQ3_S and IQ3_XXS. Weights
are reordered once at load time; the reordered MMVQ dot products then
walk dense super-blocks with dp4a and SWAR sign expansion instead of
gathering bits from the interleaved standard layout. Reorder-aware
dequantization feeds the batched prefill GEMMs so prompt processing
sees the same benefit.

Also harden the oneDNN build check: a missing DNNL package or a
CPU-only DNNL build now fails at configure time instead of silently
producing a degraded backend.

Measured on Intel Arc Pro B70 with Qwen3.8-27B-GSQ-RCO-IQ3_S (MTP),
-ngl 999 -b 4096 -ub 1024 -fa on:

tg256 @ 0 ctx      12.92 -> 23.54 tok/s (+82%)
tg256 @ 9,216      12.33 -> 21.69 tok/s (+76%)
tg256 @ 64,000      9.76 -> 14.91 tok/s (+53%)
pp8192             833.9  -> 833.6 tok/s (flat)

Qwen3.5-9B Q4_K_M / Q3_K_M are unchanged within noise (largest delta
-0.5% pp, tg flat), so standard quants are not affected.

The MMVQ GLU fusion widening and rms_norm+scale fusion are
intentionally not part of this change; master already carries them
via #28931.

* removing wait as per review request

* sycl : fix fusion.cpp line ending
2026-10-14 09:41:23 -07:00
Leebr Data Consulting 685f70894b ggml-cpu : keep the FP16->FP32 lookup table on x86 with F16C (#30342) (#30343)
* ggml-cpu : keep the FP16->FP32 lookup table on x86 with F16C (#30342)

#30270 routed both scalar conversions through F16C. For FP16->FP32, Clang 20.1.8
lowers the intrinsic to a vpinsrw merging into the live accumulator register of
the quantized dot products, which serialises the loop (19 vs 4 cycles/iteration
estimated, 45-65% slower Q8_0/Q4_0 MUL_MAT_ID kernels measured). Keep F16C for
FP32->FP16 as intended and the table for FP16->FP32.

* ggml-cpu : comment out the F16C FP16->FP32 define instead of removing it (review)
2026-10-14 19:36:06 +03:00
Aaron Teo c114bd4c58 docs: update IBM zDNN backend docs (#30527)
* docs: update zdnn lib install + add notes on running models

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* docs: change title

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* docs: add note about minimum version

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

---------

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
2026-10-14 19:17:50 +03:00
Aman Gupta 5cdadfc326 ggml-cpu : size set_rows conversion buffer for all threads (#30454)
* ggml-cpu : size set_rows conversion buffer for all threads

SET_ROWS is planned with n_tasks = 1, but every thread runs it and uses its own row of the F16 -> F32 conversion buffer, so threads > 0 wrote past the work buffer when the graph had another multi-threaded op.

* move set_rows to n_task = n_threads group
2026-10-14 18:31:53 +03:00
62df1815bc cuda, hip: GDN chunked kernel (#29353)
* cuda, hip: GDN chunked kernel

Ported-from: https://github.com/tetherto/qvac-fabric-llm.cpp/pull/257
Source-commit: 53f0027d8070715fe9ad2131f3201bb0001d4eed
Co-authored-by: Aman Gupta <aman.gupta@tether.io>

* fix older builds

* review fixes

* cuda: enable GDN MMA on tested Ampere and Ada GPUs

Enable SM86 and SM89 after RTX 3090 and RTX 4090 correctness and prefill benchmarks. Keep other untested architectures on the existing fallback.

Assisted-by: Codex

* cuda : use refactored swizzle API in GDN MMA

* ggml : GDN only guarantees the final state when n_tokens > K

* cuda : add short GDN MMA phase comments

---------

Co-authored-by: gianni-cor <gianfranco.cordella@tether.io>
Co-authored-by: Aman Gupta <aman.gupta@tether.io>
2026-10-14 22:00:58 +08:00
Aleksander Grygier 37d6d653e4 skill: Update code-review skill for UI development (#30509)
* skill : add tools/ui review checklist to code-review skill

Adds a WebUI (tools/ui) section distilled from a year of review
feedback on tools/ui PRs: constants/enums placement, layering
(stores/services/utils/hooks), DRY and dead code, Svelte 5 idioms,
comment hygiene, streaming perf, frontend security, and PR hygiene
(static build, demos, scope discipline). Scoped to tools/ui only.

* skill : expand tools/ui constants/enums ruleset

Replaces the short constants bullet list with a decision ruleset
distilled from review outcomes: when a literal becomes a string
enum vs an enum-keyed map vs a declarative config vs a grouped
as-const object vs a scalar constant, naming and suffix
conventions (_REGEX, _MS, _PX, semantic separators), placement
and composition rules, and what legitimately stays inline.

* skill : drop stale static-build-commit requirement for tools/ui

Static build output is no longer stored in the repo - the UI is
built by CI and bundled at release time. Keep the requirement that
npm run build succeeds and the UI build job is green.

* skill : address copilot review of tools/ui checklist

- scope the video-demo requirement to user-facing interactive/visual
  changes instead of every tools/ui PR
- narrow the constants trigger to reused/domain-significant literals,
  keeping the inline boundary as the escape hatch
- carve out independent literals for persisted/wire-contract test
  assertions so they don't become tautological via shared constants
- allow simple presentation-local $derived in components; only shared
  business logic, orchestration and parsing leave the component
- match the existing barrel documentation style instead of requiring
  per-export JSDoc everywhere
- keep non-obvious why/rationale comments; move only examples and
  edge-case enumerations into tests
- point at the actual interpolate-size/grid-template-rows transition
  pattern in CollapsibleRegion.svelte instead of calc-size()

* skill : use spaces for nested list indentation

EditorConfig CI requires space indentation for all files; the
scalar-constants sub-bullets used tabs. Matches the 2-space nested
list convention used in docs/.
2026-10-14 15:20:27 +02:00
Pascal 6868a52cb3 ggml: fix conv_1d result layout for batched input (#28738)
* ggml: fix conv_1d result layout for batched input

The GEMM behind conv_1d folds the batch into its rows, so the product
sits in memory as [OL, N, OC] while the reshape claims [OL, OC, N].
The shape is right and the contents are transposed, so nothing fires:
no assert, no abort, the graph just runs and the channels come out
interleaved with the batches. With N == 1 the two readings describe
the same buffer, which is why the three vocoders in the tree never
hit it, they all squeeze to 2d right after the call.

Reshape to the real layout and permute OC and N back. The N == 1 path
keeps its direct reshape, so the current callers build the same graph
and pay for no extra copy.

tests/test-conv-1d.cpp compares one batched convolution against N
single batch convolutions built from the same weight and the same
input slices. Five of the seven geometries fail before this change and
all seven pass after. N == 1 and OC == 1 pass either way, since
swapping two dimensions when one of them is 1 changes nothing.

The batched path matters to downstream audio projects where several
synthesis lanes share a single graph. Nothing in mtmd batches a
vocoder today, each one streams one flow and carries its left context
between calls, but the op is correct now if one of them grows a lane
dimension.

* ggml: build the same conv_1d graph for every batch size

Address review from ggerganov: drop the N == 1 branch of ggml_conv_1d.
The GEMM result is always reshaped to [OC, N, OL] and permuted back,
as ggml_conv_2d does, so every batch size builds the same graph. For
N == 1 this adds a cont on an already ordered tensor.
2026-10-14 14:16:08 +03:00
Georgi Gerganov 3fb47677ea common : add --spec-draft-n-ubatch (#30459)
* common : add --spec-draft-n-ubatch

Allow configuring the physical batch size of the draft context. The
drafters feed it at most the drafted tokens per sequence and chunk larger
batches by its own n_ubatch, so it does not need the ubatch of the target.
A smaller value shrinks the draft compute buffer and leaves more VRAM for
the target KV cache (the draft is fitted together with the target).

The default 0 keeps the current behavior of following --n-ubatch.

Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next

* cont : remove comment

[no ci]
2026-10-14 13:58:05 +03:00
Francisco Herrera 318a471a62 docs : clarify installation recommendation for Intel oneAPI Toolkit (#30407)
Added that the oneapi toolkit includes the sycl dependencies. This makes it clear for first time users, that the toolkit includes all necessary dependencies, so that the user is guided to install the toolkit if they do not already have the dependencies separately
2026-10-14 12:19:10 +02:00
Georgi Gerganov 8c2134a875 ggml : add note about mul_mat_id + disable mul_mat_id_dup tests (#30501) 2026-10-14 12:36:52 +03:00
Pascal df18bee11e model: split layers evenly when no device reports free memory (#29677)
When every device reports zero free memory the split points are
normalized by a zero sum, the NaN split sends layers to a device index
past the end and loading aborts with vector::_M_range_check.
2026-10-14 09:30:00 +03:00
Adrien Gallouët 3eb101360f vendor : update cpp-httplib to 0.61.0 (#30494)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-10-14 08:28:24 +02:00
Georgi Gerganov af0579c78a ggml : improve RPC tensor transfer logs (#30481)
- add rpc_log_format_size() and print byte counts in KiB/MiB/GiB across the
  RPC client, server and transport traces
- add rpc_log_format_tensor() and include tensor name, op, type and shape in
  the set/get/memset/copy/init tensor logs
- fix rpc_cmd_name() returning UNKNOWN for SET_TENSOR_2D, GET_TENSOR_2D,
  COMM_INIT, COMM_ALLREDUCE and COMM_FREE
- move the log helpers to ggml-rpc/log.{h,cpp} under a common rpc_log_ prefix

Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
2026-10-14 09:23:44 +03:00
An Longandthelittlefireman 62bacfa011 cuda : use SIMT implicit GEMM conv2d on AMD without WMMA (#30325)
* cuda : use SIMT implicit GEMM conv2d on AMD without WMMA

Assisted-by: Claude Opus

* cuda : update conv2d SIMT comment

* cuda : skip SIMT conv2d on AMD for short reductions

Co-authored-by: thelittlefireman <5165783+thelittlefireman@users.noreply.github.com>

---------

Co-authored-by: thelittlefireman <5165783+thelittlefireman@users.noreply.github.com>
2026-10-14 09:22:11 +03:00
Yuri Khrustalev 77390baa18 jinja : write floats in tojson as Python does (#30362)
tojson printed floats through a C++ stream (6 significant digits, no
".0"), so 48.858844 became 48.8588 and 1.0 became 1 in tool definitions,
tool-call arguments and other template output. Write the shortest digits
that read back as the same value, as json.dumps does. The string and safe
filters of a float use the same code.
2026-10-14 09:18:38 +03:00
lhez c5a997624a opencl: use buf+offset instead of subbuffer for adreno moe expert sort (#30418) 2026-10-14 09:17:09 +03:00
lhezandHongqiang Wang 21a19cd3be opencl: fix q8_0 dp4a mul_mat_id and mark PREC_F32 mul_mat_id unsupported for adreno (#30365)
* opencl: fix min handling for moe q8_0 dp4a GEMM

Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>

* opencl: reject PREC_F32 MUL_MAT_ID for adreno

---------

Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
2026-10-14 09:16:54 +03:00
Pascal 9cf967a04d models: drop the t_embd expand of the embedding graphs (#30461)
These graphs only produce embeddings, and build_pooling() already
expands its output for every pooling type, so the expand at the end of
the model graph is redundant.
2026-10-14 08:15:08 +02:00
liangliangchang 4dd6dd4b96 HIP: use 64-row MMQ tiles for some quant types on RDNA3.5 (#28195)
The RDNA3.5 config table selects a 64-row / 128-thread tile up to J=32 and a
128-row / 256-thread tile above that. On gfx115x the smaller tile leaves enough
LDS free for a second workgroup per CU, while the larger one does not.

Extend the 64-row tile to the (type, J) pairs where it measured faster on
gfx1151, and leave every other entry untouched. J=96 is deliberately excluded
because it did not benefit.

test-backend-ops perf on gfx1151, 100 MUL_MAT shapes over the affected types
(N in 48..512, M both padded and unpadded), 3 rounds: median +2.1%, best
+26.0%, no shape outside run-to-run noise. 5 MUL_MAT_ID shapes: median +8.0%.

Assisted-by: Claude Opus 5
2026-10-14 08:10:29 +02:00
Ruben Ortlam 14310190cd vulkan: fix compile error (#30495) 2026-10-14 09:06:09 +03:00
Georgi Gerganov e7268bf6a9 llama : build the k-pool selection in the causal mask layout (#30480)
* llama : build the k-pool selection in the causal mask layout

- the glm5-next indexer folded the causal mask in through a [n_kv, n_tokens]
  view: one reshape of the mask ahead of the add, and one of the result ahead
  of every layer reading the selection, including the shared-indexer layers
- build the selection directly in the mask layout, as qwen4exp already does,
  and drop the now-degenerate reshape in its attention as well
- assert the selection width matches the input it is taken from

Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next

* cont : remove comment
2026-10-14 08:45:55 +03:00
Masato Nakasaka cdce056a3e vulkan: Add workaround for device free memory > total memory on Intel Windows (#29835)
* add workaround for Intel Windows driver issue

* applying safeguard to all GPUs

also a minor refactor

* partial revert
2026-10-14 08:45:38 +03:00
Matt Corallo d524bb660b vulkan: Define the shader count for Intel Arc Pro B65 + B70 (#30431) 2026-10-14 08:44:22 +03:00
Ruben Ortlam 7ec420bc09 vulkan: support mulmat bf16 activations (#29069)
* vulkan: support bf16 src1 in mul_mat

* clean up mul_mat b-type selection

* fix conditions
2026-10-14 08:36:32 +03:00
Sandro Hanea 10da8c463f unicode : add opt-in NFC normalization for Unigram tokenizers (#30390)
* unicode : add opt-in NFC normalization for Unigram tokenizers

Note: used help from Codex, Sol 6.1 Medium

* Fixing initial bot comments + linter error (using existing out instead of print)

* removed test_nfc_tokenizer
2026-10-14 08:33:24 +03:00
Takamichi Ohara eca5458ce0 convert : export YaRN truncation config (#30288)
* convert : export YaRN truncation config

* rename
2026-10-14 08:31:40 +03:00
Raffaele Di DonnaandRaffaele Di Donna 091b6a7fc5 Enable OpenSSL in FP32/FP16 SYCL builds to restore HF HTTPS downloads (#30330)
Co-authored-by: Raffaele Di Donna <235519540+raffaeleedidonna@users.noreply.github.com>
2026-10-13 22:25:26 -04:00
Masashi Yoshimura fcac19079f webgpu: (Fix CI) fix mul_mat_id with duplicate expert ids (#30451)
* Fix mm-id of webgpu in the case of ids duplication

* formatting

* fix comments to avoid breaking up sentences in comments
2026-10-14 08:24:23 +09:00
Yuri Khrustalev 04f7226a2c server : escape the special tokens without std::regex (#30243)
The d1-omni and kev prompts escape "<|name|>" in the request text with
std::regex_replace. libstdc++ recurses for each char of the name, so a
state of "<|" and 30000 letters overflows the stack of the HTTP thread
and llama-server dies with SIGSEGV.

server_decision_escape_special_tokens() writes the same text with a loop.
test-server-component compares it with the regex on every string of up to
6 chars of "<|>a-".
2026-10-14 01:17:11 +02:00
Xuan-Son Nguyen ff026f8c0a chat: replace output string with parts (#30389)
* chat: replace output string with parts

* reserve first
2026-10-14 01:03:56 +02:00
Adrien Gallouët 1b70350c92 ggml : simplify backend search path error handling (#28367)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-10-13 21:22:19 +03:00
Aaron Teo d71e324a44 docs : update zDNN supported ops (#30472)
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
2026-10-13 20:20:41 +02:00
Charles Xu 45451c366d kleidiai: add SME2 Q4_K and Q6_K matmul support (#29219)
Change-Id: Ib4aa4c9814c010f9cf084783cc80612e364f9a27
Change-Id: I406a6606c3fea51f8cd32331c1ed71466845d363
Change-Id: Ib4aa4c9814c010f9cf084783cc80612e364f9a27

Signed-off-by: Charles Xu <charles.xu@arm.com>
2026-10-13 09:37:48 -07:00
Jeff Bolz 77c4540ce4 vulkan: Optimize pool_2d when the pooling window is large (#30119)
When the window is larger than a subgroup, use a whole subgroup to
compute each output.
2026-10-13 17:17:59 +02:00
Igor Matuszewski ecdd29e8b2 meta: recreate all subgraphs when the arena is reallocated (#30453)
* meta: recreate all subgraphs when the arena is reallocated

* chore: revert added test cases
2026-10-13 17:58:35 +03:00
Pranav DhinakarandMax Krasnyansky ec0bfac179 hexagon: add configurable HTP DCVS/HMX power mode (#30423)
* hexagon: add configurable HTP DCVS/HMX power mode

* hexagon: address PR review feedback (drop CLI flag, align QAIRT power modes, comment/log cleanup)

* hexagon: add --hex-power-mode flag to run.py

Hooks GGML_HEXAGON_POWER_MODE into the unified run.py runner, following
the existing --hex-nhvx pattern. The env var must be set before the
process touches the Hexagon backend registry (see comment in
ggml_hexagon_init), so this is exposed as a runner flag rather than a
common/arg.cpp CLI option.

* hexagon: keep HTP awake for DCVS power profiles

* Apply suggestion from @max-krasnyansky

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-10-13 07:47:55 -07:00
Pascal ace20f679e graph: pad the KQ bias of the no-cache attention (#30446)
The no-cache attention pads K, V and the mask to 256, but T5 also adds
a relative position bias of [n_tokens, n_tokens], so the add with KQ
fails unless n_tokens divides the padded width. Pad the bias with
zeros up to the KV dimension of the mask, like K and V.
2026-10-13 16:34:10 +02:00
Georgi Gerganov b9f3599fb1 llama : do not map foreign ranges into a backend buffer (#30405)
* llama : do not map foreign ranges into a backend buffer

A buffer created from a host pointer pins everything that it wraps. When
several contexts share the same file (lazy tensors, MoE tensors kept on the
CPU with -cmoe), the single buffer that spans the first and the last tensor
of a context also wraps the ranges owned by the other contexts, pinning data
that the backend never reads.

For models with per-layer token embeddings this pins the whole lazy tensor
into the GPU buffer (27 GiB for Qwen3.8-Flash-Next IQ4_NL), defeating the
point of reading it lazily.

Instead, compute the file ranges of each context and subtract the ranges
owned by the other contexts, then create one buffer per range. Weights that
are mapped in more than one context (e.g. token_embd.weight) keep being part
of the range of every context that uses them.

Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next

* llama : enable prefetch on macOS

The prefetch hint of the tensors marked as prefetchable is a no-op on macOS,
so their pages are faulted in one by one while the graph runs. madvise
(MADV_WILLNEED) behaves as expected here, so add the platform to the guard.

With the lazy per-layer token embeddings of Qwen3.8-Flash-Next IQ4_NL read
from a cold file cache, prompt processing goes from 356 / 375 / 475 t/s to
735 / 749 / 788 t/s for pp512 / pp2048 / pp4096.

Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next

* llama : use a plain buffer list per context

Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
2026-10-13 17:28:15 +03:00
Adam CaldwellandAdam Caldwell f441d2d9db model: load YaRN beta metadata for laguna models (#30331)
Assisted-by: Codex

Co-authored-by: Adam Caldwell <accaldwell@users.noreply.github.com>
2026-10-13 22:18:01 +08:00
Pascal cb55ffe6e2 graph: skip the LM head for pooled embeddings (#30445)
* graph: skip the LM head for pooled embeddings

A context that extracts pooled embeddings outputs one vector per
sequence, but every model still expanded its LM head, so the graph
computed n_vocab logits per token and the context kept a host buffer
for them.

The models now only declare t_logits, and set_outputs() expands it
unless the context extracts embeddings with a pooling type other than
NONE. Since the toggle changes the graph shape for a pooled context,
set_embeddings() schedules a re-reserve in that case.

* graph: drop the redundant t_embd expand

Address review from @CISC: with embeddings on, build_pooling() already
expands its output for every pooling type, and a decoder graph without
embeddings does not read t_embd, so set_outputs() only expands
t_logits.

* graph: add build_logits() for the LM head output

Address review from @ggerganov: the LM head is expanded by a dedicated
build_logits(), called from llama_model::build_graph() like
build_pooling(), and set_outputs() only marks the outputs again.

* graph: call build_logits() before build_sampling()

Address review from @ggerganov: build_sampling() reads t_logits, so the
LM head is built first. A pooled context then skips the sampling layers
too, since it has no logits to sample from.

* nit
2026-10-13 16:16:14 +02:00
Georgi Gerganov a17e94d229 tests : consolidate context state tests into test-llama-context (#30023)
* tests : consolidate the context state tests into test-llama-context

test-save-load-state, test-recurrent-state-rollback and
test-state-restore-fragmented check the same object from three places:
the KV cache and the recurrent state of a llama_context, saved, loaded,
copied, rolled back. Each had its own main, its own loop over the
models, its own copy of the token and the comparison helpers, and a run
of all three loaded the same models several times.

They are one binary now, and the case is the unit of the test:

  tests/llama-context/test-case.h   the interface a case implements
  tests/llama-context/cases-kv.cpp  the cases of the KV cache
  tests/llama-context/cases-rs.cpp  the cases of the recurrent state
  tests/llama-context/test-util.cpp tokens, generation, comparison
  tests/llama-context/test-runner.cpp the models of a case, on threads
  tests/llama-context/test-log.cpp  the log file of a model

A run is a table: a row per model, a column per case, a cell holds the
verdict of that pair. The models load on several threads (-j, --threads,
min(cores, 4) by default), the table is printed when it is complete and
the log of a model that failed is printed after it, in the order of the
rows. A case returns a verdict and the reason for it, and the runner is
the only one who reports - the test does not use tests/testing.h, a run
over 130 models is a matrix of verdicts and not a tree of subtests.

Staging no longer travels through the environment: llama_model_set_attn_rot
replaces the LLAMA_ATTN_ROT_DISABLE variable, which a model loaded in one
thread used to change for the models the other threads were loading, and
llama_memory_get_buffers is added for the cases that copy the buffers of
a memory. common_log_set_stdo keeps the console quiet while the table is
being built.

ggml_aligned_malloc allocates with posix_memalign when the thread
sanitizer is on. On macOS it allocates with the Mach VM, which the
sanitizer does not intercept, so it never sees a page range change hands
and reports the first write into recycled memory as a race - 30 of them
in this test, and every one of them false.

Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next

* cont : run single-threaded by default

* cont : try to build the tests on all platforms

* cont : trim down some body fat

* cont : fix

* cont : fix the fix
2026-10-13 16:21:55 +03:00
Georgi Gerganov b476d9a762 llama : fix log message (#30452)
[no ci]
2026-10-13 16:07:54 +03:00
Ruben Ortlam e0bcaccbe5 vulkan: allow multiple devices to access gpu-visible host memory (#29741)
* vulkan: allow multiple devices to access gpu-visible host memory

* clean up erroneous newlines

* add Windows support

* handle nullptr

* fix enable condition
2026-10-13 15:05:10 +02:00