* Windows unbuffered io
* Fix read len calculation
* Add test and model load timing
* chunked fallback read + removing filepointer semantic divergence between linux and windows
* tests : remove unused mmap test arguments
Assisted-by: Codex
* remove timing, set file pointer after chunk read, test refinement
* Update tests/test-llama-mmap.cpp
Co-authored-by: Oliver Simons <osimons@nvidia.com>
* Update src/llama-mmap.cpp
Co-authored-by: Oliver Simons <osimons@nvidia.com>
* test refinement
Assisted-by: Codex
* remove duplicate constexpr
Assisted-by: Codex
* adding label behavior to tests
Assisted-by: Codex
---------
Co-authored-by: Oliver Simons <osimons@nvidia.com>
* server : functionality test to make sure every time when we add new spec dec method the existing ones are not broken.
* server: use compact models for speculative decoding CI
* ci: avoid GHA cache for speculative models
* ci: run speculative tests on DGX Spark
* server: address speculative test review
* server: use consolidated speculative models
* server: refine speculative test calibration
* docs: record speculative Spark validation
* ci : run the server-speculative steps with bash
* server: recalibrate speculative test floors on GB10
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* server : accept top-level schema in response_format with type json_schema
The OpenAI-compatible chat completions endpoint only applied the schema
when it was nested inside response_format.json_schema.schema. When the
schema was provided directly as response_format.schema (as documented in
the server README, and used by some clients), it was silently ignored
and generation ran unconstrained.
Also mention json_schema in the error message for invalid response_format
types, document the OpenAI wrapper form in the README, and add a test
case for the top-level schema form.
Fixes#10732
* server : let the json_schema wrapper take precedence by presence
Pick the schema source based on whether response_format has a
json_schema key, instead of falling back when the parsed wrapper schema
is empty. An empty schema {} is valid (any JSON value), so the old check
let a top-level schema override an explicit {} in the wrapper. Use
json_value() for the top-level schema too, like the rest of the
function, and add tests for the precedence.
---------
Co-authored-by: linsen <251731047+linsen458-spec@users.noreply.github.com>
* ggml: implement the backward pass for TRI
GGML_OP_TRI had no case in the backward switch, so a trainable graph containing it aborted with "unsupported ggml op for backward pass: TRI". The op is linear, so its gradient is masked the same way. The incoming gradient is not necessarily contiguous (the backward of a transpose hands over a view) and ggml_tri requires contiguity, so it is made contiguous first.
test_tri gains a post_transpose variant (out = transpose(tri(a))) that aborts as above without this change.
* ggml : simplify the TRI backward pass
Pass the gradient directly to ggml_tri and drop the transpose test. The existing TRI cases cover the backward pass for contiguous F32 gradients.
The dummy-model generator fills every tensor from a plain normal pool.
`blk.N.ffn.act.scales` is a divisor in the mpt graph, so with seed
1224489258 it draws |v| ~ 1e-6, and `ffn_act = gelu(...) / scales` reaches
2.7e5. Metal stages both operands of `kernel_mul_mm_f32_f32` as half, so
those activations overflow to inf, the norm downstream turns the inf into
nan and the logits come out nan, while the CPU reference stays in f32 and
stays finite. Both NMSE values then become nan and the check fails.
Give `ffn.act` the same treatment as `rope_factors` and keep its magnitude
away from zero.
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (RTX 5090 32GB, DDR5 128GB)
* ui : fold the sidebar's conversation sections
Pinned and Recent conversations fold under their heading. The whole heading
row is the toggle; its chevron follows the title, shows on hover and stays
while the section is folded. Folded sections are remembered across reloads.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : show a sidebar section's first six chats
Pinned and Recent list their first six chats and end with a Show more
row that opens the whole section; Show less, sticking to the bottom of
the sidebar while the section scrolls under it, closes it back. The
expanded sections last for the session.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : rearrange the sidebar's sections and chats by dragging
Pinned and Recent can be dragged by their title. The real section follows
the pointer, tinted like the sidebar's resize handle and edged with a
faint border, the other sliding aside as it passes; let go, it glides from
there into its new slot. The order is remembered across reloads, and a
section never moved keeps its default place. All sections now scroll
together.
A chat dragged onto Pinned is pinned, onto Recent unpinned. A lifted copy
of the row follows the pointer while the row slides out of its section.
The section under the pointer takes the same tint and border; a section
ignores its own chats. Let go, the row slides in where its section sorts
it, the rows below making room, and the copy glides into it; anywhere
else, it slides back into its own section.
While dragging, the rows and headings passed over take no hover. Dragging
needs a mouse or a pen, so a finger still scrolls the sidebar; the sidebar
scrolls near its edges, and Escape cancels.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : set the sidebar's sections apart from their rows
A section heading now reads above its chats: larger, semibold and in the
full foreground colour. Chat titles drop to a regular weight in a slightly
softened colour, full again when the chat is open or hovered, and "Show
more" drops to a regular weight in the muted colour. Chats inside a
section sit closer together, sections further apart, and the sidebar's
actions get more room above and below.
Assisted-by: Claude (Opus 5.5 Medium)
* fix(rocm,ci): Update ci workflow to use devlab ci runners for gfx1151
* ci : move gpu-rocm job to Strix Halo devlab runners
Assisted-by: Claude
* ci : ROCm 10 wheels with per-GPU device packages for gpu-rocm; add gfx1201
The runners' system ROCm 7.2.1 cannot compile the current HIP code for
gfx1151 ("Invalid dpp_ctrl value" in mmf-instance-ncols_*.cu). Go back
to the TheRock ROCm 10 wheels with the device-<gfx> extra, so rocBLAS
has the target's kernels, and keep ci/run.sh's results and models under
$RUNNER_TEMP instead of the runners' RAM-backed home. Add a gfx1201
(Radeon AI PRO R9700) row with HIP graphs disabled.
Assisted-by: Claude
---------
Co-authored-by: David Hekimian <dhekimian@users.noreply.github.com>
* metal: make ggml_tri select instead of scaling by the comparison
The kernel multiplied by the 0/1 comparison, which turns a +-inf outside the triangle into NaN, while the CPU kernel writes a plain zero there. Select, like the CPU does.
Adds test_tri_special, which puts +-inf into the cells that have to be discarded, for all four triangle types. It fails against the previous kernel and passes with this change.
* Just trim down the repetitive comments.
* mtmd, server : add temperature to /tts, default Pocket TTS to 0.3
Assisted-by: Claude Opus 5.5
* feat: add temp parameter in llama-tts
* temp: fix default hardcoded values
* fix: remove too narrow test
* fix: addressing PR comments
The rotation angle of a position p is rounded in f32 to about
p*FLT_EPSILON radians, and the CPU reference and the GPU kernels reach
it through different rounding sequences (cumulative product versus
pow). With positions up to 8192 this puts the NMSE floor between
backends near 8e-8, so the default 1e-7 threshold fails at random on
Metal. Derive the tolerance from n_ctx instead: it stays at 1e-7 for
n_ctx=512 and becomes 9.5e-7 for the YaRN cases at n_ctx=8192.
* ggml : fix for the radix TOP_K thread timing bug
* make the key an ordered 64-bit pair
* a rewrite without shift-skip logic
* fix casts
* warp-contiguous layout
* reduce the scope removing vulkan+metal changes
* turn it off for Metal/ROCm, note about cub::DeviceTopK
* fix the hip build
* skip Vulkan as well
* popc lambda from fattn
* do block_cols math in int64_t
* make it allowlist bc SYCL looks non deterministic too
* ggml : make Metal radix Top-K tied selections deterministic
The radix top-k kernel gathered the selected indices with atomic
counters, so the order of the tied candidates at the threshold (and,
with more ties than remaining slots, the subset itself) depended on
the thread scheduling.
Replace the two atomic compaction loops with a deterministic scheme:
count the elements above and tied with the threshold in contiguous
per-thread chunks, rank them with a simdgroup scan and a serial scan
over the (now unused) radix histogram, and write each kept element
exactly once in column order. The output is now byte-identical
across runs.
This is also faster than the atomic version: the strided atomic
loops become contiguous passes, reducing top-k times by ~20-45% on
M5 Ultra, with tie-heavy shapes within a few percent.
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* tests : enable deterministic top-k check on Metal, add radix-path tie cases
Metal now produces deterministic tied selections for the radix top-k
path, so the determinism self-check runs for the MTL backend as well.
Add eval cases covering the radix path (ncols > 2048) with more ties
at the threshold than remaining slots, and tie-heavy perf cases to
monitor the cost of the deterministic tie handling.
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* ggml : use coalesced accesses in the Metal radix Top-K compaction
The deterministic compaction walked contiguous per-thread chunks with
scalar sequential loads, which the memory system of the M2 Ultra can
not overlap well: the radix Top-K times regressed by ~1.5-2.4x on the
large-ncols shapes.
Give each warp a contiguous chunk of the row instead and let the lanes
interleave within it with stride 32, so every access is fully
coalesced. The lanes write their kept elements at lane-major ranks,
which is deterministic, and only a tie overflow falls back to a
per-step cross-lane ranking to keep the lowest tied indices in column
order.
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* tests : remove the extra top-k tie and perf cases
The radix-path eval and tie-heavy perf cases were added to validate and
benchmark the Metal deterministic tie handling. Keep only the enabling
of the determinism self-check for the MTL backend.
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* metal: scan the radix bins and warp totals with a simdgroup
The radix top-k selected the threshold bucket with a serial scan of
the 256 bins by thread 0 in each of the 4 passes, and ranked the warps
of the compaction with two serial scans by thread 0. The first
simdgroup now does both: each lane sums 8 bins, an exclusive prefix
sum and a ballot find the lane that reaches the k-th value, and the
warp totals get their exclusive bases from simd_prefix_exclusive_sum.
* tests: enable the deterministic top-k check on ROCm
HIP has no cub::DeviceTopK, so every row goes through the radix or
bitonic path, which break ties in the same way in each run. The check
covers all tie cases on ROCm, including the 1 and 2 row ones that CUDA
skips.
* cuda: remove the redundant radix top-k counter reset
Address review from ORippler. top_k_radix_init already zeroes
greater_count for each chunk of rows and top_k_radix_select keeps it,
so the reset kernel before the gather is one launch with no effect.
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Co-authored-by: Pascal <admin@serveurperso.com>
Zamba2 and similar architectures use a state size of 64 in their
recurrent-scan step, which fell outside the GPU kernel's existing
fast paths (96, 128, 256). Every scan step was falling back to CPU,
bottlenecking generation to 6-7 tok/s against the 50-70 tok/s band
other architectures achieve on the same hardware.
Add a 4th template instantiation for d_state=64, following the
existing pattern; no change to the underlying math. Verified against
the existing CPU-reference correctness harness with 3 new test cases
built from Zamba2's real hyperparameters (28/28 passing).
Measured ~35% throughput improvement (11 -> 15.1 tok/s) on Zamba2-7B-
Instruct. Full parity with other architectures not yet reached - a
separate, unrelated fraction of scan calls (~28%) still falls back to
CPU for a cause not yet diagnosed.
Assisted-by: Claude Sonnet 5
Co-authored-by: Sargentmajord <331229907+sargentmajord@users.noreply.github.com>
* ggml : switch to F32 src1 in the CPU backend
Change the default computation for GGML_TYPE_F16 to use F32 for src1
data in matrix multiplications: vec_dot_type(F16) is now F32 and the dot
product goes through the new ggml_vec_dot_f16_f32 (x = F16 src0, y = F32
src1), so src1 is no longer quantized to F16 and worst-case inputs that
only fit in the F32 FP range are handled correctly.
- conv_2d/conv_3d: store im2col patches (the inner mul_mat src1) as F32
and pass separate src0/src1 types to the inner mul_mat; the previous
F16 patches made the inner mul_mat widen src1 into the conv work
buffer, corrupting the patches and crashing on large inputs
- conv_transpose_1d/2d (F16 kernel): keep src1 in F32 in the work buffer
and use ggml_vec_dot_f16_f32; update work buffer planning accordingly
- supports_op: restore MUL_MAT support for F16 src1, which is now
widened to F32
- ggml_vec_dot_f16_f32: add GGML_SIMD support by converting chunks of x
to F32 and reusing the F32 SIMD dot
- mul_mat/mul_mat_id: add TODOs for a more efficient F16 x F16 path that
can be used if ggml_prec_get_src(dst, 1) allows converting src1 to F16
MUL_MAT f16 x f32 perf on M5 Ultra (GFLOPS):
- m=2048, n=512, k=16 : 120.9 -> 101.7
- m=2048, n=512, k=64 : 150.5 -> 344.2
- m=4096, k=14336 GEMV: 63 -> 343-415
- m=4096, n=512, k=14336: 65.6 -> 770.2
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* cont : add some F16 x F32 vectorized paths
* cont : enable the fused F16 x F32 dot on AVX without FMA
The 256-bit fused path of ggml_vec_dot_f16_f32 required AVX2 and FMA,
so CPUs with AVX and F16C only fell back to the chunked conversion and
F16 decode was 4x slower than before. Enable it on __AVX__ && __F16C__
and accumulate with GGML_F32x8_FMA, which does mul + add without FMA.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* llama, mtmd: follow the reference precision of ggml_prec_set_acc calls
Requests that the reference implementation does not keep in F32 move
to GGML_PREC_BF16: the GLM4 family down and output projections, the
MiMoVL vision matmuls and the Qwen3-TTS speaker encoder conv1d. The
DeepSeek V4/V4.1 routers and the MiniMax-M3 indexer stay in F32 with
a link to the reference line that computes them in F32.
* ggml: handle GGML_PREC_BF16 the same way as GGML_PREC_F32 in backends
Backends that reject GGML_PREC_F32 on the src1 of MUL_MAT_ID now also
reject GGML_PREC_BF16 (CUDA, Hexagon, OpenCL, WebGPU, Vulkan), and
backends that accept GGML_PREC_F32 for flash attention now also accept
GGML_PREC_BF16 with the same F32 accumulation (CPU, SpaceMIT, ET).
Vulkan keeps the F32 transfer between devices for both. The CUDA
cuBLAS path already handles GGML_PREC_BF16 and is unchanged.
* tests: cover GGML_PREC_BF16 in the flash attention tests
The flash attention cases run with GGML_PREC_BF16 next to F32 and
DEFAULT, and var_to_str prints bf16 so these cases keep their own name.
* llama: request GGML_PREC_BF16 for the KQ multiplication
* cont : add TODO in clip.cpp
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* opencl: add kernel_gemv_noshuffle_q8_0_f32_mc
Assisted-by: Claude Opus 5.5
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
* opencl: small-n MoE GEMV for q5_1, q4_k
Assisted-by: Claude Opus 5.5
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
* opencl: adreno kernel k floor 256 for x2
Assisted-by: Claude Opus 5.5
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
* opencl: small-n MoE GEMV for more quants
Assisted-by: Claude Opus 5.5
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
* opencl: rename function
* opencl: disable bin FA for small verify batch
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
* hexagon: use absolute skel path
* hexagon: support non-ascii dir and long path in skel-abs path
* hex-libdir: add support for FASTRPC_MANAGE_DSP_LIBRARY_PATH and GGML_HEXAGON_LIBHTP_DIR on all platforms
* hex-libdir: keep extra fwd slash in abs path on Android and Linux
* hex-libpath: always start with absolute path then fallback to session ctl then env
* hex-libdir: add CDSP1,2,3 to the fallback, needed for the IQ10
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
* hex-nhvx: apply nhvx updates to the latest master
* hex-nhvx: set N_HVX and N_THREADS based on the arch to avoid overallocation
* hex-build: add support for enabling HTP_DEBUG via build.py
* hex-trace: move tracing state/helpers into new htp-trace.h header
* hex-dcache: cleanup dcache #defines
* hex-hnvx: minimize the diff by reflowing the macros
* hex-ci: fix python type/linter warnings
* hex-nhvx: consitent logging
* mtmd: support MageVL-Cowork model
* mtmd: keep magevl rope_freqs at model level and check its shape
The rope factors are shared by every layer, so they live in clip_model
instead of a pointer copied into each clip_layer. The loader now checks
that v.rope_freqs is F32 with d_head elements and that d_head splits
into the 4:6:6 pairs the graph expects.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* moe-cache : compute part of the misses on the CPU
--moe-cache-cpu-share F computes a share F of the MoE cache misses on the CPU
from the host experts instead of uploading them. The graph builds the expert
FFN twice: on the device from the cache slots, where the misses left to the CPU
read zero slots, and on the CPU from the host experts, where the experts
computed on the device have id -1. The two results are added before the down
bias.
- ggml-cpu: MUL_MAT_ID skips negative ids and writes zero rows
- sched: the copy callback is called for all the inputs of a split, and before
the host sync when there are no events
- the slot lookup callback reads the inputs of the CPU part together with the
selected experts, so the CPU part does not wait for the device, and its
result is copied back asynchronously
- each token uses distinct zero slots, mul_mat_id does not support an expert
selected twice by a token
- LLAMA_MOE_CACHE_TRACE writes the selected experts of each lookup, for a
simulator of the cache
* scripts : add moe-cache-tune.py to pick the MoE cache arguments
The script wraps a llama-server command: it starts the server at two cache
sizes and three CPU shares, fits the upload and CPU speeds of the machine, and
replays the routing of the runs through a copy of the cache logic to predict
other sizes and shares. It prints the recommended --moe-cache-mib and
--moe-cache-cpu-share.
docs/moe-cache.md describes the MoE cache arguments and the script.
* add am17an as codeowner for MoE cache
* moe-cache : upload all the misses until the cache is full
With a high --moe-cache-cpu-share, few misses are uploaded, so an empty cache
fills slowly and more misses than needed go to the CPU. The CPU share now
starts once all the slots of a cache group hold an expert. moe-cache-tune.py
replays the same rule.
* add backend sync for async transfer queue
* moe-cache : let the scheduler copy the results of the CPU part
* Fix MSVC compilation
* remove sync change
* llama-graph : extract the CPU share condition into can_cpu_share
* llama-graph : make llm_graph_params::moe_cache const
The graph construction no longer updates the MoE cache directly: the CPU
parts of the layers are accumulated in the graph result and applied to
the cache in llama_context::process_ubatch, after the graph inputs are
set and before the graph is computed.
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* scripts : replay the ClassLRU policy in moe-cache-tune.py
* moe-cache : pick CPU misses by CRF instead of random (#31853)
* moe-cache : remove the backend logic of the CPU part from the graph
The CPU part reads the host experts, so the scheduler places it on the
CPU without set_tensor_backend. The cache gives no CPU map for batches
that a device would offload. Graphs with a CPU part look up their own
slot map, so the graph does not register the part with the cache.
ggml-backend : the inputs of the next split that are already computed
are copied to host memory before the current split runs, so a CPU split
does not wait for the device split before it. This replaces the stash
of the cache, and the copy callback is again called for host weights
only.
* llama-graph : return the MoE cache slots and CPU ids in a struct
The scheduler prefetch no longer checks that the next split runs on
another backend: the inputs of a split are never on its own backend,
so the input loop already skips that case.
---------
Co-authored-by: Gaurav Garg <gaugarg@nvidia.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Co-authored-by: Alexey Kopytko <alexey@kopytko.com>
The kernel_group_norm_f32 kernel and its host dispatch ignored the
batch dimension (ne03) of the source tensor: only ngrp threadgroups
were dispatched and each normalized the flat range
[tgpig*gs, (tgpig+1)*gs) with gs = ne00*ne01*ceil(ne02/ngrp). For
tensors with ne03 > 1, all but the first ne00*ne01*ne02 elements were
left unwritten (exposed by the new GROUP_NORM test with
ne=[3,1,1,65537], num_groups=1).
Dispatch one threadgroup per (group, i03) pair (ngrp in x, ne03 in z,
same work distribution as the Vulkan backend) and compute per-batch
statistics, matching the CPU reference. Also clamp the group element
count for the last group when ne02 is not divisible by ngrp and use
int64_t indexing.
The kernel now only requires ggml_is_contiguous_rows(op->src[0])
instead of full contiguity: it addresses src0 and dst with their own
strides (nb03 and the dst strides are added to the kernel args - dst
can be a strided view of src0 with ggml_group_norm_inplace) and falls
back from the flat fast path to a strided path that decodes the linear
element index when rows or channels are padded. The flat/strided
selection is done on the host in
ggml_metal_library_get_pipeline_group_norm and compiled into one of
two specialized pipeline variants via a FC_group_norm_flat function
constant, so the dead path is eliminated at pipeline creation.
GROUP_NORM support is also gated on F32 src and dst in supports_op -
the kernel only implements f32 and an f16 input would silently be read
as f32.
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* server: visit parent slots before child slots
A parent slot wakes its waiting child slots once it finishes the shared
prompt prefix, while slots are visited in id order. When a child task
lands on a lower slot id than its parent, the child is visited before
the parent wakes it and waits an extra batch to process its suffix.
Non-child slots are now visited before active child slots, so every
child woken by its parent joins the same batch.
* server: check systemone test events through test tags
The systemone tests matched free-form debug messages, including the
formatted f_keep value of the prompt cache. The server now emits
__TEST_TAG_WAIT_PARENT_SLOT__ when a child slot waits for its parent,
__TEST_TAG_PROMPT_CACHE_RESTORE__ on every prompt cache restore and
__TEST_TAG_PROMPT_CACHE_FULL_RESTORE__ when the whole cached prompt is
a prefix of the new one, compared on token counts.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* Only run webgpu CI on my fork
* Add webgpu only workflow
* Add WebGPU group_norm kernel
* clean up
* resolve merge conflicts
* parameterize data type; clean up cmake changes
* add nullptr check
* clean up webkit experiments
* more cleanup
* more cleanup
* clang format
* Remove unused feature check for WebGPU
* clean up supports_op condition and update documentation
* add perf tests for group_norm
* clean up doc with script and undo change in test-backend-ops
* Do 2d dispatch of workgroups
* Do 2d dispatch of workgroups
* Unstage unrelated changes
* undo docs churn
* add new test case
* undo small change
* undo small change