The dummy-model generator fills every tensor from a plain normal pool.
`blk.N.ffn.act.scales` is a divisor in the mpt graph, so with seed
1224489258 it draws |v| ~ 1e-6, and `ffn_act = gelu(...) / scales` reaches
2.7e5. Metal stages both operands of `kernel_mul_mm_f32_f32` as half, so
those activations overflow to inf, the norm downstream turns the inf into
nan and the logits come out nan, while the CPU reference stays in f32 and
stays finite. Both NMSE values then become nan and the check fails.
Give `ffn.act` the same treatment as `rope_factors` and keep its magnitude
away from zero.
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (RTX 5090 32GB, DDR5 128GB)
* metal: make ggml_tri select instead of scaling by the comparison
The kernel multiplied by the 0/1 comparison, which turns a +-inf outside the triangle into NaN, while the CPU kernel writes a plain zero there. Select, like the CPU does.
Adds test_tri_special, which puts +-inf into the cells that have to be discarded, for all four triangle types. It fails against the previous kernel and passes with this change.
* Just trim down the repetitive comments.
The rotation angle of a position p is rounded in f32 to about
p*FLT_EPSILON radians, and the CPU reference and the GPU kernels reach
it through different rounding sequences (cumulative product versus
pow). With positions up to 8192 this puts the NMSE floor between
backends near 8e-8, so the default 1e-7 threshold fails at random on
Metal. Derive the tolerance from n_ctx instead: it stays at 1e-7 for
n_ctx=512 and becomes 9.5e-7 for the YaRN cases at n_ctx=8192.
* ggml : fix for the radix TOP_K thread timing bug
* make the key an ordered 64-bit pair
* a rewrite without shift-skip logic
* fix casts
* warp-contiguous layout
* reduce the scope removing vulkan+metal changes
* turn it off for Metal/ROCm, note about cub::DeviceTopK
* fix the hip build
* skip Vulkan as well
* popc lambda from fattn
* do block_cols math in int64_t
* make it allowlist bc SYCL looks non deterministic too
* ggml : make Metal radix Top-K tied selections deterministic
The radix top-k kernel gathered the selected indices with atomic
counters, so the order of the tied candidates at the threshold (and,
with more ties than remaining slots, the subset itself) depended on
the thread scheduling.
Replace the two atomic compaction loops with a deterministic scheme:
count the elements above and tied with the threshold in contiguous
per-thread chunks, rank them with a simdgroup scan and a serial scan
over the (now unused) radix histogram, and write each kept element
exactly once in column order. The output is now byte-identical
across runs.
This is also faster than the atomic version: the strided atomic
loops become contiguous passes, reducing top-k times by ~20-45% on
M5 Ultra, with tie-heavy shapes within a few percent.
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* tests : enable deterministic top-k check on Metal, add radix-path tie cases
Metal now produces deterministic tied selections for the radix top-k
path, so the determinism self-check runs for the MTL backend as well.
Add eval cases covering the radix path (ncols > 2048) with more ties
at the threshold than remaining slots, and tie-heavy perf cases to
monitor the cost of the deterministic tie handling.
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* ggml : use coalesced accesses in the Metal radix Top-K compaction
The deterministic compaction walked contiguous per-thread chunks with
scalar sequential loads, which the memory system of the M2 Ultra can
not overlap well: the radix Top-K times regressed by ~1.5-2.4x on the
large-ncols shapes.
Give each warp a contiguous chunk of the row instead and let the lanes
interleave within it with stride 32, so every access is fully
coalesced. The lanes write their kept elements at lane-major ranks,
which is deterministic, and only a tie overflow falls back to a
per-step cross-lane ranking to keep the lowest tied indices in column
order.
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* tests : remove the extra top-k tie and perf cases
The radix-path eval and tie-heavy perf cases were added to validate and
benchmark the Metal deterministic tie handling. Keep only the enabling
of the determinism self-check for the MTL backend.
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* metal: scan the radix bins and warp totals with a simdgroup
The radix top-k selected the threshold bucket with a serial scan of
the 256 bins by thread 0 in each of the 4 passes, and ranked the warps
of the compaction with two serial scans by thread 0. The first
simdgroup now does both: each lane sums 8 bins, an exclusive prefix
sum and a ballot find the lane that reaches the k-th value, and the
warp totals get their exclusive bases from simd_prefix_exclusive_sum.
* tests: enable the deterministic top-k check on ROCm
HIP has no cub::DeviceTopK, so every row goes through the radix or
bitonic path, which break ties in the same way in each run. The check
covers all tie cases on ROCm, including the 1 and 2 row ones that CUDA
skips.
* cuda: remove the redundant radix top-k counter reset
Address review from ORippler. top_k_radix_init already zeroes
greater_count for each chunk of rows and top_k_radix_select keeps it,
so the reset kernel before the gather is one launch with no effect.
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Co-authored-by: Pascal <admin@serveurperso.com>
Zamba2 and similar architectures use a state size of 64 in their
recurrent-scan step, which fell outside the GPU kernel's existing
fast paths (96, 128, 256). Every scan step was falling back to CPU,
bottlenecking generation to 6-7 tok/s against the 50-70 tok/s band
other architectures achieve on the same hardware.
Add a 4th template instantiation for d_state=64, following the
existing pattern; no change to the underlying math. Verified against
the existing CPU-reference correctness harness with 3 new test cases
built from Zamba2's real hyperparameters (28/28 passing).
Measured ~35% throughput improvement (11 -> 15.1 tok/s) on Zamba2-7B-
Instruct. Full parity with other architectures not yet reached - a
separate, unrelated fraction of scan calls (~28%) still falls back to
CPU for a cause not yet diagnosed.
Assisted-by: Claude Sonnet 5
Co-authored-by: Sargentmajord <331229907+sargentmajord@users.noreply.github.com>
* llama, mtmd: follow the reference precision of ggml_prec_set_acc calls
Requests that the reference implementation does not keep in F32 move
to GGML_PREC_BF16: the GLM4 family down and output projections, the
MiMoVL vision matmuls and the Qwen3-TTS speaker encoder conv1d. The
DeepSeek V4/V4.1 routers and the MiniMax-M3 indexer stay in F32 with
a link to the reference line that computes them in F32.
* ggml: handle GGML_PREC_BF16 the same way as GGML_PREC_F32 in backends
Backends that reject GGML_PREC_F32 on the src1 of MUL_MAT_ID now also
reject GGML_PREC_BF16 (CUDA, Hexagon, OpenCL, WebGPU, Vulkan), and
backends that accept GGML_PREC_F32 for flash attention now also accept
GGML_PREC_BF16 with the same F32 accumulation (CPU, SpaceMIT, ET).
Vulkan keeps the F32 transfer between devices for both. The CUDA
cuBLAS path already handles GGML_PREC_BF16 and is unchanged.
* tests: cover GGML_PREC_BF16 in the flash attention tests
The flash attention cases run with GGML_PREC_BF16 next to F32 and
DEFAULT, and var_to_str prints bf16 so these cases keep their own name.
* llama: request GGML_PREC_BF16 for the KQ multiplication
* cont : add TODO in clip.cpp
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* Only run webgpu CI on my fork
* Add webgpu only workflow
* Add WebGPU group_norm kernel
* clean up
* resolve merge conflicts
* parameterize data type; clean up cmake changes
* add nullptr check
* clean up webkit experiments
* more cleanup
* more cleanup
* clang format
* Remove unused feature check for WebGPU
* clean up supports_op condition and update documentation
* add perf tests for group_norm
* clean up doc with script and undo change in test-backend-ops
* Do 2d dispatch of workgroups
* Do 2d dispatch of workgroups
* Unstage unrelated changes
* undo docs churn
* add new test case
* undo small change
* undo small change
* model: add Kimi-K3.1-Pro text model
K3 with the full-attention MLA layers replaced by DSA: on this checkpoint 24
of the 93 layers carry a lightning indexer (32 heads x 128, top_k 2048, no
rope). The KDA (linear-attention) layers and the rest of the K3 skeleton -
cross-layer residuals, latent MoE, situ, MLA output gate, full-rank KDA gate -
are unchanged, so this reuses the K3 graph and converter and adds the indexer.
Indexer:
- the selection is an additive mask over the whole cache, not a gather of
the top-k rows, so it reuses the existing sparse flash attention path;
off that path the mask stays correct but still costs a kq over the cache
- the indexer key LayerNorm takes its eps from the model's rms_norm_eps,
because dsa_config declares no eps of its own
Graph and cache:
- new arch with four indexer tensors per DSA layer, and a hybrid memory
that keeps a K-only MLA cache for those layers, recurrent state for the
KDA layers, and the indexer key cache
- the graph node budget shares the K3 entry
Conversion:
- KimiK31Model(KimiK3Model), picked by KimiK31ForConditionalGeneration
- dsa_config becomes attention.indexer.{head_count,key_length,top_k,types}
plus the indexer LayerNorm eps; the layer list is 1-indexed like
linear_attn_config's, and the converter asserts that it covers exactly
the full-attention layers
- the routed experts stay MXFP4 and are repacked as in K3, not dequantized
- the vision tower and mm_projector are skipped: text only for now
- the arch key follows the checkpoint's own model_type ("kimi_k31") and the
loader registers KimiK31ForConditionalGeneration; the release name carries
a -Pro suffix that nothing in the config or the tensor names uses
Quantisation: the indexer's q_b, k and proj are kept out of the quant
overrides, since the top-k selection is sensitive to their precision.
Verified on an 8-layer slice against the HF checkpoint: 38 tensors checked
exactly, the indexer metadata against the config, and the selection against
full attention - identical PPL at top_k beyond the context length, divergent
once top_k is forced below it. Arch test fixture and the full arch suite pass.
* converter: avoid clutter in the k3 base
* narrow comment to match model
* update metal fusion expectations
* restore
* sync with PR 31733 refactoring
* hexagon: Generalization of CONV_2D implementation for CONV_3D
- New op HTP_OP_CONV_3D is wired through htp-ops.h, main.c (shares op_conv_2d) and the op remap in htp-kparams.cpp.
- conv_3d runs on the existing HMX tiled (implicit GEMM) conv2d path. KD and KH are folded into one virtual kernel row (KD*KH), and output rows are counted as N*OD*OH.
- htp_kparams_solve_conv3d checks the conv_3d layout and calls the tiled solver htp_kparams_conv2d_tiled, which takes an is3d flag.
- The tiled kernel takes the sizes and strides that differ between conv_2d and conv_3d from a struct htp_conv_geom. Only the input stage unfolds depth.
- 256-entry DMA queue can no longer overflow
Assisted-by: Claude Code
* hexagon: modify checking to eliminate divide-by-zero hazards
* hexagon: conv2d tiled: compute the conv_3d unfold divisors on the host
The KH and OH fastdiv values used to unfold conv_3d kh into (kd, kh) and
oh into (od, oh) were built per op on the device, each with a 64-bit
divide. Pass them in the kernel params instead; they fit in the space the
window variant already reserves, so the struct size does not change.
Assisted-by: Claude Code
* hexagon : preserve fold scheduling in conv2d for small size
hexagon : support arbitrary positive conv2d strides
Support positive horizontal strides for WHCN and CWHN inputs. Preserve
load/shuffle paths for strides 1/2 and use gather for larger strides.
hexagon : improve conv2d tile size selection
Prefer the largest feasible output-channel block and derive spatial
chunk capacity from VTCM usage. Compare capacity and thread-aligned
candidates using per-thread work and chunk overhead.
Co-authored-by: Yiwei Shao <yiwei@aizip.ai>
* hex-conv: fix compiler warning
---------
Co-authored-by: Yiwei Shao <yiwei@aizip.ai>
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
* tests : qualify isnan and signbit with std::
The unqualified calls rely on the C math.h names also being present in the
global namespace, which is not the case for the NetBSD 11.99.9/x86_64 gcc
14.4.0 build reported in the issue. Prefix the calls with std:: to select
the standard C++ overloads declared by <cmath>.
ref: https://github.com/ggml-org/ggml/issues/1681
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* tests : qualify C math calls with std::
Follow-up to the isnan / signbit fix: make the remaining math calls in
test-quantize-fns.cpp consistent with the std:: isfinite / max / numeric_limits
already used in the same file.
The float variants are overloads in C++, not distinct names, so the mapping is
fabsf -> std::fabs, copysignf -> std::copysign, nextafterf -> std::nextafter -
not std::fabsf & friends, which do not exist in libstdc++ and would break the
gcc build. NAN and INFINITY are macros and are left unqualified.
Only behavioural difference: array_rmse() fed a double accumulator to sqrtf(),
now std::sqrt() keeps the computation in double. The result is more accurate
and differs from the previous value by at most 1.4e-7 relative, three orders of
magnitude below the tightest tolerance in this test.
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
The self-hosted CPU runners are containers with 8 CPUs, but
std::thread::hardware_concurrency reports the cores of the host. The
table mode of test-llama-context sizes its pool from that count, so it
runs 4 models x 8 threads on an 8-CPU job and hangs.
Add TEST_ARG_HARDWARE_CONCURRENCY and TEST_ARG_PARALLEL_WORKERS as the
defaults for the auto-selected counts of test-llama-context and of
test-backend-ops, and set them to 8 and 4 for cpu-x64-high-perf.
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (RTX 5090 32GB, DDR5 128GB)
* feat : added ricursiveai ripple support
* fix : mandatiry SWA, scale through helper, comments back
* fix : metal
* model : make ricursive RoPE/NoPE explicit and validate the layer pattern
Write attention.rope_pattern and read it as optional, falling back to the
sliding window pattern so existing GGUFs keep loading. Refuse a
sliding_window_pattern that does not cover every layer, a model with no
sliding layer, and a gate or value residual present in only some layers.
The converter derives layer_types when the config omits it, ignores a stale
hybrid_layer_pattern and drops tensors of features the config disables.
Assisted-by: Claude Opus 5.5
* model : make ricursive swiglu_clamp per-layer
Store swiglu_clamp as a per-layer array, read it with get_key_or_arr so the
scalar in existing GGUFs still applies to every layer, and refuse a negative
or non-finite value. Add a test-llama-archs fixture with mixed per-layer
clamps and no rope_pattern, so the saver roundtrip checks both.
Assisted-by: Claude Opus 5.5
* model : use the ricursive per-tensor scales and RoPE parameters
Pass the attn_gate and output weight scales to build_lora_mm, take
ext_factor and attn_factor from the graph instead of hardcoding them, and
copy rope_freq_scale_train to the SWA layers instead of pinning it to 1.
Cite the reference lines for the value residual and the NoPE layers.
Assisted-by: Claude Opus 5.5
* convert : drop the ricursive chat_template.jinja override
The precedence between tokenizer_config.json and chat_template.jinja is to
be changed in SpecialVocab for all models instead. Until then a new
conversion stores the short tokenizer_config.json template; GGUFs that
already carry the tool-aware template are not affected.
Assisted-by: Claude Opus 5.5
* fix : remove NVFP4 checkpoint quantization block
* match #31733 refactor
---------
Co-authored-by: Piotr Wilkin <ilintar@gmail.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* metal : support scale-bias routing norm in the TOPK_MOE fusion
The DeepSeek-V4.1 graph normalizes the routing weights with a scale+bias
(sum + 1e-20) instead of the lower clamp used by the other models. Add
norm_bias/norm_scale_bias fusion chains that accept a SCALE node with a
unit scale factor at the clamp position, pass an additive bias in the
kernel args, and apply it as max(wt_sum, clamp) + bias so both variants
share one kernel. Cover the new chains in test-backend-ops.
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* metal : fuse sqrt(softplus) MoE routing with a selection bias
Add TOPK_MOE chains anchored at the softplus node (UNARY+SQRT+ADD+
ARGSORT+GET_ROWS plus optional norm/scale) so the DeepSeek-V4/V4.1
routing fuses on Metal. The kernel scores with sqrt(softplus) and ranks
selection-biased keys while the gathered weights and the norm stay
unbiased, matching the reference graph. Dispatch hooks at the softplus
node, cover the norm-bias combination in test-backend-ops and record the
new deepseek4/deepseek41 fusion counters.
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* metal : fuse sigmoid MoE routing with a selection bias
The sigmoid-gated MoE archs (GLM, DeepSeek-V2/V3.2, ...) rank
sigmoid(logits) + exp_probs_b with an optional weights-norm, but only
the softmax and sqrt(softplus) routing chains were fusable.
- add 6 sigmoid-anchored TOPK_MOE chains: sigmoid + ADD + ARGSORT +
GET_ROWS with optional SCALE and SUM_ROWS + CLAMP/SCALE + DIV norm
tail; the selection-bias ADD is required, same as the sqrt(softplus)
chains
- generalize the gating function constant from a sqrt(softplus) bool
to an int (0 = softmax, 1 = sqrt(softplus), 2 = sigmoid); sigmoid
scores are 1 / (1 + exp(-logits)), matching the CPU, Metal unary and
CUDA kernels
- anchor the unary dispatch hook at sigmoid nodes too
- test-backend-ops: 4 sigmoid routing cases with and without a
weights scale
- re-record tests/fusion/MTL.csv: the sigmoid archs (glm5-next,
deepseek2/32, kimi, minimax, ...) now fuse their routing, and
muse-spark prefill drops one MOE_REDUCE fire because the reordered
graph allocates the routing weights inside the reduce output range,
which the dst/input overlap check correctly declines
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* cont : remove low-signal comments
* ocd : valign ggml_metal_fusions
* ggml : support `truncate=false` for YaRN
YaRN calculates correction intervals to switch frequency interpolation
methods, but the setting to not round the boundaries of these intervals
to integers (`truncate=false`) was not supported.
Models that use `truncate=false` include gpt-oss and some PLaMo-3
models; to support them correctly, the GGML operations and CPU backend
are modified to handle calculations without rounding. The default
behavior is `truncate=true`, and existing calls not explicitly
configured will maintain their previous behavior.
* support CUDA backend
* add tests
* move RoPE helpers to internal header
* format
* tests: fix CI flakes of TOPK_MOE caused by near ties
With uniform random logits, two experts sometimes land within one f32
ulp of each other at a routing boundary, mostly with sqrt_softplus
gating whose output range is narrow. A backend whose gating differs by
a few ulps from the CPU then picks the other expert and the test fails.
Draw each row of logits again until the group selection and the final
top-k both have a relative margin of 1e-4, computed on the host in
double precision.
* tests: build the TOPK_MOE logits by inverting the gating function
Draw the selection scores (probs plus bias) of each row as a random
permutation of a grid, move the experts of the best groups up by half a
step, and set the logits to the inverse of the gating function at these
scores. Every expert and group decision is then one grid step apart on
all backends, and the host no longer recomputes the routing.
The softmax probs are built to sum to one so that the softmax keeps the
scores, and the views of the logits are not initialized again after the
logits are set.
* cuda: fuse NVFP4 gate/up and GLU in MMQ
* cuda: honor the requested src1 precision in fused gate/up MMQ
* cuda: share NVFP4 scale checks, skip unused fused FP4 kernels
* cuda: test NVFP4 fused MMQ at W4A8 without scales and with batches, clean up naming
* cuda: check gate_scale before loading the fused NVFP4 gate scale
* common : fix -hf tag matching inside longer quant names
":F16" picked a BF16 file when the repo had both, since "BF16"
contains "F16". The same applied to ":Q4_XS" vs "IQ4_XS". The tag
now has to start at a token boundary in the path.
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* common : accept underscore as -hf tag boundary
Some repos separate the quant with an underscore, e.g.
stepfun-ai/Step-3.5-Flash-GGUF-Q4_K_S ships
step3p5_flash_Q4_K_S-00001-of-00012.gguf, which :Q4_K_S no longer
matched. Add _ to the token boundary and pin the tag and default
lookup cases.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* hexagon: initial support for LIGHTNING_INDEXER op
* hex-lght-idxr: hmx for prefill
* hex-lght-idxr: q8_0 decode gather transpose, two-block decode compute
* hex-lght-idxr: pipelined hmx path for decode and prefill
* hex-lid: add support for K (f32)
* hex-lid: add support for K (f32)
* hex-lid: add support for q4_0
* hex-lid: remove scalar VTCM reads in the HMX q4_0 K prep and remove vector spills
* hex-lid: host/kernel kparams alighnment and remove sw devices in kernel
* hex-lit: update dma pipeline to use canonical dbl buffering in all paths
* hex-lit: add GGML_HEXAGON_LI_SELECT to force hmx/hvx paths
* hex-lit: add support for dims collapse
* hex-lit: make kparams constistent with other ops
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
* hexagon: add support for DSV4_HC_PRE
* hexagon: hsv4-hc-pre plain/gated split kernel and l2 prefetch optimization
* hex-dsv4-hc-pre: double-buffered DMA for prefill
* hex-dsv4-hc-pre: refactor DMA block setup
* hex-dsv4-hc-pre: precompute kernel params
* hex-dsv4-hc-pre: support for n_hc != 4
* hexagon: add support for DSV4_HC_POST
* hex-dsv4-hc-pre: refactor gated PRE prefill DMA to dst writeback after each block
* hex-dsv4-hc-pre: remove in-loop vector spills and redundant device-side checks
* hex-run: add support for test-llama-archs tool
* hex-dsv4: refactoring and optimizations
Remove non-dma paths.
Clean up DMA pipelines and dma_addr_t casts.
Add dims collapsing support.
Tested with test-llama-archs -a qwen4ext and test-backend-ops
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
* cuda: gather batched tensor reads into mapped host memory
Assisted-by: Codex
* add comments
Assisted-by: Codex
* simplify the implementation based on checks that exist earlier in the flow
* simplify batched tensor gather indexing
* add conditional dispatch
* address comments - make host registration raii, better test mindful of ci time, assert instead of return false
* cuda : gather small reads and use pinned DMA for large reads
* cuda : use std::vector for host gather descriptors
* cuda : add 8-byte gather copies, generalize tensor test and log registration failures
* common buffer for descriptors
* simplify stream sync + increase to 128k descriptor pinned buf
* meta: support GLM-5.3 next sm tensor
* glm5-next : cont q_absorbed for -sm tensor with non-unified KV
* meta : keep GGML_OP_ARANGE in ggml.h order
* glm5-next : split MLA heads in pairs for -sm tensor
The MLA cache has a single KV head, so the heads on a device are its GQA
ratio. The HIP tile FA kernel for D=512 needs an even GQA ratio; with 3
devices the 16 heads of the test model were split 5/5/6 and it aborted.
* meta : keep the split for views inside one slab of the split axis
With a non-unified KV cache, build_attn_mha splits q into streams with a
view. For a permuted q, e.g. the absorbed query of glm5-next, that view
aborted with "view of permuted tensor not implemented". The split is now
kept if every other dim of the source and the view stays inside one slab
of the split axis. The per-device strides are only rescaled above it.
From tetherto/qvac-fabric-llm.cpp#353.
Co-authored-by: Pratik Narola <pratik.narola@tether.io>
* glm5-next : drop the q_absorbed cont
The meta backend can now split the per-stream view of the permuted q, so
the cont is not needed for -sm tensor with a non-unified KV cache. With
the cont, the q of flash attention was not contiguous, and when FA ran on
the CPU (WebGPU) the meta get_tensor aborted.
---------
Co-authored-by: Pratik Narola <pratik.narola@tether.io>
* hexagon: add tiled HVX transpose path for permuted CONT/CPY
Permuted copies where dst dim 0 is strided in src, but some other src dim
is contiguous, used to fall back to HTP_COPY_KERNEL_RESHAPE. That issues
one DMA descriptor per element. The channel-first <-> channel-last
permutes around per-channel norms (e.g. [W,H,T,C] <-> [C,W,H,T] in video
VAEs) ht this path, and these copies ran at about 45 MB/s.
Add HTP_COPY_KERNEL_TRANSPOSE for same-type, same-shape f32/f16 copies
into a contiguous dst.
Host:
- drop extent-1 dims and merge dims that stay adjacent in src into a
canonical view, stored in the kernel params. There are no minimum
size limits.
- the copy is treated as a 2D transpose between dst dim 0 and the
contiguous src dim, batched over the remaining dims.
- tiles are sized to 64 KB per buffer and halved until they fit in
session VTCM. blkk is capped so the halfword vgather offsets fit in
16 bits.
Kernel, per tile:
- 2D DMA of blk0 strided src rows (blkk contiguous elements each)
into VTCM
- vgather transpose in 32x32 (f32) or 64x64 (f16) blocks
- 2D DMA of blkk contiguous rows out to dst
Staged and transposed buffers are double-buffered, so the next tile's
input DMA overlaps the current transpose and output DMA. Tiles are split
across threads and mdev sessions.
Small dims:
- packed mode: when dim k follows dim 0 and dim 0 divides the HVX
lanes (f32: 2/4/8/16, f16: 2/4/.../32), one vgather writes several
whole dst rows, and each tile goes out as one contiguous DMA.
- when the contiguous src dim fits in one vector, it is staged exactly
and the tile is made long along dim 0.
All other copies select the same kernels as before.
Tests: add test_cont cases for perms {1,2,3,0} and {3,0,1,2}, f32 and
f16, at various sizes.
* hexagon: cpy transpose cleanups
- fix vgather halfword offset comments: offsets are unsigned 16-bit, caps are kept lower for speed
- set packed directly in the packed case
- add dim_k == 3 CONT tests (full reversal)
Assisted-by: Claude Opus 5.5
* hexagon: use cpy transpose path for ne[0] == 1 permutes
- drop the nb[0] check in the transpose solver: size-1 dims are skipped, and the canonical tp.nb[0] check covers packed dim 0
- try the transpose path before SAMESHAPE_SAMETYPE, which did one DMA per element for these
- add ne[0] == 1 permute CPY tests
Assisted-by: Claude Opus 5.5
* hex-cpy: further improve cpy
Bring in improvements from aizip branch/PR.
Cleanup kparams and replace divs with fastdiv.
Co-authored-by: Yiwei Shao <yiwei@aizip.ai>
* hex-cpy: avoid vtcm bank conflicts
* hex-cpy: trim down the comments
* cpy-test: trim down new tests that cover permute scenarios
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
Co-authored-by: Yiwei Shao <yiwei@aizip.ai>
* llama : use llm_attn_mask and kv_rows for unified attention
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* cont : optimize `get_seq_cells`
* llama : build the kv_rows lists from per-sequence cell bitmaps
Keep a bitmap over the cells for each sequence (the transpose of `seq`), updated in seq_pos_inc/dec, and build the cell lists by walking its set bits. The lists stay in cell order, cells shared by several sequences go to every list, and nothing has to be rebuilt when cells are removed.
Also leave out of the lists the cells that the SWA hides from every token of the slice (standard and chunked SWA).
* cont : make get_kv_rows_p_min private
* cont : minor metal
* llama : use kv_rows for every ubatch of a unified cache
A ubatch made of equal-length runs with a single sequence each (f.ex. the drafts of several slots) now splits into per-sequence slices.
Any other ubatch (interleaved, uneven or coupled sequences) becomes a single slice over the cells of all its sequences instead of falling back to the dense KQ mask, so the graph keeps the topology of the reserved graph.
* llama : merge get_kv_rows_seq_ids and get_kv_rows_p_min into get_kv_rows_slices
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* hexagon : fix DMA transfers with strides > 24 bits
On v75+ the 2D DMA descriptor stores row_size and strides in 24 bits.
dma_queue_push sent oversized strides to fallback_2d, which only splits
nrows, so strides of 16 MB or more were truncated - wrong addresses.
Fix will fall back to one transfer per row in that case.
Also added two CONT tests with source strides over 16 MB.
* hex-dma: refactor the fallback paths to remove duplication
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
* chat : accept the think block in the qwen3-coder response format with reasoning_format none
Assisted-by: Claude Opus 5.5
* chat : define both qwen3-coder think block variants up front
* memory: refuse rollbacks past the snapshots of the last ubatch
The recurrent memory and the DSV4 cache accepted any rollback up to
n_rs_seq, whether or not the last ubatch wrote that snapshot. Each now
records the deepest rollback its last ubatch guarantees and seq_rm
refuses anything past it:
- recurrent: n - 1 for a ubatch of n <= n_rs_seq + 1 tokens, 0 for a
longer one, which only guarantees the final state
- DSV4: min(n, n_rs_seq), the snapshots are rebuilt from the state
before the ubatch
The dead rollback block of the rollback test becomes the assertion it
asked for, and rollback_bound checks the exact bound of each memory
after ubatches of 1, 2, n_rs_seq + 1 and n_rs_seq + 2 tokens.
* tests: address review from ggerganov
Detect architectures with llama_arch_from_string in comp_padding and
arch_reserves_single_seq, as in rollback_bound.
* WIP OCP FP8 E4M3 support
* Fix python type-checker
* Materialize input_scales also for FP8 when converting
* Fix conversion of https://huggingface.co/RedHatAI/Qwen3.8-27B-NVFP4
1. Per row FP8 wrongly fell to NVFP4 conversion, as NVFP4 checked on
dims alone. Also check on dtype for FP8
2. Need to reshape FP8 QKV projections scales in the same way that
weights are reshaped
* Allow remaining FP8 tensors to be dequantized during conversion
* Allow tensor-strategy for FP8-only compressed-tensor ckpts
* Fix and test gguf-py behavior for FP8_E4M3
* Disallow QKV-fusion for FP8 temporarily
* Fix conversion of fused qkvz & FP8 for Qwen3-Next
* Avoid fusing FP8 for --fuse-gate-up-exps as we may have different ws
* Emit warnings when not fusing FP8 during conversion
* More FP8 conversion fixes
* Add supported for strided copies to/from FP8 in CPU
* remove unneeded scratch buffer from FP8 cpu cpy/dup path
* add explanatory comments and TODOs
* Add dedicated encoder and decoder tests for FP8
* Whitelist ggml_op_get_rows in metal instead of blacklisting
required to fix failing CI, and white-listing should be more
future-proof than adding FP8 to the black-list
https://github.com/ggml-org/llama.cpp/actions/runs/38433916498/job/115422880633?pr=27512
* Fix more conversion bugs
* Defer weight materialization during conversion
* Blacklist fp8 in metal backend instead of white-list
* Adjust indents
* Fix lint + execute tests
* Add fast CPU kernel for F8_E4M3
* Conversion fixes
* Fix remaining gaps
* Enable tied weights with shared scales
* Make mandatory-F32 rules apply to FP8 during conversion
* Fix python type checks
https://github.com/ggml-org/llama.cpp/actions/runs/39765670052/job/119780379595?pr=27512
* Fix kaleidai unit-test failures
https://github.com/ggml-org/llama.cpp/actions/runs/39765670125/job/119780383150
* Relax numeric precision constraints for pre-check
* Leave fast-FP8 CPU on RISC-V for follow-up
We already exclude arm from the fast-path
* Allow output scale tensors to be located in any gguf shard
* Move test_conversion_fp8.py test execution to behavior-check.yaml
* Apply batched suggestions from code review
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Move mixed compressed-tensor logic into dequant_fp8 + add unit-tests for it
* Install all requirements also on Windows for behavior-check
Fixes https://github.com/ggml-org/llama.cpp/actions/runs/40535852590/job/122300758223?pr=27512
* Unmap lazy tensors before clean-up in Windows
Should fix https://github.com/ggml-org/llama.cpp/actions/runs/40539797824/job/122314782459?pr=27512
* LF line endings
https://github.com/ggml-org/llama.cpp/actions/runs/40539798168/job/122314791118?pr=27512
* Extract FP8 tests from llama-arch to its own test-fixture
See https://github.com/ggml-org/llama.cpp/pull/27512#discussion_r4400628221
* Update gguf-py/gguf/scripts/gguf_convert_endian.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>