Temporary carry of the open upstream PR
https://github.com/ggml-org/llama.cpp/pull/32740 at head 808fd1c4e,
its four commits (1fa93f902, d29f9d489, ff91e4f84, 808fd1c4e) squashed.
Inkling's per-query bias table is applied inside flash attention
instead of a dense [n_kv, n_tokens, n_head] bias tensor, which cuts
Inkling's compute buffers and long-context prefill time on the RX 580.
Drop this commit when the PR is merged upstream (FORK.md, "Temporary
upstream carries").
One conflict, ggml-vulkan.cpp use_mask_opt: kept the fork's GCN
`HSK >= 256 || HSV >= 256` clause and its comment (F2) and upstream's
`!use_sparse`, added the PR's `!bias &&`.
Original commit trailer: Assisted-by: Codex
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
Upstream #30591 added its own tiled-transpose concat pipelines, selected
for any dim-0 concat whose src1 is transposed. Behind the fork's fast path
(which already takes ne0 >= 32 and honours GGML_VK_CONCAT_FASTPATH) that
selector still caught the decode case (ne0 < 32), bringing back the
decode regression the TILE_DIM gate fixed (FORK.md F3), and
GGML_VK_CONCAT_FASTPATH=0 no longer gave the generic kernel.
Apply the same gate and the knob to upstream's selector.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
GGML_VK_CONCAT_FASTPATH=0 forces the generic path and =log reports each
fast-path dispatch, so validation can prove the path fires at prefill and
not at decode.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The tiled-transpose fast path in ggml_vk_concat is a large prefill win but a
loss at decode. On a hybrid model like Qwen3.6-35B (gated delta-net on 30 of
40 layers) build_conv_state emits a transposed concat per recurrent layer, so
the fast path runs 30 times per decoded token.
At one token the transposed source is a single column, so there is no
uncoalesced stride left to fix, but the path still costs two dispatches and an
unconditional ggml_vk_sync_buffers - a full pipeline barrier that serializes
the command stream. Measured -17% tg256 on RX 580; the barrier, not the
half-empty transpose dispatch, is nearly all of it.
Require the transposed source to be at least one TILE_DIM wide. That source is
qkv_mixed transposed, so ne[0] is the ubatch token count: prefill keeps the
fast path, decode and small speculative drafts take the generic one.
pp8192 +11% retained, tg256 back to parity, generation byte-identical.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
Add an optional backend interface method free_scratch (with a public
ggml_backend_free_scratch wrapper) that frees transient/scratch device
memory a backend holds outside of any allocated buffer, keeping the
backend usable - the scratch is reallocated lazily on the next compute.
Implement it for the Vulkan backend (ggml_backend_vk_free_scratch): free
the prealloc_x/y/split_k/add_rms_partials and sync_staging device buffers
and reset their sizes, so an idle/cold model does not hold the vision or
matmul compute preallocations in VRAM. All other backends leave the hook
null (no-op).
Assisted-by: Claude
The generic concat shader reads a transposed source (nb[1]==type_size) with an
uncoalesced stride, which is catastrophically slow on discrete GPUs (~5.6ms for
a 16MB concat on the RX 580 vs ~130us of memory bandwidth), a ~9% prefill
hotspot on Qwen3.5-4B (delta-net state concat).
Add a fast path for concat along dim 0 where one source is stored transposed and
the other source + dst are contiguous along dim 0: copy the contiguous source
with copy.comp and transpose the other source into the matching dst sub-region
with the existing tiled copy_transpose shader (shared-memory 32x32 transpose,
coalesced read+write). Reuses pipeline_cpy_* / pipeline_cpy_transpose_* with
custom push constants + doffset, no new shader. Falls back to the generic path
otherwise (gated on type/shape/contiguity + 16-bit doffset bound).
Assisted-by: opencode
Staging K/V through shared memory looks like an obvious win on GCN: without it
each rowgroup re-reads the whole K/V block from global memory, and with row_split
4 that pulls one 16KB block through a 16KB vector L1 four times.
Measured, it loses: -6.7% pp2048 at depth 16k and -7.4% at 32k on Polaris at head
size 128. The kvsh stride of D/4+1 dwords is 4 mod 32, which costs an 8-way LDS
bank conflict on wave64 - the +1 padding is tuned for warp32 - and the extra
shared memory eats occupancy this shader is already short of.
Comment only, no behaviour change. Leaving a note so the next person does not
spend a GPU on it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
mask_opt was disabled on AMD GCN for head sizes <= 256, but it is
beneficial there in high-context prefill: on fully-visible causal
blocks it skips the per-block mask load+add, and it skips fully-masked
blocks entirely. The attention op on GCN is compute-bound on the
softmax path (no matrix cores), so this cuts real work.
Verified lossless (perplexity bit-identical with it on vs off) and a
measured prefill win on Qwen3.5-35B-A3B (head_dim 256) on an RX 580:
pp2048 unchanged at short context, +8% @ 16k, +12% @ 32k, growing with
depth. Enable for GCN when HSK/HSV >= 256; the existing large-mask
conditions keep it off for decode.
Assisted-by: Claude
* ggml : add SIGMOID backward pass
Sigmoid is used in Qwen3.5 hybrid models and had no backward pass.
Bump RPC protocol patch version for the new op.
Signed-off-by: gianni-cor <gianfranco.cordella@tether.io>
* replace assert with ggml_assert
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* ggml : keep sigmoid backward checks in release builds
Assisted-by: Codex
* ggml : preserve existing operation IDs when adding SIGMOID_BACK
Assisted-by: Codex
---------
Signed-off-by: gianni-cor <gianfranco.cordella@tether.io>
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
Used angelX to optimize SYCL/drama 3 coordination, results are as follows
Each output reads only inputs with i*s0 <= idx < i*s0 + K, in the same ascending order as before.
Arc Pro B70, oneAPI 2026.1, test-backend-ops perf, base 9b56ccf00 vs patch, median of 3 same-boot A/B/A rounds, A/A 1.000-1.001:
| input [ne0, ne1] | kernel [K, OC, IC] | s0 | base us | patch us | speedup |
|---|---|---|---|---|---|
| [16, 1536] | [16, 768, 1536] | 8 | 3910.57 | 1348.59 | 2.90x |
| [128, 768] | [16, 384, 768] | 8 | 37735.00 | 1665.26 | 22.66x |
| [1024, 384] | [8, 192, 384] | 4 | 270371.75 | 1471.74 | 183.71x |
| [4096, 192] | [4, 96, 192] | 2 | 534614.63 | 752.74 | 710.23x |
Shapes are the drama 3 codec decoder upsampling stack, added as perf cases. Chained 4-layer stack at 1/4/24 input frames: base 66.3/847.4/28638.5 ms, patch 2.3/6.7/30.7 ms. Kernel numbers, not end-to-end TTS latency.
Correctness: test-backend-ops -o CONV_TRANSPOSE_1D 116/116 on base and patch, and 116/116 rebased on 3f03b1772. Seeded output dump (7.46 MB) byte-identical to base.
Full test-backend-ops aborts on base and patch alike at the non-F32 SWIGLU_OAI assert (element_wise.cpp:1187, see #30857). With only those cases excluded: 20145/20173 on base and patch, and every patch-run failure also fails on base.
Co-authored-by: Titaniumtown <titaniumtown@proton.me>
Fixes https://github.com/ggml-org/llama.cpp/issues/31680
The matmul_tiled() implementation in tinyBLAS_Q0_PPC requires nc to be a multiple of 8 to perform matrix multiplication correctly.
The previous alignment logic contained a special case for n == 4, which allowed n_aligned to be 4. Consequently, the tiled path could be selected with nc = 4, violating the kernel's tile-width requirement.
This caused incorrect matrix multiplication results, observed as garbled inference output when running Flint Q4_K_M and Gemma 3 27B Instruct.
* hip: add RDNA1 MMVQ parameter table (1 warp per block for ncols=1)
* hip: write the RDNA1 MMVQ values explicitly instead of recursing into the generic table
* hip: list the RDNA1 MMVQ table before RDNA2 in the enum
* Update ggml/src/ggml-cuda/mmvq.cu
---------
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* tests: add CUDA-only MUL_MAT/MUL_MAT_ID cases with results above the fp16 range
IQ1_M weights have no MMQ kernel, so once the batch is bigger than the MMVQ
limit the path is dequantize + cuBLAS. These cases produce results around
1.3e5, past the fp16 max of 65504, so an fp16 result buffer turns them into inf
while the CPU reference stays finite. They are only added for the CUDA backend.
Vulkan deliberately accumulates quantized matmuls in fp16 at default precision
and would fail them by design.
* CUDA: use an fp32 result for quantized src0 in the cuBLAS fallback
On the F16 compute path, quantized weights went through cuBLAS with fp16
accumulation and an fp16 result buffer that was converted to fp32 afterwards.
Anything above 65504 became inf and the next layer turned into NaN (see #30951:
gpt-oss-120b with IQ1_M experts at -ub 512). Set prefer_f32_output for
quantized src0 so the GEMM still uses fp16 operands but accumulates and stores
in fp32, the same mode already used on Volta, RDNA4 and CDNA. Every other
quantized path already accumulates in fp32.
Diagnosis and fix by denizyuret in #30951.
* CUDA: keep the MUL_MAT_ID precision in the per-expert fallback
The per-expert dst_slice kept op = GGML_OP_NONE, so ggml_prec_get_acc returned UNDEFINED and a requested precision was ignored. Set the op to GGML_OP_MUL_MAT. Also revert the fp32 result change and remove the tests.
---------
Co-authored-by: JonSnow1807 <JonSnow1807@users.noreply.github.com>
On Adreno, subgroupAdd() after divergent code returns wrong sums. The
fused ADD+RMS_NORM partial-sum path hits this, so the decode-step norm
is garbage and models emit stray tokens after long prompts.
Clear subgroup_arithmetic for the Qualcomm proprietary driver, which
also turns off add_rms_fusion and the other subgroupAdd paths.
test-backend-ops on Adreno 850 and X2-90: ADD_RMS_NORM 20/25 -> 25/25,
RMS_NORM_MUL_ADD 33/45 -> 45/45. llama 3B Q4_0 on Adreno 850: pp512
and tg128 unchanged.
* opencl: support int8 KQ for DK = 512
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
Assisted-by: Codex
* opencl: allow query chunk up to 96 for int8 KQ
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
Assisted-by: Codex
* opencl: extend `build_program_from_source` to allow non-fatal failure
Assisted-by: Codex
* opencl: add int8 KQ for decomposed FA with DK <= 256
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
Assisted-by: Codex
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
* ggml : add BF16 support to ROPE and SET_ROWS (CPU, CUDA)
* cuda : SET_ROWS from BF16 into quantized types and F8_E4M3
* cuda : fuse BF16 rope into a BF16 cache
* tests : trim the BF16 rope cases to one per code path
* ggml-openvino: round the GPU KV length up to a power of two
Review notes (follow-ups, not blocking):
- convert_ggml_input_to_ov pads the KQ mask into a std::vector and then
copies it into the ov::Tensor. Writing the padded rows straight into
padded.data() avoids the second allocation and copy (up to 16 MB per
call at a 16K context).
- The padding is skipped when the mask is not contiguous, and the KV
inputs are then sliced to more rows than the mask has columns. Masks
are always contiguous today; a GGML_ASSERT would make that explicit.
- compute_llm_params reads GGML_OPENVINO_STATEFUL_EXECUTION directly;
is_stateful_enabled() is local to ggml-openvino.cpp.
* ggml-openvino: Optimize model recompiles of prefill stage
Optimize the model recompiles of prefill stage when attended size
changed between prompts. This can save model recompile time for
llama-bench and llama-server.
* ggml-openvino: Optimize model recompiles of prefill stage
Optimize the model recompiles of prefill stage when attended size
changed between prompts. This can save model recompile time for
llama-bench and llama-server.
* ggml-openvino: Remove solution C and keep A as default
* ggml-openvino: match the output count before reusing a compiled NPU prefill
The static prefill model has a fixed inp_out_ids shape [1,1,1,output_len].
The new reuse check compared only the attention sizes, so llama-perplexity,
which alternates 1 and 512 outputs per ubatch, reused the model compiled
for 512 outputs on a 1-output ubatch and failed with "got [1,1,1,1]
expecting [1,1,1,512]". Store the output count of the cached prefill
model and require it to match.
* ggml-openvino: Fix LFM2 context create on GPU and DSpark speculative decoding
* ggml-openvino: remove leftovers from merging dev into the LFM2/DSpark fix
- compute_model_outputs exported OUTPUT-flagged nodes twice: dev's block
ran before the new one and still exported result_norm, so the
result_norm exclusion had no effect. Keep only the new block.
- compute_node_dynamic_dims checked is_activation_input twice in a row.
Remove the second copy.
- process_view_input_new created the dynamic Split on split_dim. With
stateful execution the input drops the leading size-1 axis, so use
split_axis, which already accounts for the rank difference.
* ggml-openvino: hash the output flags into the model-cache fingerprint
The fingerprint hashed the ops, names and weights but not the outputs, so
a blob compiled with another output set was imported. With the result_norm
change, an NPU blob written before it gave wrong output, and enabling
llama_set_embeddings_layer_inp() reused a blob without the layer-input
outputs. Hash each node's output flag and bump the cache version to 8.
* ggml-openvino: support f16 SCALE and f16/bf16 TRI
test-backend-ops now runs SCALE on f16 and TRI on f16 and bf16. Both
translators built their constants as f32, and Multiply, Add and Select
need all inputs in one element type, so building the graph threw for
these cases.
SCALE now converts a non-f32 input to f32, applies the scale and bias,
and converts back once, as the CPU backend computes it. TRI creates its
zero constant in the input's element type. f32 graphs are unchanged.
* ggml-openvino: [Qwen3.6 A3B]fix static-shape ids view of the last MoE
layer
The same-stride single-dim Slice in process_view_input_new was followed by a Reshape
whose target baked the static chunk size into the token axis. After the output-ids
GET_ROWS the last layer runs on n_outputs tokens (1 in the prefill graph), so the
Reshape failed shape validation for qwen35moe with FORCE_STATIC or on the NPU.
Skip the Reshape in static graphs when the slice already keeps the rank;
the dynamic graphs are unchanged.
* enable mamba based models on OV CPU-GPU
* revert changes in tests
* docs : add Falcon-H1 to the OpenVINO validated models
Falcon-H1 runs on CPU and GPU since Mamba SSM support was added. Add its
row and the known limitations: multi-sequence batches fail, GPU accuracy
is slightly lower than CPU, and the NPU fails with an element type
mismatch.
* ggml-openvino: simplify the SSM_SCAN build check and clean up
- Drop the SelectiveSSM configure check and its guards: OpenVINO 2026.4.1
always provides the op. Drop the no-op LANGUAGE CXX source property.
- Drop the semicolon after OPENVINO_OP("SelectiveSSM"). GCC 15 reports it
with -Wextra-semi, which fails builds with fatal warnings.
- Add ggml_openvino_is_static() for the repeated NPU or
GGML_OPENVINO_FORCE_STATIC check.
- Share the f16 and f32 KQ mask padding code.
- Drop the redundant inline on in-class static functions.
---------
Co-authored-by: Suhas Somashekar <suhas.somashekar@intel.com>
Co-authored-by: yzhou51 <yaowei.zhou@intel.com>
Co-authored-by: Ananth, Swathi <swathi.ananth@intel.com>
Co-authored-by: Beheshti <nazanin.beheshti@intel.com>
* cuda: add scatter for batched tensor writes
* update test case to mirror get_n test case
* save restore timing logs
* refactor code into batch-copy.cu file
* Adds initial scheduler stress tests
* Adds new sleep op to simplify scheduler testing
* Adds testcase addressing https://github.com/ggml-org/llama.cpp/issues/23321
* Adds experimental vulkan GGML_OP_SLEEP
* Improves structure and naming of previous tests. Adds new tests,
including one which detects the nkvo-bug correctly, and validates the
proposed hotfix.
* Cleanup: Removes CUDA specific comments and nvtx-markers
* Fix: forgot to add sleep specific test case
* Fix oversights identified by CI
* Drops default prints so that it behaves like other ctests. Adds -v flag
to toggle this behavior
* Fix: Early exit in CPU-only settings (cause for graviton CI fails)
* Improves stress_test_linked_list by removing overlap with
test_chain_all_backends and test_pair_user_inputs. The latter ones test
either activations or user inputs in isolation and just once, whereas
the stress test tests all combinations multiple rounds
* Exclude ACCEL devices from new tests due to incompatibility
* Fixes vulkan test failure in llvmpipe
* Adds asserts, improves comments and some checks
* Fixes nitpicks in white-space and variable naming
* Adds comments that certain vk drivers could time OP_SLEEP incorrectly
* Drops limitiation to CPU/GPU in test-sleep
* Aligns OP_SLEEP guards in cuda with vk guards (input shape and
contiguity)
* Implements feedback for OP_SLEEP to work in place (no copy)
* Apply non-code suggestions from JohannesGaesslers code review
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* Removes struct backend_supports; enables tests to return skip, instead
of only pass/fail
* Make clang-tidy happy
* Removes sched_backend_caps, improves create_test_scheduler according to
feedback (L149)
* Removes "name" field from struct sched_check in favor of
ggml_set_name/ggml_get_name
* Drops unnecessary shader_clock.comp test and guards
* Replaces sched_ prefixes with helper_ prefix
* Moves test suite to new RAII types for better memory management
* Adds sleep option to test_chain_all_backends
* Adds sleep option to first-run arm of the y-shaped test
* Fix switchcase order for legibility in stress_test_dag
* Simplfies stress_test_linked_list via double for-loop
* Adds pseudo-random input to stress_test_linked_list, and runs all
transitions both with and without CPU input (next to activation copy as
input)
* Fix RPC Proto patch due to GGML_OPS bump
* Adds ggml-cpp.h treatment to test-sleep.cpp
* Adds TODO pointers for subsequent additional tests.
* Apply suggestion to ggml.h
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* Apply suggestion to tests/test-backend-sched.cpp
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* Apply suggestion to tests/test-backend-sched.cpp
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* Fixes node compatibility check to not take buffer locations into account.
This led to skipped tests due to incorrectly rejected ops due to wrong buffer locations in multi-GPU (which is one of the scheduler behaviors under test)
* Apply batched suggestions from gg code review
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* Completes callsite change of gg-suggestion
* Switch test-sleep from std::chrono::steady_clock::now() to ggml_time_us
to avoid mac clock drift and spurious test failures.
This makes the test a bit nonsensical on CPU (same clock checks itself),
but the test still has merit on the other backends where
independent device clocks are used.
---------
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>