3 people bc81d47aba CUDA: route batch>=4 quantized matmul to MMQ on AMD MFMA hardware (#23227) ...

* CUDA: per-quant MMVQ/MMQ batch threshold on AMD MFMA hardware

The dispatcher uses a single global threshold (MMVQ_MAX_BATCH_SIZE = 8)
to choose between mul_mat_vec_q (per-row GEMV) and mul_mat_q (MFMA-tiled
GEMM) for quantized matmul. On AMD CDNA, the optimal crossover differs
substantially by quant family because the per-row GEMV cost is dominated
by dequantisation, not the dot-product itself: K-quants pay a heavier
super-block decode and so MMQ wins sooner; legacy and IQ quants have
lean decode and stay ahead until the batch fully populates an MFMA tile.

This patch introduces ggml_cuda_should_use_mmvq(type, cc, ne11) -> bool,
mirroring the existing ggml_cuda_should_use_mmq, and gates per-quant
thresholds on amd_mfma_available(cc):

  Q3_K, Q4_K, Q5_K  : MMVQ <= 3   (MMQ wins from batch=4: +5% .. +76%)
  Q2_K, Q6_K        : MMVQ <= 5   (MMQ wins from batch=6: +8% .. +35%)
  others            : MMVQ <= 8   (legacy & IQ regress under MMQ; unchanged)

Non-AMD-MFMA paths (NVIDIA, RDNA, CDNA1 without MFMA) are byte-identical
to master. GGML_CUDA_FORCE_MMVQ=1 restores the original global threshold
for A/B testing.

Measured on MI250X (gfx90a, ROCm 7.2.1) with Llama-3.2-3B-Instruct,
llama-bench pp512 across all 20 supported quants, ubatch 1..8, 10 reps.
Full table in PR description.

  Selected pp512 throughput (tok/s, ub=8):
    Q4_K_S:  559 -> 940  (+68%)
    Q5_K_S:  503 -> 884  (+76%)
    Q3_K_S:  629 -> 879  (+40%)
    Q2_K  :  615 -> 809  (+32%)
    Q6_K  :  582 -> 776  (+33%)

  Selected pp512 throughput (tok/s, ub=4):
    Q4_K_S:  444 -> 480  (+ 8%)
    Q4_0  :  682 -> 685  (+ 0%)   (no regression - retains MMVQ)
    IQ4_XS:  706 -> 698  (- 1%)   (no regression - retains MMVQ)

* CUDA: address review — inline MMVQ batch table, drop env hatch & doc block

* tune kernel selection logic for CDNA1

---------

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

2026-05-28 14:50:25 +02:00

..

ggml-blas

vulkan: add get/set tensor 2d functions (#22514)

2026-04-30 17:37:13 +02:00

ggml-cann

vulkan: add get/set tensor 2d functions (#22514)

2026-04-30 17:37:13 +02:00

ggml-cpu

ggml: fixed Arm SVE usage bug in vec.h, vec.cpp (#22841)

2026-05-28 10:04:21 +03:00

ggml-cuda

CUDA: route batch>=4 quantized matmul to MMQ on AMD MFMA hardware (#23227)

2026-05-28 14:50:25 +02:00

ggml-hexagon

hexagon: minor refresh for HMX FA and MM (#23796)

2026-05-28 04:49:11 -07:00

ggml-hip

ggml: backend-agnostic tensor parallelism (experimental) (#19378)

2026-04-09 16:42:19 +02:00

ggml-metal

metal : add apple device id (#23566)

2026-05-25 21:05:16 +03:00

ggml-musa

ggml-cuda: native bf16 flash attention for vec kernel (#20525)

2026-03-22 11:05:51 +01:00

ggml-opencl

opencl: OP_GATED_DELTA_NET (#23312)

2026-05-27 21:23:21 -07:00

ggml-openvino

openvino: driver setup, CI split, thread safety, and NPU optimizations (#21944)

2026-04-21 18:58:34 +03:00

ggml-rpc

rpc : keep last_graph_uid in the device context (#23273)

2026-05-19 09:42:36 +03:00

ggml-sycl

SYCL: implement ggml_sycl_pool_vmm (#22862)

2026-05-26 07:59:00 +03:00

ggml-virtgpu

ggml-virtgpu : include missing mutex header (#22810)

2026-05-10 17:32:41 +02:00

ggml-vulkan

vulkan: fast path for walsh-hadamard transform (#23687)

2026-05-28 13:18:43 +02:00

ggml-webgpu

ggml-webgpu: remove legacy constants (#23672)

2026-05-27 14:22:33 -07:00

ggml-zdnn

vulkan: add get/set tensor 2d functions (#22514)

2026-04-30 17:37:13 +02:00

ggml-zendnn

ggml-zendnn : fixed naming of matmul function (#20964)

2026-05-27 00:59:35 +02:00

CMakeLists.txt

ggml : Parallelize quant LUT init (#23595)

2026-05-25 10:15:46 +03:00

ggml-alloc.c

ggml-alloc: fix out-of-bounds read in ggml_dyn_tallocr_remove_block (ggml/1492)

2026-05-25 12:38:01 +03:00

ggml-backend-dl.cpp

hexagon: enable offloading to Hexagon on Windows on Snapdragon (#19150)

2026-01-29 12:33:21 -08:00

ggml-backend-dl.h

hexagon: enable offloading to Hexagon on Windows on Snapdragon (#19150)

2026-01-29 12:33:21 -08:00

ggml-backend-impl.h

ggml: backend-agnostic tensor parallelism (experimental) (#19378)

2026-04-09 16:42:19 +02:00

ggml-backend-meta.cpp

TP: fix ggml context size calculation (#22616)

2026-05-25 12:37:25 +03:00

ggml-backend-reg.cpp

ggml : skip already registered backends and devices (#22296)

2026-04-28 10:02:32 +03:00

ggml-backend.cpp

ggml : Check the right iface method before using the fallback 2d get (#23514)

2026-05-23 12:49:24 +02:00

ggml-common.h

ggml: add Q1_0 1-bit quantization support (CPU) (#21273)

2026-04-06 20:55:21 +02:00

ggml-impl.h

ggml: add graph_reused (#21764)

2026-04-16 17:21:28 +08:00

ggml-opt.cpp

fix: free ctx_copy in ggml_opt_free to plug per-training-session leak (#21592)

2026-04-08 17:40:15 +02:00

ggml-quants.c

ggml : Parallelize quant LUT init (#23595)

2026-05-25 10:15:46 +03:00

ggml-quants.h

ggml: add Q1_0 1-bit quantization support (CPU) (#21273)

2026-04-06 20:55:21 +02:00

ggml-threading.cpp

ggml : build backends as libraries (#10256)

2024-11-14 18:04:35 +01:00

ggml-threading.h

remove CMAKE_WINDOWS_EXPORT_ALL_SYMBOLS (#10797)

2024-12-12 19:02:49 +01:00

ggml.c

llama + spec: MTP Support (#22673)

2026-05-16 20:06:23 +08:00

ggml.cpp

ggml : Print backtrace on uncaught C++ exceptions (ggml/1232)

2025-06-01 13:43:57 +03:00

gguf.cpp

ggml: gguf_init_from_callback and gguf_init_from_buffer (#22341)

2026-05-25 11:33:29 +02:00