3 people ecbcb7ea9d CUDA: add CDNA3 MFMA support for flash attention MMA kernel (#19806) ...

* CUDA: add CDNA3 MFMA support for flash attention MMA kernel

Add MI300X (gfx942) MFMA tensor core flash attention using
v_mfma_f32_16x16x16_f16 (FP16 in, FP32 accumulate).

- Add FATTN_WARP_SIZE=64 for CDNA wavefront64
- Add CDNA config for head sizes 64, 80, 96, 112, 128
- Add FP16 MFMA intrinsic path in mma.cuh
- Add manual V transpose load for MFMA register layout
- Route CDNA to MMA for prompt processing, VEC for token generation
- Fix Q loading and combine stride granularity for non-power-of-2 heads

Benchmarks (Qwen2.5-1.5B Q4_K_M, MI300X):
  pp512  +7%,  pp1024 +13%,  pp2048 +23%,  pp4096 +39%
  tg128  -10% (FA overhead, VEC used for both)

All 2480 flash attention tests pass.

Ref: https://github.com/ggml-org/llama.cpp/issues/17917

* address review: replace FATTN_WARP_SIZE with constexpr, improve dispatch

- Replace #define FATTN_WARP_SIZE with constexpr int warp_size =
  ggml_cuda_get_physical_warp_size() in each device function
- Use ne[1]*gqa_ratio threshold for MMA vs tile dispatch. Benchmarked
  crossover on MI300X @ d32768 with power-of-2 GQA models:
    hsk=64  (Llama 1B, gqa=4): MMA wins at eff >= 128 (+11%)
    hsk=128 (Llama 3B, gqa=4): MMA wins at eff >= 128 (+4%)
  Unified threshold: eff_nq >= 128 for all head sizes.
- Remove VEC fallback; small batches fall through to tile kernel

* Update ggml/src/ggml-cuda/fattn.cu

* use ggml_cuda_info().devices warp_size instead of hardcoded check

---------

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

2026-02-27 19:37:26 +01:00

..

ggml-blas

ggml : add ggml_build_forward_select (#18550)

2026-01-19 20:03:19 +02:00

ggml-cann

CANN: Remove unnecessary wrapper for gml_backend_buft_is_cann (#18968)

2026-02-10 14:19:30 +08:00

ggml-cpu

ggml-cpu: add repack for mxfp4 (#19738)

2026-02-27 18:15:09 +08:00

ggml-cuda

CUDA: add CDNA3 MFMA support for flash attention MMA kernel (#19806)

2026-02-27 19:37:26 +01:00

ggml-hexagon

hexagon refactor all Ops to use local context struct (#19819)

2026-02-23 16:32:14 -08:00

ggml-hip

HIP: add mmf for CDNA (#18896)

2026-01-29 11:10:53 +01:00

ggml-metal

models : optimize qwen3next graph (#19375)

2026-02-14 12:57:36 +02:00

ggml-musa

CUDA: faster tile FA, add oob checks, more HSs (#16492)

2025-10-11 20:54:32 +02:00

ggml-opencl

opencl: refactor expm1 and softplus (#19404)

2026-02-17 14:47:18 -08:00

ggml-rpc

rpc : use unordered_map::reserve and emplace (#18513)

2026-01-02 12:09:36 +02:00

ggml-sycl

replace the magic nunber 768 by max work group size to support iGPU (#19920)

2026-02-27 09:26:07 +08:00

ggml-virtgpu

ggml-virtgpu: improve the reliability of the code (#19846)

2026-02-26 20:00:57 +08:00

ggml-vulkan

vulkan: fix fp16 Flash Attention on Windows AMD RDNA2 and below (#19921)

2026-02-26 19:11:04 +01:00

ggml-webgpu

ggml-webgpu: Add unary op (SQR, SQRT, SIN, COS) support. (#19700)

2026-02-19 09:18:30 -07:00

ggml-zdnn

ggml-zdnn : mark zDNN buffers as non-host (#18967)

2026-01-22 01:16:21 +01:00

ggml-zendnn

ggml-zendnn: update code for latest ZenDNN API (#19923)

2026-02-27 08:43:41 +08:00

CMakeLists.txt

hexagon: enable offloading to Hexagon on Windows on Snapdragon (#19150)

2026-01-29 12:33:21 -08:00

ggml-alloc.c

ggml : make ggml_is_view as API (#19539)

2026-02-16 17:43:34 +02:00

ggml-backend-dl.cpp

hexagon: enable offloading to Hexagon on Windows on Snapdragon (#19150)

2026-01-29 12:33:21 -08:00

ggml-backend-dl.h

hexagon: enable offloading to Hexagon on Windows on Snapdragon (#19150)

2026-01-29 12:33:21 -08:00

ggml-backend-impl.h

llama: use host memory if device reports 0 memory (#18587)

2026-01-09 05:34:56 +08:00

ggml-backend-reg.cpp

ggml : use noexcept overload for is_regular_file in backend registration (#19452)

2026-02-10 10:57:48 +01:00

ggml-backend.cpp

ggml-backend: fix async set/get fallback sync (#19179)

2026-02-02 10:00:05 +01:00

ggml-common.h

llama : add gpt-oss (#15091)

2025-08-05 22:10:36 +03:00

ggml-impl.h

ggml : make ggml_is_view as API (#19539)

2026-02-16 17:43:34 +02:00

ggml-opt.cpp

finetune: SGD optimizer, more CLI args (#13873)

2025-08-14 12:03:57 +02:00

ggml-quants.c

ggml : fix uninitialized is_on_grid in quantize_row_iq3_xxs_impl (#15928)

2025-09-23 10:25:20 +02:00

ggml-quants.h

llama : add gpt-oss (#15091)

2025-08-05 22:10:36 +03:00

ggml-threading.cpp

ggml : build backends as libraries (#10256)

2024-11-14 18:04:35 +01:00

ggml-threading.h

remove CMAKE_WINDOWS_EXPORT_ALL_SYMBOLS (#10797)

2024-12-12 19:02:49 +01:00

ggml.c

ggml/gguf : prevent integer overflows (#19856)

2026-02-24 20:17:11 +02:00

ggml.cpp

ggml : Print backtrace on uncaught C++ exceptions (ggml/1232)

2025-06-01 13:43:57 +03:00

gguf.cpp

gguf : avoid too many file size calls (#19919)

2026-02-26 12:46:32 +02:00