Files
llama.cpp/ggml/cmake
Oliver SimonsandAman Gupta 9e9619d030 CUDA: Add OCP FP8 support (#28413)
* Initial CUDA WIP version

* Use fp8 intrinsics

* Do W8A16 for FP8 in GEMV case

(base) osimons@ub-osimons:~/lcpp_fp8.git$ ./build-x64-linux-gcc-reldbg/bin/llama-bench -m /mnt/share/gguf/nvidia/Qwen3.6-27B-NVFP4/Qwen3.6-27B-NVFP4_fp8.gguf -p 8196 -ub 512,2048 -d 0,32000 -n 256 --load-mode dio
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 32165 MiB):
  Device 0: NVIDIA RTX PRO 4500 Blackwell, compute capability 12.0, VMM: yes, VRAM: 32165 MiB
| model                          |       size |     params | backend    | ngl | n_ubatch |         lm |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -------: | ---------: | --------------: | -------------------: |
| qwen35 27B NVFP4               |  19.56 GiB |    27.32 B | CUDA       |  -1 |      512 |        dio |          pp8196 |       2781.53 ± 8.48 |
| qwen35 27B NVFP4               |  19.56 GiB |    27.32 B | CUDA       |  -1 |      512 |        dio |           tg256 |         37.56 ± 0.02 |
| qwen35 27B NVFP4               |  19.56 GiB |    27.32 B | CUDA       |  -1 |      512 |        dio | pp8196 @ d32000 |       1616.31 ± 2.61 |
| qwen35 27B NVFP4               |  19.56 GiB |    27.32 B | CUDA       |  -1 |      512 |        dio |  tg256 @ d32000 |         34.11 ± 0.03 |
| qwen35 27B NVFP4               |  19.56 GiB |    27.32 B | CUDA       |  -1 |     2048 |        dio |          pp8196 |       2542.00 ± 0.51 |
| qwen35 27B NVFP4               |  19.56 GiB |    27.32 B | CUDA       |  -1 |     2048 |        dio |           tg256 |         37.36 ± 0.02 |
| qwen35 27B NVFP4               |  19.56 GiB |    27.32 B | CUDA       |  -1 |     2048 |        dio | pp8196 @ d32000 |       1762.86 ± 1.33 |
| qwen35 27B NVFP4               |  19.56 GiB |    27.32 B | CUDA       |  -1 |     2048 |        dio |  tg256 @ d32000 |         34.09 ± 0.02 |

* Optimize GEMV kernel for FP8

* Guard __nv_cvt_fp8x2_to_bf162raw on CTK 13.2

Fixes https://github.com/ggml-org/llama.cpp/pull/28413#discussion_r4044500099

* Remove trailing whitespace

* Add supported for strided copies to/from FP8 in CUDA

* Prepare Q_reg for FP8 FlashAttention in fattn-vec.cuh

* Fix odd-rows for convert/getrows, possible with FP8

* Fix FP8 fallback in cuBlas

* Gate mmvf_f8x2_e4m3_to_bf162 on FP8 availability, i.e. CTK >= 11.8

* Restrict (fused) MMVF to 2*ts alignment, and cublas FP8 fo 16-byte

* Add todo about accumulator/precision honoring

This is beyond the scope of the stacked PR

* Avoid races in fp8_fallback

* Allow Hopper and other BW SKUs for FP8

* Use intrinsic over explicit manual conversions

* Resolve TODO

As per spec, we don't have to preserve signs

* Add commend on cublaslt workspaces

* Use convert_unary_cont_cuda over block dequant in convert.cu

* Use get_rows_cuda_float for fp8 kv cache

* Make FP8 copies work on all IHV's

* More cleanup

* Simplify by falling back to shared cuBLAS helper

* Fuse weight_scale multiplication for FP8 also

* More code-cleanup

* Guard __nv_cvt_fp8x2_to_bf162raw on CTK 13.4

Only cvt.rn.bf16x2.e4m3x2 is exposed from 13.2 onwards, so
93e98b9203dfacb85f9ac665f5f0482a8c6b4192 was incorrect fix

* Use F32 activations in fallback mmvf path

* Explicitly link against cublaslt

* Move FP8 conversion intrinsics into ggml_cuda_cast

Cleaner this way

* WIP src1/prec for fp8 in MMVF

* Remove comment

* Keep only f8 x f8 KV cache for now

* Just do FP8 x FP32 x FP32 in MMVF as its fast enough

* Update ggml/src/ggml-cuda/ggml-cuda.cu

Co-authored-by: Aman Gupta <amangupta052@gmail.com>

* Add batched FP8 GEMMS in cublas as suggested by code review

https://github.com/ggml-org/llama.cpp/pull/28413#issuecomment-6676015644

* Extract fp8x2 -> float2 load into helper

* Searapte copy checks to/from fp8

* Compiler massaging to avoid repeated address offsets

---------

Co-authored-by: Aman Gupta <amangupta052@gmail.com>
2026-11-11 10:44:30 +02:00
..