llama, mtmd: follow the reference precision of ggml_prec_set_acc calls (#31851)

* llama, mtmd: follow the reference precision of ggml_prec_set_acc calls

Requests that the reference implementation does not keep in F32 move
to GGML_PREC_BF16: the GLM4 family down and output projections, the
MiMoVL vision matmuls and the Qwen3-TTS speaker encoder conv1d. The
DeepSeek V4/V4.1 routers and the MiniMax-M3 indexer stay in F32 with
a link to the reference line that computes them in F32.

* ggml: handle GGML_PREC_BF16 the same way as GGML_PREC_F32 in backends

Backends that reject GGML_PREC_F32 on the src1 of MUL_MAT_ID now also
reject GGML_PREC_BF16 (CUDA, Hexagon, OpenCL, WebGPU, Vulkan), and
backends that accept GGML_PREC_F32 for flash attention now also accept
GGML_PREC_BF16 with the same F32 accumulation (CPU, SpaceMIT, ET).
Vulkan keeps the F32 transfer between devices for both. The CUDA
cuBLAS path already handles GGML_PREC_BF16 and is unchanged.

* tests: cover GGML_PREC_BF16 in the flash attention tests

The flash attention cases run with GGML_PREC_BF16 next to F32 and
DEFAULT, and var_to_str prints bf16 so these cases keep their own name.

* llama: request GGML_PREC_BF16 for the KQ multiplication

* cont : add TODO in clip.cpp

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
This commit is contained in:
PascalandGeorgi Gerganov authored and GitHub committed 2026-11-08 08:37:46 +01:00
1 parent a2b301831d
commit d6be065572
17 files changed
+32 -27

No files matched your search

+1 -1
View File
@@ -875,7 +875,7 @@ llama_model_deepseek41::graph::graph(const llm_graph_params & params) :
cur = build_norm(cur, layer.ffn_norm, nullptr, LLM_NORM_RMS, il);
cb(cur, "ffn_norm", il);
// F32 router accumulation: expert selection is near-tie sensitive
// the reference computes the router in F32: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/blob/1bae0e1088993b9b82de08f59de5be5a34e42170/inference/model.py#L813
ggml_tensor * logits = build_lora_mm(layer.ffn_gate_inp, cur);
ggml_prec_set_acc(logits, GGML_PREC_F32);
cb(logits, "ffn_moe_logits", il);