Georgi Gerganov ea40a3a1e1 metal : fuse DeepSeek-V4/V4.1 and GLM MoE routing (#31615)
* metal : support scale-bias routing norm in the TOPK_MOE fusion

The DeepSeek-V4.1 graph normalizes the routing weights with a scale+bias
(sum + 1e-20) instead of the lower clamp used by the other models. Add
norm_bias/norm_scale_bias fusion chains that accept a SCALE node with a
unit scale factor at the clamp position, pass an additive bias in the
kernel args, and apply it as max(wt_sum, clamp) + bias so both variants
share one kernel. Cover the new chains in test-backend-ops.

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)

* metal : fuse sqrt(softplus) MoE routing with a selection bias

Add TOPK_MOE chains anchored at the softplus node (UNARY+SQRT+ADD+
ARGSORT+GET_ROWS plus optional norm/scale) so the DeepSeek-V4/V4.1
routing fuses on Metal. The kernel scores with sqrt(softplus) and ranks
selection-biased keys while the gathered weights and the norm stay
unbiased, matching the reference graph. Dispatch hooks at the softplus
node, cover the norm-bias combination in test-backend-ops and record the
new deepseek4/deepseek41 fusion counters.

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)

* metal : fuse sigmoid MoE routing with a selection bias

The sigmoid-gated MoE archs (GLM, DeepSeek-V2/V3.2, ...) rank
sigmoid(logits) + exp_probs_b with an optional weights-norm, but only
the softmax and sqrt(softplus) routing chains were fusable.

- add 6 sigmoid-anchored TOPK_MOE chains: sigmoid + ADD + ARGSORT +
  GET_ROWS with optional SCALE and SUM_ROWS + CLAMP/SCALE + DIV norm
  tail; the selection-bias ADD is required, same as the sqrt(softplus)
  chains
- generalize the gating function constant from a sqrt(softplus) bool
  to an int (0 = softmax, 1 = sqrt(softplus), 2 = sigmoid); sigmoid
  scores are 1 / (1 + exp(-logits)), matching the CPU, Metal unary and
  CUDA kernels
- anchor the unary dispatch hook at sigmoid nodes too
- test-backend-ops: 4 sigmoid routing cases with and without a
  weights scale
- re-record tests/fusion/MTL.csv: the sigmoid archs (glm5-next,
  deepseek2/32, kimi, minimax, ...) now fuse their routing, and
  muse-spark prefill drops one MOE_REDUCE fire because the reordered
  graph allocates the routing weights inside the reduce output range,
  which the dst/input overlap check correctly declines

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)

* cont : remove low-signal comments

* ocd : valign ggml_metal_fusions
2026-11-05 13:52:56 +02:00
2026-11-04 15:45:29 +02:00
2026-11-05 10:29:14 +02:00
2026-11-04 12:00:37 +02:00
2026-06-12 15:53:26 +02:00
2026-10-17 02:11:56 +02:00
2026-02-02 08:38:55 +02:00

llama.cpp

llama

Quick start

A few options to get llama.cpp installed on your machine:

# curl
curl -LsSf https://llama.app/install.sh | sh

# powershell
irm https://llama.app/install.ps1 | iex

Once installed:

# Download and run a model directly from Hugging Face
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF

# Launch OpenAI-compatible API server
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF
VLM session with `llama cli` VLM session with llama cli Built-in web UI against `llama serve` running Qwen 3.6 Built-in web UI against llama serve

Description

The main goal of llama.cpp is to enable LLM (and VLM) inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud.

  • Plain C/C++ implementation without any dependencies
  • Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks
  • AVX, AVX2, AVX512 and AMX support for x86 architectures
  • RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE support for RISC-V architectures
  • 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use
  • Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA)
  • Vulkan and SYCL backend support
  • CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity

The llama.cpp project is build on top of the ggml library.

Supported backends

Backend Target devices
BLAS All
BLIS All
CANN Huawei Ascend NPU
CUDA Nvidia GPU
HIP AMD GPU
Hexagon Snapdragon
IBM zDNN IBM Z & LinuxONE
MUSA Moore Threads GPU
Metal Apple Silicon
OpenCL Adreno GPU
OpenVINO [In Progress] Intel CPUs, GPUs, and NPUs
RPC All
SYCL Intel GPU
VirtGPU VirtGPU APIR
Vulkan GPU
WebGPU All
ZenDNN AMD CPU

Documentation

Tools

Development

Contributing

  • Contributors can open PRs
  • Collaborators will be invited based on contributions
  • Maintainers can push to branches in the llama.cpp repo and merge PRs into the master branch
  • Any help with managing issues, PRs and projects is very appreciated!
  • Read the CONTRIBUTING.md for more information

Acknowledgements

  • yhirose/cpp-httplib - Single-header HTTP server, used by llama-server - MIT license
  • nothings/stb - Single-header image format decoder, used by multimodal subsystem - Public domain
  • nlohmann/json - Single-header JSON library, used by various tools/examples - MIT License
  • mackron/miniaudio - Single-header audio format decoder, used by multimodal subsystem - Public domain
  • sheredom/subprocess.h - Single-header process launching solution for C and C++ - Public domain
S
Description
No description provided
Readme MIT
580 MiB
0 Stars 1 Watchers 0 Forks
Languages
C++ 54.9%
C 16.7%
Python 7%
Cuda 5.6%
TypeScript 3.9%
Other 11.7%