llama.cpp

LLM inference in C/C++

ggml / ops / maintainer PRs / dev stats / lib llama API / llama-server REST API

This fork - Polaris / GCN tuning for large MoE models

Changes and measurements for running large MoE models with their experts offloaded to system RAM (--n-cpu-moe) on an old GCN card. The two code changes below are auto-on, need no flag, and are token-identical to mainline. Everything else here is tuning guidance.

Findings on an RX 580 8GB (Polaris / GCN, PCIe 3.0 x16, no fp16, no matrix cores) with Qwen3.5-35B-A3B Q4_K_M, -b 2048 -ub 2048:

• Flash-attention mask_opt is enabled for GCN large head sizes (this fork's own change). Upstream disables it on GCN; it is a lossless win in high-context prefill - it skips fully-masked causal blocks and the per-block mask add on fully-visible ones, which is real work on a card whose attention is compute-bound (no matrix cores). Auto-on, no flag. On Qwen3.5-35B (head_dim 256): pp2048 +8% @ 16k, +12% @ 32k, growing with depth; perplexity bit-identical.
• -b 2048 -ub 2048 is the biggest prefill lever (the default -ub 512 roughly halves pp).
• Tune --n-cpu-moe to context length. Keep some expert layers resident in spare VRAM for short prompts (e.g. ncmoe 28 on the 35B, ~+5% over all-host); at long context the KV cache needs that VRAM, so raise it (ncmoe 40, all experts on host). Keep flash attention on (-fa 1).

Recommended RX 580 / Polaris serving command (per model):

llama-server -hf <repo>:<quant> -ngl 99 --n-cpu-moe 40 -b 2048 -ub 2048 -fa 1

Lower --n-cpu-moe (e.g. 28) if the model plus your context budget leave spare VRAM; keep it high for long-context / agentic use. At long context the bottleneck is attention compute (GPU-bound), so mask_opt (above) is where the remaining prefill wins come from, not the MoE-transfer path.

Very large MoE (experts bigger than the GTT limit)

Measured on the same RX 580 with Laguna-S-2.1 118B IQ2_M (48 layers, 256 experts, 10 used, experts 30.7 GiB of a 34.7 GiB file, all host-resident at --n-cpu-moe 48). The advice above changes in this regime:

• --no-mmap stops being an option. Its pinned host buffer is charged against the amdgpu GTT limit (~31.4 GiB here, about half of system RAM). At 30.7 GiB of experts the model no longer loads, and the allocation spike can OOM the box. Use mmap and accept the staging copy.
• The routing-ids readback is pure overhead at prefill batch sizes and this fork now skips it. To decide which experts to upload, the scheduler read the ids back from the device that had just produced them, forcing a full pipeline flush once per MoE layer per eval. With 2048 tokens x 10 experts over 256 experts every expert is used anyway. Skipping it is exact - mul_mat_id only reads the rows the ids point at. Auto-on above 4 * n_expert ids; decode keeps the old path. pp2048 +4.7% @ 32k depth, +0.9% @ 16k, tg +3%.
• Where the time actually goes (GGML_VK_PERF_LOGGER=1, depth 0, 22.65 s per 2048-token eval, 17.18 s of it GPU-busy so ~24% is H2D stall): expert MUL_MAT_ID 52%, attention projections 24%, FLASH_ATTN_EXT 16%, everything else 8%. The expert matmuls run at 1686-2139 GFLOP/s while dense MUL_MAT q5_K/q6_K in the same graph reaches 3096-3691 - the single largest remaining opportunity on this hardware is closing that gap, not the transfer path.
• Interleaved SWA keeps its own small KV cache, so a sliding-window layer costs the same at any depth (n_kv pinned at n_swa * n_seq_max + n_ubatch). On this model 36 of 48 layers are O(1) in depth and the entire high-context slowdown comes from the 12 full-attention layers.
• --parallel 1 is worth setting for a solo large model: the server otherwise auto-selects 4 slots, and the SWA cache is sized n_swa * n_seq_max + n_ubatch, so 4 slots cost 4096 cells instead of 2560. Measured 223 MiB of VRAM freed at 64k context.

Dead ends measured on this hardware, recorded so they are not retried:

Change	Result
flash-attn shmem_staging enabled for GCN	-6.7% @ 16k, -7.4% @ 32k
-b 4096 -ub 4096 (to amortize the fixed per-eval expert upload)	flat (-1%)
--n-cpu-moe 44 instead of 48	+1.5%, but does not fit at 64k ctx
mask_opt gate relaxed below head_dim 256	-18.5% @ 16k

shmem_staging looks like a certain win (without it each rowgroup re-reads the whole K/V block through a 16 KiB L1) but the kvsh stride of D/4+1 dwords is 4 mod 32, which costs an 8-way LDS bank conflict on wave64 - that +1 padding is tuned for warp32. -ub 4096 fails because halving the number of expert uploads is exactly cancelled by intra-ubatch attention growing quadratically.

Serving note that outweighs all of the above. With a model this large, anything that restarts the process is far more expensive than any kernel win: the server's prompt cache is RAM-only with no disk backing, so a restart forces a full re-prefill of the conversation. If a model swapper can evict this model to run a small helper model (chat-title generation and the like), fix that first - keeping the process alive across a swap took a repeat turn from a 21,960 ms prefill down to 225 ms.

Quick start

A few options to get llama.cpp installed on your machine:

• Visit https://llama.app and follow the instructions
• Run with Docker - see our Docker documentation
• Download pre-built binaries from the releases page
• Build from source by cloning this repository - check out our build guide

Once installed:

# Download and run a model directly from Hugging Face
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF

# Launch OpenAI-compatible API server
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

VLM session with llama cli

Built-in web UI against llama serve

Description

The main goal of llama.cpp is to enable LLM (and VLM) inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud.

• Plain C/C++ implementation without any dependencies
• Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks
• AVX, AVX2, AVX512 and AMX support for x86 architectures
• RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE support for RISC-V architectures
• 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use
• Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA)
• Vulkan and SYCL backend support
• CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity

The llama.cpp project is build on top of the ggml library.

Supported backends

Backend	Target devices
BLAS	All
BLIS	All
CANN	Ascend NPU
CUDA	Nvidia GPU
HIP	AMD GPU
Hexagon	Snapdragon
IBM zDNN	IBM Z & LinuxONE
MUSA	Moore Threads GPU
Metal	Apple Silicon
OpenCL	Adreno GPU
OpenVINO [In Progress]	Intel CPUs, GPUs, and NPUs
RPC	All
SYCL	Intel GPU
VirtGPU	VirtGPU APIR
Vulkan	GPU
WebGPU	All
ZenDNN	AMD CPU

Documentation

Tools

• cli
• completion
• server
• GBNF grammars

Development

• How to build
• Running on Docker
• Build on Android
• Multi-GPU usage
• Performance troubleshooting
• GGML tips & tricks
• XCFramework
• Completions
• Models
• Release process

Contributing

• Contributors can open PRs
• Collaborators will be invited based on contributions
• Maintainers can push to branches in the llama.cpp repo and merge PRs into the master branch
• Any help with managing issues, PRs and projects is very appreciated!
• Read the CONTRIBUTING.md for more information

Acknowledgements

• yhirose/cpp-httplib - Single-header HTTP server, used by llama-server - MIT license
• nothings/stb - Single-header image format decoder, used by multimodal subsystem - Public domain
• nlohmann/json - Single-header JSON library, used by various tools/examples - MIT License
• mackron/miniaudio - Single-header audio format decoder, used by multimodal subsystem - Public domain
• sheredom/subprocess.h - Single-header process launching solution for C and C++ - Public domain