# llama.cpp
LLM inference in C/C++
[](https://opensource.org/licenses/MIT)
[](https://github.com/ggml-org/llama.cpp/releases?q=tag:v0)
[](https://github.com/ggml-org/llama.cpp/releases?q=b)
[](https://github.com/ggml-org/llama.cpp/actions/workflows/server.yml)
[](https://github.com/ggml-org/llama.cpp/actions/workflows/docker.yml)
[](https://github.com/ggml-org/llama.cpp/actions/workflows/winget.yml)
[ggml](https://github.com/ggml-org/ggml) / [ops](https://github.com/ggml-org/llama.cpp/blob/master/docs/ops.md) / [maintainer PRs](https://github.com/ggml-org/llama.cpp/issues?q=is%3Apr%20is%3Aopen%20draft%3AFalse%20(author%3Argerganov%20OR%20author%3AKitaitiMakoto%20OR%20author%3Adanbev%20OR%20author%3Aaldehir%20OR%20author%3Amax-krasnyansky%20OR%20author%3ACISC%20OR%20author%3Aggerganov%20OR%20author%3Aam17an%20OR%20author%3Ajhen0409%20OR%20author%3Abartowski1182%20OR%20author%3Anikwen%20OR%20author%3Ahipudding%20OR%20author%3Aravi9%20OR%20author%3AServeurpersoCom%20OR%20author%3Apwilkin%20OR%20author%3Areeselevine%20OR%20author%3Angxson%20OR%20author%3Ajeffbolznv%20OR%20author%3Amarty1885%20OR%20author%3A0cc4m%20OR%20author%3ATitaniumtown%20OR%20author%3Aangt%20OR%20author%3AIMbackK%20OR%20author%3Aarthw%20OR%20author%3AJohannesGaessler%20OR%20author%3AORippler%20OR%20author%3Aruixiang63%20OR%20author%3Axctan%20OR%20author%3Aallozaur%20OR%20author%3Ayomaytk%20OR%20author%3Aaendk%20OR%20author%3Awine99%20OR%20author%3Agaugarg-nv%20OR%20author%3Ataronaeo%20OR%20author%3Aforforever73%20OR%20author%3Alhez%20OR%20author%3Anetrunnereve%20OR%20author%3Afairydreaming)%20sort%3Aupdated-desc) / [dev stats](https://github.com/ggml-org/llama.cpp-dev) / [lib llama API](https://github.com/ggml-org/llama.cpp/issues/9289) / [llama-server REST API](https://github.com/ggml-org/llama.cpp/issues/9291)
## This fork - Polaris / GCN tuning for large MoE models
Changes and measurements for running large MoE models with their experts offloaded to system RAM
(`--n-cpu-moe`) on an old GCN card. The two code changes below are auto-on, need no flag, and are
**token-identical** to mainline. Everything else here is tuning guidance.
Findings on an **RX 580 8GB** (Polaris / GCN, PCIe 3.0 x16, no fp16, no matrix cores) with
**Qwen3.5-35B-A3B Q4_K_M**, `-b 2048 -ub 2048`:
- **Flash-attention `mask_opt` is enabled for GCN large head sizes (this fork's own change).**
Upstream disables it on GCN; it is a **lossless** win in high-context prefill - it skips
fully-masked causal blocks and the per-block mask add on fully-visible ones, which is real work on
a card whose attention is compute-bound (no matrix cores). Auto-on, no flag. On Qwen3.5-35B
(head_dim 256): pp2048 **+8% @ 16k, +12% @ 32k**, growing with depth; perplexity bit-identical.
- **`-b 2048 -ub 2048` is the biggest prefill lever** (the default `-ub 512` roughly halves pp).
- **Tune `--n-cpu-moe` to context length.** Keep some expert layers resident in spare VRAM for short
prompts (e.g. `ncmoe 28` on the 35B, ~+5% over all-host); at long context the KV cache needs that
VRAM, so raise it (`ncmoe 40`, all experts on host). Keep flash attention on (`-fa 1`).
**Recommended RX 580 / Polaris serving command** (per model):
```bash
llama-server -hf : -ngl 99 --n-cpu-moe 40 -b 2048 -ub 2048 -fa 1
```
Lower `--n-cpu-moe` (e.g. 28) if the model plus your context budget leave spare VRAM; keep it high
for long-context / agentic use. At long context the bottleneck is attention compute (GPU-bound), so
`mask_opt` (above) is where the remaining prefill wins come from, not the MoE-transfer path.
### Very large MoE (experts bigger than the GTT limit)
Measured on the same RX 580 with **Laguna-S-2.1 118B IQ2_M** (48 layers, 256 experts, 10 used,
experts 30.7 GiB of a 34.7 GiB file, all host-resident at `--n-cpu-moe 48`). The advice above changes
in this regime:
- **`--no-mmap` stops being an option.** Its pinned host buffer is charged against the amdgpu GTT
limit (~31.4 GiB here, about half of system RAM). At 30.7 GiB of experts the model no longer
loads, and the allocation spike can OOM the box. Use mmap and accept the staging copy.
- **The routing-ids readback is pure overhead at prefill batch sizes** and this fork now skips it.
To decide which experts to upload, the scheduler read the ids back from the device that had just
produced them, forcing a full pipeline flush once per MoE layer per eval. With 2048 tokens x 10
experts over 256 experts every expert is used anyway. Skipping it is exact - `mul_mat_id` only
reads the rows the ids point at. Auto-on above `4 * n_expert` ids; decode keeps the old path.
**pp2048 +4.7% @ 32k depth, +0.9% @ 16k, tg +3%.**
- **Where the time actually goes** (`GGML_VK_PERF_LOGGER=1`, depth 0, 22.65 s per 2048-token eval,
17.18 s of it GPU-busy so ~24% is H2D stall): expert `MUL_MAT_ID` **52%**, attention projections
**24%**, `FLASH_ATTN_EXT` **16%**, everything else 8%. The expert matmuls run at 1686-2139 GFLOP/s
while dense `MUL_MAT` q5_K/q6_K in the same graph reaches 3096-3691 - **the single largest
remaining opportunity on this hardware is closing that gap**, not the transfer path.
- **Interleaved SWA keeps its own small KV cache**, so a sliding-window layer costs the same at any
depth (`n_kv` pinned at `n_swa * n_seq_max + n_ubatch`). On this model 36 of 48 layers are O(1) in
depth and the entire high-context slowdown comes from the 12 full-attention layers.
- **`--parallel 1`** is worth setting for a solo large model: the server otherwise auto-selects 4
slots, and the SWA cache is sized `n_swa * n_seq_max + n_ubatch`, so 4 slots cost 4096 cells
instead of 2560. Measured **223 MiB of VRAM freed** at 64k context.
Dead ends measured on this hardware, recorded so they are not retried:
| Change | Result |
| --- | ---: |
| flash-attn `shmem_staging` enabled for GCN | **-6.7% @ 16k, -7.4% @ 32k** |
| `-b 4096 -ub 4096` (to amortize the fixed per-eval expert upload) | flat (-1%) |
| `--n-cpu-moe` 44 instead of 48 | +1.5%, but does not fit at 64k ctx |
| `mask_opt` gate relaxed below head_dim 256 | -18.5% @ 16k |
`shmem_staging` looks like a certain win (without it each rowgroup re-reads the whole K/V block
through a 16 KiB L1) but the `kvsh` stride of `D/4+1` dwords is 4 mod 32, which costs an 8-way LDS
bank conflict on wave64 - that `+1` padding is tuned for warp32. `-ub 4096` fails because halving the
number of expert uploads is exactly cancelled by intra-ubatch attention growing quadratically.
**Serving note that outweighs all of the above.** With a model this large, anything that restarts the
process is far more expensive than any kernel win: the server's prompt cache is RAM-only with no disk
backing, so a restart forces a full re-prefill of the conversation. If a model swapper can evict this
model to run a small helper model (chat-title generation and the like), fix that first - keeping the
process alive across a swap took a repeat turn from a 21,960 ms prefill down to 225 ms.
## Quick start
A few options to get `llama.cpp` installed on your machine:
- Visit https://llama.app and follow the instructions
- Run with Docker - see our [Docker documentation](docs/docker.md)
- Download pre-built binaries from the [releases page](https://github.com/ggml-org/llama.cpp/releases)
- Build from source by cloning this repository - check out [our build guide](docs/build.md)
Once installed:
```sh
# Download and run a model directly from Hugging Face
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF
# Launch OpenAI-compatible API server
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF
```
VLM session with llama cli
|
Built-in web UI against llama serve
|
## Description
The main goal of `llama.cpp` is to enable LLM (and VLM) inference with minimal setup and state-of-the-art performance on
a wide range of hardware - locally and in the cloud.
- Plain C/C++ implementation without any dependencies
- Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks
- AVX, AVX2, AVX512 and AMX support for x86 architectures
- RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE support for RISC-V architectures
- 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use
- Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA)
- Vulkan and SYCL backend support
- CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity
The `llama.cpp` project is build on top of the [ggml](https://github.com/ggml-org/ggml) library.
## Supported backends
| Backend | Target devices |
| --- | --- |
| [BLAS](docs/build.md#blas-build) | All |
| [BLIS](docs/backend/BLIS.md) | All |
| [CANN](docs/build.md#cann) | Ascend NPU |
| [CUDA](docs/build.md#cuda) | Nvidia GPU |
| [HIP](docs/build.md#hip) | AMD GPU |
| [Hexagon](docs/backend/snapdragon/README.md) | Snapdragon |
| [IBM zDNN](docs/backend/zDNN.md) | IBM Z & LinuxONE |
| [MUSA](docs/build.md#musa) | Moore Threads GPU |
| [Metal](docs/build.md#metal-build) | Apple Silicon |
| [OpenCL](docs/backend/OPENCL.md) | Adreno GPU |
| [OpenVINO [In Progress]](docs/backend/OPENVINO.md) | Intel CPUs, GPUs, and NPUs |
| [RPC](https://github.com/ggml-org/llama.cpp/tree/master/tools/rpc) | All |
| [SYCL](docs/backend/SYCL.md) | Intel GPU |
| [VirtGPU](docs/backend/VirtGPU.md) | VirtGPU APIR |
| [Vulkan](docs/build.md#vulkan) | GPU |
| [WebGPU](docs/build.md#webgpu) | All |
| [ZenDNN](docs/build.md#zendnn) | AMD CPU |
## Documentation
#### Tools
- [cli](tools/cli/README.md)
- [completion](tools/completion/README.md)
- [server](tools/server/README.md)
- [GBNF grammars](grammars/README.md)
#### Development
- [How to build](docs/build.md)
- [Running on Docker](docs/docker.md)
- [Build on Android](docs/android.md)
- [Multi-GPU usage](docs/multi-gpu.md)
- [Performance troubleshooting](docs/development/token_generation_performance_tips.md)
- [GGML tips & tricks](https://github.com/ggml-org/llama.cpp/wiki/GGML-Tips-&-Tricks)
- [XCFramework](docs/xcframework.md)
- [Completions](docs/completions.md)
- [Models](docs/models.md)
- [Release process](docs/release.md)
## Contributing
- Contributors can open PRs
- Collaborators will be invited based on contributions
- Maintainers can push to branches in the `llama.cpp` repo and merge PRs into the `master` branch
- Any help with managing issues, PRs and projects is very appreciated!
- Read the [CONTRIBUTING.md](CONTRIBUTING.md) for more information
## Acknowledgements
- [yhirose/cpp-httplib](https://github.com/yhirose/cpp-httplib) - Single-header HTTP server, used by `llama-server` - MIT license
- [nothings/stb](https://github.com/nothings/stb) - Single-header image format decoder, used by multimodal subsystem - Public domain
- [nlohmann/json](https://github.com/nlohmann/json) - Single-header JSON library, used by various tools/examples - MIT License
- [mackron/miniaudio](https://github.com/mackron/miniaudio) - Single-header audio format decoder, used by multimodal subsystem - Public domain
- [sheredom/subprocess.h](https://github.com/sheredom/subprocess.h) - Single-header process launching solution for C and C++ - Public domain