203 lines
12 KiB
Markdown
203 lines
12 KiB
Markdown
# llama.cpp
|
|
|
|
|
|
|
|
<div align="center">
|
|
|
|
<b>LLM inference in C/C++</b>
|
|
|
|
[](https://opensource.org/licenses/MIT)
|
|
[](https://github.com/ggml-org/llama.cpp/releases?q=tag:v0)
|
|
[](https://github.com/ggml-org/llama.cpp/releases?q=b)
|
|
[](https://github.com/ggml-org/llama.cpp/actions/workflows/server.yml)
|
|
[](https://github.com/ggml-org/llama.cpp/actions/workflows/docker.yml)
|
|
[](https://github.com/ggml-org/llama.cpp/actions/workflows/winget.yml)
|
|
|
|
[ggml](https://github.com/ggml-org/ggml) / [ops](https://github.com/ggml-org/llama.cpp/blob/master/docs/ops.md) / [maintainer PRs](https://github.com/ggml-org/llama.cpp/issues?q=is%3Apr%20is%3Aopen%20draft%3AFalse%20(author%3Argerganov%20OR%20author%3AKitaitiMakoto%20OR%20author%3Adanbev%20OR%20author%3Aaldehir%20OR%20author%3Amax-krasnyansky%20OR%20author%3ACISC%20OR%20author%3Aggerganov%20OR%20author%3Aam17an%20OR%20author%3Ajhen0409%20OR%20author%3Abartowski1182%20OR%20author%3Anikwen%20OR%20author%3Ahipudding%20OR%20author%3Aravi9%20OR%20author%3AServeurpersoCom%20OR%20author%3Apwilkin%20OR%20author%3Areeselevine%20OR%20author%3Angxson%20OR%20author%3Ajeffbolznv%20OR%20author%3Amarty1885%20OR%20author%3A0cc4m%20OR%20author%3ATitaniumtown%20OR%20author%3Aangt%20OR%20author%3AIMbackK%20OR%20author%3Aarthw%20OR%20author%3AJohannesGaessler%20OR%20author%3AORippler%20OR%20author%3Aruixiang63%20OR%20author%3Axctan%20OR%20author%3Aallozaur%20OR%20author%3Ayomaytk%20OR%20author%3Aaendk%20OR%20author%3Awine99%20OR%20author%3Agaugarg-nv%20OR%20author%3Ataronaeo%20OR%20author%3Aforforever73%20OR%20author%3Alhez%20OR%20author%3Anetrunnereve%20OR%20author%3Afairydreaming)%20sort%3Aupdated-desc) / [dev stats](https://github.com/ggml-org/llama.cpp-dev) / [lib llama API](https://github.com/ggml-org/llama.cpp/issues/9289) / [llama-server REST API](https://github.com/ggml-org/llama.cpp/issues/9291)
|
|
|
|
</div>
|
|
|
|
## This fork - Polaris / GCN tuning for large MoE models
|
|
|
|
Changes and measurements for running large MoE models with their experts offloaded to system RAM
|
|
(`--n-cpu-moe`) on an old GCN card. The two code changes below are auto-on, need no flag, and are
|
|
**token-identical** to mainline. Everything else here is tuning guidance.
|
|
|
|
Findings on an **RX 580 8GB** (Polaris / GCN, PCIe 3.0 x16, no fp16, no matrix cores) with
|
|
**Qwen3.5-35B-A3B Q4_K_M**, `-b 2048 -ub 2048`:
|
|
|
|
- **Flash-attention `mask_opt` is enabled for GCN large head sizes (this fork's own change).**
|
|
Upstream disables it on GCN; it is a **lossless** win in high-context prefill - it skips
|
|
fully-masked causal blocks and the per-block mask add on fully-visible ones, which is real work on
|
|
a card whose attention is compute-bound (no matrix cores). Auto-on, no flag. On Qwen3.5-35B
|
|
(head_dim 256): pp2048 **+8% @ 16k, +12% @ 32k**, growing with depth; perplexity bit-identical.
|
|
- **`-b 2048 -ub 2048` is the biggest prefill lever** (the default `-ub 512` roughly halves pp).
|
|
- **Tune `--n-cpu-moe` to context length.** Keep some expert layers resident in spare VRAM for short
|
|
prompts (e.g. `ncmoe 28` on the 35B, ~+5% over all-host); at long context the KV cache needs that
|
|
VRAM, so raise it (`ncmoe 40`, all experts on host). Keep flash attention on (`-fa 1`).
|
|
|
|
**Recommended RX 580 / Polaris serving command** (per model):
|
|
|
|
```bash
|
|
llama-server -hf <repo>:<quant> -ngl 99 --n-cpu-moe 40 -b 2048 -ub 2048 -fa 1
|
|
```
|
|
|
|
Lower `--n-cpu-moe` (e.g. 28) if the model plus your context budget leave spare VRAM; keep it high
|
|
for long-context / agentic use. At long context the bottleneck is attention compute (GPU-bound), so
|
|
`mask_opt` (above) is where the remaining prefill wins come from, not the MoE-transfer path.
|
|
|
|
### Very large MoE (experts bigger than the GTT limit)
|
|
|
|
Measured on the same RX 580 with **Laguna-S-2.1 118B IQ2_M** (48 layers, 256 experts, 10 used,
|
|
experts 30.7 GiB of a 34.7 GiB file, all host-resident at `--n-cpu-moe 48`). The advice above changes
|
|
in this regime:
|
|
|
|
- **`--no-mmap` stops being an option.** Its pinned host buffer is charged against the amdgpu GTT
|
|
limit (~31.4 GiB here, about half of system RAM). At 30.7 GiB of experts the model no longer
|
|
loads, and the allocation spike can OOM the box. Use mmap and accept the staging copy.
|
|
- **The routing-ids readback is pure overhead at prefill batch sizes** and this fork now skips it.
|
|
To decide which experts to upload, the scheduler read the ids back from the device that had just
|
|
produced them, forcing a full pipeline flush once per MoE layer per eval. With 2048 tokens x 10
|
|
experts over 256 experts every expert is used anyway. Skipping it is exact - `mul_mat_id` only
|
|
reads the rows the ids point at. Auto-on above `4 * n_expert` ids; decode keeps the old path.
|
|
**pp2048 +4.7% @ 32k depth, +0.9% @ 16k, tg +3%.**
|
|
- **Where the time actually goes** (`GGML_VK_PERF_LOGGER=1`, depth 0, 22.65 s per 2048-token eval,
|
|
17.18 s of it GPU-busy so ~24% is H2D stall): expert `MUL_MAT_ID` **52%**, attention projections
|
|
**24%**, `FLASH_ATTN_EXT` **16%**, everything else 8%. The expert matmuls run at 1686-2139 GFLOP/s
|
|
while dense `MUL_MAT` q5_K/q6_K in the same graph reaches 3096-3691 - **the single largest
|
|
remaining opportunity on this hardware is closing that gap**, not the transfer path.
|
|
- **Interleaved SWA keeps its own small KV cache**, so a sliding-window layer costs the same at any
|
|
depth (`n_kv` pinned at `n_swa * n_seq_max + n_ubatch`). On this model 36 of 48 layers are O(1) in
|
|
depth and the entire high-context slowdown comes from the 12 full-attention layers.
|
|
- **`--parallel 1`** is worth setting for a solo large model: the server otherwise auto-selects 4
|
|
slots, and the SWA cache is sized `n_swa * n_seq_max + n_ubatch`, so 4 slots cost 4096 cells
|
|
instead of 2560. Measured **223 MiB of VRAM freed** at 64k context.
|
|
|
|
Dead ends measured on this hardware, recorded so they are not retried:
|
|
|
|
| Change | Result |
|
|
| --- | ---: |
|
|
| flash-attn `shmem_staging` enabled for GCN | **-6.7% @ 16k, -7.4% @ 32k** |
|
|
| `-b 4096 -ub 4096` (to amortize the fixed per-eval expert upload) | flat (-1%) |
|
|
| `--n-cpu-moe` 44 instead of 48 | +1.5%, but does not fit at 64k ctx |
|
|
| `mask_opt` gate relaxed below head_dim 256 | -18.5% @ 16k |
|
|
|
|
`shmem_staging` looks like a certain win (without it each rowgroup re-reads the whole K/V block
|
|
through a 16 KiB L1) but the `kvsh` stride of `D/4+1` dwords is 4 mod 32, which costs an 8-way LDS
|
|
bank conflict on wave64 - that `+1` padding is tuned for warp32. `-ub 4096` fails because halving the
|
|
number of expert uploads is exactly cancelled by intra-ubatch attention growing quadratically.
|
|
|
|
**Serving note that outweighs all of the above.** With a model this large, anything that restarts the
|
|
process is far more expensive than any kernel win: the server's prompt cache is RAM-only with no disk
|
|
backing, so a restart forces a full re-prefill of the conversation. If a model swapper can evict this
|
|
model to run a small helper model (chat-title generation and the like), fix that first - keeping the
|
|
process alive across a swap took a repeat turn from a 21,960 ms prefill down to 225 ms.
|
|
|
|
## Quick start
|
|
|
|
A few options to get `llama.cpp` installed on your machine:
|
|
|
|
- Visit https://llama.app and follow the instructions
|
|
- Run with Docker - see our [Docker documentation](docs/docker.md)
|
|
- Download pre-built binaries from the [releases page](https://github.com/ggml-org/llama.cpp/releases)
|
|
- Build from source by cloning this repository - check out [our build guide](docs/build.md)
|
|
|
|
Once installed:
|
|
|
|
```sh
|
|
# Download and run a model directly from Hugging Face
|
|
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF
|
|
|
|
# Launch OpenAI-compatible API server
|
|
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF
|
|
```
|
|
|
|
<table align="center">
|
|
<tr>
|
|
<td align="center" width=50%>
|
|
<img width="1310" height="888" alt="VLM session with `llama cli`" src="https://github.com/user-attachments/assets/88726b48-1713-48aa-a525-95a02e78afc4" />
|
|
<i>VLM session with <b>llama cli</b></i>
|
|
</td>
|
|
<td align="center">
|
|
<img width="1392" height="958" alt="Built-in web UI against `llama serve` running Qwen 3.6" src="https://github.com/user-attachments/assets/b402f972-2e32-4def-8771-8d849f08cf2e" />
|
|
<i>Built-in web UI against <b>llama serve</b></i>
|
|
</td>
|
|
</tr>
|
|
<table>
|
|
|
|
## Description
|
|
|
|
The main goal of `llama.cpp` is to enable LLM (and VLM) inference with minimal setup and state-of-the-art performance on
|
|
a wide range of hardware - locally and in the cloud.
|
|
|
|
- Plain C/C++ implementation without any dependencies
|
|
- Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks
|
|
- AVX, AVX2, AVX512 and AMX support for x86 architectures
|
|
- RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE support for RISC-V architectures
|
|
- 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use
|
|
- Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA)
|
|
- Vulkan and SYCL backend support
|
|
- CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity
|
|
|
|
The `llama.cpp` project is build on top of the [ggml](https://github.com/ggml-org/ggml) library.
|
|
|
|
## Supported backends
|
|
|
|
| Backend | Target devices |
|
|
| --- | --- |
|
|
| [BLAS](docs/build.md#blas-build) | All |
|
|
| [BLIS](docs/backend/BLIS.md) | All |
|
|
| [CANN](docs/build.md#cann) | Ascend NPU |
|
|
| [CUDA](docs/build.md#cuda) | Nvidia GPU |
|
|
| [HIP](docs/build.md#hip) | AMD GPU |
|
|
| [Hexagon](docs/backend/snapdragon/README.md) | Snapdragon |
|
|
| [IBM zDNN](docs/backend/zDNN.md) | IBM Z & LinuxONE |
|
|
| [MUSA](docs/build.md#musa) | Moore Threads GPU |
|
|
| [Metal](docs/build.md#metal-build) | Apple Silicon |
|
|
| [OpenCL](docs/backend/OPENCL.md) | Adreno GPU |
|
|
| [OpenVINO [In Progress]](docs/backend/OPENVINO.md) | Intel CPUs, GPUs, and NPUs |
|
|
| [RPC](https://github.com/ggml-org/llama.cpp/tree/master/tools/rpc) | All |
|
|
| [SYCL](docs/backend/SYCL.md) | Intel GPU |
|
|
| [VirtGPU](docs/backend/VirtGPU.md) | VirtGPU APIR |
|
|
| [Vulkan](docs/build.md#vulkan) | GPU |
|
|
| [WebGPU](docs/build.md#webgpu) | All |
|
|
| [ZenDNN](docs/build.md#zendnn) | AMD CPU |
|
|
|
|
## Documentation
|
|
|
|
#### Tools
|
|
|
|
- [cli](tools/cli/README.md)
|
|
- [completion](tools/completion/README.md)
|
|
- [server](tools/server/README.md)
|
|
- [GBNF grammars](grammars/README.md)
|
|
|
|
#### Development
|
|
|
|
- [How to build](docs/build.md)
|
|
- [Running on Docker](docs/docker.md)
|
|
- [Build on Android](docs/android.md)
|
|
- [Multi-GPU usage](docs/multi-gpu.md)
|
|
- [Performance troubleshooting](docs/development/token_generation_performance_tips.md)
|
|
- [GGML tips & tricks](https://github.com/ggml-org/llama.cpp/wiki/GGML-Tips-&-Tricks)
|
|
- [XCFramework](docs/xcframework.md)
|
|
- [Completions](docs/completions.md)
|
|
- [Models](docs/models.md)
|
|
- [Release process](docs/release.md)
|
|
|
|
## Contributing
|
|
|
|
- Contributors can open PRs
|
|
- Collaborators will be invited based on contributions
|
|
- Maintainers can push to branches in the `llama.cpp` repo and merge PRs into the `master` branch
|
|
- Any help with managing issues, PRs and projects is very appreciated!
|
|
- Read the [CONTRIBUTING.md](CONTRIBUTING.md) for more information
|
|
|
|
## Acknowledgements
|
|
|
|
- [yhirose/cpp-httplib](https://github.com/yhirose/cpp-httplib) - Single-header HTTP server, used by `llama-server` - MIT license
|
|
- [nothings/stb](https://github.com/nothings/stb) - Single-header image format decoder, used by multimodal subsystem - Public domain
|
|
- [nlohmann/json](https://github.com/nlohmann/json) - Single-header JSON library, used by various tools/examples - MIT License
|
|
- [mackron/miniaudio](https://github.com/mackron/miniaudio) - Single-header audio format decoder, used by multimodal subsystem - Public domain
|
|
- [sheredom/subprocess.h](https://github.com/sheredom/subprocess.h) - Single-header process launching solution for C and C++ - Public domain
|