The README fork section still said "the two code changes below are auto-on" (the fork has three prefill changes plus the VRAM sharing and server fixes) and recommended --n-cpu-moe 28 on the 35B, which FORK.md section 4 records as worse at depth and unable to hold a long context. Replace it with a short accurate summary that points to FORK.md, the runbook and the scripts. The Laguna time profile, the iSWA KV note and Laguna's --parallel 1 saving were only in the README and move to FORK.md sections 4 and 5. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
130 lines
7.8 KiB
Markdown
130 lines
7.8 KiB
Markdown
# llama.cpp
|
|
|
|

|
|
|
|
<div align="center">
|
|
|
|
<b>LLM inference in C/C++</b>
|
|
|
|
[](https://opensource.org/licenses/MIT)
|
|
[](https://github.com/ggml-org/llama.cpp/releases?q=tag:v1)
|
|
[](https://github.com/ggml-org/llama.cpp/releases?q=b)
|
|
[](https://github.com/ggml-org/llama.cpp/actions/workflows/server.yml)
|
|
[](https://github.com/ggml-org/llama.cpp/actions/workflows/docker.yml)
|
|
[](https://github.com/ggml-org/llama.cpp/actions/workflows/winget.yml)
|
|
|
|
[v1.0.0](https://huggingface.co/blog/ggml-org/llama-cpp-v1) / [ggml](https://github.com/ggml-org/ggml) / [ops](https://github.com/ggml-org/llama.cpp/blob/master/docs/ops.md) / [maintainer PRs](https://github.com/ggml-org/llama.cpp/pulls?q=is%3Apr%20is%3Aopen%20draft%3AFalse%20(author%3Asanmai%20OR%20author%3Argerganov%20OR%20author%3AKitaitiMakoto%20OR%20author%3Adanbev%20OR%20author%3Aaldehir%20OR%20author%3Amax-krasnyansky%20OR%20author%3ACISC%20OR%20author%3Aggerganov%20OR%20author%3Aam17an%20OR%20author%3Ajhen0409%20OR%20author%3Abartowski1182%20OR%20author%3Anikwen%20OR%20author%3Ahipudding%20OR%20author%3Aravi9%20OR%20author%3AServeurpersoCom%20OR%20author%3Athelittlefireman%20OR%20author%3Apwilkin%20OR%20author%3Areeselevine%20OR%20author%3Angxson%20OR%20author%3Ajeffbolznv%20OR%20author%3Amarty1885%20OR%20author%3A0cc4m%20OR%20author%3ATitaniumtown%20OR%20author%3Aangt%20OR%20author%3AIMbackK%20OR%20author%3Aarthw%20OR%20author%3AJohannesGaessler%20OR%20author%3AORippler%20OR%20author%3Aruixiang63%20OR%20author%3Axctan%20OR%20author%3AmasterFoad%20OR%20author%3Aallozaur%20OR%20author%3AAndreas-Krebbel%20OR%20author%3Ayomaytk%20OR%20author%3Aaendk%20OR%20author%3Awine99%20OR%20author%3Agaugarg-nv%20OR%20author%3Ataronaeo%20OR%20author%3Aforforever73%20OR%20author%3Achaxu01%20OR%20author%3Alhez%20OR%20author%3AAlekseiNikiforovIBM%20OR%20author%3Anetrunnereve%20OR%20author%3Afairydreaming%20OR%20author%3Aynankani)%20sort%3Aupdated-desc) / [dev stats](https://github.com/ggml-org/llama.cpp-dev) / [lib llama API](https://github.com/ggml-org/llama.cpp/issues/9289) / [llama-server REST API](https://github.com/ggml-org/llama.cpp/issues/9291)
|
|
|
|
</div>
|
|
|
|
## Description
|
|
|
|
The main goal of `llama.cpp` is to enable LLM (and MLLM) inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud.
|
|
|
|
The `llama.cpp` project is built on top of the [ggml](https://github.com/ggml-org/ggml) library.
|
|
|
|
## This fork - RX 580 (Polaris / GCN) serving of large MoE models
|
|
|
|
A private fork maintained for one machine: an AMD RX 580 8 GB (RADV Vulkan) serving Qwen3.6-35B-A3B,
|
|
gemma-4-26B-A4B and smaller models under llama-swap, with the MoE experts in host RAM
|
|
(`--n-cpu-moe`). **[FORK.md](FORK.md) is the source of truth**: every fork change with its
|
|
invariants, rebase hazards and validation, the measured dead ends, the production configuration and
|
|
the measurement traps. The rebase and deploy procedure is
|
|
[docs/fork-rebase-runbook.md](docs/fork-rebase-runbook.md); `scripts/fork-validate.sh` validates a
|
|
build against a reference; `scripts/rx580-bench/` is the benchmark harness.
|
|
|
|
In short, the fork adds:
|
|
|
|
- three Polaris prefill wins, on without a flag and token-identical to mainline: the MoE routing-ids
|
|
readback skip in `sched_copy_experts`, flash-attention `mask_opt` for GCN at head size >= 256, and
|
|
a tiled-transpose fast path for the delta-net conv-state concat;
|
|
- opt-in VRAM time-sharing between several llama-server processes (`LLAMA_SLEEP_VRAM_ONLY`,
|
|
`LLAMA_SLEEP_EVICT_KV`, `LLAMA_MMPROJ_ONDEMAND`, `LLAMA_VRAM_ARENA`): a model releases everything it
|
|
holds in VRAM and gets it back without a restart, so its RAM prompt cache survives;
|
|
- fixes to the server's sleep, wake and shutdown paths and to cold teardown that the sharing exposed;
|
|
- validation-only env knobs (`GGML_VK_CONCAT_FASTPATH`, `LLAMA_MOE_IDS_SKIP`), unset in production.
|
|
|
|
Production settings for the 35B are in FORK.md section 5 (`-lm none -t 6 -ngl 99 --n-cpu-moe 40
|
|
-b 2048 -ub 2048 -fa 1 --parallel 1`, ctx 131072). Keep all experts on the host (`--n-cpu-moe 40`)
|
|
for long contexts: lower values are faster only at short context and run out of VRAM as the KV grows
|
|
(FORK.md section 4).
|
|
|
|
## Quick start
|
|
|
|
A few options to get `llama.cpp` installed on your machine:
|
|
|
|
```bash
|
|
# curl
|
|
curl -LsSf https://llama.app/install.sh | sh
|
|
|
|
# powershell
|
|
irm https://llama.app/install.ps1 | iex
|
|
```
|
|
|
|
- Visit https://llama.app and follow the instructions
|
|
- Run with Docker - see our [Docker documentation](docs/docker.md)
|
|
- Download pre-built binaries from the [releases page](https://github.com/ggml-org/llama.cpp/releases)
|
|
- Build from source by cloning this repository - check out [our build guide](docs/build.md)
|
|
|
|
Once installed:
|
|
|
|
```sh
|
|
# Download and run a model directly from Hugging Face
|
|
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF
|
|
|
|
# Launch OpenAI-compatible API server
|
|
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF
|
|
```
|
|
|
|
<table align="center">
|
|
<tr>
|
|
<td align="center" width=50%>
|
|
<img width="1310" height="888" alt="VLM session with `llama cli`" src="https://github.com/user-attachments/assets/88726b48-1713-48aa-a525-95a02e78afc4" />
|
|
<i>VLM session with <b>llama cli</b></i>
|
|
</td>
|
|
<td align="center">
|
|
<img width="1392" height="958" alt="Llama Chat, the built-in chat app, against `llama serve` running Qwen 3.6" src="https://github.com/user-attachments/assets/b402f972-2e32-4def-8771-8d849f08cf2e" />
|
|
<i>Llama Chat, the built-in chat app, against <b>llama serve</b></i>
|
|
</td>
|
|
</tr>
|
|
<table>
|
|
|
|
## Documentation
|
|
|
|
#### Tools
|
|
|
|
- [cli](tools/cli/README.md)
|
|
- [completion](tools/completion/README.md)
|
|
- [server](tools/server/README.md)
|
|
- [Text-to-speech (TTS) models](docs/multimodal/tts.md)
|
|
- [GBNF grammars](grammars/README.md)
|
|
|
|
#### Development
|
|
|
|
- [How to build](docs/build.md)
|
|
- [Running on Docker](docs/docker.md)
|
|
- [Build on Android](docs/android.md)
|
|
- [Multi-GPU usage](docs/multi-gpu.md)
|
|
- [Performance troubleshooting](docs/development/token_generation_performance_tips.md)
|
|
- [GGML tips & tricks](https://github.com/ggml-org/llama.cpp/wiki/GGML-Tips-&-Tricks)
|
|
- [XCFramework](docs/xcframework.md)
|
|
- [Completions](docs/completions.md)
|
|
- [Models](docs/models.md)
|
|
- [Release process](docs/release.md)
|
|
|
|
## Contributing
|
|
|
|
- Contributors can open PRs
|
|
- Collaborators will be invited based on contributions
|
|
- Maintainers can push to branches in the `llama.cpp` repo and merge PRs into the `master` branch
|
|
- Any help with managing issues, PRs and projects is very appreciated!
|
|
- Read the [CONTRIBUTING.md](CONTRIBUTING.md) for more information
|
|
|
|
## Acknowledgements
|
|
|
|
- [yhirose/cpp-httplib](https://github.com/yhirose/cpp-httplib) - Single-header HTTP server, used by `llama-server` - MIT license
|
|
- [nothings/stb](https://github.com/nothings/stb) - Single-header image format decoder, used by multimodal subsystem - Public domain
|
|
- [nlohmann/json](https://github.com/nlohmann/json) - Single-header JSON library, used by various tools/examples - MIT License
|
|
- [mackron/miniaudio](https://github.com/mackron/miniaudio) - Single-header audio format decoder, used by multimodal subsystem - Public domain
|
|
- [sheredom/subprocess.h](https://github.com/sheredom/subprocess.h) - Single-header process launching solution for C and C++ - Public domain
|