Runs beside production. The model is the small hybrid (Qwen3.5-0.8B), CPU only (-dev none -ngl 0, no VRAM), with chat requests and greedy decoding. A reference server that never restarts runs turn 1 and turn 2. The candidate saves turn 1 on SIGTERM and restarts. Turn 2 must then be restored from disk with the same prompt_n as the reference (only the suffix processed) and the same answer. Cases: - restore plain, with the arbiter env warm, and forced cold by a fake waiter on want.lock (the save from a released state). Each case records the exit time and SIGTERM-to-token-free, also once without the cache; - no restore for another conversation, or with another n_ctx, KV type or model path (other key); - a truncated file is removed at startup and a byte-flipped one at load, never used; - the size limit evicts the oldest file; - a write error (RLIMIT_FSIZE 1 MiB, SIGXFSZ ignored: EFBIG) is logged, the exit stays 0 with no file left, and the next start serves; - the same restore on the GPU (-ngl 99, arbiter env, warm and cold) when >= 1.5 GiB of VRAM is free, else SKIP. start_server gains SRV_WRAP, a command prefix that execs the server. First run (2026-10-25, beside production): 11 PASS, 0 FAIL; the GPU leg SKIPped (1219 MiB free). The first run found the CPU_REPACK release crash (exit 139 in the cold case). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
llama.cpp
LLM inference in C/C++
ggml / ops / maintainer PRs / dev stats / lib llama API / llama-server REST API
This fork - RX 580 (Polaris / GCN) serving of large MoE models
A private fork maintained for one machine: an AMD RX 580 8 GB (RADV Vulkan) serving Qwen3.6-35B-A3B,
gemma-4-26B-A4B and smaller models under llama-swap, with the MoE experts in host RAM
(--n-cpu-moe). FORK.md is the source of truth: every fork change with its
invariants, rebase hazards and validation, the measured dead ends, the production configuration and
the measurement traps. The rebase and deploy procedure is
docs/fork-rebase-runbook.md; scripts/fork-validate.sh validates a
build against a reference; scripts/rx580-bench/ is the benchmark harness.
In short, the fork adds:
- three Polaris prefill wins, on without a flag and token-identical to mainline: the MoE routing-ids
readback skip in
sched_copy_experts, flash-attentionmask_optfor GCN at head size >= 256, and a tiled-transpose fast path for the delta-net conv-state concat; - opt-in VRAM time-sharing between several llama-server processes (
LLAMA_SLEEP_VRAM_ONLY,LLAMA_SLEEP_EVICT_KV,LLAMA_MMPROJ_ONDEMAND,LLAMA_VRAM_ARENA): a model releases everything it holds in VRAM and gets it back without a restart, so its RAM prompt cache survives; - fixes to the server's sleep, wake and shutdown paths and to cold teardown that the sharing exposed;
- validation-only env knobs (
GGML_VK_CONCAT_FASTPATH,LLAMA_MOE_IDS_SKIP), unset in production.
Production settings for the 35B are in FORK.md section 5 (-lm none -t 6 -ngl 99 --n-cpu-moe 40 -b 2048 -ub 2048 -fa 1 --parallel 1, ctx 131072). Keep all experts on the host (--n-cpu-moe 40)
for long contexts: lower values are faster only at short context and run out of VRAM as the KV grows
(FORK.md section 4).
Quick start
A few options to get llama.cpp installed on your machine:
# curl
curl -LsSf https://llama.app/install.sh | sh
# powershell
irm https://llama.app/install.ps1 | iex
- Visit https://llama.app and follow the instructions
- Run with Docker - see our Docker documentation
- Download pre-built binaries from the releases page
- Build from source by cloning this repository - check out our build guide
Once installed:
# Download and run a model directly from Hugging Face
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF
# Launch OpenAI-compatible API server
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF
|
|
|
Description
The main goal of llama.cpp is to enable LLM (and VLM) inference with minimal setup and state-of-the-art performance on
a wide range of hardware - locally and in the cloud.
- Plain C/C++ implementation without any dependencies
- Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks
- AVX, AVX2, AVX512 and AMX support for x86 architectures
- RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE support for RISC-V architectures
- 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use
- Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA)
- Vulkan and SYCL backend support
- CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity
The llama.cpp project is build on top of the ggml library.
Supported backends
| Backend | Target devices |
|---|---|
| BLAS | All |
| BLIS | All |
| CANN | Ascend NPU |
| CUDA | Nvidia GPU |
| HIP | AMD GPU |
| Hexagon | Snapdragon |
| IBM zDNN | IBM Z & LinuxONE |
| MUSA | Moore Threads GPU |
| Metal | Apple Silicon |
| OpenCL | Adreno GPU |
| OpenVINO [In Progress] | Intel CPUs, GPUs, and NPUs |
| RPC | All |
| SYCL | Intel GPU |
| VirtGPU | VirtGPU APIR |
| Vulkan | GPU |
| WebGPU | All |
| ZenDNN | AMD CPU |
Documentation
Tools
Development
- How to build
- Running on Docker
- Build on Android
- Multi-GPU usage
- Performance troubleshooting
- GGML tips & tricks
- XCFramework
- Completions
- Models
- Release process
Contributing
- Contributors can open PRs
- Collaborators will be invited based on contributions
- Maintainers can push to branches in the
llama.cpprepo and merge PRs into themasterbranch - Any help with managing issues, PRs and projects is very appreciated!
- Read the CONTRIBUTING.md for more information
Acknowledgements
- yhirose/cpp-httplib - Single-header HTTP server, used by
llama-server- MIT license - nothings/stb - Single-header image format decoder, used by multimodal subsystem - Public domain
- nlohmann/json - Single-header JSON library, used by various tools/examples - MIT License
- mackron/miniaudio - Single-header audio format decoder, used by multimodal subsystem - Public domain
- sheredom/subprocess.h - Single-header process launching solution for C and C++ - Public domain