LumpiastyandClaude Opus 5
48f0221c98
The doorbell was edge-triggered: a waiter wrote <arena>/doorbell/<pid> and the holder's warden turned the inotify event into a queue flag. A ring carried no notion of "still wanted", so it was lost whenever the holder was not already warm and listening - during its model load (the watch does not exist yet, and the kernel does not queue events for a watch that is not there) or inside restore_device - and any ring that survived past the point its sender had been satisfied caused a spurious release. Lost rings wedged the waiter permanently, because flock(LOCK_EX) never times out and, with idle-sleep disabled, the holder had no other reason to release. The wait blocks under mutex_tasks, so the whole server stopped answering while /health still returned 200. Express the request as kernel state instead. A waiter holds <arena>/want.lock shared while it waits and drops it once it owns the token; the sleep decision probes that lock non-blocking and releases the GPU while anyone is waiting. The probe needs its own fd - flock treats two open file descriptions of one file independently, so probing on the waiter's fd would convert our own lock rather than conflict with it. Nothing can be missed, nothing goes stale, and a waiter that dies is cleaned up by the kernel. Probe from should_sleep() on the loop thread rather than from a warden thread. Routing it through a flag is what made the first attempts fail: start_loop() holds mutex_tasks from should_sleep() through the callbacks to the wait, so a warden's request_yield() blocks on that mutex and is admitted only after the flag has been consumed, latching a release for the next wake. Reading live state where the decision is made has no edge to latch, and drops the warden's poll latency. Two sleep-path bugs this exposed: wait_until_no_sleep() waited on !sleeping but the loop clears req_stop_sleeping on the way in, so a loop that slept again before the waiter ran stranded it forever - re-ask on every wake. And a task queued after the waiter saw us awake could not wake us by itself, so sleep now also breaks on a non-empty queue. Hold the GPU for 100 ms after a wake: the request that woke us is not queued yet, and yielding at once only sends it round again. Measured on the RX 580 pod, two servers contending, 60 alternating handoffs: 0 stranded, median 0.309 s against the doorbell's 0.314 s. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
llama.cpp
LLM inference in C/C++
ggml / ops / maintainer PRs / dev stats / lib llama API / llama-server REST API
This fork - Polaris / GCN tuning for large MoE models
Changes and measurements for running large MoE models with their experts offloaded to system RAM
(--n-cpu-moe) on an old GCN card. The two code changes below are auto-on, need no flag, and are
token-identical to mainline. Everything else here is tuning guidance.
Findings on an RX 580 8GB (Polaris / GCN, PCIe 3.0 x16, no fp16, no matrix cores) with
Qwen3.5-35B-A3B Q4_K_M, -b 2048 -ub 2048:
- Flash-attention
mask_optis enabled for GCN large head sizes (this fork's own change). Upstream disables it on GCN; it is a lossless win in high-context prefill - it skips fully-masked causal blocks and the per-block mask add on fully-visible ones, which is real work on a card whose attention is compute-bound (no matrix cores). Auto-on, no flag. On Qwen3.5-35B (head_dim 256): pp2048 +8% @ 16k, +12% @ 32k, growing with depth; perplexity bit-identical. -b 2048 -ub 2048is the biggest prefill lever (the default-ub 512roughly halves pp).- Tune
--n-cpu-moeto context length. Keep some expert layers resident in spare VRAM for short prompts (e.g.ncmoe 28on the 35B, ~+5% over all-host); at long context the KV cache needs that VRAM, so raise it (ncmoe 40, all experts on host). Keep flash attention on (-fa 1).
Recommended RX 580 / Polaris serving command (per model):
llama-server -hf <repo>:<quant> -ngl 99 --n-cpu-moe 40 -b 2048 -ub 2048 -fa 1
Lower --n-cpu-moe (e.g. 28) if the model plus your context budget leave spare VRAM; keep it high
for long-context / agentic use. At long context the bottleneck is attention compute (GPU-bound), so
mask_opt (above) is where the remaining prefill wins come from, not the MoE-transfer path.
Very large MoE (experts bigger than the GTT limit)
Measured on the same RX 580 with Laguna-S-2.1 118B IQ2_M (48 layers, 256 experts, 10 used,
experts 30.7 GiB of a 34.7 GiB file, all host-resident at --n-cpu-moe 48). The advice above changes
in this regime:
--no-mmapstops being an option. Its pinned host buffer is charged against the amdgpu GTT limit (~31.4 GiB here, about half of system RAM). At 30.7 GiB of experts the model no longer loads, and the allocation spike can OOM the box. Use mmap and accept the staging copy.- The routing-ids readback is pure overhead at prefill batch sizes and this fork now skips it.
To decide which experts to upload, the scheduler read the ids back from the device that had just
produced them, forcing a full pipeline flush once per MoE layer per eval. With 2048 tokens x 10
experts over 256 experts every expert is used anyway. Skipping it is exact -
mul_mat_idonly reads the rows the ids point at. Auto-on above4 * n_expertids; decode keeps the old path. pp2048 +4.7% @ 32k depth, +0.9% @ 16k, tg +3%. - Where the time actually goes (
GGML_VK_PERF_LOGGER=1, depth 0, 22.65 s per 2048-token eval, 17.18 s of it GPU-busy so ~24% is H2D stall): expertMUL_MAT_ID52%, attention projections 24%,FLASH_ATTN_EXT16%, everything else 8%. The expert matmuls run at 1686-2139 GFLOP/s while denseMUL_MATq5_K/q6_K in the same graph reaches 3096-3691 - the single largest remaining opportunity on this hardware is closing that gap, not the transfer path. - Interleaved SWA keeps its own small KV cache, so a sliding-window layer costs the same at any
depth (
n_kvpinned atn_swa * n_seq_max + n_ubatch). On this model 36 of 48 layers are O(1) in depth and the entire high-context slowdown comes from the 12 full-attention layers. --parallel 1is worth setting for a solo large model: the server otherwise auto-selects 4 slots, and the SWA cache is sizedn_swa * n_seq_max + n_ubatch, so 4 slots cost 4096 cells instead of 2560. Measured 223 MiB of VRAM freed at 64k context.
Dead ends measured on this hardware, recorded so they are not retried:
| Change | Result |
|---|---|
flash-attn shmem_staging enabled for GCN |
-6.7% @ 16k, -7.4% @ 32k |
-b 4096 -ub 4096 (to amortize the fixed per-eval expert upload) |
flat (-1%) |
--n-cpu-moe 44 instead of 48 |
+1.5%, but does not fit at 64k ctx |
mask_opt gate relaxed below head_dim 256 |
-18.5% @ 16k |
shmem_staging looks like a certain win (without it each rowgroup re-reads the whole K/V block
through a 16 KiB L1) but the kvsh stride of D/4+1 dwords is 4 mod 32, which costs an 8-way LDS
bank conflict on wave64 - that +1 padding is tuned for warp32. -ub 4096 fails because halving the
number of expert uploads is exactly cancelled by intra-ubatch attention growing quadratically.
Serving note that outweighs all of the above. With a model this large, anything that restarts the process is far more expensive than any kernel win: the server's prompt cache is RAM-only with no disk backing, so a restart forces a full re-prefill of the conversation. If a model swapper can evict this model to run a small helper model (chat-title generation and the like), fix that first - keeping the process alive across a swap took a repeat turn from a 21,960 ms prefill down to 225 ms.
Quick start
A few options to get llama.cpp installed on your machine:
- Visit https://llama.app and follow the instructions
- Run with Docker - see our Docker documentation
- Download pre-built binaries from the releases page
- Build from source by cloning this repository - check out our build guide
Once installed:
# Download and run a model directly from Hugging Face
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF
# Launch OpenAI-compatible API server
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF
|
|
|
Description
The main goal of llama.cpp is to enable LLM (and VLM) inference with minimal setup and state-of-the-art performance on
a wide range of hardware - locally and in the cloud.
- Plain C/C++ implementation without any dependencies
- Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks
- AVX, AVX2, AVX512 and AMX support for x86 architectures
- RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE support for RISC-V architectures
- 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use
- Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA)
- Vulkan and SYCL backend support
- CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity
The llama.cpp project is build on top of the ggml library.
Supported backends
| Backend | Target devices |
|---|---|
| BLAS | All |
| BLIS | All |
| CANN | Ascend NPU |
| CUDA | Nvidia GPU |
| HIP | AMD GPU |
| Hexagon | Snapdragon |
| IBM zDNN | IBM Z & LinuxONE |
| MUSA | Moore Threads GPU |
| Metal | Apple Silicon |
| OpenCL | Adreno GPU |
| OpenVINO [In Progress] | Intel CPUs, GPUs, and NPUs |
| RPC | All |
| SYCL | Intel GPU |
| VirtGPU | VirtGPU APIR |
| Vulkan | GPU |
| WebGPU | All |
| ZenDNN | AMD CPU |
Documentation
Tools
Development
- How to build
- Running on Docker
- Build on Android
- Multi-GPU usage
- Performance troubleshooting
- GGML tips & tricks
- XCFramework
- Completions
- Models
- Release process
Contributing
- Contributors can open PRs
- Collaborators will be invited based on contributions
- Maintainers can push to branches in the
llama.cpprepo and merge PRs into themasterbranch - Any help with managing issues, PRs and projects is very appreciated!
- Read the CONTRIBUTING.md for more information
Acknowledgements
- yhirose/cpp-httplib - Single-header HTTP server, used by
llama-server- MIT license - nothings/stb - Single-header image format decoder, used by multimodal subsystem - Public domain
- nlohmann/json - Single-header JSON library, used by various tools/examples - MIT License
- mackron/miniaudio - Single-header audio format decoder, used by multimodal subsystem - Public domain
- sheredom/subprocess.h - Single-header process launching solution for C and C++ - Public domain