llama-swap SIGTERMs a model when the owner switches models (only one large model fits in RAM), and switching back re-prefills the whole conversation: about 150 s for a 30k-token chat on the 35B. With --cache-disk-path DIR, each slot with a prompt is saved to DIR on shutdown. After the restart, a request whose prompt starts with a saved prompt is restored from the file, so only the suffix is processed. Off by default. Save, in server_context::start_loop() after the loop returns: - disk_cache_collect() copies the state of each slot with >= 256 text tokens to host memory (llama_state_seq_get_data_ext, flags NONE) while the device buffers still exist. A cold server reads its released state through llama_context_restore_memory_host(), with no device use and no extra copy. - vram_share_exit() then frees the VRAM and releases the GPU token as before, so only the D2H copy delays the token release, never the disk IO. - disk_cache_flush() writes the files. A slot is skipped if its KV does not cover its token list (seq_pos_max + 1 != n_tokens: the last batch was not decoded) or if the same token list is already on disk. File (server_prompt_disk_cache, server-task.cpp): - A 48-byte header (magic, version, key hash, sizes, payload hash), then a payload of 8-byte padded sections: tokens, checkpoint table, state, the checkpoint states. The hash is FNV-1a over the 64-bit words of the payload. - Plain fwrite to <name>.tmp, fdatasync, POSIX_FADV_DONTNEED, rename. The read is a plain fread plus DONTNEED. There is no mmap: upstream #28092 livelocked the kernel on AMD + Vulkan by DMA-ing into a MAP_SHARED file. - The index is built at startup from the headers. A bad magic, a size that does not match the header, or an inconsistent table means the file is removed. A hash mismatch at load also removes it, and nothing is restored. Other keys and other versions are kept. A .tmp older than 1 h is removed. - --cache-disk-size N MiB (default 4096, 0 = no limit): before a write, the least recently used *.lpcd files of every model in DIR are removed until the new file fits; a restore updates the file mtime. Entries the new prompt extends are dropped after the write and do not count against the limit. A write also needs its size + 1 GiB free on the file system. A write error or a full disk is logged; the file is not saved and the exit stays clean. Key: the build commit, the model path + size + mtime, LoRA and control vectors, n_ctx, n_ctx_seq, n_seq_max, n_ubatch, kv_unified, swa_full, the K/V types, the flash-attn type, and the RoPE/YaRN parameters. Restore, in get_available_slot() after the RAM cache: - Only while warm (holding the GPU token), with cache_prompt on and no media tokens. The file must reuse at least 256 more tokens than the slot holds. - Reusable means one of two things. Either the saved token list is an exact prefix of the new prompt, or (for a recurrent/SWA model) a saved checkpoint at or below the common prefix can be used. A plain-KV model can be cut back to any prefix. Exact prefix alone is not enough for OpenWebUI on Qwen: the template rewrites the previous assistant turn (the reasoning is dropped), so the saved list is almost never a prefix of the next turn. The checkpoints are exact snapshots, and update_slots() rolls back to one exactly as it does for the RAM cache. They are restored with id_task -1, so one that fails to load means a re-prefill, not an abort. - The slot's current prompt goes to the RAM cache first. A set_data failure clears the slot and removes the file. Not supported with speculative decoding (warning, disabled): the draft state is not saved. Only the slots are saved, not the RAM cache entries. Measured on the pod, CPU only, Qwen3.5-0.8B (hybrid, 6 of 24 layers attention): - 573-token chat: 64.5 MiB file with 2 checkpoints, copy 18 ms, write 171 ms, exit 314 ms (125 ms without the cache). After the restart: read 166 ms, restore 3 ms, turn 2 prompt_n 58 of 596 in 462 ms against a 2913 ms full prefill. The answer is identical to a server that never restarted. - 12k tokens: 200.7 MiB, copy 110 ms, write 350 ms, exit 641 ms, peak RSS +162 MiB during the exit save. After the restart: read 307 ms, restore 18 ms, turn 2 prompt_n 35 of 12188 in 1.2 s against a 72 s prefill. - The /root PVC writes at about 0.66 GB/s with fdatasync. /tmp (the node overlay) stalled once: 5.8 s for 64.5 MiB. llama-swap v239 waits for the old process to exit before it starts the next model, and SIGKILLs a process that is still there after 30 s. So the exit save adds to the model switch time: about 1 s per GB on the PVC. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
llama.cpp
LLM inference in C/C++
ggml / ops / maintainer PRs / dev stats / lib llama API / llama-server REST API
This fork - RX 580 (Polaris / GCN) serving of large MoE models
A private fork maintained for one machine: an AMD RX 580 8 GB (RADV Vulkan) serving Qwen3.6-35B-A3B,
gemma-4-26B-A4B and smaller models under llama-swap, with the MoE experts in host RAM
(--n-cpu-moe). FORK.md is the source of truth: every fork change with its
invariants, rebase hazards and validation, the measured dead ends, the production configuration and
the measurement traps. The rebase and deploy procedure is
docs/fork-rebase-runbook.md; scripts/fork-validate.sh validates a
build against a reference; scripts/rx580-bench/ is the benchmark harness.
In short, the fork adds:
- three Polaris prefill wins, on without a flag and token-identical to mainline: the MoE routing-ids
readback skip in
sched_copy_experts, flash-attentionmask_optfor GCN at head size >= 256, and a tiled-transpose fast path for the delta-net conv-state concat; - opt-in VRAM time-sharing between several llama-server processes (
LLAMA_SLEEP_VRAM_ONLY,LLAMA_SLEEP_EVICT_KV,LLAMA_MMPROJ_ONDEMAND,LLAMA_VRAM_ARENA): a model releases everything it holds in VRAM and gets it back without a restart, so its RAM prompt cache survives; - fixes to the server's sleep, wake and shutdown paths and to cold teardown that the sharing exposed;
- validation-only env knobs (
GGML_VK_CONCAT_FASTPATH,LLAMA_MOE_IDS_SKIP), unset in production.
Production settings for the 35B are in FORK.md section 5 (-lm none -t 6 -ngl 99 --n-cpu-moe 40 -b 2048 -ub 2048 -fa 1 --parallel 1, ctx 131072). Keep all experts on the host (--n-cpu-moe 40)
for long contexts: lower values are faster only at short context and run out of VRAM as the KV grows
(FORK.md section 4).
Quick start
A few options to get llama.cpp installed on your machine:
# curl
curl -LsSf https://llama.app/install.sh | sh
# powershell
irm https://llama.app/install.ps1 | iex
- Visit https://llama.app and follow the instructions
- Run with Docker - see our Docker documentation
- Download pre-built binaries from the releases page
- Build from source by cloning this repository - check out our build guide
Once installed:
# Download and run a model directly from Hugging Face
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF
# Launch OpenAI-compatible API server
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF
|
|
|
Description
The main goal of llama.cpp is to enable LLM (and VLM) inference with minimal setup and state-of-the-art performance on
a wide range of hardware - locally and in the cloud.
- Plain C/C++ implementation without any dependencies
- Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks
- AVX, AVX2, AVX512 and AMX support for x86 architectures
- RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE support for RISC-V architectures
- 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use
- Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA)
- Vulkan and SYCL backend support
- CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity
The llama.cpp project is build on top of the ggml library.
Supported backends
| Backend | Target devices |
|---|---|
| BLAS | All |
| BLIS | All |
| CANN | Ascend NPU |
| CUDA | Nvidia GPU |
| HIP | AMD GPU |
| Hexagon | Snapdragon |
| IBM zDNN | IBM Z & LinuxONE |
| MUSA | Moore Threads GPU |
| Metal | Apple Silicon |
| OpenCL | Adreno GPU |
| OpenVINO [In Progress] | Intel CPUs, GPUs, and NPUs |
| RPC | All |
| SYCL | Intel GPU |
| VirtGPU | VirtGPU APIR |
| Vulkan | GPU |
| WebGPU | All |
| ZenDNN | AMD CPU |
Documentation
Tools
Development
- How to build
- Running on Docker
- Build on Android
- Multi-GPU usage
- Performance troubleshooting
- GGML tips & tricks
- XCFramework
- Completions
- Models
- Release process
Contributing
- Contributors can open PRs
- Collaborators will be invited based on contributions
- Maintainers can push to branches in the
llama.cpprepo and merge PRs into themasterbranch - Any help with managing issues, PRs and projects is very appreciated!
- Read the CONTRIBUTING.md for more information
Acknowledgements
- yhirose/cpp-httplib - Single-header HTTP server, used by
llama-server- MIT license - nothings/stb - Single-header image format decoder, used by multimodal subsystem - Public domain
- nlohmann/json - Single-header JSON library, used by various tools/examples - MIT License
- mackron/miniaudio - Single-header audio format decoder, used by multimodal subsystem - Public domain
- sheredom/subprocess.h - Single-header process launching solution for C and C++ - Public domain