Files
llama.cpp/README.md
T
LumpiastyandClaude Opus 5.5 5f753f7a0c docs(readme): short fork summary, FORK.md is the source of truth
The README fork section still said "the two code changes below are auto-on"
(the fork has three prefill changes plus the VRAM sharing and server fixes)
and recommended --n-cpu-moe 28 on the 35B, which FORK.md section 4 records as
worse at depth and unable to hold a long context. Replace it with a short
accurate summary that points to FORK.md, the runbook and the scripts. The
Laguna time profile, the iSWA KV note and Laguna's --parallel 1 saving were
only in the README and move to FORK.md sections 4 and 5.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:51 +01:00

7.8 KiB

llama.cpp

llama.cpp

Description

The main goal of llama.cpp is to enable LLM (and MLLM) inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud.

The llama.cpp project is built on top of the ggml library.

This fork - RX 580 (Polaris / GCN) serving of large MoE models

A private fork maintained for one machine: an AMD RX 580 8 GB (RADV Vulkan) serving Qwen3.6-35B-A3B, gemma-4-26B-A4B and smaller models under llama-swap, with the MoE experts in host RAM (--n-cpu-moe). FORK.md is the source of truth: every fork change with its invariants, rebase hazards and validation, the measured dead ends, the production configuration and the measurement traps. The rebase and deploy procedure is docs/fork-rebase-runbook.md; scripts/fork-validate.sh validates a build against a reference; scripts/rx580-bench/ is the benchmark harness.

In short, the fork adds:

  • three Polaris prefill wins, on without a flag and token-identical to mainline: the MoE routing-ids readback skip in sched_copy_experts, flash-attention mask_opt for GCN at head size >= 256, and a tiled-transpose fast path for the delta-net conv-state concat;
  • opt-in VRAM time-sharing between several llama-server processes (LLAMA_SLEEP_VRAM_ONLY, LLAMA_SLEEP_EVICT_KV, LLAMA_MMPROJ_ONDEMAND, LLAMA_VRAM_ARENA): a model releases everything it holds in VRAM and gets it back without a restart, so its RAM prompt cache survives;
  • fixes to the server's sleep, wake and shutdown paths and to cold teardown that the sharing exposed;
  • validation-only env knobs (GGML_VK_CONCAT_FASTPATH, LLAMA_MOE_IDS_SKIP), unset in production.

Production settings for the 35B are in FORK.md section 5 (-lm none -t 6 -ngl 99 --n-cpu-moe 40 -b 2048 -ub 2048 -fa 1 --parallel 1, ctx 131072). Keep all experts on the host (--n-cpu-moe 40) for long contexts: lower values are faster only at short context and run out of VRAM as the KV grows (FORK.md section 4).

Quick start

A few options to get llama.cpp installed on your machine:

# curl
curl -LsSf https://llama.app/install.sh | sh

# powershell
irm https://llama.app/install.ps1 | iex

Once installed:

# Download and run a model directly from Hugging Face
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF

# Launch OpenAI-compatible API server
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF
VLM session with `llama cli` VLM session with llama cli Llama Chat, the built-in chat app, against `llama serve` running Qwen 3.6 Llama Chat, the built-in chat app, against llama serve

Documentation

Tools

Development

Contributing

  • Contributors can open PRs
  • Collaborators will be invited based on contributions
  • Maintainers can push to branches in the llama.cpp repo and merge PRs into the master branch
  • Any help with managing issues, PRs and projects is very appreciated!
  • Read the CONTRIBUTING.md for more information

Acknowledgements

  • yhirose/cpp-httplib - Single-header HTTP server, used by llama-server - MIT license
  • nothings/stb - Single-header image format decoder, used by multimodal subsystem - Public domain
  • nlohmann/json - Single-header JSON library, used by various tools/examples - MIT License
  • mackron/miniaudio - Single-header audio format decoder, used by multimodal subsystem - Public domain
  • sheredom/subprocess.h - Single-header process launching solution for C and C++ - Public domain