9fbe30c2a0c207f9de7f38b23c43bc044b7a9981
* tests: add CUDA-only MUL_MAT/MUL_MAT_ID cases with results above the fp16 range IQ1_M weights have no MMQ kernel, so once the batch is bigger than the MMVQ limit the path is dequantize + cuBLAS. These cases produce results around 1.3e5, past the fp16 max of 65504, so an fp16 result buffer turns them into inf while the CPU reference stays finite. They are only added for the CUDA backend. Vulkan deliberately accumulates quantized matmuls in fp16 at default precision and would fail them by design. * CUDA: use an fp32 result for quantized src0 in the cuBLAS fallback On the F16 compute path, quantized weights went through cuBLAS with fp16 accumulation and an fp16 result buffer that was converted to fp32 afterwards. Anything above 65504 became inf and the next layer turned into NaN (see #30951: gpt-oss-120b with IQ1_M experts at -ub 512). Set prefer_f32_output for quantized src0 so the GEMM still uses fp16 operands but accumulates and stores in fp32, the same mode already used on Volta, RDNA4 and CDNA. Every other quantized path already accumulates in fp32. Diagnosis and fix by denizyuret in #30951. * CUDA: keep the MUL_MAT_ID precision in the per-expert fallback The per-expert dst_slice kept op = GGML_OP_NONE, so ggml_prec_get_acc returned UNDEFINED and a requested precision was ignored. Set the op to GGML_OP_MUL_MAT. Also revert the fp32 result change and remove the tests. --------- Co-authored-by: JonSnow1807 <JonSnow1807@users.noreply.github.com>
tool-call: fix Qwen 2.5 Coder support, add micro benchmarks, support trigger patterns for lazy grammars (#12034)
llama.cpp
LLM inference in C/C++
v1.0.0 / ggml / ops / maintainer PRs / dev stats / lib llama API / llama-server REST API
Description
The main goal of llama.cpp is to enable LLM (and MLLM) inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud.
The llama.cpp project is built on top of the ggml library.
Quick start
A few options to get llama.cpp installed on your machine:
# curl
curl -LsSf https://llama.app/install.sh | sh
# powershell
irm https://llama.app/install.ps1 | iex
- Visit https://llama.app and follow the instructions
- Run with Docker - see our Docker documentation
- Download pre-built binaries from the releases page
- Build from source by cloning this repository - check out our build guide
Once installed:
# Download and run a model directly from Hugging Face
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF
# Launch OpenAI-compatible API server
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF
|
|
|
Documentation
Tools
Development
- How to build
- Running on Docker
- Build on Android
- Multi-GPU usage
- Performance troubleshooting
- GGML tips & tricks
- XCFramework
- Completions
- Models
- Release process
Contributing
- Contributors can open PRs
- Collaborators will be invited based on contributions
- Maintainers can push to branches in the
llama.cpprepo and merge PRs into themasterbranch - Any help with managing issues, PRs and projects is very appreciated!
- Read the CONTRIBUTING.md for more information
Acknowledgements
- yhirose/cpp-httplib - Single-header HTTP server, used by
llama-server- MIT license - nothings/stb - Single-header image format decoder, used by multimodal subsystem - Public domain
- nlohmann/json - Single-header JSON library, used by various tools/examples - MIT License
- mackron/miniaudio - Single-header audio format decoder, used by multimodal subsystem - Public domain
- sheredom/subprocess.h - Single-header process launching solution for C and C++ - Public domain
Languages
C++
54.9%
C
16.7%
Python
7%
Cuda
5.6%
TypeScript
3.9%
Other
11.7%