Pascal 10b8b75534 tests: give the random model generator a per-tensor filler table (#30674)
* tests: give the random model generator a per-tensor filler table

Muse Spark makes test-llama-context fail on state_load, cp_h and cp_d on about
six percent of the random seeds, with an NMSE of 1.0e-05 to 1.4e-05 against a
threshold of 1.0e-05. The sliding window is what exposes it: a restored state
keeps the window alone, so the attention of an SWA layer reduces over a
different cell layout and the continuation is not bit exact. Every SWA model
carries that, at 1e-08. Muse Spark carried it at 1e-05.

The generator is what turns 1e-08 into 1e-05. A norm preserving residual gate
holds alpha^2 + beta^2 == 1 and the conversion resolves both weights from one
raw weight, while the generator draws every tensor on its own. The pair then
scaled the stream and the branch independently, the stream lost an arbitrary
factor at each layer, and the model became an amplifier. The weights the test
ran on were not a Muse Spark.

Per-tensor rules lived inside the write path, where a flag picked the values of
a tensor inside each of the two branches that store them, so a new rule meant a
new flag threaded through both. The file asks to be rid of that in a TODO of its
own. Give each rule a filler, pick one from a table keyed on a substring of the
tensor name, and keep a single write path behind it. Mamba moves into the table
and Muse Spark joins it: the gate filler draws the raw weight once per block and
resolves alpha and beta the way the conversion does.

Over a thousand seeds on a CPU build the worst case drops from 1.36e-05 to
2.48e-06 and the median from 4.10e-06 to 5.48e-07, with no seed above the
threshold. The other 131 models come out identical to the bit, since no other
architecture carries these tensors. ctest is green on a CPU build and on a CUDA
build.

* tests: draw every tensor filler from fill_normal

The fillers repeated the generator and the distribution each time. The
normal draw now takes its key and the other fillers transform its
values, so the generator lives in one place. The generated models are
identical to the bit.
2026-10-17 16:07:55 +03:00
2026-10-17 02:11:56 +02:00
2026-10-17 02:11:56 +02:00
2026-06-12 15:53:26 +02:00
2026-10-17 02:11:56 +02:00
2026-02-02 08:38:55 +02:00
2026-10-17 02:11:56 +02:00

llama.cpp

llama

Quick start

A few options to get llama.cpp installed on your machine:

# curl
curl -LsSf https://llama.app/install.sh | sh

# powershell
irm https://llama.app/install.ps1 | iex

Once installed:

# Download and run a model directly from Hugging Face
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF

# Launch OpenAI-compatible API server
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF
VLM session with `llama cli` VLM session with llama cli Built-in web UI against `llama serve` running Qwen 3.6 Built-in web UI against llama serve

Description

The main goal of llama.cpp is to enable LLM (and VLM) inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud.

  • Plain C/C++ implementation without any dependencies
  • Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks
  • AVX, AVX2, AVX512 and AMX support for x86 architectures
  • RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE support for RISC-V architectures
  • 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use
  • Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA)
  • Vulkan and SYCL backend support
  • CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity

The llama.cpp project is build on top of the ggml library.

Supported backends

Backend Target devices
BLAS All
BLIS All
CANN Ascend NPU
CUDA Nvidia GPU
HIP AMD GPU
Hexagon Snapdragon
IBM zDNN IBM Z & LinuxONE
MUSA Moore Threads GPU
Metal Apple Silicon
OpenCL Adreno GPU
OpenVINO [In Progress] Intel CPUs, GPUs, and NPUs
RPC All
SYCL Intel GPU
VirtGPU VirtGPU APIR
Vulkan GPU
WebGPU All
ZenDNN AMD CPU

Documentation

Tools

Development

Contributing

  • Contributors can open PRs
  • Collaborators will be invited based on contributions
  • Maintainers can push to branches in the llama.cpp repo and merge PRs into the master branch
  • Any help with managing issues, PRs and projects is very appreciated!
  • Read the CONTRIBUTING.md for more information

Acknowledgements

  • yhirose/cpp-httplib - Single-header HTTP server, used by llama-server - MIT license
  • nothings/stb - Single-header image format decoder, used by multimodal subsystem - Public domain
  • nlohmann/json - Single-header JSON library, used by various tools/examples - MIT License
  • mackron/miniaudio - Single-header audio format decoder, used by multimodal subsystem - Public domain
  • sheredom/subprocess.h - Single-header process launching solution for C and C++ - Public domain
S
Description
No description provided
Readme MIT
560 MiB
0 Stars 1 Watchers 0 Forks
Languages
C++ 54.9%
C 16.7%
Python 7%
Cuda 5.6%
TypeScript 3.9%
Other 11.7%