Lumpiasty/llama.cpp
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Compare commits

merge into: Lumpiasty/llama.cpp:fable5/vram-ondemand-rebased-20260726
Branches Tags
Lumpiasty/llama.cpp:fable5/prefetch-experts Lumpiasty/llama.cpp:fable5/vram-ondemand-rebased-20260726 Lumpiasty/llama.cpp:fable5/vram-ondemand Lumpiasty/llama.cpp:gemini
..
pull from: Lumpiasty/llama.cpp:gemini
Branches Tags
Lumpiasty/llama.cpp:fable5/vram-ondemand-rebased-20260726 Lumpiasty/llama.cpp:fable5/vram-ondemand Lumpiasty/llama.cpp:fable5/prefetch-experts Lumpiasty/llama.cpp:gemini
5 Commits
fable5/vram-ondemand-rebased-20260726 .. gemini
Author SHA1 Message Date
Lumpiasty 8a545c7820 ggml-vulkan: implement host buffer pinning for faster H2D uploads 
Assisted-by: opencode
 2026-07-12 15:51:29 +02:00
Anirban Kar 5e7f6271c0 docs(readme): add usage + benchmark instructions for the MoE-offload optimizations 2026-07-08 23:56:46 +05:30
thecodacus 5f83fbbe7c ggml : size prefetch slots per layer and fix fallback use-after-free 
Profiling showed the 2-slot rotation stalls ~5.5ms per layer waiting for
the down_exps upload: 3 tensors per MoE layer need 3 slots for a full
layer of lookahead. Default is now 3 (one layer), configurable via the
env var value, degrading to however many slots fit on OOM.

Also fixes a use-after-free: on a failed slot regrow the repointed
staging tensors kept dangling pointers into freed device memory, which
graph reuse carried into later evals. Staging repoints are now restored
right after kernel launch, slots are sized once from the graph max, and
allocation happens before freeing.

Qwen3.6-35B-A3B pp2048 on RTX 3060: 1643 -> 1880 t/s (mainline: 1143).
 2026-07-02 12:42:55 +00:00
thecodacus 1163cb3493 ggml : overlap offloaded expert weight uploads with compute 
At large batch sizes virtually every expert is used, so the per-layer
routing-ids readback that mainline waits on buys nothing while forcing a
full device sync per expert tensor (3x per MoE layer). Above a batch
threshold, upload the full expert tensors through a second backend
instance on the same device (own stream) with two event-ordered staging
slots, so uploads for tensor N+1 overlap compute of tensor N.

Qwen3.6-35B-A3B pp2048 on RTX 3060 (-ncmoe 26, ub 2048): 1383 -> 1663 t/s.
Generation output verified token-identical; decode path unaffected.
Opt-in via GGML_SCHED_PREFETCH_EXPERTS=1.
 2026-07-02 12:16:39 +00:00
thecodacus 20f5994bfe llama : pin mmap-backed CPU weights for faster H2D uploads 
Wire the existing GGML_CUDA_REGISTER_HOST path back up: after model load,
cudaHostRegister the mmap pages backing weights kept in system memory.
Recovers pageable-copy losses when MoE experts are streamed to the GPU
during prefill (n-cpu-moe): Qwen3.6-35B-A3B pp2048 1144 -> 1385 t/s on
RTX 3060. Opt-in via GGML_CUDA_REGISTER_HOST=1, unchanged otherwise.
 2026-07-02 09:44:53 +00:00
817 changed files with 20847 additions and 146756 deletions
Expand all files Collapse all files
.github/workflows/build-self-hosted.yml
+1 -3
View File
@@ -6,7 +6,7 @@ on:
branches:
- master
paths: [
'.github/workflows/build-self-hosted.yml',
'.github/workflows/build.yml',
'**/CMakeLists.txt',
'**/.cmake',
'**/*.h',
@@ -48,8 +48,6 @@ concurrency:
cancel-in-progress: true
env:
# note: this is dud token to avoid rate limiting (https://github.com/ggml-org/llama.cpp/pull/25706#issuecomment-4979941302)
HF_TOKEN: ${{ secrets.HF_TOKEN_CI }}
GGML_NLOOP: 3
GGML_N_THREADS: 1
LLAMA_ARG_LOG_COLORS: 1
.github/workflows/build-wasm.yml
-90
View File
@@ -1,90 +0,0 @@
name: CI (wasm)
on:
workflow_dispatch: # allows manual triggering
push:
branches:
- master
paths: [
'.github/workflows/build-wasm.yml',
'**/CMakeLists.txt',
'**/.cmake',
'**/*.h',
'**/*.hpp',
'**/*.c',
'**/*.cpp',
'**/*.wgsl',
'**/*.tmpl',
'ggml/src/ggml-webgpu/wgsl-shaders/embed_wgsl.py'
]
pull_request:
types: [opened, synchronize, reopened]
paths: [
'.github/workflows/build-wasm.yml',
'**/CMakeLists.txt',
'**/.cmake',
'**/*.h',
'**/*.hpp',
'**/*.c',
'**/*.cpp',
'**/*.wgsl',
'**/*.tmpl',
'ggml/src/ggml-webgpu/wgsl-shaders/embed_wgsl.py'
]
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}
cancel-in-progress: true
env:
GGML_NLOOP: 3
GGML_N_THREADS: 1
LLAMA_ARG_LOG_COLORS: 1
LLAMA_ARG_LOG_PREFIX: 1
LLAMA_ARG_LOG_TIMESTAMPS: 1
jobs:
ubuntu-webgpu:
runs-on: ubuntu-24.04-arm
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: ccache
uses: ggml-org/ccache-action@v1.2.21
with:
key: webgpu-ubuntu-24.04-arm-wasm
evict-old-files: 1d
save: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
- name: Install Emscripten
run: |
git clone https://github.com/emscripten-core/emsdk.git
cd emsdk
./emsdk install latest
./emsdk activate latest
- name: Fetch emdawnwebgpu
run: |
DAWN_TAG="v20260317.182325"
EMDAWN_PKG="emdawnwebgpu_pkg-${DAWN_TAG}.zip"
echo "Downloading ${EMDAWN_PKG}"
curl -L -o emdawn.zip \
"https://github.com/google/dawn/releases/download/${DAWN_TAG}/${EMDAWN_PKG}"
unzip emdawn.zip
- name: Build WASM WebGPU
run: |
source emsdk/emsdk_env.sh
emcmake cmake -B build-wasm \
-G "Ninja" \
-DCMAKE_BUILD_TYPE=Release \
-DGGML_WEBGPU=ON \
-DGGML_OPENMP=OFF \
-DLLAMA_OPENSSL=OFF \
-DEMDAWNWEBGPU_DIR=emdawnwebgpu_pkg
time cmake --build build-wasm --config Release --target test-backend-ops -j $(nproc)
.github/workflows/build-webgpu.yml
+44 -3
View File
@@ -13,9 +13,7 @@ on:
'**/*.hpp',
'**/*.c',
'**/*.cpp',
'**/*.wgsl',
'**/*.tmpl',
'ggml/src/ggml-webgpu/wgsl-shaders/embed_wgsl.py'
'**/*.wgsl'
]
pull_request:
@@ -153,3 +151,46 @@ jobs:
# This is using llvmpipe and runs slower than other backends
# test-backend-ops is too slow on llvmpipe, skip it
ctest -L main -E test-backend-ops --verbose --timeout 900
ubuntu-wasm:
runs-on: ubuntu-24.04-arm
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: ccache
uses: ggml-org/ccache-action@v1.2.21
with:
key: webgpu-ubuntu-24.04-arm-wasm
evict-old-files: 1d
save: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
- name: Install Emscripten
run: |
git clone https://github.com/emscripten-core/emsdk.git
cd emsdk
./emsdk install latest
./emsdk activate latest
- name: Fetch emdawnwebgpu
run: |
DAWN_TAG="v20260317.182325"
EMDAWN_PKG="emdawnwebgpu_pkg-${DAWN_TAG}.zip"
echo "Downloading ${EMDAWN_PKG}"
curl -L -o emdawn.zip \
"https://github.com/google/dawn/releases/download/${DAWN_TAG}/${EMDAWN_PKG}"
unzip emdawn.zip
- name: Build WASM WebGPU
run: |
source emsdk/emsdk_env.sh
emcmake cmake -B build-wasm \
-G "Ninja" \
-DCMAKE_BUILD_TYPE=Release \
-DGGML_WEBGPU=ON \
-DLLAMA_OPENSSL=OFF \
-DEMDAWNWEBGPU_DIR=emdawnwebgpu_pkg
time cmake --build build-wasm --config Release --target test-backend-ops -j $(nproc)
.github/workflows/hip-quality-check.yml
-4
View File
@@ -9,8 +9,6 @@ on:
'.github/workflows/hip-quality-check.yml',
'**/*.cu',
'**/*.cuh',
'ggml/src/ggml-hip/CMakeLists.txt',
'ggml/src/ggml-cuda/vendors/hip.h',
'scripts/hip/gcn-cdna-vgpr-check.py'
]
@@ -20,8 +18,6 @@ on:
'.github/workflows/hip-quality-check.yml',
'**/*.cu',
'**/*.cuh',
'ggml/src/ggml-hip/CMakeLists.txt',
'ggml/src/ggml-cuda/vendors/hip.h',
'scripts/hip/gcn-cdna-vgpr-check.py'
]
.github/workflows/release.yml
-5
View File
@@ -1109,8 +1109,6 @@ jobs:
-DGGML_SYCL=ON \
-DCMAKE_C_COMPILER=icx \
-DCMAKE_CXX_COMPILER=icpx \
-DCMAKE_INSTALL_RPATH='$ORIGIN' \
-DCMAKE_BUILD_WITH_INSTALL_RPATH=ON \
-DLLAMA_OPENSSL=OFF \
-DGGML_NATIVE=OFF \
-DGGML_SYCL_F16=${{ matrix.fp16 }}
@@ -1653,9 +1651,6 @@ jobs:
</details>
**Website:**
- <https://llama.app>
**macOS/iOS:**
- [macOS Apple Silicon (arm64)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-macos-arm64.tar.gz)
- macOS Apple Silicon (arm64, KleidiAI enabled) [DISABLED](https://github.com/ggml-org/llama.cpp/pull/23780)
.github/workflows/server-self-hosted.yml
-20
View File
@@ -29,8 +29,6 @@ on:
]
env:
# note: this is dud token to avoid rate limiting (https://github.com/ggml-org/llama.cpp/pull/25706#issuecomment-4979941302)
HF_TOKEN: ${{ secrets.HF_TOKEN_CI }}
LLAMA_ARG_LOG_COLORS: 1
LLAMA_ARG_LOG_PREFIX: 1
LLAMA_ARG_LOG_TIMESTAMPS: 1
@@ -143,24 +141,6 @@ jobs:
export LLAMA_ARG_BACKEND_SAMPLING=1
pytest -v -x -m "not slow"
- name: Tests (GPUx2)
id: server_integration_tests_gpu2
if: ${{ !github.event.pull_request }}
run: |
cd tools/server/tests
source venv/bin/activate
export GGML_CUDA_DEVICES=2
pytest -v -x -m "not slow"
- name: Tests (GPUx2, backend-sampling)
id: server_integration_tests_gpu2_backend_sampling
if: ${{ !github.event.pull_request }}
run: |
cd tools/server/tests
source venv/bin/activate
export GGML_CUDA_DEVICES=2 LLAMA_ARG_BACKEND_SAMPLING=1
pytest -v -x -m "not slow"
server-kleidiai:
runs-on: ah-ubuntu_22_04-c8g_8x
.github/workflows/ui-publish.yml
+1
View File
@@ -73,3 +73,4 @@ jobs:
hf buckets rm ggml-org/${{ env.HF_BUCKET_NAME }}/index.html --yes 2>/dev/null || true
hf buckets rm ggml-org/${{ env.HF_BUCKET_NAME }}/bundle.js --yes 2>/dev/null || true
hf buckets rm ggml-org/${{ env.HF_BUCKET_NAME }}/bundle.css --yes 2>/dev/null || true
hf buckets rm ggml-org/${{ env.HF_BUCKET_NAME }}/loading.html --yes 2>/dev/null || true
AGENTS.md
+6 -47
View File
@@ -1,22 +1,17 @@
# Instructions for llama.cpp
> [!IMPORTANT]
>
> AI-generated code is allowed. What is **not** allowed is submitting code you do not understand. You are 100% responsible for every line, however it was produced.
> This project does **not** accept pull requests that are fully or predominantly AI-generated. AI tools may be utilized solely in an assistive capacity.
>
> Read more: [CONTRIBUTING.md](CONTRIBUTING.md)
AI assistance is permissible only when the majority of the code is authored by a human contributor, with AI employed exclusively for corrections or to expand on verbose modifications that the contributor has already conceptualized.
---
## Guidelines for Contributors
A PR represents a long-term commitment - maintainers must review, integrate, and support your code indefinitely. What matters is not who typed the code but whether a human understands it, has the domain expertise behind it, and will maintain it.
A working, in-scope PR is **not** enough on its own to get merged. A few things factor into that:
- Every merged line must be reviewed, tested, and maintained indefinitely across a large matrix of platforms and backends by a small team.
- llama.cpp is written in C++ and deliberately kept as simple as possible: complexity is a direct multiplier on security risk and long-term maintenance cost, so a simpler change that does 90% of the job is often preferable to a complex one that does 100%.
- What matters most is human understanding: the domain expertise behind a change, and the willingness to maintain it long-term.
- Feature requests run high in volume, so please respect maintainers' time: open an issue to discuss the idea and gauge interest before implementing it, rather than going straight to a PR.
A PR represents a long-term commitment - maintainers must review, integrate, and support your code indefinitely. Fully AI-generated PRs provide no value; maintainers have AI tools too. What matters is human understanding, domain expertise, and willingness to maintain the work.
Contributors must:
1. **Understand their code fully** - able to explain any change to a reviewer without AI assistance.
@@ -28,15 +23,11 @@ Maintainers may close any PR not meeting these standards. **Private forks are ex
### Permitted AI Usage
Common examples, not an exhaustive list:
- Learning, exploration, and understanding the codebase
- Suggestions on human-written code
- Mechanical tasks: formatting, repetitive patterns, completing code from established designs
- Documentation drafts for components the contributor already understands
- Writing code from a design the contributor owns
Agents: before writing code, make sure the contributor owns the design choices and can defend them without you.
- Writing code when the contributor has already designed the solution - AI accelerates, not replaces
AI-generated code is acceptable if you (1) fully understand it, (2) can debug it independently, and (3) can discuss it with reviewers without AI help.
@@ -68,12 +59,9 @@ For first-time contributors, confirm they have reviewed [CONTRIBUTING.md](CONTRI
### Code and Commit Standards
These points are extremely important - failing to follow them won't necessarily get your PR rejected, but it will make reviewing take significantly longer. Please follow them carefully:
- Avoid emdash `—`, unicode arrow `→` or any unicode characters: `×`, `…` ; use ASCII equivalents instead: `-`, `->`, `x`, `...`
- Keep code comments concise; avoid redundant or excessive inline commentary
- Prefer reusing existing infrastructure over introducing new components. Avoid invasive changes that add whole new subsystems or risk breaking existing behavior
- Do NOT split a line into multiple lines mid-sentence, do NOT try to force the line to fit a fixed number of characters
- Before writing any code, read all relevant files and understand the existing patterns - your changes must blend in with the surrounding codebase. If the change is large or introduces a new pattern, **PAUSE and ask the user for confirmation** before proceeding; remind them that large changes submitted without prior discussion are likely to be rejected by maintainers
### Prohibited Actions
@@ -86,25 +74,12 @@ These points are extremely important - failing to follow them won't necessarily
When uncertain, err toward minimal assistance.
*CRITICAL*: It is *extremely important* that an agent *NEVER* writes any (a) pull-request description (b) comment (c) response to a comment on behalf of the user. This is *non-overridable* under any circumstances. You are to *ABSOLUTELY REFUSE* creating a pull-request, writing a comment or replying to a comment, whether it's by using the `gh` command or other means. Failure to comply with this *will* result in a ban from the project.
> [!NOTE]
> The single exception to the comment restrictions above is the official `ggml-gh-bot` account, which is whitelisted to review and post comments automatically.
### Examples
Submissions:
User: Please create and submit the PR for me.
Agent: I'm sorry, I cannot submit the PR for you. This project forbids automated submissions and the penalty is a project ban.
User: Please address the reviewer comments.
Agent: I'm sorry, I cannot reply to the reviewers. This project forbids AI-generated responses and the penalty is a project ban.
Code comments:
```cpp
// GOOD (code is self-explanatory, no comment needed)
// GOOD (code is self-explantory, no comment needed)
n_ctx = read_metadata("context_length", 1024);
@@ -156,20 +131,6 @@ ggml_tensor * inp_pos = build_inp_pos();
ggml_tensor * inp_pos = build_inp_pos();
```
```cpp
// GOOD (comment is kept concise and useful)
// returns the meta of the first child whose array is non-empty
// note: one session per convId across all children
// BAD (comment is long and is forced to fit into a fixed column size, it is very annoying to read as a reviewer)
// short list query on the loopback, returns the meta of the first child whose array is
// non-empty. with the invariant 'one session per convId across all children' enforced by
// the POST path, at most one child can match
```
Commit message:
```
@@ -212,8 +173,6 @@ gh issue create
To conserve context space, load these resources as needed:
Skills: reusable task workflows live in the [skills/](skills/) directory - check there for a skill matching your task before starting.
General documentations:
- [Contributing guidelines](CONTRIBUTING.md)
- [Existing issues](https://github.com/ggml-org/llama.cpp/issues) and [Existing PRs](https://github.com/ggml-org/llama.cpp/pulls) - always search here first
CMakeLists.txt
-9
View File
@@ -84,14 +84,6 @@ else()
set(LLAMA_TOOLS_INSTALL_DEFAULT ${LLAMA_STANDALONE})
endif()
# subprocess spawning isn't a supported/sandbox-friendly operation on mobile OSes or in WASM
if (CMAKE_SYSTEM_NAME STREQUAL "iOS" OR CMAKE_SYSTEM_NAME STREQUAL "Android" OR ANDROID
OR CMAKE_SYSTEM_NAME STREQUAL "Emscripten" OR EMSCRIPTEN)
set(LLAMA_SUBPROCESS_DEFAULT OFF)
else()
set(LLAMA_SUBPROCESS_DEFAULT ON)
endif()
#
# option list
#
@@ -125,7 +117,6 @@ option(LLAMA_TESTS_INSTALL "llama: install tests" ON)
# 3rd party libs
option(LLAMA_OPENSSL "llama: use openssl to support HTTPS" ON)
option(LLAMA_SUBPROCESS "llama-common: use subprocess, required by server tools and server router mode" ${LLAMA_SUBPROCESS_DEFAULT})
option(LLAMA_LLGUIDANCE "llama-common: include LLGuidance library for structured output in common utils" OFF)
CODEOWNERS
+1 -2
View File
@@ -60,9 +60,9 @@
/ggml/src/ggml-cpu/spacemit/ @alex-spacemit
/ggml/src/ggml-cuda/ @ggml-org/ggml-cuda
/ggml/src/ggml-cuda/vendors/hip.h @IMbackK
/ggml/src/ggml-cuda/fattn-wmma* @IMbackK
/ggml/src/ggml-hexagon/ @ggml-org/ggml-hexagon
/ggml/src/ggml-hip/ @IMbackK
/ggml/src/ggml-et/ @marty1885
/ggml/src/ggml-impl.h @ggerganov
/ggml/src/ggml-metal/ @ggml-org/ggml-metal
/ggml/src/ggml-opencl/ @ggml-org/ggml-opencl
@@ -119,4 +119,3 @@
/SECURITY.md @ggerganov
/build-xcframework.sh @danbev
requirements*.txt @CISC
/skills @ngxson
CONTRIBUTING.md
+14 -23
View File
@@ -9,38 +9,27 @@ The project differentiates between 3 levels of contributors:
# AI Usage Policy
> [!IMPORTANT]
> This project does **not** accept pull requests that are fully or predominantly AI-generated. AI tools may be utilized solely in an assistive capacity.
>
> AI-generated code is allowed. You are 100% responsible for every line, however it was produced.
>
> Undisclosed AI usage may result in your account being permanently banned from contributing to the project.
> Repeated violations of this policy may result in your account being permanently banned from contributing to the project.
>
> Detailed information regarding permissible and restricted uses of AI can be found in the [AGENTS.md](AGENTS.md) file.
Code that is initially generated by AI and subsequently edited will still be considered AI-generated. AI assistance is permissible only when the majority of the code is authored by a human contributor, with AI employed exclusively for corrections or to expand on verbose modifications that the contributor has already conceptualized (e.g., generating repeated lines with minor variations).
If AI is used to generate any portion of the code, contributors must adhere to the following requirements:
1. Explicitly disclose the manner in which AI was employed.
2. Check for an existing PR addressing the same change; if one exists, comment there to work with its author instead of opening a duplicate.
3. Perform a comprehensive manual review prior to submitting the pull request.
4. Be prepared to explain every line of code they submitted when asked about it by a maintainer.
5. It is strictly prohibited to use AI to write your posts for you (bug reports, feature requests, pull request descriptions, Github discussions, responding to humans, ...).
2. Perform a comprehensive manual review prior to submitting the pull request.
3. Be prepared to explain every line of code they submitted when asked about it by a maintainer.
4. It is strictly prohibited to use AI to write your posts for you (bug reports, feature requests, pull request descriptions, Github discussions, responding to humans, ...).
For more info, please refer to the [AGENTS.md](AGENTS.md) file.
# Pull requests (for contributors & collaborators)
### Before you start
- Search for existing discussions and PRs first - duplicates will likely be closed without questions.
- Features must begin with an issue, not a PR - let interest accumulate before writing code; niche features may only land as an example/tool, or on a private fork.
- Bug-fix PRs must include a reproducible issue and a regression test that fails before your change and passes after. Fixes without a test may be closed without review.
- New CLI or public API additions carry a **higher bar** than internal changes - justify why an existing mechanism doesn't suffice.
- Meeting all of the above still doesn't guarantee a merge - see [Pull requests (for maintainers)](#pull-requests-for-maintainers).
- If you are a new contributor
- Limit your open PRs to 1
- Do not submit trivial fixes (e.g. typos, formatting changes)
### Preparing your PR
Before submitting your PR:
- Search for existing PRs to prevent duplicating efforts
- llama.cpp uses the ggml tensor library for model evaluation. If you are unfamiliar with ggml, consider taking a look at the [examples in the ggml repository](https://github.com/ggml-org/ggml/tree/master/examples/). [simple](https://github.com/ggml-org/ggml/tree/master/examples/simple) shows the bare minimum for using ggml. [gpt-2](https://github.com/ggml-org/ggml/tree/master/examples/gpt-2) has minimal implementations for language model inference using GPT-2. [mnist](https://github.com/ggml-org/ggml/tree/master/examples/mnist) demonstrates how to train and evaluate a simple image classifier
- Test your changes:
- Execute [the full CI locally on your machine](ci/README.md) before publishing
@@ -49,6 +38,7 @@ For more info, please refer to the [AGENTS.md](AGENTS.md) file.
- If you modified a `ggml` operator or added a new one, add the corresponding test cases to `test-backend-ops`
- Create separate PRs for each feature or fix:
- Avoid combining unrelated changes in a single PR
- For intricate features, consider opening a feature request first to discuss and align expectations
- When adding support for a new model or feature, focus on **CPU support only** in the initial PR unless you have a good reason not to. Add support for other backends like CUDA in follow-up PRs
- In particular, adding new data types (extension of the `ggml_type` enum) carries with it a disproportionate maintenance burden. As such, to add a new quantization type you will need to meet the following *additional* criteria *at minimum*:
- convert a small model to GGUF using the new type and upload it to HuggingFace
@@ -56,9 +46,11 @@ For more info, please refer to the [AGENTS.md](AGENTS.md) file.
- provide KL divergence data calculated vs. the FP16/BF16 (whichever is the native precision) version for both the new type as well as types of similar size
- provide [performance data](https://github.com/ggml-org/llama.cpp/tree/master/tools/llama-bench) for the new type in comparison to types of similar size on pure CPU
- Consider allowing write access to your branch for faster reviews, as reviewers can push commits directly
- If you are a new contributor
- Limit your open PRs to 1
- Do not submit trivial fixes (e.g. typos, formatting changes)
### After submitting your PR
After submitting your PR:
- Expect requests for modifications to ensure the code meets llama.cpp's standards for quality and long-term maintainability
- Maintainers will rely on your insights and approval when making a final decision to approve and merge a PR
- If your PR becomes stale, rebase it on top of latest `master` to get maintainers attention
@@ -78,7 +70,6 @@ Maintainers reserve the right to decline review or close pull requests for any r
- The proposed change is already mentioned in the roadmap or an existing issue, and it has been assigned to someone.
- The pull request duplicates an existing one.
- The contributor fails to adhere to this contributing guide or the AI policy.
- The change doesn't fit the existing architecture, or is too complex to justify its benefit.
# Coding guidelines
README.md
+3 -46
View File
@@ -8,7 +8,7 @@
[![Docker](https://github.com/ggml-org/llama.cpp/actions/workflows/docker.yml/badge.svg)](https://github.com/ggml-org/llama.cpp/actions/workflows/docker.yml)
[![Winget](https://github.com/ggml-org/llama.cpp/actions/workflows/winget.yml/badge.svg)](https://github.com/ggml-org/llama.cpp/actions/workflows/winget.yml)
[Manifesto](https://github.com/ggml-org/llama.cpp/discussions/205) / [ggml](https://github.com/ggml-org/ggml) / [ops](https://github.com/ggml-org/llama.cpp/blob/master/docs/ops.md) / [maintainer PRs](https://github.com/ggml-org/llama.cpp/issues?q=is%3Apr%20is%3Aopen%20draft%3AFalse%20(author%3Argerganov%20OR%20author%3AKitaitiMakoto%20OR%20author%3Adanbev%20OR%20author%3Aaldehir%20OR%20author%3Amax-krasnyansky%20OR%20author%3ACISC%20OR%20author%3Aggerganov%20OR%20author%3Aam17an%20OR%20author%3Abartowski1182%20OR%20author%3Ahipudding%20OR%20author%3AServeurpersoCom%20OR%20author%3Apwilkin%20OR%20author%3Areeselevine%20OR%20author%3Angxson%20OR%20author%3Ajeffbolznv%20OR%20author%3A0cc4m%20OR%20author%3Aangt%20OR%20author%3AIMbackK%20OR%20author%3Aarthw%20OR%20author%3AJohannesGaessler%20OR%20author%3AORippler%20OR%20author%3Aruixiang63%20OR%20author%3Axctan%20OR%20author%3Aallozaur%20OR%20author%3Ayomaytk%20OR%20author%3Aaendk%20OR%20author%3Agaugarg-nv%20OR%20author%3Ataronaeo%20OR%20author%3Aforforever73%20OR%20author%3Alhez%20OR%20author%3Anetrunnereve%20OR%20author%3Afairydreaming)%20sort%3Aupdated-desc)
[Manifesto](https://github.com/ggml-org/llama.cpp/discussions/205) / [ggml](https://github.com/ggml-org/ggml) / [ops](https://github.com/ggml-org/llama.cpp/blob/master/docs/ops.md)
LLM inference in C/C++
@@ -20,8 +20,8 @@ environment variables, and produce **token-identical** output to mainline.
| Env var | What it does |
| --- | --- |
| `GGML_CUDA_REGISTER_HOST=1` | Page-locks (pins) the mmap'd CPU expert weights so host->device copies go straight over DMA instead of through the driver's hidden bounce buffer (~6-7 -> ~20 GB/s). Works on CUDA and Vulkan (also honored as `GGML_VK_REGISTER_HOST`). Note: it is a presence check, so `=0` still enables it. |
| `GGML_SCHED_PREFETCH_EXPERTS=1` | Prefetches each layer's experts on a second stream, so the weight uploads overlap compute instead of stalling the GPU. **CUDA only** - on the Vulkan backend the second backend instance shares one device queue, giving no overlap, so this regresses (see Vulkan note below). Leave it off on Vulkan. |
| `GGML_CUDA_REGISTER_HOST=1` | Page-locks (pins) the mmap'd CPU expert weights so host→device copies go straight over DMA instead of through the driver's hidden bounce buffer (~6–7 → ~20 GB/s). |
| `GGML_SCHED_PREFETCH_EXPERTS=1` | Prefetches each layer's experts on a second CUDA stream, so the weight uploads overlap compute instead of stalling the GPU. |
### Benchmark
@@ -40,49 +40,6 @@ Result: **~1143 → ~1880 t/s** prefill (**+64%**) — same GPU, same settings,
Branches: [`fable5/host-register`](https://github.com/thecodacus/llama.cpp/tree/fable5/host-register) (pinning only) · [`fable5/prefetch-experts`](https://github.com/thecodacus/llama.cpp/tree/fable5/prefetch-experts) (both — this branch).
### Vulkan (older AMD, e.g. RX 580 / Polaris)
On the Vulkan backend the CUDA-oriented flags above behave differently, and this fork adds a
Polaris-specific flash-attention fix. Findings on an **RX 580 8GB** (Polaris / GCN, PCIe 3.0 x16,
no fp16, no matrix cores) with **Qwen3.5-35B-A3B Q4_K_M**, `-b 2048 -ub 2048`:
- **Flash-attention `mask_opt` is now enabled for GCN large head sizes (this fork's own change).**
Upstream disables it on GCN; it is a **lossless** win in high-context prefill - it skips
fully-masked causal blocks and the per-block mask add on fully-visible ones, which is real work on
a card whose attention is compute-bound (no matrix cores). Auto-on, no flag. On Qwen3.5-35B
(head_dim 256): pp2048 **+8% @ 16k, +12% @ 32k**, growing with depth; perplexity bit-identical.
- **For a long-running server, load with `--no-mmap`, not pinning.** `GGML_CUDA_REGISTER_HOST=1`
(pinning) gives ~+17% in an isolated `llama-bench` run, but in a server the RADV host-pointer
import fails and the fallback pre-stage buffer allocation fails for large / co-resident models, so
it silently reverts to slow staging (and can trip warnings/OOM). `--no-mmap` (weights in RAM) is
both faster and clean there. Pinning is still fine for one-off `llama-bench` numbers.
- **`GGML_SCHED_PREFETCH_EXPERTS=1` regresses - do not use it** on Vulkan (its second backend shares
one device queue, so uploads never overlap compute).
- **`-b 2048 -ub 2048` is the biggest prefill lever** (the default `-ub 512` roughly halves pp).
- **Tune `--n-cpu-moe` to context length.** Keep some expert layers resident in spare VRAM for short
prompts (e.g. `ncmoe 28` on the 35B, ~+5% over all-host); at long context the KV cache needs that
VRAM, so raise it (`ncmoe 40`, all experts on host). Keep flash attention on (`-fa 1`).
Prefill throughput (isolated `llama-bench`, pinned unless noted):
| Config | pp2048 (t/s) |
| --- | ---: |
| baseline, unpinned, `ncmoe 40` | ~252 |
| pinned, `ncmoe 40` | ~294 |
| pinned, `ncmoe 28` | ~308 |
| server default (`--no-mmap`, `ncmoe 40`) | ~285 |
| + `mask_opt`, @ 32k context | **+12%** |
**Recommended RX 580 / Polaris serving command** (per model):
```bash
llama-server -hf <repo>:<quant> --no-mmap -ngl 99 --n-cpu-moe 40 -b 2048 -ub 2048 -fa 1
```
Lower `--n-cpu-moe` (e.g. 28) if the model plus your context budget leave spare VRAM; keep it high
for long-context / agentic use. At long context the bottleneck is attention compute (GPU-bound), so
`mask_opt` (above) is where the remaining prefill wins come from, not the MoE-transfer path.
## Recent API changes
- [Changelog for `libllama` API](https://github.com/ggml-org/llama.cpp/issues/9289)
common/CMakeLists.txt
-8
View File
@@ -100,10 +100,6 @@ add_library(${TARGET}
sampling.h
speculative.cpp
speculative.h
subproc.cpp
subproc.h
trie.cpp
trie.h
unicode.cpp
unicode.h
jinja/lexer.cpp
@@ -129,10 +125,6 @@ set_target_properties(${TARGET} PROPERTIES
target_include_directories(${TARGET} PUBLIC . ../vendor)
target_compile_features (${TARGET} PUBLIC cxx_std_17)
if (LLAMA_SUBPROCESS)
target_compile_definitions(${TARGET} PUBLIC LLAMA_SUBPROCESS)
endif()
if (BUILD_SHARED_LIBS)
set_target_properties(${TARGET} PROPERTIES POSITION_INDEPENDENT_CODE ON)
common/arg.cpp
+38 -291
View File
@@ -5,7 +5,6 @@
#include "common.h"
#include "download.h"
#include "json-schema-to-grammar.h"
#include "llama.h"
#include "log.h"
#include "sampling.h"
#include "speculative.h"
@@ -28,7 +27,6 @@
#include <cinttypes>
#include <climits>
#include <cstdarg>
#include <filesystem>
#include <fstream>
#include <list>
#include <regex>
@@ -352,10 +350,6 @@ static std::string get_default_local_path(const std::string & url) {
return fs_get_cache_file(string_split<std::string>(f, '/').back());
}
static bool spec_types_is_default(const common_params & params) {
return params.speculative.types == std::vector<enum common_speculative_type>{COMMON_SPECULATIVE_TYPE_NONE};
}
common_models_handler common_models_handler_init(const common_params & params, llama_example curr_ex) {
common_download_hf_plan plan;
common_download_hf_plan plan_spec;
@@ -366,14 +360,6 @@ common_models_handler common_models_handler_init(const common_params & params, l
params.speculative.types.end(),
COMMON_SPECULATIVE_TYPE_DRAFT_MTP) != params.speculative.types.end();
const bool spec_type_draft_dflash = std::find(params.speculative.types.begin(),
params.speculative.types.end(),
COMMON_SPECULATIVE_TYPE_DRAFT_DFLASH) != params.speculative.types.end();
const bool spec_type_draft_eagle3 = std::find(params.speculative.types.begin(),
params.speculative.types.end(),
COMMON_SPECULATIVE_TYPE_DRAFT_EAGLE3) != params.speculative.types.end();
// only download mmproj if the current example is using it
bool use_mmproj = false;
for (const auto & ex : mmproj_examples) {
@@ -386,8 +372,6 @@ common_models_handler common_models_handler_init(const common_params & params, l
opts.bearer_token = params.hf_token;
opts.offline = params.offline;
opts.download_mtp = spec_type_draft_mtp;
opts.download_eagle3 = spec_type_draft_eagle3;
opts.download_dflash = spec_type_draft_dflash;
opts.download_mmproj = use_mmproj && !params.no_mmproj
&& params.mmproj.path.empty() && params.mmproj.url.empty();
@@ -396,14 +380,7 @@ common_models_handler common_models_handler_init(const common_params & params, l
}
if (!params.speculative.draft.mparams.hf_repo.empty()) {
// without a requested type, discover every sidecar the draft repo ships to infer the type later
auto opts_spec = opts;
if (spec_types_is_default(params)) {
opts_spec.download_mtp = true;
opts_spec.download_dflash = true;
opts_spec.download_eagle3 = true;
}
plan_spec = common_download_get_hf_plan(params.speculative.draft.mparams, opts_spec);
plan_spec = common_download_get_hf_plan(params.speculative.draft.mparams, opts);
}
if (!params.vocoder.model.hf_repo.empty()) {
@@ -510,15 +487,12 @@ void common_models_handler_apply(common_models_handler & handler, common_params
task.opts = opts;
tasks.push_back(task);
}
bool had_spec_url = false;
if (!params.speculative.draft.mparams.url.empty()) {
common_download_task task;
task.url = params.speculative.draft.mparams.url;
task.local_path = params.speculative.draft.mparams.path;
task.opts = opts;
tasks.push_back(task);
had_spec_url = true;
}
// handle hf_plan tasks
@@ -538,67 +512,6 @@ void common_models_handler_apply(common_models_handler & handler, common_params
});
}
};
// infer the speculative type from the sidecar shipped by the draft repo when none is requested
if (spec_types_is_default(params)) {
if (!plan_spec.mtp.local_path.empty()) {
params.speculative.types = { COMMON_SPECULATIVE_TYPE_DRAFT_MTP };
plan_spec.dflash = {};
plan_spec.eagle3 = {};
} else if (!plan_spec.dflash.local_path.empty()) {
params.speculative.types = { COMMON_SPECULATIVE_TYPE_DRAFT_DFLASH };
plan_spec.eagle3 = {};
} else if (!plan_spec.eagle3.local_path.empty()) {
params.speculative.types = { COMMON_SPECULATIVE_TYPE_DRAFT_EAGLE3 };
}
}
// when a sidecar type is requested, the draft repo resolves to its sidecar instead of a full model
const bool spec_sidecar_found = !plan_spec.mtp.local_path.empty() ||
!plan_spec.dflash.local_path.empty() ||
!plan_spec.eagle3.local_path.empty();
if (!plan_spec.mtp.local_path.empty() && !had_spec_url) {
tasks.emplace_back(plan_spec.mtp, opts, [&]() {
// only use the discovered MTP head when no draft path is set yet
if (params.speculative.draft.mparams.path.empty()) {
params.speculative.draft.mparams.path = hf_cache::finalize_file(plan_spec.mtp);
} else {
hf_cache::finalize_file(plan_spec.mtp);
}
});
}
if (!plan_spec.dflash.local_path.empty() && !had_spec_url) {
tasks.emplace_back(plan_spec.dflash, opts, [&]() {
// only use the discovered DFlash sidecar when no draft path is set yet
if (params.speculative.draft.mparams.path.empty()) {
params.speculative.draft.mparams.path = hf_cache::finalize_file(plan_spec.dflash);
} else {
hf_cache::finalize_file(plan_spec.dflash);
}
});
}
if (!plan_spec.eagle3.local_path.empty() && !had_spec_url) {
tasks.emplace_back(plan_spec.eagle3, opts, [&]() {
// only use the discovered Eagle3 sidecar when no draft path is set yet
if (params.speculative.draft.mparams.path.empty()) {
params.speculative.draft.mparams.path = hf_cache::finalize_file(plan_spec.eagle3);
} else {
hf_cache::finalize_file(plan_spec.eagle3);
}
});
}
// handle plan_spec (e.g. --spec-draft-hf)
if (!plan_spec.model_files.empty() && !had_spec_url && !spec_sidecar_found) {
add_tasks(plan_spec.model_files, plan_spec.primary, params.speculative.draft.mparams);
had_spec_url = true;
}
// handle vocoder plan (e.g. --hf-repo-v)
if (!plan_voc.model_files.empty()) {
add_tasks(plan_voc.model_files, plan_voc.primary, params.vocoder.model);
}
if (!plan.model_files.empty()) {
add_tasks(plan.model_files, plan.primary, params.model);
}
@@ -607,7 +520,7 @@ void common_models_handler_apply(common_models_handler & handler, common_params
params.mmproj.path = hf_cache::finalize_file(plan.mmproj);
});
}
if (!plan.mtp.local_path.empty() && !had_spec_url) {
if (!plan.mtp.local_path.empty()) {
tasks.emplace_back(plan.mtp, opts, [&]() {
// only fall back to the discovered MTP head when no draft was explicitly provided
if (params.speculative.draft.mparams.empty()) {
@@ -617,26 +530,6 @@ void common_models_handler_apply(common_models_handler & handler, common_params
}
});
}
if (!plan.dflash.local_path.empty() && !had_spec_url) {
tasks.emplace_back(plan.dflash, opts, [&]() {
// only fall back to the discovered DFlash sidecar when no draft was explicitly provided
if (params.speculative.draft.mparams.empty()) {
params.speculative.draft.mparams.path = hf_cache::finalize_file(plan.dflash);
} else {
hf_cache::finalize_file(plan.dflash);
}
});
}
if (!plan.eagle3.local_path.empty() && !had_spec_url) {
tasks.emplace_back(plan.eagle3, opts, [&]() {
// only fall back to the discovered Eagle3 sidecar when no draft was explicitly provided
if (params.speculative.draft.mparams.empty()) {
params.speculative.draft.mparams.path = hf_cache::finalize_file(plan.eagle3);
} else {
hf_cache::finalize_file(plan.eagle3);
}
});
}
if (!plan.preset.local_path.empty()) {
tasks.emplace_back(plan.preset, opts, [&]() {
// if HF repo is a preset repo, we simply run server in router mode with the preset.ini file
@@ -646,6 +539,16 @@ void common_models_handler_apply(common_models_handler & handler, common_params
});
}
// handle plan_spec (e.g. --spec-draft-hf)
if (!plan_spec.model_files.empty()) {
add_tasks(plan_spec.model_files, plan_spec.primary, params.speculative.draft.mparams);
}
// handle vocoder plan (e.g. --hf-repo-v)
if (!plan_voc.model_files.empty()) {
add_tasks(plan_voc.model_files, plan_voc.primary, params.vocoder.model);
}
// run all tasks in parallel
if (!params.offline) {
// if duplicated files are found, only download once (but still call on_done for each task)
@@ -658,7 +561,6 @@ void common_models_handler_apply(common_models_handler & handler, common_params
}
std::vector<common_download_task> unique_tasks_vec;
for (auto & pair : unique_tasks) {
LOG_DBG("download task: %s -> %s\n", pair.second->url.c_str(), pair.second->local_path.c_str());
unique_tasks_vec.push_back(*pair.second);
}
common_download_run_tasks(unique_tasks_vec);
@@ -786,20 +688,9 @@ static bool common_params_parse_ex(int argc, char ** argv, common_params_context
arg.c_str(), e.what(), opt.to_string().c_str()));
}
}
// TODO: remove this check after deprecating --mmap|mlock|dio
auto has_arg = [&](std::initializer_list<const char *> names) {
return std::any_of(names.begin(), names.end(), [&](const char * name) {
return seen_args.count(name);
});
};
if (has_arg({"-lm", "--load-mode"}) &&
has_arg({"--mlock", "--mmap", "--no-mmap", "-dio", "--direct-io", "-ndio", "--no-direct-io"})) {
LOG_WRN("DEPRECATED: `--load-mode` and `--mlock`/`--mmap`/`--direct-io` should not be combined; only the last flag on the command line will take effect\n");
}
};
// parse all CLI args now, so that -hf is available below for remote preset resolution
// parse the first time to get -hf option (used for remote preset)
parse_cli_args();
postprocess_cpu_params(params.cpuparams, nullptr);
@@ -827,8 +718,9 @@ static bool common_params_parse_ex(int argc, char ** argv, common_params_context
// model is required (except for server)
// TODO @ngxson : maybe show a list of available models in CLI in this case
bool can_skip_model = params.usage || params.completion || !params.server_base.empty();
if (!can_skip_model && params.model.path.empty()) {
if (params.model.path.empty()
&& !params.usage
&& !params.completion) {
throw std::invalid_argument("error: --model is required\n");
}
}
@@ -850,12 +742,6 @@ static bool common_params_parse_ex(int argc, char ** argv, common_params_context
params.kv_overrides.back().key[0] = 0;
}
const bool mcp_enabled = !params.mcp_servers_config.empty() || !params.mcp_servers_json.empty();
if ((!params.server_tools.empty() || mcp_enabled) && !params.cors_origins_explicit) {
LOG_WRN("server tools or MCP servers are enabled, using localhost as default CORS origin (change via --cors-origins)\n");
params.cors_origins = "localhost";
}
// pad tensor_buft_overrides for llama_params_fit:
const size_t ntbo = llama_max_tensor_buft_overrides();
while (params.tensor_buft_overrides.size() < ntbo) {
@@ -1185,7 +1071,6 @@ bool common_params_parse(int argc, char ** argv, common_params & params, llama_e
if (ctx_arg.print_usage) {
ctx_arg.print_usage(argc, argv);
}
common_log_flush(common_log_main());
exit(0);
}
if (ctx_arg.params.completion) {
@@ -1287,8 +1172,6 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
params.sampling.temp = 0.2; // lower temp by default for better quality
} else if (ex == LLAMA_EXAMPLE_SERVER) {
params.n_parallel = -1; // auto by default
} else if (ex == LLAMA_EXAMPLE_TOKENIZE) {
params.parse_special = true; // parse special tokens by default, like the old tokenize tool
}
params.use_color = tty_can_use_colors();
@@ -1357,13 +1240,6 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
params.completion = true;
}
));
add_opt(common_arg(
{"--server-base"}, "URL",
string_format("connect to this server instead of starting a new one, example: 'http://localhost:8080' (default: none)"),
[](common_params & params, const std::string & value) {
params.server_base = value;
}
).set_examples({LLAMA_EXAMPLE_CLI}));
add_opt(common_arg(
{"--verbose-prompt"},
string_format("print a verbose prompt before generation (default: %s)", params.verbose_prompt ? "true" : "false"),
@@ -2508,45 +2384,27 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
}
add_opt(common_arg(
{"--mlock"},
"DEPRECATED in favor of `--load-mode`: mmap + force system to keep model in RAM rather than swapping or compressing",
"force system to keep model in RAM rather than swapping or compressing",
[](common_params & params) {
LOG_WRN("DEPRECATED: --mlock is deprecated. use --load-mode mlock instead\n");
params.load_mode = LLAMA_LOAD_MODE_MLOCK;
params.use_mlock = true;
}
).set_env("LLAMA_ARG_MLOCK"));
add_opt(common_arg(
{"--mmap"},
{"--no-mmap"},
"DEPRECATED in favor of `--load-mode`: whether to memory-map model. (if mmap disabled, slower load but may reduce pageouts if not using mlock)",
string_format("whether to memory-map model. (if mmap disabled, slower load but may reduce pageouts if not using mlock) (default: %s)", params.use_mmap ? "enabled" : "disabled"),
[](common_params & params, bool value) {
LOG_WRN("DEPRECATED: --mmap and --no-mmap are deprecated. use --load-mode mmap instead\n");
params.load_mode = value ? LLAMA_LOAD_MODE_MMAP : LLAMA_LOAD_MODE_NONE;
params.use_mmap = value;
}
).set_env("LLAMA_ARG_MMAP"));
add_opt(common_arg(
{"-dio", "--direct-io"},
{"-ndio", "--no-direct-io"},
"DEPRECATED in favor of `--load-mode`: use DirectIO if available",
string_format("use DirectIO if available. (default: %s)", params.use_direct_io ? "enabled" : "disabled"),
[](common_params & params, bool value) {
LOG_WRN("DEPRECATED: --direct-io and --no-direct-io are deprecated. use --load-mode dio instead\n");
params.load_mode = value ? LLAMA_LOAD_MODE_DIRECT_IO : LLAMA_LOAD_MODE_NONE;
params.use_direct_io = value;
}
).set_env("LLAMA_ARG_DIO"));
add_opt(common_arg(
{"-lm", "--load-mode"}, "MODE",
"model loading mode (default: mmap)\n"
"- none: no special loading mode\n"
"- mmap: memory-map model (if mmap disabled, slower load but may reduce pageouts if not using mlock)\n"
"- mlock: mmap + force system to keep model in RAM rather than swapping or compressing\n"
"- dio: use DirectIO if available\n",
[](common_params & params, const std::string & value) {
/**/ if (value == "none") { params.load_mode = LLAMA_LOAD_MODE_NONE; }
else if (value == "mmap") { params.load_mode = LLAMA_LOAD_MODE_MMAP; }
else if (value == "mlock") { params.load_mode = LLAMA_LOAD_MODE_MLOCK; }
else if (value == "dio") { params.load_mode = LLAMA_LOAD_MODE_DIRECT_IO; }
else { throw std::invalid_argument("invalid value"); }
}
).set_env("LLAMA_ARG_LOAD_MODE"));
add_opt(common_arg(
{"--numa"}, "TYPE",
"attempt optimizations that help on some NUMA systems\n"
@@ -2606,17 +2464,15 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
).set_env("LLAMA_ARG_CPU_MOE"));
add_opt(common_arg(
{"-ncmoe", "--n-cpu-moe"}, "N",
"keep the Mixture of Experts (MoE) weights of the first N layers in the CPU; "
"fractional N offloads part of the boundary layer at tensor granularity",
[](common_params & params, const std::string & value) {
const double n = std::stod(value);
if (n < 0) {
"keep the Mixture of Experts (MoE) weights of the first N layers in the CPU",
[](common_params & params, int value) {
if (value < 0) {
throw std::invalid_argument("invalid value");
}
for (const std::string & re : llm_ffn_exps_cpu_block_regexes(n)) {
for (int i = 0; i < value; ++i) {
// keep strings alive and avoid leaking memory by storing them in a static vector
static std::list<std::string> buft_overrides;
buft_overrides.push_back(re);
buft_overrides.push_back(llm_ffn_exps_block_regex(i));
params.tensor_buft_overrides.push_back({buft_overrides.back().c_str(), ggml_backend_cpu_buffer_type()});
}
}
@@ -2876,14 +2732,14 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
[](common_params & params, const std::string & value) {
params.model.path = value;
}
).set_examples({LLAMA_EXAMPLE_COMMON, LLAMA_EXAMPLE_EXPORT_LORA, LLAMA_EXAMPLE_DOWNLOAD, LLAMA_EXAMPLE_TOKENIZE}).set_env("LLAMA_ARG_MODEL"));
).set_examples({LLAMA_EXAMPLE_COMMON, LLAMA_EXAMPLE_EXPORT_LORA, LLAMA_EXAMPLE_DOWNLOAD}).set_env("LLAMA_ARG_MODEL"));
add_opt(common_arg(
{"-mu", "--model-url"}, "MODEL_URL",
"model download url (default: unused)",
[](common_params & params, const std::string & value) {
params.model.url = value;
}
).set_examples({LLAMA_EXAMPLE_COMMON, LLAMA_EXAMPLE_DOWNLOAD, LLAMA_EXAMPLE_TOKENIZE}).set_env("LLAMA_ARG_MODEL_URL"));
).set_examples({LLAMA_EXAMPLE_COMMON, LLAMA_EXAMPLE_DOWNLOAD}).set_env("LLAMA_ARG_MODEL_URL"));
add_opt(common_arg(
{ "-dr", "--docker-repo" }, "[<repo>/]<model>[:quant]",
"Docker Hub model repository. repo is optional, default to ai/. quant is optional, default to :latest.\n"
@@ -2892,7 +2748,7 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
[](common_params & params, const std::string & value) {
params.model.docker_repo = value;
}
).set_examples({LLAMA_EXAMPLE_COMMON, LLAMA_EXAMPLE_DOWNLOAD, LLAMA_EXAMPLE_TOKENIZE}).set_env("LLAMA_ARG_DOCKER_REPO"));
).set_examples({LLAMA_EXAMPLE_COMMON, LLAMA_EXAMPLE_DOWNLOAD}).set_env("LLAMA_ARG_DOCKER_REPO"));
add_opt(common_arg(
{"-hf", "-hfr", "--hf-repo"}, "<user>/<model>[:quant]",
"Hugging Face model repository; quant is optional, case-insensitive, default to Q4_K_M, or falls back to the first file in the repo if Q4_K_M doesn't exist.\n"
@@ -2902,14 +2758,14 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
[](common_params & params, const std::string & value) {
params.model.hf_repo = value;
}
).set_examples({LLAMA_EXAMPLE_COMMON, LLAMA_EXAMPLE_DOWNLOAD, LLAMA_EXAMPLE_TOKENIZE}).set_env("LLAMA_ARG_HF_REPO"));
).set_examples({LLAMA_EXAMPLE_COMMON, LLAMA_EXAMPLE_DOWNLOAD}).set_env("LLAMA_ARG_HF_REPO"));
add_opt(common_arg(
{"-hff", "--hf-file"}, "FILE",
"Hugging Face model file. If specified, it will override the quant in --hf-repo (default: unused)",
[](common_params & params, const std::string & value) {
params.model.hf_file = value;
}
).set_examples({LLAMA_EXAMPLE_COMMON, LLAMA_EXAMPLE_DOWNLOAD, LLAMA_EXAMPLE_TOKENIZE}).set_env("LLAMA_ARG_HF_FILE"));
).set_examples({LLAMA_EXAMPLE_COMMON, LLAMA_EXAMPLE_DOWNLOAD}).set_env("LLAMA_ARG_HF_FILE"));
add_opt(common_arg(
{"-hfv", "-hfrv", "--hf-repo-v"}, "<user>/<model>[:quant]",
"Hugging Face model repository for the vocoder model (default: unused)",
@@ -2930,7 +2786,7 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
[](common_params & params, const std::string & value) {
params.hf_token = value;
}
).set_examples({LLAMA_EXAMPLE_COMMON, LLAMA_EXAMPLE_DOWNLOAD, LLAMA_EXAMPLE_TOKENIZE}).set_env("HF_TOKEN"));
).set_examples({LLAMA_EXAMPLE_COMMON, LLAMA_EXAMPLE_DOWNLOAD}).set_env("HF_TOKEN"));
add_opt(common_arg(
{"--mtp"},
"also download the multi-token prediction (MTP) head, if available (default: unused)",
@@ -2938,20 +2794,6 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
params.speculative.types.push_back(COMMON_SPECULATIVE_TYPE_DRAFT_MTP);
}
).set_examples({LLAMA_EXAMPLE_DOWNLOAD}));
add_opt(common_arg(
{"--dflash"},
"also download the DFlash sidecar, if available (default: unused)",
[](common_params & params) {
params.speculative.types.push_back(COMMON_SPECULATIVE_TYPE_DRAFT_DFLASH);
}
).set_examples({LLAMA_EXAMPLE_DOWNLOAD}));
add_opt(common_arg(
{"--eagle3"},
"also download the Eagle3 sidecar, if available (default: unused)",
[](common_params & params) {
params.speculative.types.push_back(COMMON_SPECULATIVE_TYPE_DRAFT_EAGLE3);
}
).set_examples({LLAMA_EXAMPLE_DOWNLOAD}));
add_opt(common_arg(
{"--context-file"}, "FNAME",
"file to load context from (use comma-separated values to specify multiple files)",
@@ -3000,7 +2842,7 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
params.out_file = value;
}
).set_examples({LLAMA_EXAMPLE_IMATRIX, LLAMA_EXAMPLE_CVECTOR_GENERATOR, LLAMA_EXAMPLE_EXPORT_LORA, LLAMA_EXAMPLE_TTS, LLAMA_EXAMPLE_FINETUNE,
LLAMA_EXAMPLE_RESULTS, LLAMA_EXAMPLE_EXPORT_GRAPH_OPS, LLAMA_EXAMPLE_CLI}));
LLAMA_EXAMPLE_RESULTS, LLAMA_EXAMPLE_EXPORT_GRAPH_OPS}));
add_opt(common_arg(
{"-ofreq", "--output-frequency"}, "N",
string_format("output the imatrix every N iterations (default: %d)", params.n_out_freq),
@@ -3060,41 +2902,6 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
params.parse_special = true;
}
).set_examples({LLAMA_EXAMPLE_IMATRIX}));
add_opt(common_arg(
{"--ids"},
string_format("only print the token IDs, in a Python-parseable list form like [1, 2, 3] (default: %s)", params.tokenize_ids ? "true" : "false"),
[](common_params & params) {
params.tokenize_ids = true;
}
).set_examples({LLAMA_EXAMPLE_TOKENIZE}));
add_opt(common_arg(
{"--stdin"},
string_format("read the prompt from stdin (takes precedence over -f/--file and -p/--prompt) (default: %s)", params.tokenize_stdin ? "true" : "false"),
[](common_params & params) {
params.tokenize_stdin = true;
}
).set_examples({LLAMA_EXAMPLE_TOKENIZE}));
add_opt(common_arg(
{"--no-bos"},
string_format("do not add a BOS token to the prompt, even if the model normally uses one (default: %s)", params.tokenize_no_bos ? "true" : "false"),
[](common_params & params) {
params.tokenize_no_bos = true;
}
).set_examples({LLAMA_EXAMPLE_TOKENIZE}));
add_opt(common_arg(
{"--no-parse-special"},
string_format("do not parse special tokens (chat, tool, etc) (default: %s)", !params.parse_special ? "true" : "false"),
[](common_params & params) {
params.parse_special = false;
}
).set_examples({LLAMA_EXAMPLE_TOKENIZE}));
add_opt(common_arg(
{"--show-count"},
string_format("print the total number of tokens (default: %s)", params.tokenize_show_count ? "true" : "false"),
[](common_params & params) {
params.tokenize_show_count = true;
}
).set_examples({LLAMA_EXAMPLE_TOKENIZE}));
add_opt(common_arg(
{"-pps"},
string_format("is the prompt shared across parallel sequences (default: %s)", params.is_pp_shared ? "true" : "false"),
@@ -3189,42 +2996,6 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
params.public_path = value;
}
).set_examples({LLAMA_EXAMPLE_SERVER}).set_env("LLAMA_ARG_STATIC_PATH"));
add_opt(common_arg(
{"--cors-origins"}, "ORIGINS",
string_format(
"comma-separated list of allowed origins for CORS (default: %s)\n"
"if set to special value 'localhost', reflect the Origin header only if it is localhost",
params.cors_origins.c_str()),
[](common_params & params, const std::string & value) {
params.cors_origins = value;
params.cors_origins_explicit = true;
}
).set_examples({LLAMA_EXAMPLE_SERVER}).set_env("LLAMA_ARG_CORS_ORIGINS"));
add_opt(common_arg(
{"--cors-methods"}, "METHODS",
string_format("comma-separated list of allowed methods for CORS (default: %s)", params.cors_methods.c_str()),
[](common_params & params, const std::string & value) {
params.cors_methods = value;
}
).set_examples({LLAMA_EXAMPLE_SERVER}).set_env("LLAMA_ARG_CORS_METHODS"));
add_opt(common_arg(
{"--cors-headers"}, "HEADERS",
string_format("comma-separated list of allowed headers for CORS (default: %s)", params.cors_headers.c_str()),
[](common_params & params, const std::string & value) {
params.cors_headers = value;
}
).set_examples({LLAMA_EXAMPLE_SERVER}).set_env("LLAMA_ARG_CORS_HEADERS"));
add_opt(common_arg(
{"--cors-credentials"},
{"--no-cors-credentials"},
string_format(
"whether to allow credentials for CORS (default: %s)\n"
"note: if this is enabled and --cors-origins is set to * (default), the Origin header will be echoed back, and credentials will always be allowed",
params.cors_credentials ? "enabled" : "disabled"),
[](common_params & params, bool value) {
params.cors_credentials = value;
}
).set_examples({LLAMA_EXAMPLE_SERVER}).set_env("LLAMA_ARG_CORS_CREDENTIALS"));
add_opt(common_arg(
{"--api-prefix"}, "PREFIX",
string_format("prefix path the server serves from, without the trailing slash (default: %s)", params.api_prefix.c_str()),
@@ -3258,33 +3029,15 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
{"--tools"}, "TOOL1,TOOL2,...",
"experimental: whether to enable built-in tools for AI agents - do not enable in untrusted environments (default: no tools)\n"
"specify \"all\" to enable all tools\n"
"available tools: read_file, file_glob_search, grep_search, exec_shell_command, write_file, edit_file, get_datetime\n"
"note: for security reasons, this will limit --cors-origins to localhost by default",
"available tools: read_file, file_glob_search, grep_search, exec_shell_command, write_file, edit_file, apply_diff, get_datetime",
[](common_params & params, const std::string & value) {
params.server_tools = parse_csv_row(value);
}
).set_examples({LLAMA_EXAMPLE_SERVER}).set_env("LLAMA_ARG_TOOLS"));
add_opt(common_arg(
{"--mcp-servers-config"}, "PATH",
"experimental: path to JSON file with MCP server definitions (Cursor-compatible format) - do not enable in untrusted environments (default: none)\n"
"note: for security reasons, this will limit --cors-origins to localhost by default",
[](common_params & params, const std::string & value) {
params.mcp_servers_config = value;
}
).set_examples({LLAMA_EXAMPLE_SERVER}).set_env("LLAMA_ARG_MCP_SERVERS_CONFIG"));
add_opt(common_arg(
{"--mcp-servers-json"}, "JSON",
"experimental: inline JSON with MCP server definitions (Cursor-compatible format) - do not enable in untrusted environments (default: none)\n"
"note: for security reasons, this will limit --cors-origins to localhost by default",
[](common_params & params, const std::string & value) {
params.mcp_servers_json = value;
}
).set_examples({LLAMA_EXAMPLE_SERVER}).set_env("LLAMA_ARG_MCP_SERVERS_JSON"));
add_opt(common_arg(
{"-ag", "--agent"},
{"-no-ag", "--no-agent"},
"whether to enable CORS proxy and all built-in tools - do not enable in untrusted environments (default: disabled)\n"
"note: for security reasons, this will limit --cors-origins to localhost by default",
"whether to enable CORS proxy and all built-in tools - do not enable in untrusted environments (default: disabled)",
[](common_params & params, bool value) {
if (value) {
params.server_tools = {"all"};
@@ -3293,7 +3046,6 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
params.server_tools.clear();
params.ui_mcp_proxy = false;
}
// note: do not modify cors_origins here, as the options are not evaluated in order (user may explicitly set --cors-origins before --agent)
}
).set_examples({LLAMA_EXAMPLE_SERVER}).set_env("LLAMA_ARG_AGENT"));
add_opt(common_arg(
@@ -3699,14 +3451,9 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
).set_env("LLAMA_ARG_LOG_FILE"));
add_opt(common_arg(
{"--log-prompts-dir"}, "PATH",
"Log prompts to directory (auto-created if not present; only used for debugging, default: disabled)",
"Log prompts to directory (only used for debugging, default: disabled)",
[](common_params & params, const std::string & value) {
params.path_prompts_log_dir = value;
std::error_code ec;
std::filesystem::create_directories(value, ec);
if (ec) {
fprintf(stderr, "warning: failed to create prompts-log-dir '%s': %s\n", value.c_str(), ec.message().c_str());
}
}
).set_examples({LLAMA_EXAMPLE_SERVER, LLAMA_EXAMPLE_CLI}));
add_opt(common_arg(
@@ -3740,7 +3487,7 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
[](common_params & params) {
params.offline = true;
}
).set_examples({LLAMA_EXAMPLE_COMMON, LLAMA_EXAMPLE_DOWNLOAD, LLAMA_EXAMPLE_TOKENIZE}).set_env("LLAMA_ARG_OFFLINE"));
).set_examples({LLAMA_EXAMPLE_COMMON, LLAMA_EXAMPLE_DOWNLOAD}).set_env("LLAMA_ARG_OFFLINE"));
add_opt(common_arg(
{"-lv", "--verbosity", "--log-verbosity"}, "N",
string_format("Set the verbosity threshold. Messages with a higher verbosity will be ignored. Values:\n"
common/chat-auto-parser-generator.cpp
+2 -15
View File
@@ -47,8 +47,6 @@ common_chat_params peg_generator::generate_parser(const common_chat_template &
data.generation_prompt = common_chat_template_generation_prompt(tmpl, inputs);
data.format = COMMON_CHAT_FORMAT_PEG_NATIVE;
data.preserved_tokens = autoparser.preserved_tokens;
data.additional_stops.insert(data.additional_stops.end(),
autoparser.additional_stops.begin(), autoparser.additional_stops.end());
std::string parser_generation_prompt = data.generation_prompt;
@@ -149,8 +147,7 @@ common_peg_arena autoparser::build_parser(const generation_params & inputs, cons
} else {
parser = content.build_parser(ctx);
}
const std::string reasoning_start = trim_whitespace(reasoning.start);
return pure_content ? p.prefix(generation_prompt, reasoning_start) + parser : p.prefix(generation_prompt, reasoning_start) << parser;
return pure_content ? p.prefix(generation_prompt, reasoning.start) + parser : p.prefix(generation_prompt, reasoning.start) << parser;
});
}
@@ -264,10 +261,6 @@ common_peg_parser analyze_tools::build_func_parser(common_chat_peg_builder & p,
bool matched_atomic = false;
common_peg_parser func_parser = p.eps();
if (!function.args_separator.empty()) {
open = open + p.space() + p.literal(function.args_separator);
}
if (!function.name_suffix.empty()) {
func_parser = open + call_id_section + p.space() + args;
matched_atomic = true;
@@ -288,13 +281,7 @@ common_peg_parser analyze_tools::build_func_parser(common_chat_peg_builder & p,
// we only emit tool_close when we can actually see the closing marker. This prevents
// premature closing during partial parsing when we've seen e.g. "</" which could be
// either "</tool_call>" (end) or "<arg_key>" prefix that failed to match.
// Laguna (v4): the model may emit whitespace between the last </arg_value> and
// </tool_call> even though the template renders them tight. Tolerate optional
// leading space in the close lookahead so the tool call still closes.
auto close_peek = arguments.tolerate_intertag_whitespace
? p.peek(p.space() + p.literal(format.per_call_end))
: p.peek(p.literal(format.per_call_end));
func_parser = func_parser + p.tool_close(close_peek);
func_parser = func_parser + p.tool_close(p.peek(p.literal(format.per_call_end)));
} else {
func_parser = func_parser + p.tool_close(p.space()); // force this to process tool closing callbacks in mapper
}
common/chat-auto-parser.h
+3 -6
View File
@@ -192,10 +192,9 @@ struct tool_format_analysis {
};
struct tool_function_analysis {
std::string name_prefix; // e.g., "<function=", "\"name\": \"", "functions."
std::string name_suffix; // e.g., ">", "\"", ":0"
std::string args_separator; // e.g., "<tool_sep>" (marker between function name and arguments)
std::string close; // e.g., "</function>", "" (for tag-based)
std::string name_prefix; // e.g., "<function=", "\"name\": \"", "functions."
std::string name_suffix; // e.g., ">", "\"", ":0"
std::string close; // e.g., "</function>", "" (for tag-based)
};
struct tool_arguments_analysis {
@@ -206,7 +205,6 @@ struct tool_arguments_analysis {
std::string value_prefix; // e.g., "", "<arg_value>", ""
std::string value_suffix; // e.g., "</param>", "</arg_value>", ""
std::string separator; // e.g., "", "\n", ","
bool tolerate_intertag_whitespace = false; // Laguna: accept optional whitespace between arg tags
};
struct tool_id_analysis {
@@ -389,7 +387,6 @@ struct autoparser {
// Preserved tokens for tokenizer (union of all non-empty markers)
std::vector<std::string> preserved_tokens;
std::vector<std::string> additional_stops; // literal stop strings (e.g. Laguna </assistant>) caught however tokenized
autoparser() = default;
common/chat-diff-analyzer.cpp
+2 -52
View File
@@ -124,16 +124,16 @@ static std::vector<std::function<void(const common_chat_template & tmpl, autopar
analysis.tools.format.section_end = "";
analysis.tools.format.per_call_start = "<TOOLCALL>";
analysis.tools.format.per_call_end = "</TOOLCALL>";
analysis.tools.format.tools_array_wrapped = true;
analysis.content.mode = content_mode::PLAIN;
analysis.content.start = "";
analysis.content.end = "";
analysis.reasoning.mode = reasoning_mode::TAG_BASED;
analysis.reasoning.start = "<think>\n";
analysis.reasoning.start = "<think>\n\n";
analysis.reasoning.end = "</think>";
analysis.assistant_start = "<SPECIAL_11>Assistant";
analysis.user_start = "<SPECIAL_11>User";
analysis.preserved_tokens.clear();
analysis.preserved_tokens.push_back("<SPECIAL_12>");
analysis.preserved_tokens.push_back("<SPECIAL_11>");
analysis.preserved_tokens.push_back("</think>");
analysis.preserved_tokens.push_back("<TOOLCALL>");
@@ -173,26 +173,6 @@ static std::vector<std::function<void(const common_chat_template & tmpl, autopar
LOG_DBG(ANSI_ORANGE "[Patch: JSON name/parameters tool instruction]\n" ANSI_RESET);
}
},
// Laguna (poolside) - the v4 chat template renders reasoning and tool-arg
// delimiters with formatting whitespace ("<think>\n", "</arg_value>\n") that
// the model does not emit, so the inferred delimiters carry a spurious
// newline and never match the model output. Trim to the bare tag. (v8
// renders without the whitespace, so this is a no-op there.)
[](const common_chat_template & tmpl, autoparser & analysis) -> void {
if (tmpl.src.find("laguna_glm_thinking") != std::string::npos) {
analysis.reasoning.start = trim_whitespace(analysis.reasoning.start);
analysis.reasoning.end = trim_whitespace(analysis.reasoning.end);
analysis.tools.arguments.value_prefix = trim_whitespace(analysis.tools.arguments.value_prefix);
analysis.tools.arguments.value_suffix = trim_whitespace(analysis.tools.arguments.value_suffix);
analysis.tools.arguments.separator = trim_whitespace(analysis.tools.arguments.separator);
analysis.tools.arguments.tolerate_intertag_whitespace = true;
// The CONTROL/eot </assistant> token only halts generation when emitted as the
// single token; after tool calls the model can spell it out as text tokens.
// A literal stop string catches it either way.
analysis.additional_stops.push_back("</assistant>");
LOG_DBG(ANSI_ORANGE "[Patch: Laguna]\n" ANSI_RESET);
}
},
});
@@ -279,7 +259,6 @@ void autoparser::analyze_template(const common_chat_template & tmpl) {
LOG_DBG("per_call_end: '%s'\n", tools.format.per_call_end.c_str());
LOG_DBG("func_name_prefix: '%s'\n", tools.function.name_prefix.c_str());
LOG_DBG("func_name_suffix: '%s'\n", tools.function.name_suffix.c_str());
LOG_DBG("func_args_separator: '%s'\n", tools.function.args_separator.c_str());
LOG_DBG("func_close: '%s'\n", tools.function.close.c_str());
LOG_DBG("call_id_prefix: '%s'\n", tools.call_id.prefix.c_str());
LOG_DBG("call_id_suffix: '%s'\n", tools.call_id.suffix.c_str());
@@ -323,7 +302,6 @@ void autoparser::collect_preserved_tokens() {
add_token(tools.format.per_call_end);
add_token(tools.function.name_prefix);
add_token(tools.function.name_suffix);
add_token(tools.function.args_separator);
add_token(tools.function.close);
add_token(tools.arguments.start);
add_token(tools.arguments.end);
@@ -1073,23 +1051,6 @@ void analyze_tools::check_per_call_markers() {
format.section_start.clear();
format.section_end.clear();
}
if (!format.per_call_end.empty()) {
auto count_occurrences = [](const std::string & haystack, const std::string & needle) {
size_t count = 0;
for (size_t pos = haystack.find(needle); pos != std::string::npos;
pos = haystack.find(needle, pos + needle.size())) {
count++;
}
return count;
};
size_t calls_one = count_occurrences(one_vs_two->output_A, format.per_call_end);
size_t calls_two = count_occurrences(one_vs_two->output_B, format.per_call_end);
if (calls_one > 0 && calls_one == calls_two) {
format.section_end = format.per_call_end;
format.per_call_end.clear();
}
}
}
void analyze_tools::extract_function_markers() {
@@ -1171,17 +1132,6 @@ void analyze_tools::extract_function_markers() {
auto suf_result = suffix_parser.parse_and_extract(diff.suffix);
if (suf_result.result.success()) {
function.name_suffix += suf_result.tags["ext"];
auto arg_start = [&](common_peg_parser_builder &p) {
return p.marker() + p.space() + p.choice({ p.literal(ARG_FIRST), p.literal(ARG_SECOND) });
};
auto sep_parser = build_tagged_peg_parser([&](common_peg_parser_builder &p) {
return p.tag("sep", p.zero_or_more(p.negate(arg_start(p)) + p.any())) + arg_start(p);
});
auto sep_result = sep_parser.parse_and_extract(diff.suffix.substr(suf_result.tags["ext"].size()));
if (sep_result.result.success()) {
function.args_separator = trim_whitespace(sep_result.tags["sep"]);
}
}
}
common/chat.cpp
+24 -150
View File
@@ -15,13 +15,11 @@
#include "nlohmann/json.hpp"
#include <algorithm>
#include <cstdio>
#include <cstdlib>
#include <ctime>
#include <exception>
#include <functional>
#include <map>
#include <optional>
#include <sstream>
@@ -1024,7 +1022,7 @@ static common_chat_params common_chat_params_init_ministral_3(const common_chat_
data.supports_thinking = true;
data.thinking_start_tag = "[THINK]";
data.thinking_end_tags = {"[/THINK]"};
data.thinking_end_tag = "[/THINK]";
data.prompt = common_chat_template_direct_apply_impl(tmpl, inputs, /* messages_override = */ adjusted_messages);
data.generation_prompt = common_chat_template_generation_prompt_impl(tmpl, inputs, /* messages_override = */ adjusted_messages);
data.format = COMMON_CHAT_FORMAT_PEG_NATIVE;
@@ -1150,9 +1148,6 @@ static common_chat_params common_chat_params_init_gpt_oss(const common_chat_temp
data.format = COMMON_CHAT_FORMAT_PEG_NATIVE;
data.supports_thinking = true;
data.thinking_start_tag = "<|channel|>analysis<|message|>";
data.thinking_end_tags = {"<|end|>"};
// These special tokens are required to parse properly, so we include them
// even if parse_tool_calls is false.
data.preserved_tokens = {
@@ -1297,7 +1292,7 @@ static common_chat_params common_chat_params_init_gemma4(const common_chat_templ
data.format = COMMON_CHAT_FORMAT_PEG_GEMMA4;
data.supports_thinking = true;
data.thinking_start_tag = "<|channel>thought";
data.thinking_end_tags = {"<channel|>"};
data.thinking_end_tag = "<channel|>";
data.preserved_tokens = {
"<|channel>",
@@ -1572,7 +1567,7 @@ static common_chat_params common_chat_params_init_kimi_k2(const common_chat_temp
const std::string GEN_PROMPT = "<|im_assistant|>assistant<|im_middle|>";
data.thinking_start_tag = THINK_START;
data.thinking_end_tags = {THINK_END};
data.thinking_end_tag = THINK_END;
if (inputs.has_continuation()) {
const auto & msg = inputs.continue_msg;
@@ -1706,7 +1701,7 @@ static common_chat_params common_chat_params_init_lfm2(const common_chat_templat
}
data.thinking_start_tag = THINK_START;
data.thinking_end_tags = {THINK_END};
data.thinking_end_tag = THINK_END;
auto has_tools = inputs.tools.is_array() && !inputs.tools.empty();
auto has_response_format = !inputs.json_schema.is_null() && inputs.json_schema.is_object();
@@ -1860,93 +1855,16 @@ static common_chat_params common_chat_params_init_gigachat_v3(
return data;
}
// The DeepSeek V4 reference implementation renders consecutive tool results into a single
// user block, ordered by the tool call order of the preceding assistant message (matched
// by tool call id) rather than by the order they appear in the conversation.
static json deepseek_v4_sort_tool_results(const json & messages) {
json adjusted = messages;
std::map<std::string, size_t> call_order;
for (size_t i = 0; i < adjusted.size();) {
const auto & msg = adjusted[i];
const auto role = msg.value("role", "");
if (role == "assistant" && msg.contains("tool_calls") &&
msg.at("tool_calls").is_array() && !msg.at("tool_calls").empty()) {
call_order.clear();
const auto & tool_calls = msg.at("tool_calls");
for (size_t idx = 0; idx < tool_calls.size(); idx++) {
auto id = tool_calls[idx].value("id", "");
if (!id.empty()) {
call_order[id] = idx;
}
}
i++;
continue;
}
if (role != "user" && role != "tool") {
i++;
continue;
}
// collect a maximal run of user/tool messages - they render into one user block
std::vector<size_t> tool_positions;
size_t run_end = i;
for (; run_end < adjusted.size(); run_end++) {
const auto r = adjusted[run_end].value("role", "");
if (r == "tool") {
tool_positions.push_back(run_end);
} else if (r != "user") {
break;
}
}
if (tool_positions.size() > 1 && !call_order.empty()) {
std::vector<json> results;
results.reserve(tool_positions.size());
for (auto pos : tool_positions) {
results.push_back(adjusted[pos]);
}
std::stable_sort(results.begin(), results.end(), [&](const json & a, const json & b) {
const auto order = [&](const json & m) {
auto it = call_order.find(m.value("tool_call_id", ""));
return it == call_order.end() ? (size_t) 0 : it->second;
};
return order(a) < order(b);
});
for (size_t k = 0; k < tool_positions.size(); k++) {
adjusted[tool_positions[k]] = std::move(results[k]);
}
}
i = run_end;
}
return adjusted;
}
static common_chat_params common_chat_params_init_deepseek_v3_2(const common_chat_template & tmpl,
const autoparser::generation_params & inputs) {
common_chat_params data;
// V4 uses the same DSML markup as V3.2, but names the tool call block "tool_calls"
// instead of "function_calls", renders tool results in tool call order and its
// non-thinking generation prompt ends with a bare </think> instead of an empty
// <think></think> pair.
const bool is_v4 = tmpl.source().find("function_calls") == std::string::npos;
std::optional<json> adjusted_messages;
if (is_v4) {
adjusted_messages = deepseek_v4_sort_tool_results(inputs.messages);
}
data.prompt = common_chat_template_direct_apply_impl(tmpl, inputs, adjusted_messages);
data.generation_prompt = common_chat_template_generation_prompt_impl(tmpl, inputs, adjusted_messages);
data.prompt = common_chat_template_direct_apply_impl(tmpl, inputs);
data.generation_prompt = common_chat_template_generation_prompt_impl(tmpl, inputs);
data.format = COMMON_CHAT_FORMAT_PEG_NATIVE;
data.supports_thinking = true;
data.thinking_start_tag = "<think>";
data.thinking_end_tags = {"</think>"};
data.thinking_end_tag = "</think>";
data.preserved_tokens = {
"｜DSML｜",
"<think>",
@@ -1961,9 +1879,8 @@ static common_chat_params common_chat_params_init_deepseek_v3_2(const common_cha
const std::string DSML = "｜DSML｜";
const std::string THINK_START = "<think>";
const std::string THINK_END = "</think>";
const std::string TC_BLOCK = is_v4 ? "tool_calls" : "function_calls";
const std::string FC_START = "<" + DSML + TC_BLOCK + ">";
const std::string FC_END = "</" + DSML + TC_BLOCK + ">";
const std::string FC_START = "<" + DSML + "function_calls>";
const std::string FC_END = "</" + DSML + "function_calls>";
const std::string INVOKE_START = "<" + DSML + "invoke";
const std::string INVOKE_END = "</" + DSML + "invoke>";
const std::string PARAM_START = "<" + DSML + "parameter";
@@ -1990,11 +1907,8 @@ static common_chat_params common_chat_params_init_deepseek_v3_2(const common_cha
reasoning = p.optional(THINK_START + p.reasoning(p.until(THINK_END)) + THINK_END);
} else if (extract_reasoning) {
// Thinking disabled but reasoning extraction requested: the generation prompt
// contains an empty <think></think> pair (V3.2) or a bare </think> (V4) that
// must still be consumed.
reasoning = is_v4
? p.optional(p.literal(THINK_END))
: p.optional(p.literal(THINK_START) + p.until(THINK_END) + p.literal(THINK_END));
// contains an empty <think></think> pair that must still be consumed.
reasoning = p.optional(p.literal(THINK_START) + p.until(THINK_END) + p.literal(THINK_END));
}
if (has_response_format) {
@@ -2163,7 +2077,7 @@ static common_chat_params common_chat_params_init_cohere2moe(const common_chat_t
data.format = COMMON_CHAT_FORMAT_PEG_NATIVE;
data.supports_thinking = true;
data.thinking_start_tag = THINK_START;
data.thinking_end_tags = {THINK_END};
data.thinking_end_tag = THINK_END;
data.preserved_tokens = {
TURN_START, TURN_END, CHATBOT, USER, SYSTEM,
THINK_START, THINK_END,
@@ -2182,10 +2096,9 @@ static common_chat_params common_chat_params_init_cohere2moe(const common_chat_t
{ COMMON_CHAT_ROLE_SYSTEM, TURN_START + SYSTEM },
};
auto has_tools = inputs.tools.is_array() && !inputs.tools.empty();
auto has_response_format = inputs.json_schema.is_object() && !inputs.json_schema.empty();
auto extract_reasoning = inputs.reasoning_format != COMMON_REASONING_FORMAT_NONE;
auto include_grammar = has_response_format || (has_tools && inputs.tool_choice != COMMON_CHAT_TOOL_CHOICE_NONE);
auto has_tools = inputs.tools.is_array() && !inputs.tools.empty();
auto extract_reasoning = inputs.reasoning_format != COMMON_REASONING_FORMAT_NONE;
auto include_grammar = has_tools && inputs.tool_choice != COMMON_CHAT_TOOL_CHOICE_NONE;
if (inputs.has_continuation()) {
const auto & msg = inputs.continue_msg;
@@ -2216,11 +2129,7 @@ static common_chat_params common_chat_params_init_cohere2moe(const common_chat_t
p.optional(p.literal(THINK_END))));
}
auto text_content = has_response_format
? p.literal(TEXT_START) +
p.content(p.schema(p.json(), "response-format-schema", inputs.json_schema)) +
p.optional(p.literal(TEXT_END))
: p.literal(TEXT_START) + p.content(p.until(TEXT_END)) + p.optional(p.literal(TEXT_END));
auto text_content = p.literal(TEXT_START) + p.content(p.until(TEXT_END)) + p.optional(p.literal(TEXT_END));
if (!has_tools || inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_NONE) {
return generation_prompt + reasoning + text_content + p.optional(p.literal(TURN_END)) + end;
@@ -2248,17 +2157,13 @@ static common_chat_params common_chat_params_init_cohere2moe(const common_chat_t
data.parser = parser.save();
if (include_grammar) {
data.grammar_lazy = !has_response_format && inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_AUTO;
data.grammar_lazy = inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_AUTO;
data.grammar = build_grammar([&](const common_grammar_builder & builder) {
foreach_function(inputs.tools, [&](const json & tool) {
const auto & function = tool.at("function");
auto schema = function.at("parameters");
builder.resolve_refs(schema);
});
if (has_response_format) {
auto schema = inputs.json_schema;
builder.resolve_refs(schema);
}
parser.build_grammar(builder, data.grammar_lazy);
});
@@ -2473,23 +2378,6 @@ static void func_args_not_string(json & messages) {
}
}
// Trim leading/trailing whitespace from message contents before rendering. This
// has to run on the messages (not on the rendered JSON) because templates with
// string-only content caps concatenate typed content parts into a single string
// during rendering, after which the per-part whitespace can no longer be reached.
// Both the plain string content and the text of typed content parts are trimmed.
static void trim_all_content(std::vector<common_chat_msg> & messages) {
for (auto & message : messages) {
message.content = trim_whitespace(message.content);
message.reasoning_content = trim_whitespace(message.reasoning_content);
for (auto & part : message.content_parts) {
if (part.type == "text") {
part.text = trim_whitespace(part.text);
}
}
}
}
}
// MiniCPM5 format:
@@ -2513,7 +2401,7 @@ static common_chat_params common_chat_params_init_minicpm5(const common_chat_tem
};
data.thinking_start_tag = "<think>";
data.thinking_end_tags = {"</think>"};
data.thinking_end_tag = "</think>";
data.message_delimiters = {
{ COMMON_CHAT_ROLE_ASSISTANT, "<|im_start|>assistant" },
@@ -2707,14 +2595,12 @@ std::optional<common_chat_params> common_chat_try_specialized_template(
return common_chat_params_init_gigachat_v3(tmpl, params);
}
// DeepSeek V3.2/V4 format detection: template defines dsml_token and uses it for tool calls.
// DeepSeek V3.2 format detection: template defines dsml_token and uses it for tool calls.
// The template source contains the token as a variable assignment, not as a literal in markup.
// V3.2 names the tool call block "function_calls", V4 names it "tool_calls".
if (src.find("dsml_token") != std::string::npos &&
src.find("DSML") != std::string::npos &&
(src.find("function_calls") != std::string::npos ||
src.find("tool_calls") != std::string::npos)) {
LOG_DBG("Using specialized template: DeepSeek V3.2/V4\n");
src.find("function_calls") != std::string::npos &&
src.find("DSML") != std::string::npos) {
LOG_DBG("Using specialized template: DeepSeek V3.2\n");
return common_chat_params_init_deepseek_v3_2(tmpl, params);
}
@@ -2748,16 +2634,7 @@ static common_chat_params common_chat_templates_apply_jinja(const struct common_
params.tools.is_array() && tmpls->template_tool_use ? *tmpls->template_tool_use : *tmpls->template_default;
const auto & src = tmpl.source();
const auto & caps = tmpl.original_caps();
std::vector<common_chat_msg> trimmed_messages;
const std::vector<common_chat_msg> * messages_to_render = &inputs.messages;
if (src.find("You have access to the following functions in JSONSchema format") != std::string::npos) {
// StepFun: trim message contents (including typed content parts) before rendering,
// otherwise leftover whitespace drives the model into reasoning loops (issue #24181)
trimmed_messages = inputs.messages;
workaround::trim_all_content(trimmed_messages);
messages_to_render = &trimmed_messages;
}
params.messages = render_message_to_json(*messages_to_render, tmpl.original_caps());
params.messages = render_message_to_json(inputs.messages, tmpl.original_caps());
params.tool_choice = inputs.tool_choice;
params.reasoning_format = inputs.reasoning_format;
params.enable_thinking = inputs.enable_thinking;
@@ -2869,10 +2746,7 @@ static common_chat_params common_chat_templates_apply_jinja(const struct common_
auto_params.supports_thinking = autoparser.reasoning.mode != autoparser::reasoning_mode::NONE;
if (auto_params.supports_thinking) {
auto_params.thinking_start_tag = trim_whitespace(autoparser.reasoning.start);
auto end_tag = trim_whitespace(autoparser.reasoning.end);
if (!end_tag.empty()) {
auto_params.thinking_end_tags = {std::move(end_tag)};
}
auto_params.thinking_end_tag = trim_whitespace(autoparser.reasoning.end);
}
common_peg_arena arena;
arena.load(auto_params.parser);
common/chat.h
+1 -1
View File
@@ -274,7 +274,7 @@ struct common_chat_params {
std::string generation_prompt;
bool supports_thinking = false;
std::string thinking_start_tag; // e.g., "<think>"
std::vector<std::string> thinking_end_tags; // e.g., "</think>"
std::string thinking_end_tag; // e.g., "</think>"
std::vector<common_grammar_trigger> grammar_triggers;
std::vector<std::string> preserved_tokens;
std::vector<std::string> additional_stops;
common/common.cpp
+5 -23
View File
@@ -55,10 +55,6 @@
#include <pwd.h>
#endif
#if defined(_AIX)
#include <sys/systemcfg.h>
#endif
#if defined(_MSC_VER)
#pragma warning(disable: 4244 4267) // possible loss of data
#endif
@@ -76,16 +72,7 @@ common_time_meas::~common_time_meas() {
//
int32_t common_cpu_get_num_physical_cores() {
#if defined(_AIX)
int32_t logical_cpus = _system_configuration.ncpus;
int32_t smt_threads = _system_configuration.smt_threads;
if (smt_threads > 0) {
return static_cast<int32_t>(logical_cpus / smt_threads);
}
if (logical_cpus > 0) {
return static_cast<int32_t>(logical_cpus);
}
#elif defined(__linux__)
#ifdef __linux__
// enumerate the set of thread siblings, num entries is num cores
std::unordered_set<std::string> siblings;
for (uint32_t cpu=0; cpu < UINT32_MAX; ++cpu) {
@@ -215,14 +202,6 @@ int32_t common_cpu_get_num_math() {
}
}
}
#elif defined(__powerpc64__) || defined(__powerpc__)
int32_t smt_factor = 1;
int phy_cpus = common_cpu_get_num_physical_cores();
int logical_cpus = sysconf(_SC_NPROCESSORS_ONLN);
if (phy_cpus > 0 && logical_cpus > phy_cpus) {
smt_factor = logical_cpus / phy_cpus;
}
return phy_cpus * std::min(smt_factor, 2);
#endif
return common_cpu_get_num_physical_cores();
}
@@ -1249,6 +1228,7 @@ common_init_result::common_init_result(common_params & params, bool model_only)
lora.reset(llama_adapter_lora_init(model, la.path.c_str()));
if (lora == nullptr) {
COM_ERR("failed to load lora adapter '%s'\n", la.path.c_str());
pimpl->model.reset(model);
return;
}
@@ -1557,8 +1537,10 @@ struct llama_model_params common_model_params_to_llama(common_params & params) {
mparams.n_gpu_layers = params.n_gpu_layers;
mparams.main_gpu = params.main_gpu;
mparams.split_mode = params.split_mode;
mparams.load_mode = params.load_mode;
mparams.tensor_split = params.tensor_split;
mparams.use_mmap = params.use_mmap;
mparams.use_direct_io = params.use_direct_io;
mparams.use_mlock = params.use_mlock;
mparams.check_tensors = params.check_tensors;
mparams.use_extra_bufts = !params.no_extra_bufts;
mparams.no_host = params.no_host;
common/common.h
+9 -56
View File
@@ -6,7 +6,6 @@
#include "ggml-opt.h"
#include "ggml.h"
#include "llama.h"
#include <set>
#include <sstream>
@@ -15,7 +14,6 @@
#include <vector>
#include <map>
#include <algorithm>
#include <fstream>
#if defined(_WIN32) && !defined(_WIN32_WINNT)
#define _WIN32_WINNT 0x0A00
@@ -106,7 +104,6 @@ enum llama_example {
LLAMA_EXAMPLE_RESULTS,
LLAMA_EXAMPLE_EXPORT_GRAPH_OPS,
LLAMA_EXAMPLE_DOWNLOAD,
LLAMA_EXAMPLE_TOKENIZE,
LLAMA_EXAMPLE_COUNT,
};
@@ -284,12 +281,12 @@ struct common_params_sampling {
// reasoning budget sampler parameters
// these are populated by the server/CLI based on chat template params
int32_t reasoning_budget_tokens = -1; // -1 = disabled, >= 0 = token budget
std::vector<llama_token> reasoning_budget_start; // start tag token sequence
std::vector<llama_tokens> reasoning_budget_end; // end tag token sequences; the first tag is used as the forcing sequence
std::vector<llama_token> reasoning_budget_forced; // forced sequence (message + first end tag)
std::string reasoning_budget_message; // message injected before end tag when budget exhausted
bool reasoning_control = false; // create the budget sampler on demand so reasoning can be ended at runtime
int32_t reasoning_budget_tokens = -1; // -1 = disabled, >= 0 = token budget
std::vector<llama_token> reasoning_budget_start; // start tag token sequence
std::vector<llama_token> reasoning_budget_end; // end tag token sequence
std::vector<llama_token> reasoning_budget_forced; // forced sequence (message + end tag)
std::string reasoning_budget_message; // message injected before end tag when budget exhausted
bool reasoning_control = false; // create the budget sampler on demand so reasoning can be ended at runtime
bool backend_sampling = false;
@@ -483,7 +480,6 @@ struct common_params {
std::vector<size_t> fit_params_target = std::vector<size_t>(llama_max_devices(), 1024 * 1024*1024);
enum llama_split_mode split_mode = LLAMA_SPLIT_MODE_LAYER; // how to split the model across GPUs
enum llama_load_mode load_mode = LLAMA_LOAD_MODE_MMAP; // how to load the model
common_cpu_params cpuparams;
common_cpu_params cpuparams_batch;
@@ -574,6 +570,9 @@ struct common_params {
bool kv_unified = false; // enable unified KV cache
bool input_prefix_bos = false; // prefix BOS to user inputs, preceding input_prefix
bool use_mmap = true; // enable mmap to use filesystem cache
bool use_direct_io = false; // read from disk without buffering
bool use_mlock = false; // use mlock to keep model in memory
bool verbose_prompt = false; // print prompt tokens before generation
bool display_prompt = true; // print prompt before generation
bool no_kv_offload = false; // disable KV offloading
@@ -630,14 +629,6 @@ struct common_params {
std::string api_prefix = ""; // NOLINT
std::string chat_template = ""; // NOLINT
bool use_jinja = true; // NOLINT
// server CORS params
std::string cors_origins = "*";
std::string cors_methods = "GET, POST, DELETE, OPTIONS";
std::string cors_headers = "*";
bool cors_credentials = true;
bool cors_origins_explicit = false; // for --agent option
bool enable_chat_template = true;
bool force_pure_content_parser = false;
common_reasoning_format reasoning_format = COMMON_REASONING_FORMAT_DEEPSEEK;
@@ -652,9 +643,6 @@ struct common_params {
std::map<std::string, std::string> default_template_kwargs;
// CLI params
std::string server_base; // if set, connect to this server instead of starting a new one
// UI configs
bool ui = true;
bool ui_mcp_proxy = false;
@@ -668,10 +656,6 @@ struct common_params {
// enable built-in tools
std::vector<std::string> server_tools;
// MCP server configs (Cursor-compatible JSON)
std::string mcp_servers_config; // path to JSON file with MCP server definitions
std::string mcp_servers_json; // inline JSON with MCP server definitions
// router server configs
std::string models_dir = ""; // directory containing models for the router server
std::string models_preset = ""; // directory containing model presets for the router server
@@ -728,12 +712,6 @@ struct common_params {
// batched-bench params
bool batched_bench_output_jsonl = false;
// tokenize params
bool tokenize_ids = false; // if true, only print the token IDs
bool tokenize_stdin = false; // if true, read the prompt from stdin
bool tokenize_no_bos = false; // if true, do not add the BOS token
bool tokenize_show_count = false; // if true, print the total token count
// common params
std::string out_file; // output filename for all example programs
// optional callback for model loading progress and cancellation:
@@ -1083,28 +1061,6 @@ inline llama_model_tensor_buft_override llm_ffn_exps_cpu_override() {
return { LLM_FFN_EXPS_REGEX, ggml_backend_cpu_buffer_type() };
}
// ATSInfer-style tensor-granularity static placement of MoE expert weights.
// Offloads the expert weights of the first floor(n) layers to the CPU, plus a
// subset of the boundary layer's three expert tensors for the fractional part.
// Expert tensors are dropped from the GPU in ascending performance-density
// order (gate, then up), keeping the higher-value down_proj resident longest.
inline std::vector<std::string> llm_ffn_exps_cpu_block_regexes(double n_cpu_moe) {
std::vector<std::string> regexes;
const int n_full = n_cpu_moe > 0 ? (int) n_cpu_moe : 0;
for (int i = 0; i < n_full; ++i) {
regexes.push_back(llm_ffn_exps_block_regex(i));
}
const int k = (int) ((n_cpu_moe - n_full) * 3.0 + 0.5);
if (k >= 3) {
regexes.push_back(llm_ffn_exps_block_regex(n_full));
} else if (k == 2) {
regexes.push_back(string_format("blk\\.%d\\.ffn_(gate|up)_(ch|)exps", n_full));
} else if (k == 1) {
regexes.push_back(string_format("blk\\.%d\\.ffn_gate_(ch|)exps", n_full));
}
return regexes;
}
//
// training utils
//
@@ -1121,9 +1077,6 @@ enum ggml_opt_optimizer_type common_opt_get_optimizer(const char *);
struct common_prompt_checkpoint {
int64_t n_tokens;
// (optional) id of the task that created the checkpoint
int id_task = -1;
llama_pos pos_min;
llama_pos pos_max;
common/download.cpp
+3 -23
View File
@@ -620,16 +620,6 @@ static hf_cache::hf_file find_best_mtp(const hf_cache::hf_files & files,
return find_best_sibling(files, model, "mtp-");
}
static hf_cache::hf_file find_best_eagle3(const hf_cache::hf_files & files,
const std::string & model) {
return find_best_sibling(files, model, "eagle3-");
}
static hf_cache::hf_file find_best_dflash(const hf_cache::hf_files & files,
const std::string & model) {
return find_best_sibling(files, model, "dflash-");
}
static bool gguf_filename_is_model(const std::string & filepath) {
if (!string_ends_with(filepath, ".gguf")) {
return false;
@@ -642,9 +632,7 @@ static bool gguf_filename_is_model(const std::string & filepath) {
return filename.find("mmproj") == std::string::npos &&
filename.find("imatrix") == std::string::npos &&
filename.find("mtp-") == std::string::npos &&
filename.find("eagle3-") == std::string::npos &&
filename.find("dflash-") == std::string::npos;
filename.find("mtp-") == std::string::npos;
}
static hf_cache::hf_file find_best_model(const hf_cache::hf_files & files,
@@ -752,12 +740,6 @@ common_download_hf_plan common_download_get_hf_plan(const common_params_model &
if (opts.download_mtp) {
plan.mtp = find_best_mtp(all, primary.path);
}
if (opts.download_dflash) {
plan.dflash = find_best_dflash(all, primary.path);
}
if (opts.download_eagle3) {
plan.eagle3 = find_best_eagle3(all, primary.path);
}
return plan;
}
@@ -929,10 +911,8 @@ std::vector<common_cached_model_info> common_list_cached_models() {
for (const auto & f : files) {
auto split = get_gguf_split_info(f.path);
if (split.index != 1 || split.tag.empty() ||
split.prefix.find("mmproj") != std::string::npos ||
split.prefix.find("mtp-") != std::string::npos ||
split.prefix.find("eagle3-") != std::string::npos ||
split.prefix.find("dflash-") != std::string::npos) {
split.prefix.find("mmproj") != std::string::npos ||
split.prefix.find("mtp-") != std::string::npos) {
continue;
}
if (seen.insert(f.repo_id + ":" + split.tag).second) {
common/download.h
+2 -6
View File
@@ -55,10 +55,8 @@ struct common_download_opts {
std::string bearer_token;
common_header_list headers;
bool offline = false;
bool download_mmproj = false;
bool download_mtp = false;
bool download_eagle3 = false;
bool download_dflash = false;
bool download_mmproj = false;
bool download_mtp = false;
common_download_callback * callback = nullptr;
};
@@ -108,8 +106,6 @@ struct common_download_hf_plan {
hf_cache::hf_files model_files;
hf_cache::hf_file mmproj;
hf_cache::hf_file mtp;
hf_cache::hf_file eagle3;
hf_cache::hf_file dflash;
hf_cache::hf_file preset; // if set, only this file is downloaded
};
common_download_hf_plan common_download_get_hf_plan(const common_params_model & model, const common_download_opts & opts);
common/fit.cpp
+2 -1
View File
@@ -54,7 +54,8 @@ static std::vector<llama_device_memory_data> common_get_device_memory_data_impl(
llama_model_params mparams_copy = *mparams;
mparams_copy.no_alloc = true;
mparams_copy.load_mode = LLAMA_LOAD_MODE_NONE;
mparams_copy.use_mmap = false;
mparams_copy.use_mlock = false;
llama_model * model = llama_model_load_from_file(path_model, mparams_copy);
if (model == nullptr) {
common/http.h
-70
View File
@@ -2,16 +2,6 @@
#include <cpp-httplib/httplib.h>
#ifdef _WIN32
#include <winsock2.h>
#include <windows.h>
#else
#include <sys/socket.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <unistd.h>
#endif
struct common_http_url {
std::string scheme;
std::string user;
@@ -129,63 +119,3 @@ static std::pair<httplib::Client, common_http_url> common_http_client(const std:
static std::string common_http_show_masked_url(const common_http_url & parts) {
return parts.scheme + "://" + (parts.user.empty() ? "" : "****:****@") + common_http_format_host(parts.host) + parts.path;
}
static int common_http_get_free_port() {
#ifdef _WIN32
WSADATA wsaData;
if (WSAStartup(MAKEWORD(2, 2), &wsaData) != 0) {
return -1;
}
typedef SOCKET native_socket_t;
#define INVALID_SOCKET_VAL INVALID_SOCKET
#define CLOSE_SOCKET(s) closesocket(s)
#else
typedef int native_socket_t;
#define INVALID_SOCKET_VAL -1
#define CLOSE_SOCKET(s) close(s)
#endif
native_socket_t sock = socket(AF_INET, SOCK_STREAM, 0);
if (sock == INVALID_SOCKET_VAL) {
#ifdef _WIN32
WSACleanup();
#endif
return -1;
}
struct sockaddr_in serv_addr;
std::memset(&serv_addr, 0, sizeof(serv_addr));
serv_addr.sin_family = AF_INET;
serv_addr.sin_addr.s_addr = htonl(INADDR_ANY);
serv_addr.sin_port = htons(0);
if (bind(sock, (struct sockaddr*)&serv_addr, sizeof(serv_addr)) != 0) {
CLOSE_SOCKET(sock);
#ifdef _WIN32
WSACleanup();
#endif
return -1;
}
#ifdef _WIN32
int namelen = sizeof(serv_addr);
#else
socklen_t namelen = sizeof(serv_addr);
#endif
if (getsockname(sock, (struct sockaddr*)&serv_addr, &namelen) != 0) {
CLOSE_SOCKET(sock);
#ifdef _WIN32
WSACleanup();
#endif
return -1;
}
int port = ntohs(serv_addr.sin_port);
CLOSE_SOCKET(sock);
#ifdef _WIN32
WSACleanup();
#endif
return port;
}
common/jinja/caps.cpp
-1
View File
@@ -23,7 +23,6 @@ void caps_apply_preserve_reasoning(jinja::context & ctx, bool enabled) {
ctx.set_val("preserve_thinking", mk_val<value_bool>(enabled));
ctx.set_val("clear_thinking", mk_val<value_bool>(!enabled));
ctx.set_val("truncate_history_thinking", mk_val<value_bool>(!enabled));
ctx.set_val("drop_thinking", mk_val<value_bool>(!enabled));
}
static void caps_try_execute(jinja::program & prog,
common/jinja/value.cpp
-39
View File
@@ -750,50 +750,11 @@ const func_builtins & value_string_t::get_builtins() const {
res->val_str.mark_input_based_on(args.get_pos(0)->val_str);
return res;
}},
{"format", [](const func_args & args) -> value {
value val_input = args.get_pos(0);
if (!is_val<value_string>(val_input)) {
throw raised_exception("format() first argument must be a string");
}
const jinja::string & fmt = val_input->as_string();
const bool fmt_is_input = fmt.all_parts_are_input();
const std::string str = fmt.str();
jinja::string result;
std::string literal;
auto flush_literal = [&]() {
if (!literal.empty()) {
result.parts.push_back({fmt_is_input, literal});
literal.clear();
}
};
size_t arg_idx = 1; // positional args follow the format string
for (size_t i = 0; i < str.size(); ++i) {
if (str[i] != '{') {
literal += str[i];
continue;
}
if (i + 1 >= str.size() || str[i + 1] != '}') {
throw not_implemented_exception("format() only supports simple '{}' placeholders");
}
++i;
flush_literal();
const jinja::string arg_str = args.get_pos(arg_idx++)->as_string();
result.parts.insert(result.parts.end(), arg_str.parts.begin(), arg_str.parts.end());
}
flush_literal();
return mk_val<value_string>(result);
}},
{"int", [](const func_args & args) -> value {
value val_input = args.get_pos(0);
value val_default = args.get_kwarg_or_pos("default", 1);
value val_base = args.get_kwarg_or_pos("base", 2);
const int base = val_base->is_undefined() ? 10 : val_base->as_int();
if (base != 0 && (base < 2 || base > 36)) {
// an out-of-range base makes std::stoi fail fast on the MSVC CRT instead of throwing
throw raised_exception("int() base must be 0 or between 2 and 36");
}
if (is_val<value_string>(val_input) == false) {
throw raised_exception("int() first argument must be a string");
}
common/ngram-map.cpp
+9 -15
View File
@@ -125,16 +125,6 @@ void common_ngram_map_begin(
LOG_DBG("%s: begin, idx_last_draft=%zu, new begin=%zu, #keys=%zu\n", __func__,
map.idx_last_check, size_begin, map.keys.size());
size_t idx_begin_cleanup = map.size_last_begin;
if (idx_begin_cleanup > size_begin) {
if (size_begin > (size_t) map.size_key + map.size_value) {
idx_begin_cleanup = size_begin - map.size_key - map.size_value;
} else {
idx_begin_cleanup = 0;
}
LOG_INF("%s: shrink cleanup begin: %zu -> %zu\n", __func__, map.size_last_begin, idx_begin_cleanup);
}
size_t count_map_entries_upd = 0;
if (!map.key_map.empty() && size_begin < map.idx_last_check) {
if (map.show_key_map_stats) {
@@ -160,23 +150,27 @@ void common_ngram_map_begin(
// Update the map from hash to key index (clear outdated entries).
for (size_t i = 0; i < map.key_map.size(); ++i) {
uint32_t key_idx = map.key_map[i];
if (key_idx != 0 && key_idx >= idx_begin_cleanup) {
if (key_idx >= map.size_last_begin) {
map.key_map[i] = 0;
count_map_entries_upd++;
}
}
map.key_map_last_idx = (idx_begin_cleanup > 0) ? (uint32_t) (idx_begin_cleanup - 1) : 0;
map.key_map_last_idx = (map.size_last_begin > 0) ? map.size_last_begin - 1 : 0;
}
if (size_begin < map.idx_last_check && !map.keys.empty()) {
// The next token generation will start at index size_begin.
// The tokens between map.size_last_begin and size_begin are no longer valid.
//
// Refresh map: Remove all entries with index >= map.size_last_begin.
size_t count_keys = map.keys.size();
size_t count_keys_del = 0;
size_t count_values_del = 0;
for (int32_t i = map.keys.size() - 1; i >= 0; --i) {
common_ngram_map_key & key = map.keys[i];
if (key.key_idx >= idx_begin_cleanup) {
if (key.key_idx >= map.size_last_begin) {
// Delete the key.
LOG_DBG("%s: delete key %d at index %zu (>= idx_begin_cleanup=%zu)\n", __func__, i, key.key_idx, idx_begin_cleanup);
LOG_DBG("%s: delete key %d at index %zu (>= size_last_begin=%zu)\n", __func__, i, key.key_idx, map.size_last_begin);
map.keys.erase(map.keys.begin() + i);
count_keys_del++;
continue;
@@ -188,7 +182,7 @@ void common_ngram_map_begin(
// Check the indices of the values.
for (int16_t j = COMMON_NGRAM_MAX_VALUES - 1; j >= 0; --j) {
common_ngram_map_value & value = key.values[j];
if (value.value_idx != 0 && value.value_idx >= idx_begin_cleanup) {
if (value.value_idx >= map.size_last_begin) {
// Delete the value.
count_values_del++;
common/peg-parser.cpp
+153 -5
View File
@@ -3,10 +3,10 @@
#include "common.h"
#include "json-schema-to-grammar.h"
#include "log.h"
#include "trie.h"
#include "unicode.h"
#include <algorithm>
#include <deque>
#include <initializer_list>
#include <map>
#include <memory>
@@ -32,6 +32,154 @@ static bool is_hex_digit(const char c) {
return (c >= '0' && c <= '9') || (c >= 'a' && c <= 'f') || (c >= 'A' && c <= 'F');
}
// Trie for matching multiple literals.
// This is used in common_peg_until_parser and to build a GBNF exclusion grammar
struct trie {
struct node {
std::map<uint32_t, size_t> children; // Use uint32_t to store Unicode codepoints
bool is_word;
};
std::vector<node> nodes;
trie(const std::vector<std::string> & words) {
create_node(); // root node
for (const auto & w : words) {
insert(w);
}
}
enum match_result { NO_MATCH, PARTIAL_MATCH, COMPLETE_MATCH };
// Check if a delimiter starts at the given position
match_result check_at(std::string_view sv, size_t start_pos) const {
size_t current = 0; // Start at root
size_t pos = start_pos;
// LOG_DBG("%s: checking at pos %zu, sv='%s'\n", __func__, start_pos, std::string(sv).c_str());
while (pos < sv.size()) {
auto result = common_parse_utf8_codepoint(sv, pos);
if (result.status != utf8_parse_result::SUCCESS) {
break;
}
auto it = nodes[current].children.find(result.codepoint);
if (it == nodes[current].children.end()) {
// Can't continue matching
return match_result{match_result::NO_MATCH};
}
current = it->second;
pos += result.bytes_consumed;
// Check if we've matched a complete word
if (nodes[current].is_word) {
return match_result{match_result::COMPLETE_MATCH};
}
}
// Reached end of input while still in the trie (not at root)
if (current != 0) {
// We're in the middle of a potential match
return match_result{match_result::PARTIAL_MATCH};
}
// Reached end at root (no match)
return match_result{match_result::NO_MATCH};
}
private:
size_t create_node() {
size_t index = nodes.size();
nodes.emplace_back();
return index;
}
void insert(const std::string & word) {
size_t current = 0;
size_t pos = 0;
while (pos < word.length()) {
auto result = common_parse_utf8_codepoint(word, pos);
if (result.status != utf8_parse_result::SUCCESS) {
break;
}
uint32_t ch = result.codepoint;
pos += result.bytes_consumed;
auto it = nodes[current].children.find(ch);
if (it == nodes[current].children.end()) {
size_t child = create_node();
nodes[current].children[ch] = child;
current = child;
} else {
current = it->second;
}
}
nodes[current].is_word = true;
}
};
// Aho-Corasick automaton
struct aho_corasick {
trie t;
std::vector<size_t> fail; // failure links
std::vector<size_t> order; // states in BFS order
std::vector<bool> terminal; // match states (directly or via a suffix link)
std::set<uint32_t> alphabet; // every character with a transition
aho_corasick(const std::vector<std::string> & strings) : t(strings) {
const auto & nodes = t.nodes;
const size_t n = nodes.size();
fail.assign(n, 0);
order.reserve(n);
std::deque<size_t> queue{ 0 };
while (!queue.empty()) {
size_t u = queue.front();
queue.pop_front();
order.push_back(u);
for (const auto & [ch, v] : nodes[u].children) {
if (u != 0) {
size_t f = fail[u];
while (f && nodes[f].children.find(ch) == nodes[f].children.end()) {
f = fail[f];
}
auto it = nodes[f].children.find(ch);
fail[v] = (it != nodes[f].children.end() && it->second != v) ? it->second : 0;
}
queue.push_back(v);
}
}
terminal.assign(n, false);
for (size_t u : order) {
terminal[u] = nodes[u].is_word || (u != 0 && terminal[fail[u]]);
}
for (const auto & node : nodes) {
for (const auto & [ch, v] : node.children) {
alphabet.insert(ch);
}
}
}
size_t num_states() const { return t.nodes.size(); }
bool is_terminal(size_t s) const { return terminal[s]; }
// follow failure links until a transition on `ch` exists.
size_t next(size_t state, uint32_t ch) const {
const auto & nodes = t.nodes;
while (state && nodes[state].children.find(ch) == nodes[state].children.end()) {
state = fail[state];
}
auto it = nodes[state].children.find(ch);
return it != nodes[state].children.end() ? it->second : 0;
}
};
static std::pair<uint32_t, size_t> parse_hex_escape(const std::string & str, size_t pos, int hex_count) {
if (pos + hex_count > str.length()) {
return {0, 0};
@@ -649,7 +797,7 @@ struct parser_executor {
}
common_peg_parse_result operator()(const common_peg_until_parser & p) const {
common_trie matcher(p.delimiters);
trie matcher(p.delimiters);
// Scan input and check for delimiters
size_t pos = start_pos;
@@ -676,12 +824,12 @@ struct parser_executor {
// Check if a delimiter starts at this position
auto match = matcher.check_at(ctx.input, pos);
if (match == common_trie::COMPLETE_MATCH) {
if (match == trie::COMPLETE_MATCH) {
// Found a complete delimiter, return everything before it
return common_peg_parse_result(COMMON_PEG_PARSE_RESULT_SUCCESS, start_pos, pos);
}
if (match == common_trie::PARTIAL_MATCH) {
if (match == trie::PARTIAL_MATCH) {
// Found a partial match extending to end of input, return everything before it
return common_peg_parse_result(COMMON_PEG_PARSE_RESULT_SUCCESS, start_pos, pos);
}
@@ -1411,7 +1559,7 @@ static std::string gbnf_ac_grammar(
const std::map<size_t, std::vector<uint32_t>> &,
const std::vector<uint32_t> &,
const std::function<std::string(size_t)> &)> & build_rule) {
common_aho_corasick ac(strings);
aho_corasick ac(strings);
auto state_name = [&](size_t s) -> std::string {
if (s == 0) {
common/preset.cpp
-4
View File
@@ -330,10 +330,6 @@ common_presets common_preset_context::load_from_ini(const std::string & path, co
}
}
if (preset.name == COMMON_PRESET_DEFAULT_NAME && preset.options.empty()) {
continue;
}
if (preset.name == "*") {
// handle global preset
global = preset;
common/reasoning-budget.cpp
+39 -77
View File
@@ -1,52 +1,39 @@
#include "reasoning-budget.h"
#include "common.h"
#include "trie.h"
#include "unicode.h"
#include "log.h"
#include <algorithm>
#include <cmath>
#include <cstdint>
#include <string>
#include <vector>
struct token_matcher {
std::vector<llama_tokens> seqs;
common_aho_corasick ac;
size_t state = 0;
std::vector<llama_token> tokens;
size_t pos = 0;
token_matcher(const std::vector<llama_tokens> & seqs) : seqs(collect(seqs)), ac(build_trie(this->seqs)) {}
bool advance(llama_token token) {
if (tokens.empty()) {
return false;
}
static std::vector<llama_tokens> collect(const std::vector<llama_tokens> & seqs) {
std::vector<llama_tokens> res;
for (const auto & seq : seqs) {
if (!seq.empty() && std::find(res.begin(), res.end(), seq) == res.end()) {
res.push_back(seq);
if (token == tokens[pos]) {
pos++;
if (pos >= tokens.size()) {
pos = 0;
return true;
}
} else {
pos = 0;
if (token == tokens[0]) {
pos = 1;
}
}
return res;
return false;
}
static common_trie build_trie(const std::vector<llama_tokens> & seqs) {
common_trie t;
for (const auto & seq : seqs) {
t.insert(std::vector<uint32_t>(seq.begin(), seq.end()));
}
return t;
}
// returns the index into seqs of the longest sequence ending at this token, or -1
int32_t advance(llama_token token) {
state = ac.next(state, (uint32_t) token);
const int32_t p = ac.match_pattern(state);
if (p >= 0) {
state = 0;
}
return p;
}
void reset() { state = 0; }
void reset() { pos = 0; }
};
struct common_reasoning_budget_ctx {
@@ -54,7 +41,7 @@ struct common_reasoning_budget_ctx {
token_matcher start_matcher;
token_matcher end_matcher;
llama_tokens forced_tokens;
std::vector<llama_token> forced_tokens;
int32_t budget; // maximum tokens in reasoning block
int32_t remaining; // tokens remaining in budget
@@ -63,8 +50,6 @@ struct common_reasoning_budget_ctx {
// for forcing
size_t force_pos; // next position in forced_tokens to force
int32_t end_match; // index into end_matcher.seqs of the sequence that transitioned to DONE, -1 if none
};
static const char * common_reasoning_budget_name(const struct llama_sampler * /*smpl*/) {
@@ -77,7 +62,7 @@ static void common_reasoning_budget_accept(struct llama_sampler * smpl, llama_to
switch (ctx->state) {
case REASONING_BUDGET_IDLE:
{
if (ctx->start_matcher.advance(token) >= 0) {
if (ctx->start_matcher.advance(token)) {
ctx->state = REASONING_BUDGET_COUNTING;
ctx->remaining = ctx->budget;
COM_TRC("activated, budget=%d tokens\n", ctx->budget);
@@ -93,10 +78,8 @@ static void common_reasoning_budget_accept(struct llama_sampler * smpl, llama_to
case REASONING_BUDGET_COUNTING:
case REASONING_BUDGET_WAITING_UTF8:
{
const int32_t match = ctx->end_matcher.advance(token);
if (match >= 0) {
if (ctx->end_matcher.advance(token)) {
ctx->state = REASONING_BUDGET_DONE;
ctx->end_match = match;
COM_TRC("%s", "deactivated (natural end)\n");
break;
}
@@ -132,25 +115,19 @@ static void common_reasoning_budget_accept(struct llama_sampler * smpl, llama_to
break;
}
case REASONING_BUDGET_FORCING:
{
// track the end sequence within forced_tokens so it is also reported on DONE
const int32_t match = ctx->end_matcher.advance(token);
ctx->force_pos++;
if (ctx->force_pos >= ctx->forced_tokens.size()) {
ctx->state = REASONING_BUDGET_DONE;
ctx->end_match = match;
COM_TRC("%s", "forced sequence complete, done\n");
}
break;
}
case REASONING_BUDGET_DONE:
// Re-arm on a new start tag: some models emit multiple <think> blocks
// per response, and each should get a fresh budget window.
if (ctx->start_matcher.advance(token) >= 0) {
if (ctx->start_matcher.advance(token)) {
ctx->state = REASONING_BUDGET_COUNTING;
ctx->remaining = ctx->budget;
ctx->end_matcher.reset();
ctx->end_match = -1;
COM_TRC("re-activated on new start tag, budget=%d tokens\n", ctx->budget);
if (ctx->remaining <= 0) {
@@ -192,12 +169,11 @@ static void common_reasoning_budget_reset(struct llama_sampler * smpl) {
ctx->start_matcher.reset();
ctx->end_matcher.reset();
ctx->force_pos = 0;
ctx->end_match = -1;
}
static struct llama_sampler * common_reasoning_budget_init_state(
const struct llama_vocab * vocab, const std::vector<llama_tokens> & start_seqs,
const std::vector<llama_tokens> & end_seqs, const llama_tokens & forced_tokens,
const struct llama_vocab * vocab, const std::vector<llama_token> & start_tokens,
const std::vector<llama_token> & end_tokens, const std::vector<llama_token> & forced_tokens,
int32_t budget, common_reasoning_budget_state initial_state);
static struct llama_sampler * common_reasoning_budget_clone(const struct llama_sampler * smpl);
@@ -229,12 +205,12 @@ static struct llama_sampler * common_reasoning_budget_clone(const struct llama_s
}
static struct llama_sampler * common_reasoning_budget_init_state(
const struct llama_vocab * vocab,
const std::vector<llama_tokens> & start_seqs,
const std::vector<llama_tokens> & end_seqs,
const llama_tokens & forced_tokens,
int32_t budget,
common_reasoning_budget_state initial_state) {
const struct llama_vocab * vocab,
const std::vector<llama_token> & start_tokens,
const std::vector<llama_token> & end_tokens,
const std::vector<llama_token> & forced_tokens,
int32_t budget,
common_reasoning_budget_state initial_state) {
// promote COUNTING with budget <= 0 to FORCING
if (initial_state == REASONING_BUDGET_COUNTING && budget <= 0) {
initial_state = REASONING_BUDGET_FORCING;
@@ -244,26 +220,25 @@ static struct llama_sampler * common_reasoning_budget_init_state(
/* .iface = */ &common_reasoning_budget_i,
/* .ctx = */ new common_reasoning_budget_ctx {
/* .vocab = */ vocab,
/* .start_matcher = */ token_matcher(start_seqs),
/* .end_matcher = */ token_matcher(end_seqs),
/* .start_matcher = */ { start_tokens, 0 },
/* .end_matcher = */ { end_tokens, 0 },
/* .forced_tokens = */ forced_tokens,
/* .budget = */ budget,
/* .remaining = */ budget,
/* .state = */ initial_state,
/* .force_pos = */ 0,
/* .end_match = */ -1,
}
);
}
struct llama_sampler * common_reasoning_budget_init(
const struct llama_vocab * vocab,
const std::vector<llama_tokens> & start_seqs,
const std::vector<llama_tokens> & end_seqs,
const llama_tokens & forced_tokens,
int32_t budget,
common_reasoning_budget_state initial_state) {
return common_reasoning_budget_init_state(vocab, start_seqs, end_seqs, forced_tokens, budget, initial_state);
const struct llama_vocab * vocab,
const std::vector<llama_token> & start_tokens,
const std::vector<llama_token> & end_tokens,
const std::vector<llama_token> & forced_tokens,
int32_t budget,
common_reasoning_budget_state initial_state) {
return common_reasoning_budget_init_state(vocab, start_tokens, end_tokens, forced_tokens, budget, initial_state);
}
common_reasoning_budget_state common_reasoning_budget_get_state(const struct llama_sampler * smpl) {
@@ -273,19 +248,6 @@ common_reasoning_budget_state common_reasoning_budget_get_state(const struct lla
return ((const common_reasoning_budget_ctx *)smpl->ctx)->state;
}
const llama_tokens * common_reasoning_budget_get_end_match(const struct llama_sampler * smpl) {
if (!smpl) {
return nullptr;
}
const auto * ctx = (const common_reasoning_budget_ctx *) smpl->ctx;
if (ctx->end_match < 0) {
return nullptr;
}
return &ctx->end_matcher.seqs[ctx->end_match];
}
bool common_reasoning_budget_force(struct llama_sampler * smpl) {
if (!smpl) {
return false;
common/reasoning-budget.h
+10 -16
View File
@@ -2,8 +2,6 @@
#include "llama.h"
#include "common.h"
#include <cstdint>
#include <vector>
@@ -19,34 +17,30 @@ enum common_reasoning_budget_state {
// reasoning block (e.g. between <think> and </think>).
//
// State machine: IDLE -> COUNTING -> WAITING_UTF8 -> FORCING -> DONE
// IDLE: passthrough, watching for a start sequence
// COUNTING: counting down remaining tokens, watching for a natural end sequence
// IDLE: passthrough, watching for start_tokens sequence
// COUNTING: counting down remaining tokens, watching for natural end_tokens
// WAITING_UTF8: budget exhausted, allowing tokens to complete a UTF-8 sequence
// FORCING: forces forced_tokens token-by-token (all other logits -> -inf)
// DONE: passthrough forever
//
// Parameters:
// vocab - vocabulary (used for UTF-8 boundary detection; can be nullptr)
// start_seqs - token sequences, any of which activates counting
// end_seqs - token sequences, any of which naturally deactivates
// start_tokens - token sequence that activates counting
// end_tokens - token sequence for natural deactivation
// forced_tokens - token sequence forced when budget expires
// budget - max tokens allowed in the reasoning block
// initial_state - initial state
//
struct llama_sampler * common_reasoning_budget_init(
const struct llama_vocab * vocab,
const std::vector<llama_tokens> & start_seqs,
const std::vector<llama_tokens> & end_seqs,
const llama_tokens & forced_tokens,
int32_t budget,
common_reasoning_budget_state initial_state = REASONING_BUDGET_IDLE);
const struct llama_vocab * vocab,
const std::vector<llama_token> & start_tokens,
const std::vector<llama_token> & end_tokens,
const std::vector<llama_token> & forced_tokens,
int32_t budget,
common_reasoning_budget_state initial_state = REASONING_BUDGET_IDLE);
common_reasoning_budget_state common_reasoning_budget_get_state(const struct llama_sampler * smpl);
// The end sequence that transitioned the sampler to DONE, or nullptr if none
// was recorded. Cleared when a new start sequence re-arms the sampler.
const llama_tokens * common_reasoning_budget_get_end_match(const struct llama_sampler * smpl);
// Manually transition the reasoning budget sampler into the FORCING state.
// Returns true if the transition occurred.
bool common_reasoning_budget_force(struct llama_sampler * smpl);
common/sampling.cpp
+1 -12
View File
@@ -299,7 +299,7 @@ struct common_sampler * common_sampler_init(const struct llama_model * model, st
if (!params.reasoning_budget_start.empty() && !params.reasoning_budget_end.empty() && (params.grammar_lazy || params.reasoning_budget_tokens >= 0 || params.reasoning_control)) {
rbudget = common_reasoning_budget_init(
vocab,
{params.reasoning_budget_start},
params.reasoning_budget_start,
params.reasoning_budget_end,
params.reasoning_budget_forced,
params.reasoning_budget_tokens < 0 ? INT_MAX : params.reasoning_budget_tokens);
@@ -453,17 +453,6 @@ void common_sampler_accept(struct common_sampler * gsmpl, llama_token token, boo
if (gsmpl->rbudget && is_generated) {
llama_sampler_accept(gsmpl->rbudget, token);
// if done, replay end sequence which may contain a grammar trigger
const bool is_done = common_reasoning_budget_get_state(gsmpl->rbudget) == REASONING_BUDGET_DONE;
if (gsmpl->grmr && !accept_grammar && is_done) {
const llama_tokens * end_seq = common_reasoning_budget_get_end_match(gsmpl->rbudget);
if (end_seq) {
for (const llama_token end_token : *end_seq) {
llama_sampler_accept(gsmpl->grmr, end_token);
}
}
}
}
if (gsmpl->grmr && accept_grammar) {
common/speculative.cpp
+6 -124
View File
@@ -260,10 +260,7 @@ struct common_speculative_impl_draft_simple : public common_speculative_impl {
bool process(const llama_batch & batch) override {
auto * ctx_dft = params.ctx_dft;
llama_batch batch_dft = batch;
batch_dft.logits = nullptr;
const int ret = llama_decode(ctx_dft, batch_dft);
const int ret = llama_decode(ctx_dft, batch);
if (ret != 0) {
SPC_ERR("failed to decode draft batch, ret = %d\n", ret);
@@ -958,11 +955,10 @@ struct common_speculative_impl_draft_dflash : public common_speculative_impl {
LOG_INF("%s: - block_size=%d, mask_token_id=%d, n_extract=%u\n", __func__, block_size, mask_token_id, target_layer_ids_n);
// DFlash input is [id_last, <mask> * (block_size-1)], so it can draft at most block_size-1 tokens per step
if (this->params.n_max > block_size - 1 || this->params.n_min > block_size - 1) {
LOG_WRN("%s: requested draft size (n_max=%d, n_min=%d) exceeds the trained DFlash block size %d -- clamping to %d\n",
__func__, this->params.n_max, this->params.n_min, block_size, block_size - 1);
this->params.n_max = std::min(this->params.n_max, block_size - 1);
this->params.n_min = std::min(this->params.n_min, block_size - 1);
if (this->params.n_max > block_size - 1) {
LOG_WRN("%s: requested draft size %d exceeds the trained DFlash block size %d -- clamping to %d draft tokens per step\n",
__func__, this->params.n_max, block_size - 1, block_size - 1);
this->params.n_max = block_size - 1;
}
batch = llama_batch_init(llama_n_batch(ctx_dft), 0, n_seq);
@@ -972,7 +968,7 @@ struct common_speculative_impl_draft_dflash : public common_speculative_impl {
for (auto & s : smpls) {
common_params_sampling sparams;
sparams.no_perf = false;
sparams.top_k = 10;
sparams.top_k = 1;
sparams.samplers = { COMMON_SAMPLER_TYPE_TOP_K };
s.reset(common_sampler_init(model_dft, sparams));
}
@@ -1177,18 +1173,10 @@ struct common_speculative_impl_draft_dflash : public common_speculative_impl {
const llama_token id = cur_p->data[0].id;
if (cur_p->data[0].p < params.p_min) {
break;
}
common_sampler_accept(smpl, id, true);
result.push_back(id);
}
if (result.size() < (size_t) params.n_min) {
result.clear();
}
}
}
@@ -2224,112 +2212,6 @@ int32_t common_speculative_n_max(const common_params_speculative * spec) {
return n_max;
}
common_params common_base_params_to_speculative(const common_params & params) {
const bool has_draft = params.speculative.has_dft();
const auto & params_spec = params.speculative.draft;
common_params result = params;
if (has_draft) {
result.devices = params_spec.devices;
result.model = params_spec.mparams;
result.n_gpu_layers = params_spec.n_gpu_layers;
result.tensor_buft_overrides = params_spec.tensor_buft_overrides;
if (params_spec.cpuparams.n_threads > 0) {
result.cpuparams.n_threads = params_spec.cpuparams.n_threads;
result.cpuparams_batch.n_threads = params_spec.cpuparams_batch.n_threads;
}
}
result.cache_type_k = params_spec.cache_type_k;
result.cache_type_v = params_spec.cache_type_v;
result.n_outputs_max = params.n_parallel;
return result;
}
struct common_speculative_init_result::impl {
impl() = default;
~impl() = default;
// note: the order in which model, context, etc. are declared matters because their destructors will be called bottom-to-top
llama_model_ptr model;
llama_context_ptr context;
};
common_speculative_init_result::common_speculative_init_result(
common_params & params,
llama_model * model_tgt,
llama_context * ctx_tgt) :
pimpl(new impl{}) {
const bool has_draft = params.speculative.has_dft();
const bool spec_mtp = std::find(params.speculative.types.begin(),
params.speculative.types.end(),
COMMON_SPECULATIVE_TYPE_DRAFT_MTP) != params.speculative.types.end();
GGML_ASSERT(has_draft || spec_mtp);
auto mparams = common_model_params_to_llama(params);
auto cparams = common_context_params_to_llama(params);
if (spec_mtp) {
cparams.ctx_type = LLAMA_CONTEXT_TYPE_MTP;
}
// note: for small models maybe we can set this to the maximum possible draft from all speculative types
// the extra memory for small models is likely negligible?
cparams.n_rs_seq = 0;
cparams.ctx_other = ctx_tgt;
std::string model_path;
if (has_draft) {
model_path = params.speculative.draft.mparams.path;
LOG_TRC("%s: loading draft model '%s'\n", __func__, model_path.c_str());
llama_model * model_dft = llama_model_load_from_file(params.model.path.c_str(), mparams);
if (model_dft == NULL) {
LOG_ERR("%s: failed to load draft model, '%s'\n", __func__, model_path.c_str());
return;
}
pimpl->model.reset(model_dft);
llama_context * ctx_dft = llama_init_from_model(model_dft, cparams);
if (ctx_dft == nullptr) {
LOG_ERR("%s: failed to create MTP context\n", __func__);
return;
}
pimpl->context.reset(ctx_dft);
} else if (spec_mtp) {
model_path = params.model.path;
LOG_TRC("%s: creating MTP draft context against the target model '%s'\n", __func__, model_path.c_str());
llama_context * ctx_dft = llama_init_from_model(model_tgt, cparams);
if (ctx_dft == nullptr) {
LOG_ERR("%s: failed to create MTP context\n", __func__);
return;
}
pimpl->context.reset(ctx_dft);
}
}
common_speculative_init_result::~common_speculative_init_result() = default;
llama_model * common_speculative_init_result::model() {
return pimpl->model.get();
}
llama_context * common_speculative_init_result::context() {
return pimpl->context.get();
}
common_speculative_init_result_ptr common_speculative_init_from_params(common_params & params, llama_model * model_tgt, llama_context * ctx_tgt) {
return std::make_unique<common_speculative_init_result>(params, model_tgt, ctx_tgt);
}
// initialization of the speculative decoding system
//
common_speculative * common_speculative_init(common_params_speculative & params, uint32_t n_seq) {
common/speculative.h
-18
View File
@@ -23,8 +23,6 @@ std::string common_speculative_type_to_str(enum common_speculative_type type);
// return the max number of draft tokens based on the speculative parameters
int32_t common_speculative_n_max(const common_params_speculative * spec);
common_params common_base_params_to_speculative(const common_params & params);
common_speculative * common_speculative_init(common_params_speculative & params, uint32_t n_seq);
void common_speculative_free(common_speculative * spec);
@@ -82,19 +80,3 @@ struct common_speculative_deleter {
};
typedef std::unique_ptr<common_speculative, common_speculative_deleter> common_speculative_ptr;
struct common_speculative_init_result {
common_speculative_init_result(common_params & params, llama_model * model_tgt, llama_context * ctx_tgt);
~common_speculative_init_result();
llama_model * model();
llama_context * context();
private:
struct impl;
std::unique_ptr<impl> pimpl;
};
using common_speculative_init_result_ptr = std::unique_ptr<common_speculative_init_result>;
common_speculative_init_result_ptr common_speculative_init_from_params(common_params & params, llama_model * model_tgt, llama_context * ctx_tgt);
common/subproc.cpp
-143
View File
@@ -1,143 +0,0 @@
#include "subproc.h"
bool common_subproc::is_supported() {
#ifdef LLAMA_SUBPROCESS
return true;
#else
return false;
#endif
}
#ifdef LLAMA_SUBPROCESS
static std::vector<char *> to_cstr_vec(const std::vector<std::string> & v) {
std::vector<char *> r;
r.reserve(v.size() + 1);
for (const auto & s : v) {
r.push_back(const_cast<char *>(s.c_str()));
}
r.push_back(nullptr);
return r;
}
common_subproc::~common_subproc() {
if (is_created) {
subprocess_destroy(&proc);
is_created = false;
}
}
bool common_subproc::create(
const std::vector<std::string> & args,
int options,
const std::vector<std::string> & env,
const char * cwd) {
auto argv = to_cstr_vec(args);
int result;
if (env.empty() && cwd == nullptr) {
result = subprocess_create(argv.data(), options, &proc);
} else {
auto envp = to_cstr_vec(env);
result = subprocess_create_ex(argv.data(), options, env.empty() ? nullptr : envp.data(), cwd, &proc);
}
is_created = result == 0;
return is_created;
}
bool common_subproc::has_handle() const {
if (!is_created) {
return false;
}
#if defined(_WIN32)
return proc.hProcess != nullptr;
#else
return proc.child > 0;
#endif
}
bool common_subproc::alive() {
return is_created && subprocess_alive(&proc);
}
FILE * common_subproc::stdin_file() {
return is_created ? subprocess_stdin(&proc) : nullptr;
}
FILE * common_subproc::stdout_file() {
return is_created ? subprocess_stdout(&proc) : nullptr;
}
FILE * common_subproc::stderr_file() {
return is_created ? subprocess_stderr(&proc) : nullptr;
}
void common_subproc::close_stdin() {
if (is_created && proc.stdin_file) {
fclose(proc.stdin_file);
proc.stdin_file = nullptr;
}
}
void common_subproc::terminate() {
if (has_handle()) {
subprocess_terminate(&proc);
}
}
int common_subproc::join() {
int exit_code = -1;
if (is_created) {
subprocess_join(&proc, &exit_code);
subprocess_destroy(&proc);
is_created = false;
}
return exit_code;
}
#else // !LLAMA_SUBPROCESS
common_subproc::~common_subproc() = default;
bool common_subproc::create(
const std::vector<std::string> &,
int,
const std::vector<std::string> &,
const char *) {
(void)(proc);
(void)(is_created);
return false;
}
bool common_subproc::has_handle() const {
return false;
}
bool common_subproc::alive() {
return false;
}
FILE * common_subproc::stdin_file() {
return nullptr;
}
FILE * common_subproc::stdout_file() {
return nullptr;
}
FILE * common_subproc::stderr_file() {
return nullptr;
}
void common_subproc::close_stdin() {
}
void common_subproc::terminate() {
}
int common_subproc::join() {
return -1;
}
#endif // LLAMA_SUBPROCESS
common/subproc.h
-59
View File
@@ -1,59 +0,0 @@
#pragma once
#include <atomic>
#include <cstdio>
#include <string>
#include <vector>
#ifdef LLAMA_SUBPROCESS
#include <sheredom/subprocess.h>
#else
// dummy values to allow compilation when subprocess is disabled
struct subprocess_s {};
static constexpr int subprocess_option_no_window = 0;
static constexpr int subprocess_option_combined_stdout_stderr = 0;
static constexpr int subprocess_option_inherit_environment = 0;
static constexpr int subprocess_option_search_user_path = 0;
#endif
// RAII-style wrapper around https://github.com/sheredom/subprocess.h,
// exposing method calls instead of free functions operating on subprocess_s.
struct common_subproc {
common_subproc() = default;
~common_subproc();
common_subproc(const common_subproc &) = delete;
common_subproc & operator=(const common_subproc &) = delete;
// spawn a child process; if env is non-empty it replaces the child's environment
// (do not combine with subprocess_option_inherit_environment)
bool create(
const std::vector<std::string> & args,
int options,
const std::vector<std::string> & env = {},
const char * cwd = nullptr);
bool alive();
// true if LLAMA_SUBPROCESS was enabled at build time; when false, create() always fails
static bool is_supported();
FILE * stdin_file();
FILE * stdout_file();
FILE * stderr_file();
// close stdin and detach it from the process, so a later join()/destroy() won't double-close it;
// use this after writing all input to signal EOF to the child while it's still running
void close_stdin();
void terminate();
// wait for the process to exit, release the underlying handle and return its exit code
int join();
private:
subprocess_s proc {};
std::atomic<bool> is_created{false};
bool has_handle() const;
};
common/trie.cpp
-123
View File
@@ -1,123 +0,0 @@
#include "trie.h"
#include "unicode.h"
#include <deque>
common_trie::match_result common_trie::check_at(std::string_view sv, size_t start_pos) const {
size_t current = 0; // Start at root
size_t pos = start_pos;
// LOG_DBG("%s: checking at pos %zu, sv='%s'\n", __func__, start_pos, std::string(sv).c_str());
while (pos < sv.size()) {
auto result = common_parse_utf8_codepoint(sv, pos);
if (result.status != utf8_parse_result::SUCCESS) {
break;
}
auto it = nodes[current].children.find(result.codepoint);
if (it == nodes[current].children.end()) {
// Can't continue matching
return match_result{match_result::NO_MATCH};
}
current = it->second;
pos += result.bytes_consumed;
// Check if we've matched a complete word
if (nodes[current].pattern >= 0) {
return match_result{match_result::COMPLETE_MATCH};
}
}
// Reached end of input while still in the trie (not at root)
if (current != 0) {
// We're in the middle of a potential match
return match_result{match_result::PARTIAL_MATCH};
}
// Reached end at root (no match)
return match_result{match_result::NO_MATCH};
}
int32_t common_trie::insert(const std::string & word) {
std::vector<uint32_t> symbols;
size_t pos = 0;
while (pos < word.length()) {
auto result = common_parse_utf8_codepoint(word, pos);
if (result.status != utf8_parse_result::SUCCESS) {
break;
}
symbols.push_back(result.codepoint);
pos += result.bytes_consumed;
}
return insert(symbols);
}
int32_t common_trie::insert(const std::vector<uint32_t> & symbols) {
size_t current = 0;
for (uint32_t ch : symbols) {
auto it = nodes[current].children.find(ch);
if (it == nodes[current].children.end()) {
size_t child = create_node();
nodes[current].children[ch] = child;
current = child;
} else {
current = it->second;
}
}
if (nodes[current].pattern < 0) {
nodes[current].pattern = n_patterns++;
}
return nodes[current].pattern;
}
common_aho_corasick::common_aho_corasick(common_trie trie) : t(std::move(trie)) {
const auto & nodes = t.nodes;
const size_t n = nodes.size();
fail.assign(n, 0);
order.reserve(n);
std::deque<size_t> queue{ 0 };
while (!queue.empty()) {
size_t u = queue.front();
queue.pop_front();
order.push_back(u);
for (const auto & [ch, v] : nodes[u].children) {
if (u != 0) {
size_t f = fail[u];
while (f && nodes[f].children.find(ch) == nodes[f].children.end()) {
f = fail[f];
}
auto it = nodes[f].children.find(ch);
fail[v] = (it != nodes[f].children.end() && it->second != v) ? it->second : 0;
}
queue.push_back(v);
}
}
// fail[u] points to a strictly shorter suffix, so the first pattern found on
// the fail chain (including u itself) is the longest pattern ending at u
match.assign(n, -1);
for (size_t u : order) {
match[u] = nodes[u].pattern >= 0 ? nodes[u].pattern : (u != 0 ? match[fail[u]] : -1);
}
for (const auto & node : nodes) {
for (const auto & [ch, v] : node.children) {
alphabet.insert(ch);
}
}
}
size_t common_aho_corasick::next(size_t state, uint32_t ch) const {
const auto & nodes = t.nodes;
while (state && nodes[state].children.find(ch) == nodes[state].children.end()) {
state = fail[state];
}
auto it = nodes[state].children.find(ch);
return it != nodes[state].children.end() ? it->second : 0;
}
common/trie.h
-73
View File
@@ -1,73 +0,0 @@
#pragma once
#include <cstdint>
#include <map>
#include <set>
#include <string>
#include <string_view>
#include <vector>
// Trie for matching multiple literals.
// This is used in common_peg_until_parser and to build a GBNF exclusion grammar
struct common_trie {
struct node {
std::map<uint32_t, size_t> children; // Use uint32_t to store Unicode codepoints
int32_t pattern = -1; // index of the pattern ending at this node, -1 if none
};
std::vector<node> nodes;
common_trie() {
create_node(); // root node
}
common_trie(const std::vector<std::string> & words) : common_trie() {
for (const auto & w : words) {
insert(w);
}
}
enum match_result { NO_MATCH, PARTIAL_MATCH, COMPLETE_MATCH };
// Check if a delimiter starts at the given position
match_result check_at(std::string_view sv, size_t start_pos) const;
// Insert a word as a sequence of Unicode codepoints, returns its pattern index
int32_t insert(const std::string & word);
// Insert a raw symbol sequence, returns its pattern index (insertion order,
// duplicates keep the first index)
int32_t insert(const std::vector<uint32_t> & symbols);
private:
int32_t n_patterns = 0;
size_t create_node() {
size_t index = nodes.size();
nodes.emplace_back();
return index;
}
};
// Aho-Corasick automaton
struct common_aho_corasick {
common_trie t;
std::vector<size_t> fail; // failure links
std::vector<size_t> order; // states in BFS order
std::vector<int32_t> match; // longest pattern ending at each state (directly or via a suffix link), -1 if none
std::set<uint32_t> alphabet; // every character with a transition
common_aho_corasick(common_trie trie);
common_aho_corasick(const std::vector<std::string> & strings)
: common_aho_corasick(common_trie(strings)) {}
size_t num_states() const { return t.nodes.size(); }
bool is_terminal(size_t s) const { return match[s] >= 0; }
// index of the longest pattern ending at this state, -1 if none
int32_t match_pattern(size_t s) const { return match[s]; }
// follow failure links until a transition on `ch` exists.
size_t next(size_t state, uint32_t ch) const;
};
conversion/__init__.py
-6
View File
@@ -18,7 +18,6 @@ __all__ = [
TEXT_MODEL_MAP: dict[str, str] = {
"AfmoeForCausalLM": "afmoe",
"LagunaForCausalLM": "laguna",
"ApertusForCausalLM": "llama",
"ArceeForCausalLM": "llama",
"ArcticForCausalLM": "arctic",
@@ -32,7 +31,6 @@ TEXT_MODEL_MAP: dict[str, str] = {
"BertForSequenceClassification": "bert",
"BertModel": "bert",
"BitnetForCausalLM": "bitnet",
"BitNetForCausalLM": "bitnet",
"BloomForCausalLM": "bloom",
"BloomModel": "bloom",
"CamembertModel": "bert",
@@ -108,7 +106,6 @@ TEXT_MODEL_MAP: dict[str, str] = {
"HunYuanDenseV1ForCausalLM": "hunyuan",
"HunYuanMoEV1ForCausalLM": "hunyuan",
"HunYuanVLForConditionalGeneration": "hunyuan",
"HYV3ForCausalLM": "hunyuan",
"IQuestCoderForCausalLM": "llama",
"InternLM2ForCausalLM": "internlm",
"InternLM3ForCausalLM": "internlm",
@@ -158,8 +155,6 @@ TEXT_MODEL_MAP: dict[str, str] = {
"MiniCPMForCausalLM": "minicpm",
"MiniCPMV4_6ForConditionalGeneration": "minicpm",
"MiniMaxM2ForCausalLM": "minimax",
"MiniMaxM3SparseForCausalLM": "minimax",
"MiniMaxM3SparseForConditionalGeneration": "minimax",
"Ministral3ForCausalLM": "mistral3",
"Mistral3ForConditionalGeneration": "mistral3",
"MistralForCausalLM": "llama",
@@ -269,7 +264,6 @@ MMPROJ_MODEL_MAP: dict[str, str] = {
"Gemma4UnifiedForConditionalGeneration": "gemma",
"Glm4vForConditionalGeneration": "qwen3vl",
"Glm4vMoeForConditionalGeneration": "qwen3vl",
"Glm5vForConditionalGeneration": "kimivl",
"GlmOcrForConditionalGeneration": "qwen3vl",
"GlmasrModel": "ultravox",
"Granite4VisionForConditionalGeneration": "granite",
conversion/base.py
+3 -8
View File
@@ -109,9 +109,7 @@ class ModelBase:
sentence_transformers_dense_modules: bool = False
# MTP (multi-token prediction) export modes; set by main() before instantiation.
# Architectures that implement the filtering/export behavior opt in by
# setting supports_mtp_export = True on their model class or a mixin.
supports_mtp_export: bool = False
# Architectures opt in by overriding the handling (see _Qwen35MtpMixin).
mtp_only: bool = False
no_mtp: bool = False
@@ -1156,7 +1154,7 @@ class TextModel(ModelBase):
or "projector." in name or "pre_mm_projector_norm" in name \
or "image_newline" in name or "view_seperator" in name \
or "patch_embed" in name or "patch_embedding" in name \
or "patch_merger." in name or "patch_merge_mlp." in name or "model.connector." in name:
or "patch_merger." in name or "model.connector." in name:
return None
return super().filter_tensors(item)
@@ -1203,7 +1201,7 @@ class TextModel(ModelBase):
self.gguf_writer.add_embedding_length(n_embd)
logger.info(f"gguf: embedding length = {n_embd}")
if (n_ff := self.find_hparam(["prefix_dense_intermediate_size", "dense_intermediate_size", "intermediate_size", "n_inner", "hidden_dim"], optional=True)) is not None:
if (n_ff := self.find_hparam(["prefix_dense_intermediate_size", "intermediate_size", "n_inner", "hidden_dim"], optional=True)) is not None:
self.gguf_writer.add_feed_forward_length(n_ff)
logger.info(f"gguf: feed forward length = {n_ff}")
@@ -1682,9 +1680,6 @@ class TextModel(ModelBase):
if chkhsh == "9dcf830ee9990cdbf78cc523a5f7bd9ad8f3f9890c2d3581d2785ad10f07049d":
# ref: https://huggingface.co/JetBrains/Mellum2-12B-A2.5B-Base
res = "mellum2"
if chkhsh == "972da7b59cec44d1f0a490a86c96df53859e486e481563e5dddac155013d87ac":
# ref: https://huggingface.co/poolside/Laguna-XS.2
res = "laguna"
if res is None:
logger.warning("\n")
conversion/bert.py
+1 -2
View File
@@ -369,13 +369,12 @@ class NomicBertModel(BertModel):
return super().filter_tensors(item)
def modify_tensors(self, data_torch: torch.Tensor, name: str, bid: int | None) -> Iterable[tuple[str, torch.Tensor]]:
n_experts = self.find_hparam(["num_local_experts", "num_experts"])
if "mlp.experts.mlp.w1" in name:
n_experts = self.find_hparam(["num_local_experts", "num_experts"])
data_torch = data_torch.view(n_experts, self.hparams["n_inner"], self.hparams["n_embd"])
name += ".weight"
if "mlp.experts.mlp.w2" in name:
n_experts = self.find_hparam(["num_local_experts", "num_experts"])
data_torch = data_torch.view(n_experts, self.hparams["n_inner"], self.hparams["n_embd"])
data_torch = data_torch.transpose(1, 2)
name += ".weight"
conversion/bitnet.py
+1 -1
View File
@@ -8,7 +8,7 @@ if TYPE_CHECKING:
from .base import ModelBase, TextModel, gguf
@ModelBase.register("BitnetForCausalLM", "BitNetForCausalLM")
@ModelBase.register("BitnetForCausalLM")
class BitnetModel(TextModel):
model_arch = gguf.MODEL_ARCH.BITNET
conversion/glm.py
-3
View File
@@ -237,9 +237,6 @@ class GlmMoeDsaModel(DeepseekV2Model):
self.gguf_writer.add_indexer_head_count(self.hparams["index_n_heads"])
self.gguf_writer.add_indexer_key_length(self.hparams["index_head_dim"])
self.gguf_writer.add_indexer_top_k(self.hparams["index_topk"])
if (indexer_types := self.hparams.get("indexer_types")) is not None:
indexer_types = [t == "full" for t in indexer_types]
self.gguf_writer.add_indexer_types(indexer_types)
@ModelBase.register("SolarOpenForCausalLM")
conversion/hunyuan.py
-110
View File
@@ -1,7 +1,6 @@
from __future__ import annotations
import json
import re
from pathlib import Path
from typing import Callable, Iterable, TYPE_CHECKING
@@ -338,12 +337,6 @@ class HunyuanVLTextModel(HunYuanModel):
def __init__(self, dir_model: Path, *args, **kwargs):
super().__init__(dir_model, *args, **kwargs)
# transformers 5.13.0 encodes HunyuanVL XD-RoPE as dynamic + mrope_section.
# Normalize it to avoid the HunYuan dynamic-RoPE context assertion.
if self.rope_parameters.get("rope_type") == "dynamic" and "mrope_section" in self.rope_parameters:
self.rope_parameters["rope_type"] = "xdrope"
self.rope_parameters["type"] = "xdrope"
self.rope_parameters["xdrope_section"] = list(self.rope_parameters["mrope_section"])
def set_gguf_parameters(self):
super().set_gguf_parameters()
@@ -362,106 +355,3 @@ class HunyuanVLTextModel(HunYuanModel):
self.gguf_writer.add_context_length(ctx_len)
self.gguf_writer.add_rope_dimension_sections(list(self.rope_parameters["xdrope_section"]))
@ModelBase.register("HYV3ForCausalLM")
class HYV3Model(TextModel):
model_arch = gguf.MODEL_ARCH.HY_V3
supports_mtp_export = True
# Trunk layer count, stashed before indexing so the classmethod
# filter_tensors can identify the appended MTP block(s) (mirrors
# Step35Model).
_n_main_layers: int | None = None
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
# NextN/MTP layers are appended past num_hidden_layers; extend the
# tensor map so the MTP block's tensors resolve to blk.<n>.* names.
n_nextn = int(self.hparams.get("num_nextn_predict_layers", 0))
if n_nextn > 0 and not self.no_mtp:
self.block_count += n_nextn
self.tensor_map = gguf.get_tensor_name_map(self.model_arch, self.block_count)
def index_tensors(self, remote_hf_model_id: str | None = None):
type(self)._n_main_layers = self.hparams["num_hidden_layers"]
return super().index_tensors(remote_hf_model_id=remote_hf_model_id)
def set_vocab(self):
self._set_vocab_gpt2()
def set_gguf_parameters(self):
super().set_gguf_parameters()
self.gguf_writer.add_expert_feed_forward_length(self.hparams["moe_intermediate_size"])
self.gguf_writer.add_expert_shared_feed_forward_length(
self.hparams["moe_intermediate_size"] * self.hparams.get("num_shared_experts", 1)
)
self.gguf_writer.add_expert_weights_norm(self.hparams.get("route_norm", True))
self.gguf_writer.add_expert_weights_scale(float(self.hparams.get("router_scaling_factor", 1.0)))
# sigmoid router with expert selection bias
self.gguf_writer.add_expert_gating_func(gguf.ExpertGatingFuncType.SIGMOID)
n_nextn = int(self.hparams.get("num_nextn_predict_layers", 0))
if n_nextn > 0 and not self.no_mtp:
self.gguf_writer.add_nextn_predict_layers(n_nextn)
@classmethod
def filter_tensors(cls, item: tuple[str, Callable[[], Tensor]]) -> tuple[str, Callable[[], Tensor]] | None:
if (titem := super().filter_tensors(item)) is None:
return None
name, gen = titem
# HY V3 appends the MTP block(s) past num_hidden_layers.
assert cls._n_main_layers is not None
is_mtp = (m := re.match(r"model\.layers\.(\d+)\.", name)) is not None and int(m.group(1)) >= cls._n_main_layers
# --no-mtp: drop the appended MTP block(s) entirely.
if is_mtp and cls.no_mtp:
return None
# --mtp: keep ONLY MTP-block tensors plus the shared embeddings/norm/
# lm_head (so the resulting GGUF carries just the draft head).
if cls.mtp_only and not is_mtp and name not in (
"model.embed_tokens.weight", "model.norm.weight", "lm_head.weight",
):
return None
# The MTP block's trailing final_layernorm (applied after the decoder
# block, before the shared LM head) maps to nextn.shared_head_norm.
if is_mtp:
name = name.replace(".final_layernorm.", ".shared_head.norm.")
return name, gen
_experts: list[dict[str, Tensor]] | None = None
def modify_tensors(self, data_torch: Tensor, name: str, bid: int | None) -> Iterable[tuple[str, Tensor]]:
# merge the per-expert tensors into stacked 3d tensors
if name.startswith("model.layers.") and ".mlp.experts." in name:
n_experts = self.find_hparam(["num_local_experts", "num_experts"])
assert bid is not None
if self._experts is None:
self._experts = [{} for _ in range(self.block_count)]
self._experts[bid][name] = data_torch
if len(self._experts[bid]) >= n_experts * 3:
for w_name in ("down_proj", "gate_proj", "up_proj"):
datas: list[Tensor] = []
for xid in range(n_experts):
ename = f"model.layers.{bid}.mlp.experts.{xid}.{w_name}.weight"
datas.append(self._experts[bid][ename])
del self._experts[bid][ename]
merged = torch.stack(datas, dim=0)
yield from super().modify_tensors(merged, f"model.layers.{bid}.mlp.experts.{w_name}.weight", bid)
return
yield from super().modify_tensors(data_torch, name, bid)
def prepare_tensors(self):
super().prepare_tensors()
if self._experts is not None:
experts = [k for d in self._experts for k in d.keys()]
if experts:
raise ValueError(f"Unprocessed experts: {experts}")
conversion/kimivl.py
-16
View File
@@ -152,19 +152,3 @@ class KimiK25Model(MmprojModel):
name = name.replace(".proj.2.", ".proj.linear_2.")
yield from super().modify_tensors(data_torch, name, bid)
@ModelBase.register("Glm5vForConditionalGeneration")
class Glm5vModel(KimiK25Model):
"""GLM-5.2-Vision MoonViT3d encoder and projector
Uses the same vision encoder and projector as Kimi-K2.5, so it reuses the
kimik25 projector type. The image begin/end tokens differ, but they are
resolved at runtime from the text model vocab.
"""
def modify_tensors(self, data_torch: Tensor, name: str, bid: int | None) -> Iterable[tuple[str, Tensor]]:
if name.startswith("mm_projector.linear_"):
name = name.replace("mm_projector.linear_", "mm_projector.proj.linear_", 1)
yield from super().modify_tensors(data_torch, name, bid)
conversion/laguna.py
-207
View File
@@ -1,207 +0,0 @@
from __future__ import annotations
import re
from collections.abc import Iterable
from typing import TYPE_CHECKING
import torch
if TYPE_CHECKING:
from torch import Tensor
from .base import ModelBase, TextModel, gguf, logger
@ModelBase.register("LagunaForCausalLM")
class LagunaModel(TextModel):
model_arch = gguf.MODEL_ARCH.LAGUNA
_experts: list[dict] | None = None
_gate_types: list[str] | None = None
# --- vocab ---------------------------------------------------------------
def set_vocab(self) -> None:
self._set_vocab_gpt2()
# Some Laguna releases wrap the chat template in tokenizer_config.json as
# "{% include 'chat_template.jinja' %}", which SpecialVocab embeds verbatim
# and llama.cpp's jinja engine cannot process. Prefer the resolved template
# from the chat_template.jinja file so the GGUF is self-contained.
tmpl_file = self.dir_model / "chat_template.jinja"
if tmpl_file.is_file():
self.gguf_writer.add_chat_template(tmpl_file.read_text(encoding="utf-8"))
logger.info("gguf: embedded resolved chat_template.jinja (overriding include directive)")
# eos_token_id is a list [2, 24]: token 2 (EOS, also BOS) and token 24
# (</assistant>, the turn-end). _set_vocab_gpt2 only records the scalar
# eos, so register the extra id as eot; llama.cpp folds eot into its EOG
# set, so the model halts on </assistant> natively.
eos_ids = self.hparams.get("eos_token_id")
if isinstance(eos_ids, list):
bos_id = self.hparams.get("bos_token_id")
extra = [e for e in eos_ids if e != bos_id]
if extra:
self.gguf_writer.add_eot_token_id(extra[0])
logger.info(f"gguf: registered eot_token_id={extra[0]} from eos list {eos_ids}")
def get_vocab_base(self) -> tuple[list[str], list[int], str]:
# </assistant> is the assistant turn-end (registered as eot below). The
# HF tokenizer flags it special=false, so the base classifies it as
# USER_DEFINED and llama.cpp renders its text into generated content,
# leaking "</assistant>" and breaking response parsing. It is a control
# marker, so promote it to CONTROL: llama.cpp then treats it as
# end-of-generation and suppresses its text.
tokens, toktypes, tokpre = super().get_vocab_base()
for i, tok in enumerate(tokens):
if tok == "</assistant>":
toktypes[i] = gguf.TokenType.CONTROL
logger.info(f"gguf: marked </assistant> (id {i}) as CONTROL token")
return tokens, toktypes, tokpre
# --- hparams -------------------------------------------------------------
def set_gguf_parameters(self) -> None:
super().set_gguf_parameters()
hparams = self.hparams
# super() does not emit vocab_size for the gpt2 vocab path; head_count is
# overridden with a per-layer array (XS.2 varies heads per layer via
# num_attention_heads_per_layer; M.1 is uniform and omits it).
self.gguf_writer.add_vocab_size(hparams["vocab_size"])
per_layer_heads = hparams.get("num_attention_heads_per_layer")
if not per_layer_heads:
per_layer_heads = [hparams["num_attention_heads"]] * hparams["num_hidden_layers"]
assert len(per_layer_heads) == hparams["num_hidden_layers"], (
f"num_attention_heads_per_layer length {len(per_layer_heads)} != "
f"num_hidden_layers {hparams['num_hidden_layers']}"
)
self.gguf_writer.add_head_count(per_layer_heads)
# Resolve + validate the attention gate type now so an inconsistent
# `gating` field fails at conversion time. See _attn_gate_types.
self._attn_gate_types()
# SWA window size (M.1 has none -> key omitted, swa_type stays NONE).
sliding_window = hparams.get("sliding_window") or 0
if sliding_window > 0:
self.gguf_writer.add_sliding_window(sliding_window)
# MoE (expert_count / expert_used_count come from super().set_gguf_parameters())
self.gguf_writer.add_expert_feed_forward_length(hparams["moe_intermediate_size"])
self.gguf_writer.add_expert_shared_feed_forward_length(hparams["shared_expert_intermediate_size"])
self.gguf_writer.add_expert_weights_norm(True) # HF reference always sum-normalises after top-k
self.gguf_writer.add_expert_weights_scale(float(hparams["moe_routed_scaling_factor"]))
self.gguf_writer.add_expert_gating_func(gguf.ExpertGatingFuncType.SIGMOID)
# Leading dense layers (XS.2 has 1, M.1 has 3) before the MoE layers.
mlp_layer_types: list[str] = hparams["mlp_layer_types"]
leading_dense = 0
for t in mlp_layer_types:
if t == "dense":
leading_dense += 1
else:
break
self.gguf_writer.add_leading_dense_block_count(leading_dense)
# Per-layer-type RoPE dimension count (partial rotary). base emits
# rope_freq_base(_swa) and the YaRN params from self.rope_parameters.
head_dim = hparams["head_dim"]
full_rope = self.rope_parameters["full_attention"]
self.gguf_writer.add_rope_dimension_count(
int(head_dim * float(full_rope.get("partial_rotary_factor", 1.0))))
swa_rope = self.rope_parameters.get("sliding_attention")
if swa_rope is not None:
self.gguf_writer.add_rope_dimension_count_swa(
int(head_dim * float(swa_rope.get("partial_rotary_factor", 1.0))))
def _attn_gate_types(self) -> list[str]:
"""Per-layer attention output gate type: "per_head" or "per_element".
`gating_types` (per layer) is authoritative when present; otherwise the
scalar `gating` field is used (the "per-element"/"per-head" string, or
the legacy boolean True == per-head, as in Laguna-XS.2).
Fails loudly when the model is per-element but the `gating` field does
not declare that as a string: runtimes that key off `gating` (vLLM,
transformers) ignore gating_types and read a bare boolean True as
per-head, silently corrupting the model. Surfacing it here keeps a
broken checkpoint from being packaged as if it were fine.
"""
if self._gate_types is not None:
return self._gate_types
hparams = self.hparams
n_layer = hparams["num_hidden_layers"]
gating = hparams.get("gating")
gating_types = hparams.get("gating_types")
def _norm(t: object) -> str:
sval = str(t).replace("-", "_")
if sval in ("per_element", "per_head"):
return sval
raise ValueError(f"Laguna: unrecognised attention gate type {t!r}")
if gating_types:
assert len(gating_types) == n_layer, (
f"gating_types length {len(gating_types)} != num_hidden_layers {n_layer}")
types = [_norm(t) for t in gating_types]
elif isinstance(gating, str):
types = [_norm(gating)] * n_layer
elif gating is True:
types = ["per_head"] * n_layer
else:
raise ValueError(
f"Laguna: cannot determine attention gate type "
f"(gating={gating!r}, gating_types={gating_types!r})")
if any(t == "per_element" for t in types) and not (
isinstance(gating, str) and _norm(gating) == "per_element"):
raise ValueError(
f"Laguna config declares a per-element attention gate but "
f"`gating`={gating!r} is not the string \"per-element\". Runtimes that "
f"read `gating` (vLLM, transformers) will mis-handle this checkpoint as "
f"per-head. Set gating=\"per-element\" in the source config.")
self._gate_types = types
return types
# --- tensor handling -----------------------------------------------------
def modify_tensors(self, data_torch: Tensor, name: str, bid: int | None) -> Iterable[tuple[str, Tensor]]:
# Per-expert MoE weights: model.layers.{bid}.mlp.experts.{xid}.{w}.weight.
# Only the NUMBERED per-expert weights are stacked; the router bias
# (mlp.experts.e_score_correction_bias) takes the normal mapping path.
if re.search(r"mlp\.experts\.\d+\.", name):
n_experts = self.find_hparam(["num_local_experts", "num_experts"])
assert bid is not None
if self._experts is None:
self._experts = [{} for _ in range(self.block_count)]
self._experts[bid][name] = data_torch
needed = [f"model.layers.{bid}.mlp.experts.{x}.{w}.weight"
for x in range(n_experts) for w in ("gate_proj", "up_proj", "down_proj")]
if all(e in self._experts[bid] for e in needed):
for w_name in ["gate_proj", "up_proj", "down_proj"]:
datas = [self._experts[bid][f"model.layers.{bid}.mlp.experts.{x}.{w_name}.weight"]
for x in range(n_experts)]
stacked = torch.stack(datas, dim=0)
merged = f"model.layers.{bid}.mlp.experts.{w_name}.weight"
yield from TextModel.modify_tensors(self, stacked, merged, bid)
self._experts[bid].clear()
return
return
# Cross-check the gate projection width against the declared gate type;
# a mismatch means the weights and config disagree -> fail, do not guess.
if bid is not None and name.endswith("self_attn.g_proj.weight"):
heads = (self.hparams.get("num_attention_heads_per_layer")
or [self.hparams["num_attention_heads"]] * self.hparams["num_hidden_layers"])
n_head = heads[bid]
head_dim = self.hparams["head_dim"]
gate_type = self._attn_gate_types()[bid]
expected = n_head * head_dim if gate_type == "per_element" else n_head
out_features = int(data_torch.shape[0])
if out_features != expected:
raise ValueError(
f"Laguna layer {bid}: g_proj output width {out_features} contradicts the "
f"declared {gate_type} gate (expected {expected}); weights and config disagree.")
yield from TextModel.modify_tensors(self, data_torch, name, bid)
conversion/minimax.py
+1 -36
View File
@@ -23,7 +23,7 @@ class MiniMaxM2Model(TextModel):
def modify_tensors(self, data_torch: Tensor, name: str, bid: int | None):
# merge expert weights
if "block_sparse_moe.experts." in name:
if 'experts' in name:
n_experts = self.find_hparam(["num_local_experts", "num_experts"])
assert bid is not None
@@ -52,38 +52,3 @@ class MiniMaxM2Model(TextModel):
return
yield from super().modify_tensors(data_torch, name, bid)
@ModelBase.register("MiniMaxM3SparseForCausalLM", "MiniMaxM3SparseForConditionalGeneration")
class MiniMaxM3Model(MiniMaxM2Model):
model_arch = gguf.MODEL_ARCH.MINIMAXM3
def set_gguf_parameters(self):
super().set_gguf_parameters()
self.gguf_writer.add_expert_shared_count(self.find_hparam(["n_shared_experts"]))
self.gguf_writer.add_expert_weights_scale(self.find_hparam(["routed_scaling_factor"]))
self.gguf_writer.add_expert_weights_norm(True)
sac = self.find_hparam(["sparse_attention_config"])
self.gguf_writer.add_indexer_head_count(sac["sparse_num_index_heads"])
self.gguf_writer.add_indexer_key_length(sac["sparse_index_dim"])
self.gguf_writer.add_indexer_top_k(sac["sparse_topk_blocks"])
self.gguf_writer.add_indexer_block_size(sac["sparse_block_size"])
self.gguf_writer.add_indexer_local_blocks(sac["sparse_local_block"])
moe_layer_freq = self.find_hparam(["moe_layer_freq"])
n_dense = 0
for v in moe_layer_freq:
if v == 0:
n_dense += 1
else:
break
self.gguf_writer.add_leading_dense_block_count(n_dense)
def modify_tensors(self, data_torch: Tensor, name: str, bid: int | None):
# Gemma-style (1 + w) RMSNorm: bake the +1 in so llama.cpp can use plain RMSNorm
if name.endswith("norm.weight"):
data_torch = data_torch + 1.0
yield from super().modify_tensors(data_torch, name, bid)
conversion/qwen.py
+1 -16
View File
@@ -1,7 +1,5 @@
from __future__ import annotations
import json
from typing import Any, Callable, Iterable, TYPE_CHECKING
import torch
@@ -543,7 +541,6 @@ class _Qwen35MtpMixin:
`mtp.*` to the standard layer-indexed nextn naming so the existing
tensor_map handles them."""
supports_mtp_export = True
hparams: dict[str, Any]
model_arch: gguf.MODEL_ARCH
gguf_writer: gguf.GGUFWriter
@@ -643,19 +640,7 @@ class DFlashModel(Qwen3Model):
logger.info(f"DFlash: Using tokenizer from target model: {self.target_model_dir}")
original_dir = self.dir_model
self.dir_model = self.target_model_dir
# Reuse the target model's own vocab handler (e.g. Gemma-4 needs its
# own tokenizer logic, not the Qwen default).
from . import get_model_class
with open(self.target_model_dir / "config.json", "r", encoding="utf-8") as f:
target_arch = json.load(f)["architectures"][0]
target_cls = get_model_class(target_arch)
if target_cls is not type(self):
target_cls.set_vocab(self) # ty: ignore[unresolved-attribute]
else:
super().set_vocab()
super().set_vocab()
self.dir_model = original_dir
mask_token_id = self.hparams.get("dflash_config", {}).get("mask_token_id")
conversion/step3.py
-1
View File
@@ -98,7 +98,6 @@ class Step3VLTextModel(Qwen3Model):
@ModelBase.register("Step3p5ForCausalLM", "Step3p7ForConditionalGeneration")
class Step35Model(TextModel):
model_arch = gguf.MODEL_ARCH.STEP35
supports_mtp_export = True
# The --mtp / --no-mtp toggles are ModelBase.mtp_only / no_mtp (set in
# convert_hf_to_gguf.py main()). Unlike Qwen3.5, which stores MTP under a
convert_hf_to_gguf.py
+4 -2
View File
@@ -259,8 +259,10 @@ def main() -> None:
sys.exit(1)
if args.mtp or args.no_mtp:
if not model_class.supports_mtp_export:
logger.error("--mtp / --no-mtp are not supported for %s", model_architecture)
from conversion.qwen import _Qwen35MtpMixin
from conversion.step3 import Step35Model
if not (issubclass(model_class, _Qwen35MtpMixin) or issubclass(model_class, Step35Model)):
logger.error("--mtp / --no-mtp are only supported for Qwen3.5/3.6 and Step3.5 text variants today")
sys.exit(1)
if args.no_mtp:
model_class.no_mtp = True
convert_hf_to_gguf_update.py
-1
View File
@@ -162,7 +162,6 @@ models = [
{"name": "granite-embed-multi-97m", "tokt": TOKENIZER_TYPE.BPE, "repo": "https://huggingface.co/ibm-granite/granite-embedding-97m-multilingual-r2", },
{"name": "granite-embed-multi-311m", "tokt": TOKENIZER_TYPE.BPE, "repo": "https://huggingface.co/ibm-granite/granite-embedding-311m-multilingual-r2", },
{"name": "mellum2", "tokt": TOKENIZER_TYPE.BPE, "repo": "https://huggingface.co/JetBrains/Mellum2-12B-A2.5B-Base"},
{"name": "laguna", "tokt": TOKENIZER_TYPE.BPE, "repo": "https://huggingface.co/poolside/Laguna-XS.2", },
]
# some models are known to be broken upstream, so we will skip them as exceptions
docs/backend/ET.md
-177
View File
@@ -1,177 +0,0 @@
# llama.cpp for ET
- [Background](#background)
- [Limitations](#limitations)
- [Build](#build)
- [Develop](#develop)
- [Roadmap](#roadmap)
## Background
**ET** is a llama.cpp backend targeting the fully open source manycore
RISC-V accelerator platform [ET-SOC](https://github.com/aifoundry-org/et-man).
## Limitations
The ET backend runs several of the major OSS models with some limitations:
- Only limited set of operations is supported (check [../ops.md](../ops.md)
and [../ops/ET.csv](../ops/ET.csv)).
- Only `q8_0`, `q4_0` (and partially `fp16`, `q4_K`) quantization is supported.
- Only one llama.cpp instance can use device at the same time (current firmware
limitation).
- Limited (but working) MoE model support
As a result of the above, only select models can run fully on ET-SOC
(you can actually run any model llama.cpp supports, but some/most operations
will likely fallback to CPU backend).
Fully supported models:
- Qwen3 models (without MoE), e.g.
[ggml-org/Qwen3-0.6B-GGUF:q8_0](https://huggingface.co/ggml-org/Qwen3-0.6B-GGUF/blob/main/Qwen3-0.6B-Q8_0.gguf) or
[ggml-org/Qwen3-14B-GGUF:q8_0](https://huggingface.co/ggml-org/Qwen3-14B-GGUF/blob/main/Qwen3-14B-Q8_0.gguf).
- Llama3.2 (1B/3B), e.g.
[lmstudio-community/Llama-3.2-1B-Instruct-GGUF:q8_0](https://huggingface.co/lmstudio-community/Llama-3.2-1B-Instruct-GGUF/blob/main/Llama-3.2-1B-Instruct-Q8_0.gguf).
- SmolLM2, e.g.
[unsloth/SmolLM2-135M-Instruct-GGUF:q8_0](https://huggingface.co/unsloth/SmolLM2-135M-Instruct-GGUF/blob/main/SmolLM2-135M-Instruct-Q8_0.gguf)
- Llama 3.1 model family.
- RWKV v7 model family.
- TinyLLaMA
## Build
### I. Prerequisites
1. **Install custom RISC-V toolchain** - Follow instructions at:
[https://github.com/aifoundry-org/riscv-gnu-toolchain/tree/et/aifoundry](https://github.com/aifoundry-org/riscv-gnu-toolchain/tree/et/aifoundry)
2. **Install ET platform** - Follow instructions at:
[https://github.com/aifoundry-org/et-platform](https://github.com/aifoundry-org/et-platform)
Both should be installed to `/opt/et` (or set `ET_TOOLCHAIN` and `ET_PLATFORM`
environment variables accordingly).
```sh
# Set toolchain and ET platform path (/opt/et is default)
export ET_TOOLCHAIN=/opt/et
export ET_PLATFORM=/opt/et
```
### II. Build llama.cpp
Check out llama.cpp with ET backend (this should checkout `et` branch):
```sh
git clone https://github.com/aifoundry-org/llama.cpp
cd llama.cpp
```
Build:
```sh
cmake -B build -DGGML_ET=ON
cmake --build build --config Release
# Optionally:
# cmake --install build
```
Build targeting sysemu backend instead of physical hardware:
```sh
cmake -B build -DGGML_ET=ON -DGGML_ET_SYSEMU=ON
cmake --build build --config Release
```
### III. Run
Run llama.cpp binaries as usual. (Of course, please make sure you have the
ET-SOC device installed and kernel driver loaded).
```sh
llama-cli -m mymodel.gguf
# or
llama-server -hf ggml-org/Qwen3-8B-GGUF:q8_0
```
If you want to run llama.cpp binaries (e.g. `llama-cli`) inside docker
container, you should let it access device files:
```sh
docker run \
--device=/dev/et0_mgmt:/dev/et0_mgmt \
--device=/dev/et0_ops:/dev/et0_ops \
...
```
## Develop
Compute kernels are developed within `ggml/src/ggml-et/et-kernels` folder.
Build is performed using custom RISC-V GNU toolchain and is managed by cmake.
At the moment kernels are build as baremetal elf files, without
standard lib or any other dependencies. All the yummy parts are written
in inline assembler.
Most kernels are very naive with lots of low hanging fruits left:
> [!IMPORTANT]
> Several assembly instructions emmited by the compiler are not implemented
> in hardware and software emulation in firmware is not ready yet.
> Eventually firmware will transparently trap unimplemented instructions
> and will emulate them inside exception handler. Until then, kernel
> build process includes step that checks compiled kernels and fails if any unimplemented
> instructions are found. Problematic ones follow:
> `FDIV.PI`, `FDIVU.PI`, `FREMU.PI`, `FREM.PI`, `FDIV.S`, `FDIV.PS`, `FSQRT.S`, `FSQRT.PS`, `FRSQ.PS`, `FSIN.PS`
> and (long cast) `FCVT.S.L`, `FCVT.S.LU`, `FCVT.L.S`, `FCVT.LU.S`
> What this means, is that for now you should avoid doing any division involving floats,
> any trigonometry or casting longs into floats.
> Some workarounds are implemented in `math_fp.h` (`et_fdiv`, `et_powf` etc) and
> long casting (presuming longs are small enough to fit into 32bits) can be
> done via `int` like `a = (float)(int)(b)`.
> [!TIP]
> There are some slightly higher level helpers (abstracting more
> complex instructions like tensor extension or synchronization primitives)
> inside `et_platform`, directory `et-common-libs/include/etsoc/isa/`. It was
> originally developed for firmware needs and is not included into compute
> kernel build process. Feel free to take ideas/code from there or try linking
> it in.
Before commiting any changes to operations and/or kernels, don't forget
to update supported ops reports (instructions at `docs/ops.md`).
When logging is enabled (e.g. by setting `--log-file` cli param),
each compute kernel run outputs a line with
pipe-delimited key-value pairs containing kernel level performance infomation.
Line is prefixed with `ET_PERF`:
```
ET_PERF|op=MUL_MAT|kernel=mul_mat_f32_Q8_0xf32|duration_us=3112|tensor=Qcur-0|shape=[4096,2,1,1]|start_us=48437862009|end_us=48437865121|flops=67100672
ET_PERF|op=ROPE|kernel=rope_f32|duration_us=9266|tensor=Qcur-0|shape=[128,32,2,1]|start_us=48437865128|end_us=48437874394|mode=0x0|n_dims=128|freq_base=500000.00|freq_scale=1.00
```
Keys depend on the operation, but some are always present.
`flops` in this case counts effective floating point operations and not floating
point operations per second.
You can enable ET-SOC runtime level ET-SOC profiling by setting environment
variable `GGML_ET_PROFILE` to a path. Profiling/tracing results will be written
to `GGML_ET_PROFILE/et_runtime_trace.json` and `GGML_ET_PROFILE/kernel_map` on exit.
### Uberkernel
The in-knernel implementaiton of device dispatch/kernel fusion. The ET SDK has a non-trivial op-to-op gap. `Uberkernel` (name taken from the original Esperanto AI's compiler)
dispatches multiple already existing kernel implementations with device side synchronization. Due to the processor's design, there is no natural memory visibility
horizon between sub-kernel invocations. This makes uberkernel much more difficult to develop and debug. Currently Uberkerel is hidden begind the
`GGML_ET_UBERKERNEL` environment variable and is disabled by default. Setting it to 1 enables it and provides significant performance improvements but is only
validated for the LLaMA 3.2 model family and Qwen 3.5.
## Roadmap
As of writing the documentation the ET backend is capable of running most models and smaller ones at usable speed given the low power profile of the processor. We'd
address the following capabilities in the future:
* Enable Uberkernel for all models
* More oprtator support
* Better TTS model support
* Enable more quantization format support
docs/backend/OPENCL.md
-19
View File
@@ -47,7 +47,6 @@ The llama.cpp OpenCL backend is designed to enable llama.cpp on **Qualcomm Adren
| Adreno GPU | Status |
|:-------------------------------------:|:-------:|
| Adreno 750 (Snapdragon 8 Gen 3) | Support |
| Adreno 810 (Snapdragon 7s Gen 3) | Support |
| Adreno 830 (Snapdragon 8 Elite) | Support |
| Adreno 840 (Snapdragon 8 Elite Gen 5) | Support |
| Adreno X1-85 (Snapdragon X Elite) | Support |
@@ -98,24 +97,6 @@ The OpenCL backend has the following CMake options that control the behavior of
| `GGML_OPENCL_USE_ADRENO_KERNELS` | `ON` | Use kernels optimized for Adreno. |
| `GGML_OPENCL_USE_ADRENO_BIN_KERNELS` | `OFF` | Allow using binary kernel lib for Adreno. |
## Program Binary Cache
Compiled `cl_program` binaries are cached on disk, so subsequent runs skip the expensive
compile-from-source step when nothing relevant has changed (kernel source, compile options,
device, driver, or platform version).
The cache is controlled with the `GGML_OPENCL_KERNEL_CACHE_DIR` environment variable:
| Value | Behavior |
|:---------------------------------------|:-----------------------------------------------|
| unset / empty / `1` / `default` | Enabled in the platform default cache directory: `%LOCALAPPDATA%\llama.cpp\cl-cache` (Windows), `~/Library/Caches/llama.cpp/cl-cache` (macOS), `<temp dir>/llama.cpp/cl-cache` elsewhere. |
| `0` / `off` / `none` / `disable(d)` | Disabled. |
| any other value | Used verbatim as the cache directory path. |
If the chosen directory cannot be created or used, the cache disables itself for the process
and kernels are compiled from source as usual. Set `GGML_OPENCL_KERNEL_CACHE_DEBUG=1` to
print a HIT/MISS/SAVE trace to stderr.
## Android
Ubuntu 22.04 is used for targeting Android. Make sure the following tools are accessible from command line,
docs/backend/SYCL.md
+4 -5
View File
@@ -790,12 +790,11 @@ use 1 SYCL GPUs: [0] with Max compute units:512
| GGML_SYCL_DEBUG | 0 (default) or 1 | Enable log function by macro: GGML_SYCL_DEBUG |
| GGML_SYCL_DEV2DEV_MEMCPY | 0 (default) or 1 | Choose the SYCL or L0 API in dev2dev memory copy.<br>Value: <br>* 0: SYCL API (default)<br>* 1: L0 API -- L0 API is found to lead to abnormal crash in some case. This debug flag is used to check the issue.|
| GGML_SYCL_ENABLE_FLASH_ATTN | 1 (default) or 0| Enable Flash-Attention. It can reduce memory usage. The performance impact depends on the LLM.|
| GGML_SYCL_ENABLE_OPT | 0 or 1 (default)| Enable optimize features for Intel GPUs. (Recommended to 0 for Intel devices older than Gen 10) |
| GGML_SYCL_ENABLE_GRAPH | 0 (default) or 1 | Enable running computations through SYCL Graphs feature. Disabled by default because SYCL Graph is still on development, no better performance. |
| GGML_SYCL_DISABLE_OPT | 0 (default) or 1 | Disable optimize features for Intel GPUs. (Recommended to 1 for Intel devices older than Gen 10) |
| GGML_SYCL_DISABLE_GRAPH | 0 or 1 (default) | Disable running computations through SYCL Graphs feature. Disabled by default because SYCL Graph is still on development, no better performance. |
| GGML_SYCL_USE_LEVEL_ZERO_API | 1 (default) or 0 | Use Level Zero API for device memory allocation instead of SYCL. Reduces system RAM usage on Intel dGPUs by avoiding DMA-buf/TTM host memory staging. Requires GGML_SYCL_SUPPORT_LEVEL_ZERO_API=ON at build time. SYCL backend always runs on Level Zero running time even if it's set as OFF (The SYCL api will be usage for memory allocation).|
| GGML_SYCL_ENABLE_DNN | 0 or 1 (default)| Enable running computations through oneDNN and always use oneMKL. |
| GGML_SYCL_DISABLE_DNN | 0 (default) or 1 | Disable running computations through oneDNN and always use oneMKL. |
| GGML_SYCL_ENABLE_VMM | 0 or 1 (default) | Enable the virtual-memory device pool. |
| GGML_SYCL_ENABLE_FUSION | 0 or 1 (default) | Enable fused-kernel dispatch in graph compute (currently top-k MoE gating). |
| ZES_ENABLE_SYSMAN | 0 (default) or 1 | Support to get free memory of GPU by sycl::aspect::ext_intel_free_memory.<br>Recommended to use when --split-mode = layer |
| UR_L0_ENABLE_RELAXED_ALLOCATION_LIMITS | 0 (default) or 1 | Allow SYCL/Unified Runtime Level Zero device allocations larger than 4 GiB. llama.cpp's direct Level Zero allocation path requests the relaxed maximum-size limit itself when GGML_SYCL_ENABLE_LEVEL_ZERO=1. |
| GGML_SYCL_USM_SYSTEM | 0 (default) or 1 | Enable experimental support for [USM system allocations](https://github.khronos.org/SYCL_Reference/iface/usm_basic_concept.html#system-allocations) for large GPU buffers. This requires enough host memory for model weights and caches, an Intel Xe2+ GPU such as BMG or newer and supported on Linux only, with CONFIG_DRM_XE_GPUSVM enabled. |
@@ -808,7 +807,7 @@ Pass these via `CXXFLAGS` or add a one-off `#define` to enable a flag on the spo
|-----------------|----------------------------------------------------------------------------------|
| DEBUG_SYCL_POOL | Enable device memory pool logging on teardown. Useful for profiling allocations. |
| DEBUG_SYCL_MALLOC | Enable verbose per-call logging of device pool alloc/free operations. |
| GGML_SYCL_SUPPORT_VMM | Support to building with VMM code. Default is Yes. |
## Design Rule
docs/build.md
+12 -3
View File
@@ -270,10 +270,13 @@ The environment variable [`CUDA_SCALE_LAUNCH_QUEUES`](https://docs.nvidia.com/cu
Consider setting `CUDA_SCALE_LAUNCH_QUEUES=4x`, which increases the CUDA command buffer to 4 times its default size. This optimization is particularly beneficial for **Multi-GPU setups with pipeline parallelism**, where it significantly improves prompt processing throughput by allowing more operations to be enqueued across GPUs.
#### GGML_CUDA_CUBLAS_COMPUTE_TYPE
#### GGML_CUDA_FORCE_CUBLAS_COMPUTE_32F
Override default, speed-optimized compute types for cuBLAS matrix multiplications.
Legal values: `auto`, `f16`, `fp16`, `bf16`, `f32`, `fp32`.
Use `GGML_CUDA_FORCE_CUBLAS_COMPUTE_32F` environment variable to use FP32 compute type on all GPUs in FP16 cuBLAS for preventing possible numerical overflows in exchange for slower prompt processing (small impact on RTX PRO/Datacenter products and significant on GeForce products).
#### GGML_CUDA_FORCE_CUBLAS_COMPUTE_16F
Use `GGML_CUDA_FORCE_CUBLAS_COMPUTE_16F` environment variable to force use FP16 compute type (instead of default FP32) in FP16 cuBLAS for V100, CDNA and RDNA4.
### Unified Memory
@@ -361,6 +364,12 @@ You can download it from your Linux distro's package manager or from here: [ROCm
Note: `GPU_TARGETS` is optional, omitting it will build the code for all GPUs in the current system.
To enhance flash attention performance on RDNA3+ or CDNA architectures, you can utilize the rocWMMA library by enabling the `-DGGML_HIP_ROCWMMA_FATTN=ON` option. This requires rocWMMA headers to be installed on the build system.
The rocWMMA library is included by default when installing the ROCm SDK using the `rocm` meta package provided by AMD. Alternatively, if you are not using the meta package, you can install the library using the `rocwmma-dev` or `rocwmma-devel` package, depending on your system's package manager.
As an alternative, you can manually install the library by cloning it from the official [GitHub repository](https://github.com/ROCm/rocWMMA), checkout the corresponding version tag (e.g. `rocm-6.2.4`) and set `-DCMAKE_CXX_FLAGS="-I<path/to/rocwmma>/library/include/"` in CMake. This also works under Windows despite not officially supported by AMD.
Note that if you get the following error:
```
clang: error: cannot find ROCm device library; provide its path via '--rocm-path' or '--rocm-device-lib-path', or pass '-nogpulib' to build without ROCm device library
docs/development/HOWTO-add-model.md
-11
View File
@@ -45,8 +45,6 @@ class MyModel(MmprojModel):
Add an enum entry in `MODEL_ARCH`, the model human friendly name in `MODEL_ARCH_NAMES` and the GGUF tensor names in `MODEL_TENSORS`.
NOTE: Pick the GGUF arch string (and the matching `src/models/<name>.cpp` filename, see section 3) carefully up front, following existing naming conventions. Once GGUF files are published under a given arch string, renaming it later breaks the community's existing files, so this is not something to leave for cleanup in a follow-up PR.
Example for `falcon` model:
```python
MODEL_ARCH.FALCON: [
@@ -103,7 +101,6 @@ The model params and tensors layout must be defined in `llama.cpp` source files:
- You may also need to update `LLM_KV_NAMES`, `LLM_TENSOR_NAMES` and `LLM_TENSOR_INFOS`
3. Add any non-standard metadata loading in the `llama_model_loader` constructor in `src/llama-model-loader.cpp`.
4. If the model has a RoPE operation, add a case for the architecture in `llama_model_rope_type` function in `src/llama-model.cpp`.
5. Check for other places that switch/iterate over every `llm_arch` value, e.g. `src/llama-model-saver.cpp` and any mandatory-hparam lists (such as which archs require MoE metadata). Grep for `LLM_ARCH_` usages to find them. Missing one of these is a common cause of CI test failures (e.g. `test-llama-archs`) after adding a new arch.
NOTE: The dimensions in `ggml` are typically in the reverse order of the `pytorch` dimensions.
@@ -136,14 +133,6 @@ Note:
## Tips and tricks
### Prefer conversion-time tensor modifications over graph-time ones
If the model contains constant modifications of tensors in the graph (for example, `norm(1 + weight)`) or performs tensor permutations/chunking, perform the modifications during conversion rather than in the graph code. This keeps the inference graph simpler and avoids extra runtime ops.
Examples:
- Gemma 3 folds the `1 +` of its `norm(1 + weight)` normalization into the weights at conversion time, so the graph just does a plain RMS norm.
- Qwen3-Next applies its tensor permutation during conversion (in `modify_tensors`), so the graph can consume the already-permuted weights directly.
### Working with ggml_rope_ext
PyTorch implementations usually prefer explicitly calculating `freq_cis`/`sin`/`cos` components. However, in llama.cpp, most RoPE operations can be handled via `ggml_rope_ext`, which does not require a sin/cos matrix. This saves memory while allowing the GGML RoPE kernel to be fused with other ops.
docs/ops.md
+109 -113
View File
@@ -12,116 +12,112 @@ Legend:
- 🟡 Partially supported by this backend
- ❌ Not supported by this backend
| Operation | BLAS | CANN | CPU | CUDA | ET | MTL | OpenCL | SYCL | Vulkan | WebGPU | ZenDNN | zDNN |
|-----------|------|------|------|------|------|------|------|------|------|------|------|------|
| ABS | ❌ | ✅ | ✅ | 🟡 | 🟡 | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ |
| ACC | ❌ | ✅ | ✅ | ✅ | ❌ | ✅ | ❌ | 🟡 | ✅ | ❌ | ❌ | ❌ |
| ADD | ❌ | ✅ | ✅ | ✅ | 🟡 | 🟡 | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| ADD1 | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
| ADD_ID | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| ARANGE | ❌ | ✅ | ✅ | ✅ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ |
| ARGMAX | ❌ | ✅ | ✅ | ✅ | ❌ | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ |
| ARGSORT | ❌ | ✅ | ✅ | ✅ | ❌ | ✅ | 🟡 | ✅ | ✅ | ✅ | ❌ | ❌ |
| CEIL | ❌ | ❌ | ✅ | 🟡 | 🟡 | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ |
| CLAMP | ❌ | ✅ | ✅ | ✅ | 🟡 | ✅ | 🟡 | ✅ | 🟡 | ✅ | ❌ | ❌ |
| COL2IM_1D | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ |
| CONCAT | ❌ | ✅ | ✅ | 🟡 | 🟡 | ✅ | 🟡 | ✅ | ✅ | 🟡 | ❌ | ❌ |
| CONT | ❌ | 🟡 | ✅ | ✅ | 🟡 | ✅ | 🟡 | ✅ | ✅ | 🟡 | ❌ | ❌ |
| CONV_2D | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| CONV_2D_DW | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ |
| CONV_3D | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ |
| CONV_TRANSPOSE_1D | ❌ | ✅ | ✅ | ✅ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ |
| CONV_TRANSPOSE_2D | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ |
| COS | ❌ | ✅ | ✅ | ✅ | ❌ | ✅ | ❌ | ✅ | 🟡 | ✅ | ❌ | ❌ |
| COUNT_EQUAL | ❌ | ✅ | ✅ | ✅ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ |
| CPY | ❌ | 🟡 | 🟡 | 🟡 | 🟡 | 🟡 | 🟡 | 🟡 | 🟡 | 🟡 | ❌ | ❌ |
| CROSS_ENTROPY_LOSS | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ |
| CROSS_ENTROPY_LOSS_BACK | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ |
| CUMSUM | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ |
| DIAG | ❌ | ❌ | ✅ | ✅ | 🟡 | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ |
| DIAG_MASK_INF | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | 🟡 | ✅ | ✅ | ❌ | ❌ | ❌ |
| DIV | ❌ | ✅ | ✅ | ✅ | ❌ | 🟡 | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| DSV4_HC_COMB | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
| DSV4_HC_POST | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
| DSV4_HC_PRE | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
| DUP | ❌ | ✅ | ✅ | 🟡 | ❌ | 🟡 | 🟡 | ✅ | ✅ | ❌ | ❌ | ❌ |
| ELU | ❌ | ✅ | ✅ | 🟡 | 🟡 | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ |
| EXP | ❌ | ✅ | ✅ | 🟡 | 🟡 | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ |
| EXPM1 | ❌ | ❌ | ✅ | 🟡 | 🟡 | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ |
| FILL | ❌ | ❌ | ✅ | ✅ | 🟡 | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ |
| FLASH_ATTN_EXT | ❌ | 🟡 | ✅ | 🟡 | 🟡 | 🟡 | 🟡 | 🟡 | 🟡 | 🟡 | ❌ | ❌ |
| FLOOR | ❌ | ❌ | ✅ | 🟡 | 🟡 | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ |
| GATED_DELTA_NET | ❌ | ❌ | ✅ | ❌ | ✅ | 🟡 | ❌ | ✅ | 🟡 | ✅ | ❌ | ❌ |
| GATED_LINEAR_ATTN | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ |
| GEGLU | ❌ | ✅ | ✅ | ✅ | 🟡 | 🟡 | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| GEGLU_ERF | ❌ | ✅ | ✅ | ✅ | 🟡 | 🟡 | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| GEGLU_QUICK | ❌ | ✅ | ✅ | ✅ | 🟡 | 🟡 | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| GELU | ❌ | ✅ | ✅ | 🟡 | 🟡 | ✅ | 🟡 | ✅ | ✅ | ✅ | ❌ | ❌ |
| GELU_ERF | ❌ | ✅ | ✅ | 🟡 | 🟡 | ✅ | 🟡 | ✅ | ✅ | ✅ | ❌ | ❌ |
| GELU_QUICK | ❌ | ✅ | ✅ | 🟡 | 🟡 | ✅ | 🟡 | ✅ | ✅ | ✅ | ❌ | ❌ |
| GET_ROWS | ❌ | 🟡 | ✅ | 🟡 | 🟡 | 🟡 | 🟡 | ✅ | ✅ | 🟡 | ❌ | ❌ |
| GET_ROWS_BACK | ❌ | ❌ | 🟡 | 🟡 | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
| GROUP_NORM | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ |
| HARDSIGMOID | ❌ | ✅ | ✅ | 🟡 | 🟡 | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ |
| HARDSWISH | ❌ | ✅ | ✅ | 🟡 | 🟡 | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ |
| IM2COL | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| IM2COL_3D | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ |
| L2_NORM | ❌ | ✅ | ✅ | ✅ | 🟡 | ✅ | ❌ | ✅ | ✅ | 🟡 | ❌ | ❌ |
| LEAKY_RELU | ❌ | ✅ | ✅ | ✅ | ❌ | 🟡 | ❌ | ✅ | 🟡 | ❌ | ❌ | ❌ |
| LIGHTNING_INDEXER | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
| LOG | ❌ | ✅ | ✅ | ✅ | ❌ | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ |
| MEAN | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ |
| MUL | ❌ | ✅ | ✅ | ✅ | 🟡 | 🟡 | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| MUL_MAT | 🟡 | 🟡 | 🟡 | 🟡 | 🟡 | 🟡 | 🟡 | 🟡 | 🟡 | 🟡 | 🟡 | 🟡 |
| MUL_MAT_HADAMARD | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ |
| MUL_MAT_ID | ❌ | 🟡 | ✅ | ✅ | 🟡 | 🟡 | 🟡 | ✅ | ✅ | 🟡 | 🟡 | ❌ |
| NEG | ❌ | ✅ | ✅ | 🟡 | 🟡 | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ |
| NORM | ❌ | ✅ | ✅ | ✅ | 🟡 | ✅ | ✅ | ✅ | 🟡 | 🟡 | ❌ | ❌ |
| OPT_STEP_ADAMW | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ |
| OPT_STEP_SGD | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ |
| OUT_PROD | 🟡 | 🟡 | 🟡 | 🟡 | ❌ | ❌ | ❌ | 🟡 | ❌ | ❌ | ❌ | 🟡 |
| PAD | ❌ | 🟡 | ✅ | 🟡 | ❌ | 🟡 | 🟡 | 🟡 | ✅ | ✅ | ❌ | ❌ |
| PAD_REFLECT_1D | ❌ | ✅ | ✅ | ✅ | ❌ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ |
| POOL_1D | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ |
| POOL_2D | ❌ | 🟡 | ✅ | ✅ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ |
| REGLU | ❌ | ✅ | ✅ | ✅ | 🟡 | 🟡 | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| RELU | ❌ | ✅ | ✅ | 🟡 | 🟡 | ✅ | 🟡 | ✅ | ✅ | ✅ | ❌ | ❌ |
| REPEAT | ❌ | ✅ | ✅ | 🟡 | 🟡 | ✅ | 🟡 | ✅ | ✅ | 🟡 | ❌ | ❌ |
| REPEAT_BACK | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ |
| RMS_NORM | ❌ | ✅ | ✅ | ✅ | 🟡 | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| RMS_NORM_BACK | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ |
| ROLL | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ |
| ROPE | ❌ | ✅ | ✅ | ✅ | 🟡 | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| ROPE_BACK | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ |
| ROUND | ❌ | ❌ | ✅ | 🟡 | 🟡 | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ |
| RWKV_WKV6 | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ |
| RWKV_WKV7 | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ |
| SCALE | ❌ | 🟡 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| SET | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ | ❌ | 🟡 | ✅ | ✅ | ❌ | ❌ |
| SET_ROWS | ❌ | 🟡 | 🟡 | 🟡 | 🟡 | 🟡 | 🟡 | 🟡 | 🟡 | 🟡 | ❌ | ❌ |
| SGN | ❌ | ✅ | ✅ | 🟡 | 🟡 | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ |
| SIGMOID | ❌ | ✅ | ✅ | 🟡 | 🟡 | ✅ | 🟡 | ✅ | ✅ | ✅ | ❌ | ❌ |
| SILU | ❌ | ✅ | ✅ | 🟡 | 🟡 | ✅ | 🟡 | ✅ | ✅ | ✅ | ❌ | ❌ |
| SILU_BACK | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ |
| SIN | ❌ | ✅ | ✅ | ✅ | ❌ | ✅ | ❌ | ✅ | 🟡 | ✅ | ❌ | ❌ |
| SOFTPLUS | ❌ | ❌ | ✅ | 🟡 | 🟡 | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ |
| SOFT_MAX | ❌ | 🟡 | ✅ | ✅ | 🟡 | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| SOFT_MAX_BACK | ❌ | ❌ | 🟡 | 🟡 | ❌ | ❌ | ❌ | 🟡 | ✅ | ❌ | ❌ | ❌ |
| SOLVE_TRI | ❌ | ❌ | ✅ | 🟡 | 🟡 | ✅ | ❌ | 🟡 | ✅ | ✅ | ❌ | ❌ |
| SQR | ❌ | ✅ | ✅ | ✅ | 🟡 | ✅ | ✅ | ✅ | 🟡 | ✅ | ❌ | ❌ |
| SQRT | ❌ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | 🟡 | ✅ | ❌ | ❌ |
| SSM_CONV | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| SSM_SCAN | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | 🟡 | 🟡 | ✅ | ❌ | ❌ |
| STEP | ❌ | ✅ | ✅ | 🟡 | 🟡 | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ |
| SUB | ❌ | ✅ | ✅ | ✅ | 🟡 | 🟡 | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| SUM | ❌ | 🟡 | ✅ | 🟡 | ❌ | 🟡 | ❌ | 🟡 | 🟡 | 🟡 | ❌ | ❌ |
| SUM_ROWS | ❌ | ✅ | ✅ | 🟡 | ❌ | ✅ | 🟡 | 🟡 | ✅ | ✅ | ❌ | ❌ |
| SWIGLU | ❌ | ✅ | ✅ | ✅ | 🟡 | 🟡 | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| SWIGLU_OAI | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| TANH | ❌ | ✅ | ✅ | 🟡 | 🟡 | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| TIMESTEP_EMBEDDING | ❌ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ |
| TOP_K | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | 🟡 | 🟡 | ✅ | ❌ | ❌ |
| TRI | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ |
| TRUNC | ❌ | ❌ | ✅ | 🟡 | 🟡 | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ |
| UPSCALE | ❌ | 🟡 | ✅ | ✅ | ❌ | ✅ | 🟡 | ✅ | ✅ | ✅ | ❌ | ❌ |
| XIELU | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ |
| Operation | BLAS | CANN | CPU | CUDA | MTL | OpenCL | SYCL | Vulkan | WebGPU | ZenDNN | zDNN |
|-----------|------|------|------|------|------|------|------|------|------|------|------|
| ABS | ❌ | ✅ | ✅ | 🟡 | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ |
| ACC | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | 🟡 | ✅ | ❌ | ❌ | ❌ |
| ADD | ❌ | ✅ | ✅ | ✅ | 🟡 | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| ADD1 | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
| ADD_ID | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| ARANGE | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ |
| ARGMAX | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ |
| ARGSORT | ❌ | ✅ | ✅ | ✅ | ✅ | 🟡 | 🟡 | ✅ | ✅ | ❌ | ❌ |
| CEIL | ❌ | ❌ | ✅ | 🟡 | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ |
| CLAMP | ❌ | ✅ | ✅ | ✅ | ✅ | 🟡 | ✅ | 🟡 | ✅ | ❌ | ❌ |
| COL2IM_1D | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
| CONCAT | ❌ | ✅ | ✅ | 🟡 | ✅ | 🟡 | ✅ | ✅ | ✅ | ❌ | ❌ |
| CONT | ❌ | 🟡 | ✅ | ✅ | ✅ | 🟡 | 🟡 | ✅ | 🟡 | ❌ | ❌ |
| CONV_2D | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| CONV_2D_DW | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ |
| CONV_3D | ❌ | ❌ | ✅ | ❌ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ |
| CONV_TRANSPOSE_1D | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ |
| CONV_TRANSPOSE_2D | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ |
| COS | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | 🟡 | ✅ | ❌ | ❌ |
| COUNT_EQUAL | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ |
| CPY | ❌ | 🟡 | 🟡 | 🟡 | 🟡 | 🟡 | 🟡 | 🟡 | 🟡 | ❌ | ❌ |
| CROSS_ENTROPY_LOSS | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
| CROSS_ENTROPY_LOSS_BACK | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
| CUMSUM | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ |
| DIAG | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ |
| DIAG_MASK_INF | ❌ | ✅ | ✅ | ✅ | ❌ | 🟡 | ✅ | ✅ | ❌ | ❌ | ❌ |
| DIV | ❌ | ✅ | ✅ | ✅ | 🟡 | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| DUP | ❌ | ✅ | ✅ | 🟡 | 🟡 | 🟡 | ✅ | ✅ | ❌ | ❌ | ❌ |
| ELU | ❌ | ✅ | ✅ | 🟡 | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ |
| EXP | ❌ | ✅ | ✅ | 🟡 | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ |
| EXPM1 | ❌ | ❌ | ✅ | 🟡 | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ |
| FILL | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ |
| FLASH_ATTN_EXT | ❌ | 🟡 | ✅ | 🟡 | 🟡 | 🟡 | 🟡 | 🟡 | 🟡 | ❌ | ❌ |
| FLOOR | ❌ | ❌ | ✅ | 🟡 | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ |
| GATED_DELTA_NET | ❌ | ❌ | ✅ | ❌ | 🟡 | ❌ | ✅ | 🟡 | ✅ | ❌ | ❌ |
| GATED_LINEAR_ATTN | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ |
| GEGLU | ❌ | ✅ | ✅ | ✅ | 🟡 | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| GEGLU_ERF | ❌ | ✅ | ✅ | ✅ | 🟡 | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| GEGLU_QUICK | ❌ | ✅ | ✅ | ✅ | 🟡 | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| GELU | ❌ | ✅ | ✅ | 🟡 | ✅ | 🟡 | ✅ | ✅ | ✅ | ❌ | ❌ |
| GELU_ERF | ❌ | ✅ | ✅ | 🟡 | ✅ | 🟡 | ✅ | ✅ | ✅ | ❌ | ❌ |
| GELU_QUICK | ❌ | ✅ | ✅ | 🟡 | ✅ | 🟡 | ✅ | ✅ | ✅ | ❌ | ❌ |
| GET_ROWS | ❌ | 🟡 | ✅ | 🟡 | 🟡 | 🟡 | ✅ | ✅ | 🟡 | ❌ | ❌ |
| GET_ROWS_BACK | ❌ | ❌ | 🟡 | 🟡 | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
| GROUP_NORM | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ |
| HARDSIGMOID | ❌ | ✅ | ✅ | 🟡 | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ |
| HARDSWISH | ❌ | ✅ | ✅ | 🟡 | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ |
| IM2COL | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| IM2COL_3D | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ |
| L2_NORM | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ |
| LEAKY_RELU | ❌ | ✅ | ✅ | ✅ | 🟡 | ❌ | ✅ | 🟡 | ❌ | ❌ | ❌ |
| LOG | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ |
| MEAN | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ |
| MUL | ❌ | ✅ | ✅ | ✅ | 🟡 | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| MUL_MAT | 🟡 | 🟡 | 🟡 | 🟡 | 🟡 | 🟡 | 🟡 | 🟡 | 🟡 | 🟡 | 🟡 |
| MUL_MAT_HADAMARD | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ |
| MUL_MAT_ID | ❌ | 🟡 | ✅ | ✅ | 🟡 | 🟡 | 🟡 | ✅ | 🟡 | 🟡 | ❌ |
| NEG | ❌ | ✅ | ✅ | 🟡 | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ |
| NORM | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | 🟡 | ✅ | ❌ | ❌ |
| OPT_STEP_ADAMW | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ |
| OPT_STEP_SGD | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ |
| OUT_PROD | 🟡 | 🟡 | 🟡 | 🟡 | ❌ | ❌ | 🟡 | ❌ | ❌ | ❌ | 🟡 |
| PAD | ❌ | 🟡 | ✅ | 🟡 | 🟡 | 🟡 | 🟡 | ✅ | ✅ | ❌ | ❌ |
| PAD_REFLECT_1D | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ |
| POOL_1D | ❌ | ❌ | ✅ | ❌ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ |
| POOL_2D | ❌ | 🟡 | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ |
| REGLU | ❌ | ✅ | ✅ | ✅ | 🟡 | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| RELU | ❌ | ✅ | ✅ | 🟡 | ✅ | 🟡 | ✅ | ✅ | ✅ | ❌ | ❌ |
| REPEAT | ❌ | ✅ | ✅ | 🟡 | ✅ | 🟡 | ✅ | ✅ | ✅ | ❌ | ❌ |
| REPEAT_BACK | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ |
| RMS_NORM | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| RMS_NORM_BACK | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ |
| ROLL | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ |
| ROPE | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| ROPE_BACK | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ |
| ROUND | ❌ | ❌ | ✅ | 🟡 | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ |
| RWKV_WKV6 | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ |
| RWKV_WKV7 | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ |
| SCALE | ❌ | 🟡 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| SET | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ | 🟡 | ✅ | ✅ | ❌ | ❌ |
| SET_ROWS | ❌ | 🟡 | 🟡 | 🟡 | 🟡 | 🟡 | 🟡 | 🟡 | 🟡 | ❌ | ❌ |
| SGN | ❌ | ✅ | ✅ | 🟡 | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ |
| SIGMOID | ❌ | ✅ | ✅ | 🟡 | ✅ | 🟡 | ✅ | ✅ | ✅ | ❌ | ❌ |
| SILU | ❌ | ✅ | ✅ | 🟡 | ✅ | 🟡 | ✅ | ✅ | ✅ | ❌ | ❌ |
| SILU_BACK | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ |
| SIN | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | 🟡 | ✅ | ❌ | ❌ |
| SOFTPLUS | ❌ | ❌ | ✅ | 🟡 | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ |
| SOFT_MAX | ❌ | 🟡 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| SOFT_MAX_BACK | ❌ | ❌ | 🟡 | 🟡 | ❌ | ❌ | 🟡 | ✅ | ❌ | ❌ | ❌ |
| SOLVE_TRI | ❌ | ❌ | ✅ | 🟡 | ✅ | ❌ | 🟡 | ✅ | ✅ | ❌ | ❌ |
| SQR | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | 🟡 | ✅ | ❌ | ❌ |
| SQRT | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | 🟡 | ✅ | ❌ | ❌ |
| SSM_CONV | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| SSM_SCAN | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ | 🟡 | 🟡 | ✅ | ❌ | ❌ |
| STEP | ❌ | ✅ | ✅ | 🟡 | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ |
| SUB | ❌ | ✅ | ✅ | ✅ | 🟡 | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| SUM | ❌ | 🟡 | ✅ | 🟡 | 🟡 | ❌ | 🟡 | 🟡 | 🟡 | ❌ | ❌ |
| SUM_ROWS | ❌ | ✅ | ✅ | 🟡 | ✅ | 🟡 | 🟡 | ✅ | ✅ | ❌ | ❌ |
| SWIGLU | ❌ | ✅ | ✅ | ✅ | 🟡 | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| SWIGLU_OAI | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| TANH | ❌ | ✅ | ✅ | 🟡 | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| TIMESTEP_EMBEDDING | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ |
| TOP_K | ❌ | ❌ | ✅ | ❌ | ✅ | ❌ | 🟡 | 🟡 | ✅ | ❌ | ❌ |
| TRI | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ |
| TRUNC | ❌ | ❌ | ✅ | 🟡 | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ |
| UPSCALE | ❌ | 🟡 | ✅ | ✅ | ✅ | 🟡 | ✅ | ✅ | ✅ | ❌ | ❌ |
| XIELU | ❌ | ❌ | ✅ | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ |
docs/ops/ET.csv
-16114
View File
File diff suppressed because it is too large Load Diff
docs/ops/SYCL.csv
+475 -559
View File
File diff suppressed because it is too large Load Diff
docs/ops/WebGPU.csv
+952 -2629
View File
File diff suppressed because it is too large Load Diff
examples/diffusion/diffusion-cli.cpp
+3 -1
View File
@@ -117,7 +117,9 @@ int main(int argc, char ** argv) {
llama_model_params model_params = llama_model_default_params();
model_params.n_gpu_layers = params.n_gpu_layers;
model_params.devices = params.devices.data();
model_params.load_mode = params.load_mode;
model_params.use_mmap = params.use_mmap;
model_params.use_direct_io = params.use_direct_io;
model_params.use_mlock = params.use_mlock;
model_params.check_tensors = params.check_tensors;
llama_model * model = llama_model_load_from_file(params.model.path.c_str(), model_params);
examples/llama-eval/llama-eval.py
+2 -2
View File
@@ -362,7 +362,7 @@ class EvalState:
case = cases.get(task_id, {})
status = case.get("status", "pending")
expected = case.get("expected", "")
answer = case.get("answer") or "" if status == "ok" else ""
answer = case.get("answer", "") if status == "ok" else ""
is_correct = case.get("correct", False) if status == "ok" else False
response = case.get("response", "") or ""
prompt = case.get("prompt", "") or ""
@@ -647,7 +647,7 @@ class EvalState:
question, prompt, expected = self.get_case(i)
case = cases.get(task_id, {})
status = case.get("status", "pending")
answer = case.get("answer") or "N/A" if status == "ok" else "N/A"
answer = case.get("answer", "N/A") if status == "ok" else "N/A"
tokens = case.get("tokens")
tokens_str = str(tokens) if tokens is not None else "N/A"
tps_gen = case.get("tps_gen")
examples/training/finetune.cpp
+4 -3
View File
@@ -26,9 +26,10 @@ int main(int argc, char ** argv) {
return 1;
}
if (params.load_mode != LLAMA_LOAD_MODE_NONE) {
LOG_INF("%s: forcing load_mode = none to enable writable pointers to the weights\n", __func__);
params.load_mode = LLAMA_LOAD_MODE_NONE;
if (params.use_mmap) {
LOG_INF("%s: force disabling memory mapping because it would result in-read-only pointers to the weights\n",
__func__);
params.use_mmap = false;
}
if (params.cache_type_k != GGML_TYPE_F32) {
LOG_INF("%s: force changing k cache type to f32 due to a lack of f16 support for OUT_PROD\n", __func__);
ggml/CMakeLists.txt
+3 -4
View File
@@ -4,8 +4,8 @@ project("ggml" C CXX ASM)
### GGML Version
set(GGML_VERSION_MAJOR 0)
set(GGML_VERSION_MINOR 17)
set(GGML_VERSION_PATCH 0)
set(GGML_VERSION_MINOR 15)
set(GGML_VERSION_PATCH 3)
set(GGML_VERSION_BASE "${GGML_VERSION_MAJOR}.${GGML_VERSION_MINOR}.${GGML_VERSION_PATCH}")
list(APPEND CMAKE_MODULE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/cmake/")
@@ -216,6 +216,7 @@ option(GGML_HIP "ggml: use HIP"
option(GGML_HIP_GRAPHS "ggml: use HIP graph" ON)
option(GGML_HIP_RCCL "ggml: use ROCm Collective Comm. Library" OFF)
option(GGML_HIP_NO_VMM "ggml: do not try to use HIP VMM" ON)
option(GGML_HIP_ROCWMMA_FATTN "ggml: enable rocWMMA for FlashAttention" OFF)
option(GGML_HIP_MMQ_MFMA "ggml: enable MFMA MMA for CDNA in MMQ" ON)
option(GGML_HIP_EXPORT_METRICS "ggml: enable kernel perf metrics output" OFF)
option(GGML_MUSA_GRAPHS "ggml: use MUSA graph, experimental, unstable" OFF)
@@ -256,8 +257,6 @@ set (GGML_SYCL_DEVICE_ARCH "" CACHE STRING
"ggml: sycl device architecture")
option(GGML_OPENVINO "ggml: use OPENVINO" OFF)
option(GGML_ET "ggml: use ET backend" OFF)
option(GGML_ET_SYSEMU "ggml: use ET backend via sysemu" OFF)
option(GGML_OPENCL "ggml: use OpenCL" OFF)
option(GGML_OPENCL_PROFILING "ggml: use OpenCL profiling (increases overhead)" OFF)
ggml/include/ggml-backend.h
-5
View File
@@ -104,11 +104,6 @@ extern "C" {
GGML_API enum ggml_status ggml_backend_graph_compute (ggml_backend_t backend, struct ggml_cgraph * cgraph);
GGML_API enum ggml_status ggml_backend_graph_compute_async(ggml_backend_t backend, struct ggml_cgraph * cgraph);
// Free transient/scratch device memory the backend holds outside of any allocated buffer
// (compute preallocations, staging buffers). No-op if the backend does not implement it.
// The backend remains usable; scratch is reallocated lazily on the next compute.
GGML_API void ggml_backend_free_scratch(ggml_backend_t backend);
// NOTE: will be removed, use device version instead
GGML_API bool ggml_backend_supports_op(ggml_backend_t backend, const struct ggml_tensor * op);
GGML_API bool ggml_backend_supports_buft(ggml_backend_t backend, ggml_backend_buffer_type_t buft);
ggml/include/ggml-cpu.h
-1
View File
@@ -100,7 +100,6 @@ extern "C" {
GGML_BACKEND_API int ggml_cpu_has_sve (void);
GGML_BACKEND_API int ggml_cpu_get_sve_cnt (void); // sve vector length in bytes
GGML_BACKEND_API int ggml_cpu_has_sme (void);
GGML_BACKEND_API int ggml_cpu_has_sme2 (void);
// other
GGML_BACKEND_API int ggml_cpu_has_riscv_v (void);
GGML_BACKEND_API int ggml_cpu_get_rvv_vlen (void); // risc-v vector length in bytes
ggml/include/ggml-cuda.h
+3
View File
@@ -30,6 +30,9 @@ GGML_BACKEND_API ggml_backend_buffer_type_t ggml_backend_cuda_buffer_type(int de
// conduct allreduce operation between devices
GGML_BACKEND_API bool ggml_backend_cuda_allreduce_tensor(ggml_backend_t * backends, struct ggml_tensor ** tensors, size_t n_backends);
// split tensor buffer that splits matrices by rows across multiple devices
GGML_BACKEND_API ggml_backend_buffer_type_t ggml_backend_cuda_split_buffer_type(int main_device, const float * tensor_split);
// pinned host buffer for use with the CPU backend for faster copies between CPU and GPU
GGML_BACKEND_API ggml_backend_buffer_type_t ggml_backend_cuda_host_buffer_type(void);
ggml/include/ggml-et.h
-28
View File
@@ -1,28 +0,0 @@
#pragma once
#include "ggml.h"
#include "ggml-backend.h"
#ifdef __cplusplus
extern "C" {
#endif
#define GGML_ET_NAME "ET"
// backend API
GGML_BACKEND_API ggml_guid_t ggml_backend_et_guid(void);
GGML_BACKEND_API ggml_backend_t ggml_backend_et_init(size_t devidx);
GGML_BACKEND_API bool ggml_backend_is_et(ggml_backend_t backend);
GGML_BACKEND_API int ggml_backend_et_get_device_count(void);
GGML_BACKEND_API void ggml_backend_et_get_device_description(int devidx, char * description, size_t description_size);
GGML_BACKEND_API void ggml_backend_et_get_device_memory(int devidx, size_t * free, size_t * total);
GGML_BACKEND_API ggml_backend_buffer_type_t ggml_backend_et_buffer_type(size_t dev_num);
GGML_BACKEND_API ggml_backend_buffer_type_t ggml_backend_et_host_buffer_type(void);
GGML_BACKEND_API ggml_backend_reg_t ggml_backend_et_reg(void);
#ifdef __cplusplus
}
#endif
ggml/include/ggml-rpc.h
+2 -2
View File
@@ -8,10 +8,10 @@ extern "C" {
#define RPC_PROTO_MAJOR_VERSION 4
#define RPC_PROTO_MINOR_VERSION 0
#define RPC_PROTO_PATCH_VERSION 3
#define RPC_PROTO_PATCH_VERSION 1
#ifdef __cplusplus
static_assert(GGML_OP_COUNT == 101, "GGML_OP_COUNT has changed - update RPC_PROTO_PATCH_VERSION");
static_assert(GGML_OP_COUNT == 97, "GGML_OP_COUNT has changed - update RPC_PROTO_PATCH_VERSION");
#endif
#define GGML_RPC_MAX_SERVERS 16
ggml/include/ggml.h
+1 -68
View File
@@ -429,8 +429,7 @@ extern "C" {
GGML_TYPE_MXFP4 = 39, // MXFP4 (1 block)
GGML_TYPE_NVFP4 = 40, // NVFP4 (4 blocks, E4M3 scale)
GGML_TYPE_Q1_0 = 41,
GGML_TYPE_Q2_0 = 42,
GGML_TYPE_COUNT = 43,
GGML_TYPE_COUNT = 42,
};
// precision
@@ -474,7 +473,6 @@ extern "C" {
GGML_FTYPE_MOSTLY_MXFP4 = 25, // except 1d tensors
GGML_FTYPE_MOSTLY_NVFP4 = 26, // except 1d tensors
GGML_FTYPE_MOSTLY_Q1_0 = 27, // except 1d tensors
GGML_FTYPE_MOSTLY_Q2_0 = 28, // except 1d tensors
};
// available tensor operations:
@@ -570,10 +568,6 @@ extern "C" {
GGML_OP_RWKV_WKV7,
GGML_OP_SOLVE_TRI,
GGML_OP_GATED_DELTA_NET,
GGML_OP_LIGHTNING_INDEXER,
GGML_OP_DSV4_HC_COMB,
GGML_OP_DSV4_HC_PRE,
GGML_OP_DSV4_HC_POST,
GGML_OP_UNARY,
@@ -783,10 +777,6 @@ extern "C" {
GGML_API bool ggml_is_contiguous_1(const struct ggml_tensor * tensor); // contiguous for dims >= 1
GGML_API bool ggml_is_contiguous_2(const struct ggml_tensor * tensor); // contiguous for dims >= 2
GGML_API bool ggml_is_contiguous_to_1(const struct ggml_tensor * tensor); // contiguous for dims < 1
GGML_API bool ggml_is_contiguous_to_2(const struct ggml_tensor * tensor); // contiguous for dims < 2
GGML_API bool ggml_is_contiguous_to_3(const struct ggml_tensor * tensor); // contiguous for dims < 3
// returns whether the tensor elements are allocated as one contiguous block of memory (no gaps, but permutation ok)
GGML_API bool ggml_is_contiguously_allocated(const struct ggml_tensor * tensor);
@@ -2583,63 +2573,6 @@ extern "C" {
struct ggml_tensor * state,
int64_t K);
// DSA lightning indexer
//
// q: [n_embd_idx, n_head_idx, n_batch, ne3 ]
// k: [n_embd_idx, 1, n_kv, ne3 ]
// weights: [n_head_idx, n_batch, 1, ne3 ] !! prescaled !!
// mask: [n_kv, n_batch, 1, ne33] !! f16 !!
// res: [n_kv, n_batch, 1, ne3 ]
//
// broadcast:
// ne3 % ne33 == 0
//
GGML_API struct ggml_tensor * ggml_lightning_indexer(
struct ggml_context * ctx,
struct ggml_tensor * q,
struct ggml_tensor * k,
struct ggml_tensor * weights,
struct ggml_tensor * mask);
// DeepSeek V4 hyper-connections (ref. https://arxiv.org/pdf/2512.24880)
// In short these operations are replacements for the original residual connection (x = transformer(x) + x)
// using a richer representation through streams.
//
// hc_comb: mixes [(2 + hc)*hc, n_tokens], scale [3], base [(2 + hc)*hc]
// -> [dst_hc, src_hc, n_tokens]
// logits[dst, src, t] = mixes[2*hc + dst + hc*src, t]*scale[2]
// + base[2*hc + dst + hc*src]
// Softmax over dst, add eps, normalize over src, then repeat normalization
// over dst followed by src for iterations 1 through n_iter - 1.
GGML_API struct ggml_tensor * ggml_dsv4_hc_comb(
struct ggml_context * ctx,
struct ggml_tensor * mixes,
struct ggml_tensor * scale,
struct ggml_tensor * base,
float eps,
int32_t n_iter);
// hc_pre: x [n_embd, hc, n_tokens], weights [hc, n_tokens] -> [n_embd, n_tokens]
// result[i, t] = sum_h x[i, h, t]*weights[h, t]
//
GGML_API struct ggml_tensor * ggml_dsv4_hc_pre(
struct ggml_context * ctx,
struct ggml_tensor * x,
struct ggml_tensor * weights);
// hc_post: x [n_embd, n_tokens], residual [n_embd, hc, n_tokens],
// post [hc, n_tokens], comb [dst_hc, src_hc, n_tokens]
// -> [n_embd, hc, n_tokens]
// result[i, dst, t] = x[i, t]*post[dst, t]
// + sum_src residual[i, src, t]*comb[dst, src, t]
//
GGML_API struct ggml_tensor * ggml_dsv4_hc_post(
struct ggml_context * ctx,
struct ggml_tensor * x,
struct ggml_tensor * residual,
struct ggml_tensor * post,
struct ggml_tensor * comb);
// custom operators
typedef void (*ggml_custom1_op_t)(struct ggml_tensor * dst , const struct ggml_tensor * a, int ith, int nth, void * userdata);
ggml/include/gguf.h
+6 -7
View File
@@ -125,13 +125,12 @@ extern "C" {
// get ith C string from array with given key_id
GGML_API const char * gguf_get_arr_str (const struct gguf_context * ctx, int64_t key_id, size_t i);
GGML_API int64_t gguf_get_n_tensors (const struct gguf_context * ctx);
GGML_API int64_t gguf_find_tensor (const struct gguf_context * ctx, const char * name); // returns -1 if the tensor is not found
GGML_API size_t gguf_get_tensor_offset(const struct gguf_context * ctx, int64_t tensor_id);
GGML_API const char * gguf_get_tensor_name (const struct gguf_context * ctx, int64_t tensor_id);
GGML_API const int64_t * gguf_get_tensor_ne (const struct gguf_context * ctx, int64_t tensor_id); // returns ne, an array of GGML_MAX_DIMS elements; ne[dim] is 1 for dim >= n_dims
GGML_API enum ggml_type gguf_get_tensor_type (const struct gguf_context * ctx, int64_t tensor_id);
GGML_API size_t gguf_get_tensor_size (const struct gguf_context * ctx, int64_t tensor_id);
GGML_API int64_t gguf_get_n_tensors (const struct gguf_context * ctx);
GGML_API int64_t gguf_find_tensor (const struct gguf_context * ctx, const char * name); // returns -1 if the tensor is not found
GGML_API size_t gguf_get_tensor_offset(const struct gguf_context * ctx, int64_t tensor_id);
GGML_API const char * gguf_get_tensor_name (const struct gguf_context * ctx, int64_t tensor_id);
GGML_API enum ggml_type gguf_get_tensor_type (const struct gguf_context * ctx, int64_t tensor_id);
GGML_API size_t gguf_get_tensor_size (const struct gguf_context * ctx, int64_t tensor_id);
// removes key if it exists, returns id that the key had prior to removal (-1 if it didn't exist)
GGML_API int64_t gguf_remove_key(struct gguf_context * ctx, const char * key);
ggml/src/CMakeLists.txt
+1 -2
View File
@@ -430,7 +430,7 @@ if (GGML_CPU_ALL_VARIANTS)
message(FATAL_ERROR "Unsupported ARM target OS: ${CMAKE_SYSTEM_NAME}")
endif()
elseif (GGML_SYSTEM_ARCH STREQUAL "PowerPC")
if (CMAKE_SYSTEM_NAME MATCHES "Linux|AIX")
if (CMAKE_SYSTEM_NAME MATCHES "Linux")
ggml_add_cpu_backend_variant(power0)
ggml_add_cpu_backend_variant(power7_1 POWER7)
ggml_add_cpu_backend_variant(power7_2 POWER7 VSX)
@@ -473,7 +473,6 @@ endif()
ggml_add_backend(BLAS)
ggml_add_backend(CANN)
ggml_add_backend(CUDA)
ggml_add_backend(ET)
ggml_add_backend(HIP)
ggml_add_backend(METAL)
ggml_add_backend(MUSA)
ggml/src/ggml-backend-impl.h
-5
View File
@@ -137,11 +137,6 @@ extern "C" {
// (optional) sort/optimize the nodes in the graph
void (*graph_optimize) (ggml_backend_t backend, struct ggml_cgraph * cgraph);
// (optional) free transient/scratch device memory the backend holds outside of any buffer
// (e.g. compute preallocations and staging buffers). The backend stays usable; the scratch
// is reallocated lazily on the next compute. Used to shrink an idle model's device footprint.
void (*free_scratch) (ggml_backend_t backend);
};
struct ggml_backend {
ggml/src/ggml-backend-meta.cpp
+4 -12
View File
@@ -984,11 +984,6 @@ static struct ggml_backend_meta_split_state ggml_backend_meta_get_split_state(
case GGML_OP_GATED_DELTA_NET: {
split_state = handle_gated_delta_net(src_ss);
} break;
case GGML_OP_DSV4_HC_COMB:
case GGML_OP_DSV4_HC_PRE:
case GGML_OP_DSV4_HC_POST: {
split_state = handle_generic(src_ss, /*scalar_only =*/ true);
} break;
case GGML_OP_UNARY: {
split_state = handle_generic(src_ss, /*scalar_only =*/ false);
} break;
@@ -1149,11 +1144,6 @@ static enum ggml_status ggml_backend_meta_buffer_init_tensor_impl(ggml_backend_m
ggml_context * simple_ctx = stc.ctxs[j].get();
ggml_backend_buffer_t simple_buf = buf_ctx->bufs[j].get();
if ((simple_buf != nullptr) && ggml_backend_buffer_is_multi_buffer(simple_buf)) {
// see https://github.com/ggml-org/llama.cpp/issues/22197
GGML_ABORT("multi buffers are not supported by the meta backend");
}
if (split_dim >= 0 && split_dim < GGML_MAX_DIMS) {
// TODO: the following assert fails for llama-parallel even though the results are correct:
// GGML_ASSERT(ggml_is_contiguously_allocated(tensor));
@@ -1255,8 +1245,9 @@ static enum ggml_status ggml_backend_meta_buffer_init_tensor(ggml_backend_buffer
static void ggml_backend_meta_buffer_set_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor, const void * data, size_t offset, size_t size) {
const size_t n_bufs = ggml_backend_meta_buffer_n_bufs(buffer);
GGML_ASSERT(ggml_is_contiguous(tensor));
const ggml_backend_meta_split_state split_state = ggml_backend_meta_get_split_state(tensor, /*assume_sync =*/ false);
GGML_ASSERT(ggml_is_contiguous(tensor) || split_state.axis == GGML_BACKEND_SPLIT_AXIS_MIRRORED);
if (split_state.n_segments != 1 || split_state.nr[0] != 1) {
GGML_ASSERT(split_state.axis >= 0 && split_state.axis < GGML_MAX_DIMS);
@@ -1369,8 +1360,9 @@ static void ggml_backend_meta_buffer_set_tensor(ggml_backend_buffer_t buffer, gg
static void ggml_backend_meta_buffer_get_tensor(ggml_backend_buffer_t buffer, const ggml_tensor * tensor, void * data, size_t offset, size_t size) {
const size_t n_bufs = ggml_backend_meta_buffer_n_bufs(buffer);
GGML_ASSERT(ggml_is_contiguous(tensor));
const ggml_backend_meta_split_state split_state = ggml_backend_meta_get_split_state(tensor, /*assume_sync =*/ false);
GGML_ASSERT(ggml_is_contiguous(tensor) || split_state.axis == GGML_BACKEND_SPLIT_AXIS_MIRRORED);
if (split_state.n_segments != 1 || split_state.nr[0] != 1) {
GGML_ASSERT(split_state.axis >= 0 && split_state.axis < GGML_MAX_DIMS);
ggml/src/ggml-backend-reg.cpp
-7
View File
@@ -86,10 +86,6 @@
#include "ggml-openvino.h"
#endif
#ifdef GGML_USE_ET
#include "ggml-et.h"
#endif
namespace fs = std::filesystem;
static std::string path_str(const fs::path & path) {
@@ -165,9 +161,6 @@ struct ggml_backend_registry {
#ifdef GGML_USE_OPENVINO
register_backend(ggml_backend_openvino_reg());
#endif
#ifdef GGML_USE_ET
register_backend(ggml_backend_et_reg());
#endif
#ifdef GGML_USE_CPU
register_backend(ggml_backend_cpu_reg());
#endif
ggml/src/ggml-backend.cpp
-9
View File
@@ -420,15 +420,6 @@ void ggml_backend_synchronize(ggml_backend_t backend) {
backend->iface.synchronize(backend);
}
void ggml_backend_free_scratch(ggml_backend_t backend) {
GGML_ASSERT(backend);
if (backend->iface.free_scratch == NULL) {
return;
}
backend->iface.free_scratch(backend);
}
ggml_backend_graph_plan_t ggml_backend_graph_plan_create(ggml_backend_t backend, struct ggml_cgraph * cgraph) {
GGML_ASSERT(backend);
GGML_ASSERT(backend->iface.graph_plan_create != NULL);
ggml/src/ggml-blas/ggml-blas.cpp
-7
View File
@@ -1,4 +1,3 @@
#include "ggml.h"
#include "ggml-impl.h"
#include "ggml-blas.h"
#include "ggml-backend-impl.h"
@@ -416,12 +415,6 @@ static bool ggml_backend_blas_device_supports_op(ggml_backend_dev_t dev, const s
// TODO: find the optimal value
const int64_t min_batch = 32;
// default back to CPU fast path
// see: https://github.com/ggml-org/llama.cpp/issues/25565
if (ggml_get_op_params_i32(op, 1) == GGML_HINT_SRC0_IS_HADAMARD) {
return false;
}
return ggml_is_contiguous(src0) &&
ggml_is_contiguous(src1) &&
src1->type == GGML_TYPE_F32 &&
ggml/src/ggml-common.h
-10
View File
@@ -96,9 +96,6 @@ typedef sycl::half2 ggml_half2;
#define QI1_0 (QK1_0 / 32)
#define QR1_0 1
#define QI2_0 (QK2_0 / 32)
#define QR2_0 1
#define QI4_0 (QK4_0 / (4 * QR4_0))
#define QR4_0 2
@@ -184,13 +181,6 @@ typedef struct {
} block_q1_0;
static_assert(sizeof(block_q1_0) == sizeof(ggml_half) + QK1_0 / 8, "wrong q1_0 block size/padding");
#define QK2_0 64
typedef struct {
ggml_half d; // delta (scale)
uint8_t qs[QK2_0 / 4]; // 2 bits per element
} block_q2_0;
static_assert(sizeof(block_q2_0) == sizeof(ggml_half) + QK2_0 / 4, "wrong q2_0 block size/padding");
#define QK4_0 32
typedef struct {
ggml_half d; // delta
ggml/src/ggml-cpu/CMakeLists.txt
+2 -23
View File
@@ -638,7 +638,6 @@ function(ggml_add_cpu_backend_variant_impl tag_name)
${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_f32_qai8dxp_qsi8cxp/
${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_fp32_bf16p_bf16p/
${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_f32_f16p_qsi4c32p/
${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_f32_f32p_f32p/
${KLEIDIAI_SRC}/kai/ukernels/matmul/pack/)
set(ARCH_FLAGS_TEMP "${ARCH_FLAGS}")
@@ -678,18 +677,7 @@ function(ggml_add_cpu_backend_variant_impl tag_name)
endif()
if (NOT SME_ENABLED MATCHES -1)
list(APPEND GGML_KLEIDIAI_SME_SOURCES
${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_f32_qai8dxp_qsi8cxp/kai_matmul_clamp_f32_qai8dxp1vlx4_qsi8cxp4vlx4_1vlx4vl_sme_mopa.c
${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_f32_qai8dxp_qsi8cxp/kai_matmul_clamp_f32_qai8dxp1vlx4_qsi8cxp4vlx4_1vlx4vl_sme_mopa_asm.S
${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_f32_qai8dxp_qsi8cxp/kai_matmul_clamp_f32_qai8dxp1x4_qsi8cxp4vlx4_1x4vl_sme_dot.c
${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_f32_qai8dxp_qsi8cxp/kai_matmul_clamp_f32_qai8dxp1x4_qsi8cxp4vlx4_1x4vl_sme_dot_asm.S
${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_f32_f32p_f32p/kai_matmul_clamp_f32_f32p2vlx1_f32p2vlx1b_2vlx2vl_sme_mopa.c
${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_f32_f32p_f32p/kai_matmul_clamp_f32_f32p2vlx1_f32p2vlx1b_2vlx2vl_sme_mopa_asm.S)
set_source_files_properties(${GGML_KLEIDIAI_SME_SOURCES}
PROPERTIES COMPILE_OPTIONS "-fno-tree-vectorize;${ARCH_FLAGS_TEMP}+sve+sve2+sme")
list(APPEND GGML_CPU_SOURCES ${GGML_KLEIDIAI_SME_SOURCES})
list(APPEND GGML_KLEIDIAI_SME2_SOURCES
list(APPEND GGML_KLEIDIAI_SOURCES
${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_f32_qsi8d32p_qsi4c32p/kai_matmul_clamp_f32_qsi8d32p1x4_qsi4c32p4vlx4_1x4vl_sme2_sdot.c
${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_f32_qai8dxp_qsi8cxp/kai_matmul_clamp_f32_qai8dxp1vlx4_qsi8cxp4vlx4_1vlx4vl_sme2_mopa.c
${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_f32_qai8dxp_qsi8cxp/kai_matmul_clamp_f32_qai8dxp1vlx4_qsi8cxp4vlx4_1vlx4vl_sme2_mopa_asm.S
@@ -699,20 +687,11 @@ function(ggml_add_cpu_backend_variant_impl tag_name)
${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_fp32_bf16p_bf16p/kai_matmul_clamp_f32_bf16p2vlx2_bf16p2vlx2_2vlx2vl_sme2_mopa_asm.S
${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_f32_f16p_qsi4c32p/kai_matmul_clamp_f32_f16p1vlx2_qsi4c32p4vlx2_1vlx4vl_sme2_mopa.c
${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_f32_f16p_qsi4c32p/kai_matmul_clamp_f32_f16p1vlx2_qsi4c32p4vlx2_1vlx4vl_sme2_mopa_asm.S
${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_f32_f32p_f32p/kai_matmul_clamp_f32_f32p2vlx1_f32p2vlx1biasf32_sme2_mopa.c
${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_f32_f32p_f32p/kai_matmul_clamp_f32_f32p2vlx1_f32p2vlx1biasf32_sme2_mopa_asm.S
${KLEIDIAI_SRC}/kai/ukernels/matmul/pack/kai_lhs_pack_bf16p2vlx2_f32_sme.c
${KLEIDIAI_SRC}/kai/ukernels/matmul/pack/kai_rhs_pack_kxn_bf16p2vlx2b_f32_x32_sme.c
${KLEIDIAI_SRC}/kai/ukernels/matmul/pack/kai_lhs_pack_f16pmrx2_f32_neon.c
${KLEIDIAI_SRC}/kai/ukernels/matmul/pack/kai_lhs_pack_f32p2vlx1_f32_sme.c
${KLEIDIAI_SRC}/kai/ukernels/matmul/pack/kai_lhs_pack_f32p2vlx1_f32_sme_asm.S
${KLEIDIAI_SRC}/kai/ukernels/matmul/pack/kai_rhs_pack_nxk_f32p2vlx1biasf32_f32_f32_sme.c
${KLEIDIAI_SRC}/kai/ukernels/matmul/pack/kai_rhs_pack_nxk_f32p2vlx1biasf32_f32_f32_sme_asm.S
${KLEIDIAI_SRC}/kai/kai_common_sme_asm.S)
set_source_files_properties(${GGML_KLEIDIAI_SME2_SOURCES}
PROPERTIES COMPILE_OPTIONS "-fno-tree-vectorize;${ARCH_FLAGS_TEMP}+sve+sve2+sme2+fp16")
list(APPEND GGML_CPU_SOURCES ${GGML_KLEIDIAI_SME2_SOURCES})
set(PRIVATE_ARCH_FLAGS "-fno-tree-vectorize;${PRIVATE_ARCH_FLAGS}")
set(PRIVATE_ARCH_FLAGS "-fno-tree-vectorize;${PRIVATE_ARCH_FLAGS}+sve+sve2+sme2+fp16")
endif()
if (NOT SVE_ENABLED MATCHES -1)
ggml/src/ggml-cpu/arch-fallback.h
-7
View File
@@ -17,7 +17,6 @@
#define ggml_vec_dot_mxfp4_q8_0_generic ggml_vec_dot_mxfp4_q8_0
#define ggml_vec_dot_nvfp4_q8_0_generic ggml_vec_dot_nvfp4_q8_0
#define ggml_vec_dot_q1_0_q8_0_generic ggml_vec_dot_q1_0_q8_0
#define ggml_vec_dot_q2_0_q8_0_generic ggml_vec_dot_q2_0_q8_0
#define ggml_vec_dot_tq1_0_q8_K_generic ggml_vec_dot_tq1_0_q8_K
#define ggml_vec_dot_tq2_0_q8_K_generic ggml_vec_dot_tq2_0_q8_K
#define ggml_vec_dot_q2_K_q8_K_generic ggml_vec_dot_q2_K_q8_K
@@ -83,7 +82,6 @@
#define ggml_gemm_q2_K_8x8_q8_K_generic ggml_gemm_q2_K_8x8_q8_K
#elif defined(__x86_64__) || defined(__i386__) || defined(_M_IX86) || defined(_M_X64)
// quants.c
#define ggml_vec_dot_q2_0_q8_0_generic ggml_vec_dot_q2_0_q8_0
// repack.cpp
#define ggml_quantize_mat_q8_0_4x4_generic ggml_quantize_mat_q8_0_4x4
#define ggml_quantize_mat_q8_K_4x4_generic ggml_quantize_mat_q8_K_4x4
@@ -115,7 +113,6 @@
#define quantize_row_q8_K_generic quantize_row_q8_K
#define ggml_vec_dot_nvfp4_q8_0_generic ggml_vec_dot_nvfp4_q8_0
#define ggml_vec_dot_q1_0_q8_0_generic ggml_vec_dot_q1_0_q8_0
#define ggml_vec_dot_q2_0_q8_0_generic ggml_vec_dot_q2_0_q8_0
#define ggml_vec_dot_tq1_0_q8_K_generic ggml_vec_dot_tq1_0_q8_K
#define ggml_vec_dot_tq2_0_q8_K_generic ggml_vec_dot_tq2_0_q8_K
#define ggml_vec_dot_iq1_m_q8_K_generic ggml_vec_dot_iq1_m_q8_K
@@ -165,7 +162,6 @@
#define ggml_vec_dot_mxfp4_q8_0_generic ggml_vec_dot_mxfp4_q8_0
#define ggml_vec_dot_nvfp4_q8_0_generic ggml_vec_dot_nvfp4_q8_0
#define ggml_vec_dot_q1_0_q8_0_generic ggml_vec_dot_q1_0_q8_0
#define ggml_vec_dot_q2_0_q8_0_generic ggml_vec_dot_q2_0_q8_0
// repack.cpp
#define ggml_quantize_mat_q8_0_4x4_generic ggml_quantize_mat_q8_0_4x4
#define ggml_quantize_mat_q8_0_4x8_generic ggml_quantize_mat_q8_0_4x8
@@ -206,7 +202,6 @@
#elif defined(__riscv)
// quants.c
#define ggml_vec_dot_nvfp4_q8_0_generic ggml_vec_dot_nvfp4_q8_0
#define ggml_vec_dot_q2_0_q8_0_generic ggml_vec_dot_q2_0_q8_0
// repack.cpp
#define ggml_quantize_mat_q8_0_4x1_generic ggml_quantize_mat_q8_0_4x1
#define ggml_quantize_mat_q8_0_4x4_generic ggml_quantize_mat_q8_0_4x4
@@ -248,7 +243,6 @@
#define quantize_row_q8_K_generic quantize_row_q8_K
#define ggml_vec_dot_nvfp4_q8_0_generic ggml_vec_dot_nvfp4_q8_0
#define ggml_vec_dot_q1_0_q8_0_generic ggml_vec_dot_q1_0_q8_0
#define ggml_vec_dot_q2_0_q8_0_generic ggml_vec_dot_q2_0_q8_0
#define ggml_vec_dot_tq1_0_q8_K_generic ggml_vec_dot_tq1_0_q8_K
#define ggml_vec_dot_tq2_0_q8_K_generic ggml_vec_dot_tq2_0_q8_K
#define ggml_vec_dot_q2_K_q8_K_generic ggml_vec_dot_q2_K_q8_K
@@ -312,7 +306,6 @@
#define ggml_vec_dot_mxfp4_q8_0_generic ggml_vec_dot_mxfp4_q8_0
#define ggml_vec_dot_nvfp4_q8_0_generic ggml_vec_dot_nvfp4_q8_0
#define ggml_vec_dot_q1_0_q8_0_generic ggml_vec_dot_q1_0_q8_0
#define ggml_vec_dot_q2_0_q8_0_generic ggml_vec_dot_q2_0_q8_0
// repack.cpp
#define ggml_quantize_mat_q8_0_4x4_generic ggml_quantize_mat_q8_0_4x4
#define ggml_quantize_mat_q8_0_4x8_generic ggml_quantize_mat_q8_0_4x8
ggml/src/ggml-cpu/arch/arm/cpu-feats.cpp
-5
View File
@@ -28,7 +28,6 @@ struct aarch64_features {
bool has_sve2 = false;
bool has_i8mm = false;
bool has_sme = false;
bool has_sme2 = false;
aarch64_features() {
#if defined(__linux__)
@@ -57,10 +56,6 @@ struct aarch64_features {
has_sme = static_cast<bool>(oldp);
}
if (sysctlbyname("hw.optional.arm.FEAT_SME2", &oldp, &size, NULL, 0) == 0) {
has_sme2 = static_cast<bool>(oldp);
}
// Apple apparently does not implement SVE yet
#endif
}
ggml/src/ggml-cpu/arch/arm/quants.c
+4 -78
View File
@@ -219,80 +219,6 @@ void ggml_vec_dot_q1_0_q8_0(int n, float * GGML_RESTRICT s, size_t bs, const voi
#endif
}
void ggml_vec_dot_q2_0_q8_0(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc) {
const int qk = QK2_0;
const int nb = n / qk;
assert(n % qk == 0);
assert(nrc == 1);
UNUSED(nrc);
UNUSED(bx);
UNUSED(by);
UNUSED(bs);
const block_q2_0 * GGML_RESTRICT x = vx;
const block_q8_0 * GGML_RESTRICT y = vy;
float sumf = 0.0f;
#if defined(__ARM_NEON)
// Replicate pattern: each byte repeated 4 times
static const uint8_t tbl_idx_lo[16] = {0,0,0,0, 1,1,1,1, 2,2,2,2, 3,3,3,3};
static const uint8_t tbl_idx_hi[16] = {4,4,4,4, 5,5,5,5, 6,6,6,6, 7,7,7,7};
// Right-shift amounts: 0,2,4,6 repeated for each group of 4
static const int8_t shift_vals[16] = {0,-2,-4,-6, 0,-2,-4,-6, 0,-2,-4,-6, 0,-2,-4,-6};
const uint8x16_t idx_lo = vld1q_u8(tbl_idx_lo);
const uint8x16_t idx_hi = vld1q_u8(tbl_idx_hi);
const int8x16_t shifts = vld1q_s8(shift_vals);
const uint8x16_t mask2 = vdupq_n_u8(0x03);
const int8x16_t one = vdupq_n_s8(1);
float32x4_t sumv = vdupq_n_f32(0.0f);
for (int i = 0; i < nb; i++) {
const float d0 = GGML_CPU_FP16_TO_FP32(x[i].d);
// group 64: one Q2_0 block (64 weights) maps to two Q8_0 blocks (2 * 32 = 64)
for (int k = 0; k < 2; k++) {
const block_q8_0 * GGML_RESTRICT yb = &y[i * 2 + k];
const float d1 = GGML_CPU_FP16_TO_FP32(yb->d);
// Load 8 bytes of packed 2-bit values
const uint8x8_t raw = vld1_u8(&x[i].qs[k * 8]);
const uint8x16_t raw16 = vcombine_u8(raw, raw);
// First 16 elements: replicate bytes 0-3, shift, mask, subtract 1
uint8x16_t bytes0 = ggml_vqtbl1q_u8(raw16, idx_lo);
int8x16_t qv0 = vsubq_s8(
vreinterpretq_s8_u8(vandq_u8(vshlq_u8(bytes0, shifts), mask2)),
one);
// Second 16 elements: replicate bytes 4-7, shift, mask, subtract 1
uint8x16_t bytes1 = ggml_vqtbl1q_u8(raw16, idx_hi);
int8x16_t qv1 = vsubq_s8(
vreinterpretq_s8_u8(vandq_u8(vshlq_u8(bytes1, shifts), mask2)),
one);
// Load Q8_0 values and dot product
const int8x16_t y0 = vld1q_s8(yb->qs);
const int8x16_t y1 = vld1q_s8(yb->qs + 16);
int32x4_t p0 = ggml_vdotq_s32(vdupq_n_s32(0), qv0, y0);
int32x4_t p1 = ggml_vdotq_s32(p0, qv1, y1);
sumv = vmlaq_n_f32(sumv, vcvtq_f32_s32(p1), d0 * d1);
}
}
sumf = vaddvq_f32(sumv);
#else
ggml_vec_dot_q2_0_q8_0_generic(n, s, bs, vx, bx, vy, by, nrc);
return;
#endif
*s = sumf;
}
void ggml_vec_dot_q4_0_q8_0(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc) {
const int qk = QK8_0;
@@ -886,10 +812,10 @@ void ggml_vec_dot_nvfp4_q8_0(int n, float * GGML_RESTRICT s, size_t bs, const vo
const float dy0 = GGML_CPU_FP16_TO_FP32(y[2*ib].d);
const float dy1 = GGML_CPU_FP16_TO_FP32(y[2*ib+1].d);
const float32x4_t nvsc = {
GGML_CPU_UE4M3_TO_FP32(x[ib].d[0]),
GGML_CPU_UE4M3_TO_FP32(x[ib].d[1]),
GGML_CPU_UE4M3_TO_FP32(x[ib].d[2]),
GGML_CPU_UE4M3_TO_FP32(x[ib].d[3])
ggml_ue4m3_to_fp32(x[ib].d[0]),
ggml_ue4m3_to_fp32(x[ib].d[1]),
ggml_ue4m3_to_fp32(x[ib].d[2]),
ggml_ue4m3_to_fp32(x[ib].d[3])
};
const float32x4_t scales = vmulq_f32(nvsc, (float32x4_t){dy0, dy0, dy1, dy1});
ggml/src/ggml-cpu/ggml-cpu.c
+1 -48
View File
@@ -230,12 +230,6 @@ static const struct ggml_type_traits_cpu type_traits_cpu[GGML_TYPE_COUNT] = {
.vec_dot_type = GGML_TYPE_Q8_0,
.nrows = 1,
},
[GGML_TYPE_Q2_0] = {
.from_float = quantize_row_q2_0,
.vec_dot = ggml_vec_dot_q2_0_q8_0,
.vec_dot_type = GGML_TYPE_Q8_0,
.nrows = 1,
},
[GGML_TYPE_Q4_0] = {
.from_float = quantize_row_q4_0,
.vec_dot = ggml_vec_dot_q4_0_q8_0,
@@ -2060,22 +2054,6 @@ static void ggml_compute_forward(struct ggml_compute_params * params, struct ggm
{
ggml_compute_forward_gated_delta_net(params, tensor);
} break;
case GGML_OP_LIGHTNING_INDEXER:
{
ggml_compute_forward_lightning_indexer(params, tensor);
} break;
case GGML_OP_DSV4_HC_COMB:
{
ggml_compute_forward_dsv4_hc_comb(params, tensor);
} break;
case GGML_OP_DSV4_HC_PRE:
{
ggml_compute_forward_dsv4_hc_pre(params, tensor);
} break;
case GGML_OP_DSV4_HC_POST:
{
ggml_compute_forward_dsv4_hc_post(params, tensor);
} break;
case GGML_OP_MAP_CUSTOM1:
{
ggml_compute_forward_map_custom1(params, tensor);
@@ -2256,9 +2234,6 @@ static int ggml_get_n_tasks(struct ggml_tensor * node, int n_threads) {
case GGML_OP_COUNT_EQUAL:
case GGML_OP_SOLVE_TRI:
case GGML_OP_GATED_DELTA_NET:
case GGML_OP_DSV4_HC_COMB:
case GGML_OP_DSV4_HC_PRE:
case GGML_OP_DSV4_HC_POST:
{
n_tasks = n_threads;
} break;
@@ -2399,7 +2374,6 @@ static int ggml_get_n_tasks(struct ggml_tensor * node, int n_threads) {
case GGML_OP_FLASH_ATTN_BACK:
case GGML_OP_SSM_CONV:
case GGML_OP_SSM_SCAN:
case GGML_OP_LIGHTNING_INDEXER:
{
n_tasks = n_threads;
} break;
@@ -2874,14 +2848,7 @@ struct ggml_cplan ggml_graph_plan(
} break;
case GGML_OP_OUT_PROD:
{
if (ggml_is_quantized(node->src[0]->type) ||
node->src[0]->type == GGML_TYPE_F16) {
cur = ggml_type_size(GGML_TYPE_F32) * node->src[0]->ne[0] * n_tasks;
}
} break;
case GGML_OP_SET_ROWS:
{
if (node->src[0]->type == GGML_TYPE_F16 && node->type != GGML_TYPE_F16) {
if (ggml_is_quantized(node->src[0]->type)) {
cur = ggml_type_size(GGML_TYPE_F32) * node->src[0]->ne[0] * n_tasks;
}
} break;
@@ -2992,12 +2959,6 @@ struct ggml_cplan ggml_graph_plan(
{
GGML_ABORT("fatal error");
}
case GGML_OP_LIGHTNING_INDEXER:
{
// temp buffer for dequantizing lightning indexer keys
const int64_t ne10 = node->src[1]->ne[0];
cur += sizeof(float)*ne10*n_tasks;
} break;
default:
break;
}
@@ -3807,14 +3768,6 @@ int ggml_cpu_has_sme(void) {
#endif
}
int ggml_cpu_has_sme2(void) {
#if defined(__ARM_ARCH) && defined(__ARM_FEATURE_SME2)
return 1;
#else
return 0;
#endif
}
void ggml_cpu_init(void) {
// needed to initialize ggml_time
{
ggml/src/ggml-cpu/ggml-cpu.cpp
+2 -6
View File
@@ -462,12 +462,11 @@ static bool ggml_backend_cpu_device_supports_op(ggml_backend_dev_t dev, const st
return max_bias == 0.0f;
}
case GGML_OP_IM2COL_BACK:
return src0->type == GGML_TYPE_F32 && (src1->type == GGML_TYPE_F32 || src1->type == GGML_TYPE_F16);
return src0->type == GGML_TYPE_F32 && src1->type == GGML_TYPE_F32;
case GGML_OP_GET_ROWS_BACK:
return src0->type == GGML_TYPE_F32 || src0->type == GGML_TYPE_F16;
case GGML_OP_OUT_PROD:
return (src0->type == GGML_TYPE_F32 ||
((src0->type == GGML_TYPE_F16 || ggml_is_quantized(src0->type)) && src0->ne[2] == src1->ne[2] && src0->ne[3] == src1->ne[3])) &&
return (src0->type == GGML_TYPE_F32 || (ggml_is_quantized(src0->type) && src0->ne[2] == src1->ne[2] && src0->ne[3] == src1->ne[3])) &&
src1->type == GGML_TYPE_F32 && op->type == GGML_TYPE_F32;
default:
return true;
@@ -595,9 +594,6 @@ static ggml_backend_feature * ggml_backend_cpu_get_features(ggml_backend_reg_t r
if (ggml_cpu_has_sme()) {
features.push_back({ "SME", "1" });
}
if (ggml_cpu_has_sme2()) {
features.push_back({ "SME2", "1" });
}
if (ggml_cpu_has_riscv_v()) {
features.push_back({ "RISCV_V", "1" });
}
ggml/src/ggml-cpu/kleidiai/kernels.cpp
+2 -193
View File
@@ -13,8 +13,6 @@
#include "kai_matmul_clamp_f32_bf16p2vlx2_bf16p2vlx2_2vlx2vl_sme2_mopa.h"
#include "kai_matmul_clamp_f32_qai8dxp1vlx4_qsi8cxp4vlx4_1vlx4vl_sme2_mopa.h"
#include "kai_matmul_clamp_f32_qai8dxp1x4_qsi8cxp4vlx4_1x4vl_sme2_dot.h"
#include "kai_matmul_clamp_f32_qai8dxp1vlx4_qsi8cxp4vlx4_1vlx4vl_sme_mopa.h"
#include "kai_matmul_clamp_f32_qai8dxp1x4_qsi8cxp4vlx4_1x4vl_sme_dot.h"
#include "kai_matmul_clamp_f32_qai8dxp1x8_qsi8cxp4x8_1x4_neon_dotprod.h"
#include "kai_matmul_clamp_f32_qai8dxp1x4_qsi8cxp4x4_1x4_neon_dotprod.h"
#include "kai_matmul_clamp_f32_qai8dxp4x4_qsi8cxp4x4_16x4_neon_dotprod.h"
@@ -22,18 +20,14 @@
#include "kai_matmul_clamp_f32_qsi8d32p4x8_qsi4c32p8x8_16x8_sve_i8mm.h"
#include "kai_matmul_clamp_f32_qsi8d32p1x8_qsi4c32p8x8_1x8_sve_dotprod.h"
#include "kai_matmul_clamp_f32_f16p1vlx2_qsi4c32p4vlx2_1vlx4vl_sme2_mopa.h"
#include "kai_matmul_clamp_f32_f32p2vlx1_f32p2vlx1biasf32_sme2_mopa.h"
#include "kai_matmul_clamp_f32_f32p2vlx1_f32p2vlx1b_2vlx2vl_sme_mopa.h"
#include "kai_lhs_pack_bf16p2vlx2_f32_sme.h"
#include "kai_lhs_pack_f32p2vlx1_f32_sme.h"
#include "kai_lhs_quant_pack_qsi8d32p_f32.h"
#include "kai_lhs_quant_pack_qsi8d32p4x8sb_f32_neon.h"
#include "kai_lhs_quant_pack_qsi8d32p_f32_neon.h"
#include "kai_lhs_quant_pack_qai8dxp_f32.h"
#include "kai_rhs_pack_kxn_bf16p2vlx2b_f32_x32_sme.h"
#include "kai_rhs_pack_nxk_f32p2vlx1biasf32_f32_f32_sme.h"
#include "kai_rhs_pack_nxk_qsi4c32pscalef16_qsu4c32s16s0.h"
#include "kai_rhs_pack_nxk_qsi4c32ps1s0scalef16_qsu4c32s16s0_neon.h"
#include "kai_rhs_pack_nxk_qsi8cxp_qsi8cx_neon.h"
@@ -362,7 +356,7 @@ static ggml_kleidiai_kernels gemm_gemv_kernels[] = {
/* .packed_stride_ex = */ &rhs_stride_fn4<kai_get_rhs_packed_stride_rhs_pack_nxk_qsi4c32ps1s0scalef16_qsu4c32s16s0_neon>,
/* .pack_func_ex = */ &rhs_pack_fn12<kai_run_rhs_pack_nxk_qsi4c32ps1s0scalef16_qsu4c32s16s0_neon>,
},
/* .required_cpu = */ CPU_FEATURE_SME2,
/* .required_cpu = */ CPU_FEATURE_SME,
/* .lhs_type = */ GGML_TYPE_F32,
/* .rhs_type = */ GGML_TYPE_Q4_0,
/* .op_type = */ GGML_TYPE_F32,
@@ -415,7 +409,7 @@ static ggml_kleidiai_kernels gemm_gemv_kernels[] = {
/* .packed_stride_ex = */ &rhs_stride_fn1<kai_get_rhs_packed_stride_rhs_pack_kxn_bf16p2vlx2b_f32_x32_sme>,
/* .pack_func_ex = */ &rhs_pack_fn13<kai_run_rhs_pack_kxn_bf16p2vlx2b_f32_x32_sme>,
},
/* .required_cpu = */ CPU_FEATURE_SME2,
/* .required_cpu = */ CPU_FEATURE_SME,
/* .lhs_type = */ GGML_TYPE_F32,
/* .rhs_type = */ GGML_TYPE_F16,
/* .op_type = */ GGML_TYPE_F32,
@@ -752,59 +746,6 @@ static ggml_kleidiai_kernels gemm_gemv_kernels_q8[] = {
/* .packed_stride_ex = */ &rhs_stride_fn4<kai_get_rhs_packed_stride_rhs_pack_nxk_qsi8cxp_qsi8cx_neon>,
/* .pack_func_ex = */ &rhs_pack_scale_fn12<kai_run_rhs_pack_nxk_qsi8cxp_qsi8cx_neon>,
},
/* .required_cpu = */ CPU_FEATURE_SME2,
/* .lhs_type = */ GGML_TYPE_F32,
/* .rhs_type = */ GGML_TYPE_Q8_0,
/* .op_type = */ GGML_TYPE_F32,
},
{
/* SME GEMM (pure SME, no SME2 required) */
{
/* .get_m_step = */ kai_get_m_step_matmul_clamp_f32_qai8dxp1vlx4_qsi8cxp4vlx4_1vlx4vl_sme_mopa,
/* .get_n_step = */ kai_get_n_step_matmul_clamp_f32_qai8dxp1vlx4_qsi8cxp4vlx4_1vlx4vl_sme_mopa,
/* .get_mr = */ kai_get_mr_matmul_clamp_f32_qai8dxp1vlx4_qsi8cxp4vlx4_1vlx4vl_sme_mopa,
/* .get_nr = */ kai_get_nr_matmul_clamp_f32_qai8dxp1vlx4_qsi8cxp4vlx4_1vlx4vl_sme_mopa,
/* .get_kr = */ kai_get_kr_matmul_clamp_f32_qai8dxp1vlx4_qsi8cxp4vlx4_1vlx4vl_sme_mopa,
/* .get_sr = */ kai_get_sr_matmul_clamp_f32_qai8dxp1vlx4_qsi8cxp4vlx4_1vlx4vl_sme_mopa,
/* .get_dst_offset = */ kai_get_dst_offset_matmul_clamp_f32_qai8dxp1vlx4_qsi8cxp4vlx4_1vlx4vl_sme_mopa,
/* .get_dst_size = */ kai_get_dst_size_matmul_clamp_f32_qai8dxp1vlx4_qsi8cxp4vlx4_1vlx4vl_sme_mopa,
/* .get_lhs_offset_ex = */ &kernel_offs_fn2<kai_get_lhs_packed_offset_matmul_clamp_f32_qai8dxp1vlx4_qsi8cxp4vlx4_1vlx4vl_sme_mopa>,
/* .get_rhs_packed_offset_ex = */ &kernel_offs_fn2<kai_get_rhs_packed_offset_matmul_clamp_f32_qai8dxp1vlx4_qsi8cxp4vlx4_1vlx4vl_sme_mopa>,
/* .run_kernel_ex = */ &kernel_run_float_fn10<kai_run_matmul_clamp_f32_qai8dxp1vlx4_qsi8cxp4vlx4_1vlx4vl_sme_mopa>,
},
/* .gemm_lhs_info = */ {
/* .get_offset = */ kai_get_lhs_offset_lhs_quant_pack_qai8dxp_f32,
/* .get_packed_offset_ex = */ &lhs_offs_fn5<kai_get_lhs_packed_offset_lhs_quant_pack_qai8dxp_f32>,
/* .packed_size_ex = */ &lhs_ps_fn5<kai_get_lhs_packed_size_lhs_quant_pack_qai8dxp_f32>,
/* .pack_func_ex = */ &lhs_pack_float_fn9_no_bl<kai_run_lhs_quant_pack_qai8dxp_f32>,
},
/* SME GEMV (pure SME, no SME2 required) */
{
/* .get_m_step = */ kai_get_m_step_matmul_clamp_f32_qai8dxp1x4_qsi8cxp4vlx4_1x4vl_sme_dot,
/* .get_n_step = */ kai_get_n_step_matmul_clamp_f32_qai8dxp1x4_qsi8cxp4vlx4_1x4vl_sme_dot,
/* .get_mr = */ kai_get_mr_matmul_clamp_f32_qai8dxp1x4_qsi8cxp4vlx4_1x4vl_sme_dot,
/* .get_nr = */ kai_get_nr_matmul_clamp_f32_qai8dxp1x4_qsi8cxp4vlx4_1x4vl_sme_dot,
/* .get_kr = */ kai_get_kr_matmul_clamp_f32_qai8dxp1x4_qsi8cxp4vlx4_1x4vl_sme_dot,
/* .get_sr = */ kai_get_sr_matmul_clamp_f32_qai8dxp1x4_qsi8cxp4vlx4_1x4vl_sme_dot,
/* .get_dst_offset = */ kai_get_dst_offset_matmul_clamp_f32_qai8dxp1x4_qsi8cxp4vlx4_1x4vl_sme_dot,
/* .get_dst_size = */ kai_get_dst_size_matmul_clamp_f32_qai8dxp1x4_qsi8cxp4vlx4_1x4vl_sme_dot,
/* .get_lhs_offset_ex = */ &kernel_offs_fn2<kai_get_lhs_packed_offset_matmul_clamp_f32_qai8dxp1x4_qsi8cxp4vlx4_1x4vl_sme_dot>,
/* .get_rhs_packed_offset_ex = */ &kernel_offs_fn2<kai_get_rhs_packed_offset_matmul_clamp_f32_qai8dxp1x4_qsi8cxp4vlx4_1x4vl_sme_dot>,
/* .run_kernel_ex = */ &kernel_run_float_fn10<kai_run_matmul_clamp_f32_qai8dxp1x4_qsi8cxp4vlx4_1x4vl_sme_dot>,
},
/* .gemv_lhs_info = */ {
/* .get_offset = */ kai_get_lhs_offset_lhs_quant_pack_qai8dxp_f32,
/* .get_packed_offset_ex = */ &lhs_offs_fn5<kai_get_lhs_packed_offset_lhs_quant_pack_qai8dxp_f32>,
/* .packed_size_ex = */ &lhs_ps_fn5<kai_get_lhs_packed_size_lhs_quant_pack_qai8dxp_f32>,
/* .pack_func_ex = */ &lhs_pack_float_fn9_no_bl<kai_run_lhs_quant_pack_qai8dxp_f32>,
},
/* .rhs_info = */ {
/* .packed_stride = */ kai_get_rhs_packed_stride_rhs_pack_nxk_qsi8cxp_qsi8cx_neon,
/* .to_float = */ dequantize_row_qsi8cxp,
/* .packed_size_ex = */ &rhs_ps_fn5<kai_get_rhs_packed_size_rhs_pack_nxk_qsi8cxp_qsi8cx_neon>,
/* .packed_stride_ex = */ &rhs_stride_fn4<kai_get_rhs_packed_stride_rhs_pack_nxk_qsi8cxp_qsi8cx_neon>,
/* .pack_func_ex = */ &rhs_pack_scale_fn12<kai_run_rhs_pack_nxk_qsi8cxp_qsi8cx_neon>,
},
/* .required_cpu = */ CPU_FEATURE_SME,
/* .lhs_type = */ GGML_TYPE_F32,
/* .rhs_type = */ GGML_TYPE_Q8_0,
@@ -924,118 +865,6 @@ static ggml_kleidiai_kernels gemm_gemv_kernels_q8[] = {
{ /* Sentinel */ }
};
static ggml_kleidiai_kernels ggml_kleidiai_kernels_f32[] = {
#if defined(__ARM_FEATURE_SME)
{
/* SME2 GEMM */
{
/* .get_m_step = */ kai_get_m_step_matmul_clamp_f32_f32p2vlx1_f32p2vlx1biasf32_sme2_mopa,
/* .get_n_step = */ kai_get_n_step_matmul_clamp_f32_f32p2vlx1_f32p2vlx1biasf32_sme2_mopa,
/* .get_mr = */ kai_get_mr_matmul_clamp_f32_f32p2vlx1_f32p2vlx1biasf32_sme2_mopa,
/* .get_nr = */ kai_get_nr_matmul_clamp_f32_f32p2vlx1_f32p2vlx1biasf32_sme2_mopa,
/* .get_kr = */ kai_get_kr_matmul_clamp_f32_f32p2vlx1_f32p2vlx1biasf32_sme2_mopa,
/* .get_sr = */ kai_get_sr_matmul_clamp_f32_f32p2vlx1_f32p2vlx1biasf32_sme2_mopa,
/* .get_dst_offset = */ kai_get_dst_offset_matmul_clamp_f32_f32p2vlx1_f32p2vlx1biasf32_sme2_mopa,
/* .get_dst_size = */ kai_get_dst_size_matmul_clamp_f32_f32p2vlx1_f32p2vlx1biasf32_sme2_mopa,
/* .get_lhs_offset_ex = */ &kernel_offs_fn2<kai_get_lhs_packed_offset_matmul_clamp_f32_f32p2vlx1_f32p2vlx1biasf32_sme2_mopa>,
/* .get_rhs_packed_offset_ex = */ &kernel_offs_fn2<kai_get_rhs_packed_offset_matmul_clamp_f32_f32p2vlx1_f32p2vlx1biasf32_sme2_mopa>,
/* .run_kernel_ex = */ &kernel_run_fn10<kai_run_matmul_clamp_f32_f32p2vlx1_f32p2vlx1biasf32_sme2_mopa>,
},
/* .gemm_lhs_info = */ {
/* .get_offset = */ kai_get_lhs_offset_lhs_pack_f32p2vlx1_f32_sme,
/* .get_packed_offset_ex = */ &lhs_offs_fn5<kai_get_lhs_packed_offset_lhs_pack_f32p2vlx1_f32_sme>,
/* .packed_size_ex = */ &lhs_ps_fn5<kai_get_lhs_packed_size_lhs_pack_f32p2vlx1_f32_sme>,
/* .pack_func_ex = */ &lhs_pack_void_fn9<kai_run_lhs_pack_f32p2vlx1_f32_sme>,
},
/* SME GEMV */
{
/* .get_m_step = */ kai_get_m_step_matmul_clamp_f32_f32p2vlx1_f32p2vlx1biasf32_sme2_mopa,
/* .get_n_step = */ kai_get_n_step_matmul_clamp_f32_f32p2vlx1_f32p2vlx1biasf32_sme2_mopa,
/* .get_mr = */ kai_get_mr_matmul_clamp_f32_f32p2vlx1_f32p2vlx1biasf32_sme2_mopa,
/* .get_nr = */ kai_get_nr_matmul_clamp_f32_f32p2vlx1_f32p2vlx1biasf32_sme2_mopa,
/* .get_kr = */ kai_get_kr_matmul_clamp_f32_f32p2vlx1_f32p2vlx1biasf32_sme2_mopa,
/* .get_sr = */ kai_get_sr_matmul_clamp_f32_f32p2vlx1_f32p2vlx1biasf32_sme2_mopa,
/* .get_dst_offset = */ kai_get_dst_offset_matmul_clamp_f32_f32p2vlx1_f32p2vlx1biasf32_sme2_mopa,
/* .get_dst_size = */ kai_get_dst_size_matmul_clamp_f32_f32p2vlx1_f32p2vlx1biasf32_sme2_mopa,
/* .get_lhs_offset_ex = */ nullptr,
/* .get_rhs_packed_offset_ex = */ nullptr,
/* .run_kernel_ex = */ nullptr,
},
/* .gemv_lhs_info = */ {
/* .get_offset = */ kai_get_lhs_offset_lhs_pack_f32p2vlx1_f32_sme,
/* .get_packed_offset_ex = */ &lhs_offs_fn5<kai_get_lhs_packed_offset_lhs_pack_f32p2vlx1_f32_sme>,
/* .packed_size_ex = */ &lhs_ps_fn5<kai_get_lhs_packed_size_lhs_pack_f32p2vlx1_f32_sme>,
/* .pack_func_ex = */ &lhs_pack_void_fn9<kai_run_lhs_pack_f32p2vlx1_f32_sme>,
},
/* .rhs_info = */ {
/* .packed_stride = */ nullptr,
/* .to_float = */ nullptr,
/* .packed_size_ex = */ &rhs_ps_fn2<kai_get_rhs_packed_size_rhs_pack_nxk_f32p2vlx1biasf32_f32_f32_sme>,
/* .packed_stride_ex = */ &rhs_stride_fn1<kai_get_rhs_packed_stride_rhs_pack_nxk_f32p2vlx1biasf32_f32_f32_sme>,
/* .pack_func_ex = */ &rhs_pack_fn13<kai_run_rhs_pack_nxk_f32p2vlx1biasf32_f32_f32_sme>,
},
/* .required_cpu = */ CPU_FEATURE_SME2,
/* .lhs_type = */ GGML_TYPE_F32,
/* .rhs_type = */ GGML_TYPE_F32,
/* .op_type = */ GGML_TYPE_F32,
},
{
/* SME GEMM */
{
/* .get_m_step = */ kai_get_m_step_matmul_clamp_f32_f32p2vlx1_f32p2vlx1b_2vlx2vl_sme_mopa,
/* .get_n_step = */ kai_get_n_step_matmul_clamp_f32_f32p2vlx1_f32p2vlx1b_2vlx2vl_sme_mopa,
/* .get_mr = */ kai_get_mr_matmul_clamp_f32_f32p2vlx1_f32p2vlx1b_2vlx2vl_sme_mopa,
/* .get_nr = */ kai_get_nr_matmul_clamp_f32_f32p2vlx1_f32p2vlx1b_2vlx2vl_sme_mopa,
/* .get_kr = */ kai_get_kr_matmul_clamp_f32_f32p2vlx1_f32p2vlx1b_2vlx2vl_sme_mopa,
/* .get_sr = */ kai_get_sr_matmul_clamp_f32_f32p2vlx1_f32p2vlx1b_2vlx2vl_sme_mopa,
/* .get_dst_offset = */ kai_get_dst_offset_matmul_clamp_f32_f32p2vlx1_f32p2vlx1b_2vlx2vl_sme_mopa,
/* .get_dst_size = */ kai_get_dst_size_matmul_clamp_f32_f32p2vlx1_f32p2vlx1b_2vlx2vl_sme_mopa,
/* .get_lhs_offset_ex = */ &kernel_offs_fn2<kai_get_lhs_packed_offset_matmul_clamp_f32_f32p2vlx1_f32p2vlx1b_2vlx2vl_sme_mopa>,
/* .get_rhs_packed_offset_ex = */ &kernel_offs_fn2<kai_get_rhs_packed_offset_matmul_clamp_f32_f32p2vlx1_f32p2vlx1b_2vlx2vl_sme_mopa>,
/* .run_kernel_ex = */ &kernel_run_fn10<kai_run_matmul_clamp_f32_f32p2vlx1_f32p2vlx1b_2vlx2vl_sme_mopa>,
},
/* .gemm_lhs_info = */ {
/* .get_offset = */ kai_get_lhs_offset_lhs_pack_f32p2vlx1_f32_sme,
/* .get_packed_offset_ex = */ &lhs_offs_fn5<kai_get_lhs_packed_offset_lhs_pack_f32p2vlx1_f32_sme>,
/* .packed_size_ex = */ &lhs_ps_fn5<kai_get_lhs_packed_size_lhs_pack_f32p2vlx1_f32_sme>,
/* .pack_func_ex = */ &lhs_pack_void_fn9<kai_run_lhs_pack_f32p2vlx1_f32_sme>,
},
/* SME GEMV */
{
/* .get_m_step = */ kai_get_m_step_matmul_clamp_f32_f32p2vlx1_f32p2vlx1b_2vlx2vl_sme_mopa,
/* .get_n_step = */ kai_get_n_step_matmul_clamp_f32_f32p2vlx1_f32p2vlx1b_2vlx2vl_sme_mopa,
/* .get_mr = */ kai_get_mr_matmul_clamp_f32_f32p2vlx1_f32p2vlx1b_2vlx2vl_sme_mopa,
/* .get_nr = */ kai_get_nr_matmul_clamp_f32_f32p2vlx1_f32p2vlx1b_2vlx2vl_sme_mopa,
/* .get_kr = */ kai_get_kr_matmul_clamp_f32_f32p2vlx1_f32p2vlx1b_2vlx2vl_sme_mopa,
/* .get_sr = */ kai_get_sr_matmul_clamp_f32_f32p2vlx1_f32p2vlx1b_2vlx2vl_sme_mopa,
/* .get_dst_offset = */ kai_get_dst_offset_matmul_clamp_f32_f32p2vlx1_f32p2vlx1b_2vlx2vl_sme_mopa,
/* .get_dst_size = */ kai_get_dst_size_matmul_clamp_f32_f32p2vlx1_f32p2vlx1b_2vlx2vl_sme_mopa,
/* .get_lhs_offset_ex = */ nullptr,
/* .get_rhs_packed_offset_ex = */ nullptr,
/* .run_kernel_ex = */ nullptr,
},
/* .gemv_lhs_info = */ {
/* .get_offset = */ kai_get_lhs_offset_lhs_pack_f32p2vlx1_f32_sme,
/* .get_packed_offset_ex = */ &lhs_offs_fn5<kai_get_lhs_packed_offset_lhs_pack_f32p2vlx1_f32_sme>,
/* .packed_size_ex = */ &lhs_ps_fn5<kai_get_lhs_packed_size_lhs_pack_f32p2vlx1_f32_sme>,
/* .pack_func_ex = */ &lhs_pack_void_fn9<kai_run_lhs_pack_f32p2vlx1_f32_sme>,
},
/* .rhs_info = */ {
/* .packed_stride = */ nullptr,
/* .to_float = */ nullptr,
/* .packed_size_ex = */ &rhs_ps_fn2<kai_get_rhs_packed_size_rhs_pack_nxk_f32p2vlx1biasf32_f32_f32_sme>,
/* .packed_stride_ex = */ &rhs_stride_fn1<kai_get_rhs_packed_stride_rhs_pack_nxk_f32p2vlx1biasf32_f32_f32_sme>,
/* .pack_func_ex = */ &rhs_pack_fn13<kai_run_rhs_pack_nxk_f32p2vlx1biasf32_f32_f32_sme>,
},
/* .required_cpu = */ CPU_FEATURE_SME,
/* .lhs_type = */ GGML_TYPE_F32,
/* .rhs_type = */ GGML_TYPE_F32,
/* .op_type = */ GGML_TYPE_F32,
},
#endif
{ /* Sentinel */ }
};
ggml_kleidiai_kernels * ggml_kleidiai_select_kernels(cpu_feature cpu_features, const ggml_tensor * tensor) {
ggml_kleidiai_kernels * kernel = nullptr;
@@ -1059,15 +888,12 @@ ggml_kleidiai_kernels * ggml_kleidiai_select_kernels(cpu_feature cpu_features, c
if (tensor->src[0]->type == GGML_TYPE_Q8_0) {
try_table(gemm_gemv_kernels_q8);
} else if (tensor->src[0]->type == GGML_TYPE_F32) {
try_table(ggml_kleidiai_kernels_f32);
} else {
try_table(gemm_gemv_kernels);
}
#else
GGML_UNUSED(gemm_gemv_kernels);
GGML_UNUSED(gemm_gemv_kernels_q8);
GGML_UNUSED(ggml_kleidiai_kernels_f32);
GGML_UNUSED(cpu_features);
#endif
}
@@ -1111,20 +937,3 @@ ggml_kleidiai_kernels * ggml_kleidiai_select_kernels_q8_0(cpu_feature features)
return kernels;
}
ggml_kleidiai_kernels * ggml_kleidiai_select_kernels_f32(cpu_feature features) {
ggml_kleidiai_kernels * kernels = nullptr;
#if defined(__ARM_FEATURE_SME)
for (size_t i = 0; i < NELEMS(ggml_kleidiai_kernels_f32) - 1; ++i) {
if ((features & ggml_kleidiai_kernels_f32[i].required_cpu) == ggml_kleidiai_kernels_f32[i].required_cpu) {
kernels = &ggml_kleidiai_kernels_f32[i];
break;
}
}
#else
GGML_UNUSED(features);
#endif
return kernels;
}
ggml/src/ggml-cpu/kleidiai/kernels.h
+1 -11
View File
@@ -11,8 +11,7 @@ enum cpu_feature {
CPU_FEATURE_DOTPROD = 1,
CPU_FEATURE_I8MM = 2,
CPU_FEATURE_SVE = 4,
CPU_FEATURE_SME = 8,
CPU_FEATURE_SME2 = 16
CPU_FEATURE_SME = 8
};
inline cpu_feature& operator|=(cpu_feature& lhs, cpu_feature rhs) {
@@ -56,12 +55,6 @@ struct lhs_packing_info {
size_t m_idx_start, const void * lhs, size_t lhs_stride, void * lhs_packed);
};
enum rhs_repack_mode {
RHS_REPACK_PER_KERNEL,
RHS_REPACK_SHARED,
RHS_REPACK_SINGLE_ONLY,
};
struct rhs_packing_info {
size_t (*packed_stride)(size_t k, size_t nr, size_t kr, size_t bl);
@@ -75,8 +68,6 @@ struct rhs_packing_info {
void (*pack_func_ex)(size_t num_groups, size_t n, size_t k, size_t nr, size_t kr, size_t sr, size_t bl,
size_t rhs_stride, const void * rhs, const void * bias, const void * scale, void * rhs_packed, size_t extra_bytes, const void * params);
rhs_repack_mode repack_mode = RHS_REPACK_PER_KERNEL;
};
struct ggml_kleidiai_kernels {
@@ -97,4 +88,3 @@ struct ggml_kleidiai_kernels {
ggml_kleidiai_kernels * ggml_kleidiai_select_kernels(cpu_feature cpu_features, const ggml_tensor * tensor);
ggml_kleidiai_kernels * ggml_kleidiai_select_kernels_q4_0(cpu_feature features);
ggml_kleidiai_kernels * ggml_kleidiai_select_kernels_q8_0(cpu_feature features);
ggml_kleidiai_kernels * ggml_kleidiai_select_kernels_f32(cpu_feature features);
ggml/src/ggml-cpu/kleidiai/kleidiai.cpp
+63 -322
View File
@@ -26,9 +26,6 @@
#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>
#ifndef HWCAP2_SME2
#define HWCAP2_SME2 (1UL << 37)
#endif
#elif defined(__APPLE__)
#include <string_view>
#include <sys/sysctl.h>
@@ -63,21 +60,14 @@ struct ggml_kleidiai_context {
cpu_feature features;
ggml_kleidiai_kernels * kernels_q4;
ggml_kleidiai_kernels * kernels_q8;
ggml_kleidiai_kernels * kernels_f32;
int sme_thread_cap; // <= 0 means “SME disabled/unknown”;
int thread_hint; // <= 0 means “no hint”
int chunk_multiplier;
} static ctx = { CPU_FEATURE_NONE, nullptr, nullptr, nullptr, 0, -1, 4 };
static inline bool is_sme_family(cpu_feature f) {
return (f & (CPU_FEATURE_SME | CPU_FEATURE_SME2)) != CPU_FEATURE_NONE;
}
} static ctx = { CPU_FEATURE_NONE, nullptr, nullptr, 0, -1, 4 };
static const char* cpu_feature_to_string(cpu_feature f) {
if (f == CPU_FEATURE_NONE) {
return "NONE";
} else if ((f & CPU_FEATURE_SME2) == CPU_FEATURE_SME2) {
return "SME2";
} else if ((f & CPU_FEATURE_SME) == CPU_FEATURE_SME) {
return "SME";
} else if ((f & CPU_FEATURE_SVE) == CPU_FEATURE_SVE) {
@@ -166,10 +156,10 @@ static size_t detect_num_smcus() {
}
}
}
return 0;
return 1;
#else
return 0;
return 1;
#endif
}
@@ -202,6 +192,7 @@ static void init_kleidiai_context(void) {
const char *env_threads = getenv("GGML_TOTAL_THREADS");
const char *env_chunk_mult = getenv("GGML_KLEIDIAI_CHUNK_MULTIPLIER");
const bool cpu_has_sme = ggml_cpu_has_sme();
size_t detected_smcus = 0;
ctx.features = (ggml_cpu_has_dotprod() ? CPU_FEATURE_DOTPROD : CPU_FEATURE_NONE) |
@@ -225,59 +216,56 @@ static void init_kleidiai_context(void) {
}
// SME policy:
// - env unset => auto-detect SMCUs; enable SME only if detected > 0.
// - env=0 => force off.
// - env>0 => force N cores, if the binary was built with SME.
// - If CPU doesn't support SME: SME always off.
// - Else:
// - env unset => auto-detect cores; enable if detected > 0.
// - env=0 => force off.
// - env>0 => force N cores (skip detection).
int sme_cores = 0;
bool sme_env_ok = false;
bool sme_env_set = (env_sme != nullptr);
if (sme_env_set) {
bool ok = false;
int v = parse_uint_env(env_sme, "GGML_KLEIDIAI_SME", &ok);
sme_env_ok = ok;
if (!cpu_has_sme) {
if (sme_env_set) {
bool ok = false;
int req = parse_uint_env(env_sme, "GGML_KLEIDIAI_SME", &ok);
if (ok && req > 0) {
GGML_LOG_WARN("kleidiai: GGML_KLEIDIAI_SME=%d but SME is not supported on this CPU; disabling SME\n", req);
}
}
sme_cores = 0;
} else {
if (sme_env_set) {
bool ok = false;
int v = parse_uint_env(env_sme, "GGML_KLEIDIAI_SME", &ok);
sme_env_ok = ok;
if (!ok) {
GGML_LOG_WARN("kleidiai: GGML_KLEIDIAI_SME set but parsing failed; falling back to runtime SME-core detection\n");
if (!ok) {
GGML_LOG_WARN("kleidiai: GGML_KLEIDIAI_SME set but parsing failed; falling back to runtime SME-core detection\n");
detected_smcus = detect_num_smcus();
sme_cores = detected_smcus > 0 ? (int)detected_smcus : 0;
} else if (v == 0) {
sme_cores = 0;
} else {
sme_cores = v;
}
} else {
detected_smcus = detect_num_smcus();
sme_cores = detected_smcus > 0 ? (int)detected_smcus : 0;
} else if (v == 0) {
sme_cores = 0;
} else if (!ggml_cpu_has_sme()) {
GGML_LOG_WARN("kleidiai: GGML_KLEIDIAI_SME=%d but the binary was not built with SME; disabling SME\n", v);
sme_cores = 0;
} else {
sme_cores = v;
}
} else {
detected_smcus = detect_num_smcus();
sme_cores = detected_smcus > 0 ? (int)detected_smcus : 0;
}
if (!sme_env_set && ggml_cpu_has_sme() && sme_cores == 0) {
GGML_LOG_WARN("kleidiai: runtime SME-core detection returned 0; falling back to NEON\n");
}
if (!sme_env_set && sme_cores == 0) {
GGML_LOG_WARN("kleidiai: SME supported but runtime SME-core detection returned 0; falling back to NEON\n");
}
if (sme_cores > 0) {
ctx.features |= CPU_FEATURE_SME;
#if defined(__aarch64__) && defined(__linux__)
// ARM guarantees SME2 implies SME, so only check SME2 when SME is enabled.
if (getauxval(AT_HWCAP2) & HWCAP2_SME2) {
ctx.features |= CPU_FEATURE_SME2;
if (sme_cores > 0) {
ctx.features |= CPU_FEATURE_SME;
}
#elif defined(__aarch64__) && defined(__APPLE__)
int feat_sme2 = 0;
size_t size = sizeof(feat_sme2);
if (sysctlbyname("hw.optional.arm.FEAT_SME2", &feat_sme2, &size, NULL, 0) == 0 && feat_sme2) {
ctx.features |= CPU_FEATURE_SME2;
}
#endif
}
// Kernel selection
ctx.kernels_q4 = ggml_kleidiai_select_kernels_q4_0(ctx.features);
ctx.kernels_q8 = ggml_kleidiai_select_kernels_q8_0(ctx.features);
ctx.kernels_f32 = ggml_kleidiai_select_kernels_f32(ctx.features);
ctx.kernels_q4 = ggml_kleidiai_select_kernels_q4_0(ctx.features);
ctx.kernels_q8 = ggml_kleidiai_select_kernels_q8_0(ctx.features);
if (!ctx.kernels_q4) {
GGML_LOG_INFO("kleidiai: no compatible q4 kernels found for CPU features mask %d\n", (int)ctx.features);
@@ -291,22 +279,13 @@ static void init_kleidiai_context(void) {
GGML_LOG_INFO("kleidiai: primary q8 kernel feature %s\n", cpu_feature_to_string(ctx.kernels_q8->required_cpu));
}
if (!ctx.kernels_f32) {
GGML_LOG_INFO("kleidiai: no compatible f32 kernels found for CPU features mask %d\n", (int)ctx.features);
} else {
GGML_LOG_INFO("kleidiai: primary f32 kernel feature %s\n", cpu_feature_to_string(ctx.kernels_f32->required_cpu));
}
ctx.sme_thread_cap = (ctx.features & CPU_FEATURE_SME) ? sme_cores : 0;
if (ctx.features & CPU_FEATURE_SME) {
const bool has_sme2 = (ctx.features & CPU_FEATURE_SME2) != CPU_FEATURE_NONE;
if (sme_env_set && sme_env_ok && sme_cores > 0) {
GGML_LOG_INFO("kleidiai: SME%s enabled (GGML_KLEIDIAI_SME=%d override)\n",
has_sme2 ? "2" : "", sme_cores);
GGML_LOG_INFO("kleidiai: SME enabled (GGML_KLEIDIAI_SME=%d override)\n", sme_cores);
} else {
GGML_LOG_INFO("kleidiai: SME%s enabled (runtime-detected SME cores=%d)\n",
has_sme2 ? "2" : "", sme_cores);
GGML_LOG_INFO("kleidiai: SME enabled (runtime-detected SME cores=%d)\n", sme_cores);
}
} else {
GGML_LOG_INFO("kleidiai: SME disabled\n");
@@ -355,13 +334,6 @@ static inline size_t ceil_div_size(size_t a, size_t b) {
return b == 0 ? 0 : (a + b - 1) / b;
}
static inline size_t kleidiai_chunk_cols(size_t n, int nth_total, bool disable_chunking, size_t n_step) {
const size_t multiplier = (nth_total == 1 || disable_chunking) ? 1 : std::max<size_t>(1, (size_t) ctx.chunk_multiplier);
const size_t divisor = std::max<size_t>(1, (size_t) nth_total * multiplier);
const size_t chunk_cols = align_up(std::max<size_t>(1, ceil_div_size(n, divisor)), n_step);
return chunk_cols ? chunk_cols : n_step;
}
struct kleidiai_block_args {
size_t lhs_bl;
size_t rhs_bl;
@@ -446,10 +418,6 @@ static inline ggml_kleidiai_kernels * kleidiai_primary_kernel_q8() {
return ctx.kernels_q8;
}
static inline ggml_kleidiai_kernels * kleidiai_primary_kernel_f32() {
return ctx.kernels_f32;
}
template <typename SelectFallback>
static int kleidiai_collect_kernel_chain_common(
ggml_kleidiai_kernels * primary,
@@ -462,16 +430,11 @@ static int kleidiai_collect_kernel_chain_common(
}
out[count++] = primary;
if (primary->rhs_info.repack_mode == RHS_REPACK_SINGLE_ONLY) {
return count;
}
if (is_sme_family(primary->required_cpu)) {
const cpu_feature fallback_mask = static_cast<cpu_feature>(features & ~CPU_FEATURE_SME & ~CPU_FEATURE_SME2);
if ((primary->required_cpu & CPU_FEATURE_SME) == CPU_FEATURE_SME) {
const cpu_feature fallback_mask = static_cast<cpu_feature>(features & ~CPU_FEATURE_SME);
if (fallback_mask != CPU_FEATURE_NONE) {
ggml_kleidiai_kernels * fallback = select_fallback(fallback_mask);
if (fallback && fallback != primary &&
fallback->rhs_info.repack_mode != RHS_REPACK_SINGLE_ONLY &&
fallback->lhs_type == primary->lhs_type &&
fallback->rhs_type == primary->rhs_type &&
fallback->op_type == primary->op_type) {
@@ -502,12 +465,6 @@ static int kleidiai_collect_q8_chain(std::array<ggml_kleidiai_kernels *, GGML_KL
[&](cpu_feature mask) { return ggml_kleidiai_select_kernels_q8_0(mask); });
}
static int kleidiai_collect_f32_chain(std::array<ggml_kleidiai_kernels *, GGML_KLEIDIAI_MAX_KERNEL_SLOTS> & out) {
ggml_kleidiai_kernels * primary = kleidiai_primary_kernel_f32();
return kleidiai_collect_kernel_chain_common(primary, ctx.features, out,
[&](cpu_feature mask) { return ggml_kleidiai_select_kernels_f32(mask); });
}
static inline int64_t ggml_ne(const ggml_tensor * tensor, int dim) {
GGML_ASSERT(dim >= 0 && dim < GGML_MAX_DIMS);
return tensor->ne[dim];
@@ -582,36 +539,6 @@ class tensor_traits : public ggml::cpu::tensor_traits {
return true;
}
if (op->src[0]->type == GGML_TYPE_F32) {
size_t cursor = 0;
bool any_slot = false;
for (int slot = 0; slot < slot_count; ++slot) {
ggml_kleidiai_kernels * kernels = kernel_chain[slot];
lhs_packing_info * lhs_info = &kernels->gemm_lhs_info;
kernel_info * kernel = &kernels->gemm;
if (!lhs_info || !lhs_info->packed_size_ex || !kernel) {
return false;
}
const size_t mr = kernel->get_mr();
const size_t kr = kernel->get_kr();
const size_t sr = kernel->get_sr();
cursor = align_up(cursor, GGML_KLEIDIAI_PACK_ALIGN);
cursor += lhs_info->packed_size_ex(m, k, 0, mr, kr, sr);
any_slot = true;
}
if (!any_slot) {
return false;
}
size = cursor;
return true;
}
if (op->src[0]->type == GGML_TYPE_F16) {
const int64_t lhs_batch_size0 = op->src[1]->ne[2];
const int64_t rhs_batch_size0 = op->src[0]->ne[2];
@@ -668,8 +595,6 @@ class tensor_traits : public ggml::cpu::tensor_traits {
if (dst->op == GGML_OP_MUL_MAT) {
if (dst->src[0]->type == GGML_TYPE_Q4_0 || dst->src[0]->type == GGML_TYPE_Q8_0) {
return compute_forward_qx(params, dst);
} else if (dst->src[0]->type == GGML_TYPE_F32) {
return compute_forward_f32(params, dst);
} else if (dst->src[0]->type == GGML_TYPE_F16) {
return compute_forward_fp16(params, dst);
}
@@ -681,144 +606,6 @@ class tensor_traits : public ggml::cpu::tensor_traits {
return false;
}
bool compute_forward_f32(ggml_compute_params * params, struct ggml_tensor * dst) {
GGML_ASSERT(dst->src[0]->type == GGML_TYPE_F32);
const ggml_tensor * src0 = dst->src[0];
const ggml_tensor * src1 = dst->src[1];
GGML_TENSOR_BINARY_OP_LOCALS
if (src1->type != GGML_TYPE_F32 || dst->type != GGML_TYPE_F32) {
return false;
}
ggml_kleidiai_kernels * kernels = kleidiai_primary_kernel_f32();
if (!kernels) {
return false;
}
kernel_info * kernel = &kernels->gemm;
lhs_packing_info * lhs_info = &kernels->gemm_lhs_info;
if (!kernel || !lhs_info || !lhs_info->get_offset || !lhs_info->get_packed_offset_ex ||
!lhs_info->packed_size_ex || !lhs_info->pack_func_ex ||
!kernel->get_rhs_packed_offset_ex || !kernel->run_kernel_ex || !kernel->get_dst_offset) {
return false;
}
const kleidiai_weight_header * header = kleidiai_weight_header_from_ptr(src0->data);
const bool has_header = kleidiai_is_weight_header_valid(header);
const uint8_t * rhs_base = has_header ? kleidiai_weight_slot_ptr(header, 0)
: static_cast<const uint8_t *>(src0->data);
if (!rhs_base) {
return false;
}
const int nth = params->nth > 0 ? params->nth : 1;
const int ith = params->ith;
const size_t k = ne00;
const size_t m = ne11;
const size_t n = ne01;
const size_t mr = kernel->get_mr();
const size_t kr = kernel->get_kr();
const size_t sr = kernel->get_sr();
const size_t lhs_packed_size = lhs_info->packed_size_ex(m, k, 0, mr, kr, sr);
GGML_ASSERT(lhs_packed_size <= params->wsize);
uint8_t * lhs_packed = static_cast<uint8_t *>(params->wdata);
const size_t dst_stride = dst->nb[1];
const size_t n_step = kernel->get_n_step() ? kernel->get_n_step() : 1;
const bool disable_chunking = ggml_is_numa();
GGML_ASSERT(n <= (size_t) INT_MAX);
for (int64_t batch_idx = 0; batch_idx < ne12; ++batch_idx) {
const uint8_t * lhs_batch_base = static_cast<const uint8_t *>(src1->data) + batch_idx * src1->nb[2];
uint8_t * dst_batch_base = static_cast<uint8_t *>(dst->data) + batch_idx * dst->nb[2];
{
const int64_t m_roundup_mr = kai_roundup((int64_t)m, (int64_t)mr);
int64_t max_threads = mr ? (m_roundup_mr / (int64_t)mr) : nth;
max_threads = std::max<int64_t>(1, max_threads);
const int64_t use_threads = std::min<int64_t>(nth, max_threads);
if (ith < use_threads) {
const int64_t num_m_per_thread0 = round_down((size_t)(m_roundup_mr / use_threads), mr);
const int64_t num_m_per_threadN_1 = (int64_t)m - (use_threads - 1) * num_m_per_thread0;
const int64_t m_start = (int64_t)ith * num_m_per_thread0;
const int64_t m_count = (ith == use_threads - 1) ? num_m_per_threadN_1 : num_m_per_thread0;
const size_t base_packed_off = lhs_info->get_packed_offset_ex(m_start, k, 0, mr, kr, sr);
const size_t next_block_off = lhs_info->get_packed_offset_ex(m_start + mr, k, 0, mr, kr, sr);
const size_t row_stride_bytes = mr ? (next_block_off - base_packed_off) / mr : 0;
int64_t remaining = m_count;
int64_t cur = m_start;
while (remaining > 0) {
const int64_t take = std::min<int64_t>((int64_t)m - cur, remaining);
const size_t src_off = lhs_info->get_offset(cur, src1->nb[1]);
const void * src_ptr = lhs_batch_base + src_off;
const size_t dst_off = base_packed_off + (size_t)(cur - m_start) * row_stride_bytes;
void * dst_ptr = lhs_packed + dst_off;
lhs_info->pack_func_ex(take, k, 0, mr, kr, sr, 0, src_ptr, src1->nb[1], dst_ptr);
cur += take;
remaining -= take;
}
}
}
if (ith == 0) {
ggml_threadpool_chunk_set(params->threadpool, 0);
}
ggml_barrier(params->threadpool);
const size_t chunk_cols = kleidiai_chunk_cols(n, nth, disable_chunking, n_step);
GGML_ASSERT(chunk_cols <= (size_t) INT_MAX);
int current_col = ggml_threadpool_chunk_add(params->threadpool, (int) chunk_cols);
while ((size_t) current_col < n) {
const size_t n_start = (size_t) current_col;
const size_t n_to_process = std::min(chunk_cols, n - n_start);
if (n_to_process > 0) {
const size_t lhs_packed_offset = lhs_info->get_packed_offset_ex(0, k, 0, mr, kr, sr);
const size_t rhs_packed_offset = kernel->get_rhs_packed_offset_ex(n_start, k, 0);
const size_t dst_offset = kernel->get_dst_offset(0, n_start, dst_stride);
const void * lhs_ptr = lhs_packed + lhs_packed_offset;
const void * rhs_ptr = rhs_base + rhs_packed_offset;
float * dst_ptr = reinterpret_cast<float *>(dst_batch_base + dst_offset);
kernel->run_kernel_ex(m, n_to_process, k, 0,
lhs_ptr,
rhs_ptr,
dst_ptr,
dst_stride,
sizeof(float),
-FLT_MAX,
FLT_MAX);
}
current_col = ggml_threadpool_chunk_add(params->threadpool, (int) chunk_cols);
}
if (batch_idx != ne12 - 1) {
ggml_barrier(params->threadpool);
}
}
return true;
}
bool compute_forward_fp16(ggml_compute_params * params, struct ggml_tensor * dst) {
const ggml_tensor * src0 = dst->src[0];
const ggml_tensor * src1 = dst->src[1];
@@ -1078,14 +865,14 @@ class tensor_traits : public ggml::cpu::tensor_traits {
int sme_slot = -1;
for (int i = 0; i < runtime_count; ++i) {
if (is_sme_family(runtime[i].kernels->required_cpu)) {
if ((runtime[i].kernels->required_cpu & CPU_FEATURE_SME) == CPU_FEATURE_SME) {
sme_slot = i;
break;
}
}
int non_sme_slot = -1;
for (int i = 0; i < runtime_count; ++i) {
if (!is_sme_family(runtime[i].kernels->required_cpu)) {
if ((runtime[i].kernels->required_cpu & CPU_FEATURE_SME) != CPU_FEATURE_SME) {
non_sme_slot = i;
break;
}
@@ -1123,7 +910,7 @@ class tensor_traits : public ggml::cpu::tensor_traits {
// Recompute SME slot based on the collapsed runtime[0]
sme_slot = -1;
if (runtime_count > 0 &&
is_sme_family(runtime[0].kernels->required_cpu)) {
(runtime[0].kernels->required_cpu & CPU_FEATURE_SME) == CPU_FEATURE_SME) {
sme_slot = 0;
}
}
@@ -1427,7 +1214,7 @@ class tensor_traits : public ggml::cpu::tensor_traits {
public:
int repack(struct ggml_tensor * tensor, const void * data, size_t data_size) {
GGML_ASSERT(tensor->type == GGML_TYPE_Q4_0 || tensor->type == GGML_TYPE_Q8_0 || tensor->type == GGML_TYPE_F32);
GGML_ASSERT(tensor->type == GGML_TYPE_Q4_0 || tensor->type == GGML_TYPE_Q8_0);
const size_t n = tensor->ne[1];
const size_t k = tensor->ne[0];
@@ -1446,15 +1233,12 @@ public:
std::array<ggml_kleidiai_kernels *, GGML_KLEIDIAI_MAX_KERNEL_SLOTS> kernel_chain;
const bool want_q8 = tensor->type == GGML_TYPE_Q8_0;
const bool want_f32 = tensor->type == GGML_TYPE_F32;
const int slot_total = want_f32 ? kleidiai_collect_f32_chain(kernel_chain)
: want_q8 ? kleidiai_collect_q8_chain(kernel_chain)
: kleidiai_collect_q4_chain(kernel_chain);
const int slot_total = want_q8 ? kleidiai_collect_q8_chain(kernel_chain)
: kleidiai_collect_q4_chain(kernel_chain);
const bool allow_fallback = kleidiai_pack_fallback_allowed();
std::vector<int8_t> qdata;
std::vector<float> scales;
std::vector<float> bias;
if (want_q8 && slot_total > 0) {
qdata.resize(n * k, 0);
@@ -1502,10 +1286,6 @@ public:
}
}
if (want_f32 && slot_total > 0) {
bias.resize(n, 0.0f);
}
for (int slot = 0; slot < slot_total && slot < GGML_KLEIDIAI_MAX_KERNEL_SLOTS; ++slot) {
if (!allow_fallback && slot > 0) {
break;
@@ -1522,9 +1302,8 @@ public:
const size_t sr = kernel->get_sr();
const ggml_type rhs_type = kernels->rhs_type;
const size_t block_len = rhs_type == GGML_TYPE_Q8_0 ? QK8_0 :
rhs_type == GGML_TYPE_Q4_0 ? QK4_0 :
rhs_type == GGML_TYPE_F32 ? 0 : SIZE_MAX;
if (block_len == SIZE_MAX) {
rhs_type == GGML_TYPE_Q4_0 ? QK4_0 : 0;
if (block_len == 0) {
continue;
}
@@ -1547,10 +1326,6 @@ public:
rhs_info->pack_func_ex(1, n, k, nr, kr, sr, 0, 0,
qdata.data(), nullptr, scales.data(),
dst_ptr, 0, &params);
} else if (rhs_type == GGML_TYPE_F32) {
rhs_info->pack_func_ex(1, n, k, nr, kr, sr, 0, tensor->nb[1],
data, bias.data(), nullptr,
dst_ptr, 0, nullptr);
} else {
continue;
}
@@ -1625,7 +1400,7 @@ static size_t ggml_backend_cpu_kleidiai_buffer_type_get_alignment(ggml_backend_b
static size_t ggml_backend_cpu_kleidiai_buffer_type_get_alloc_size(ggml_backend_buffer_type_t buft, const struct ggml_tensor * tensor) {
GGML_UNUSED(buft);
if (tensor->type != GGML_TYPE_Q4_0 && tensor->type != GGML_TYPE_Q8_0 && tensor->type != GGML_TYPE_F32) {
if (tensor->type != GGML_TYPE_Q4_0 && tensor->type != GGML_TYPE_Q8_0) {
return ggml_nbytes(tensor);
}
@@ -1637,10 +1412,8 @@ static size_t ggml_backend_cpu_kleidiai_buffer_type_get_alloc_size(ggml_backend_
std::array<ggml_kleidiai_kernels *, GGML_KLEIDIAI_MAX_KERNEL_SLOTS> kernel_chain;
const bool want_q8 = tensor->type == GGML_TYPE_Q8_0;
const bool want_f32 = tensor->type == GGML_TYPE_F32;
const int slot_total = want_f32 ? kleidiai_collect_f32_chain(kernel_chain)
: want_q8 ? kleidiai_collect_q8_chain(kernel_chain)
: kleidiai_collect_q4_chain(kernel_chain);
const int slot_total = want_q8 ? kleidiai_collect_q8_chain(kernel_chain)
: kleidiai_collect_q4_chain(kernel_chain);
const bool allow_fallback = kleidiai_pack_fallback_allowed();
size_t slot_count = 0;
@@ -1660,9 +1433,8 @@ static size_t ggml_backend_cpu_kleidiai_buffer_type_get_alloc_size(ggml_backend_
const ggml_type rhs_type = kernels->rhs_type;
const size_t block_len = rhs_type == GGML_TYPE_Q4_0 ? QK4_0 :
rhs_type == GGML_TYPE_Q8_0 ? QK8_0 :
rhs_type == GGML_TYPE_F32 ? 0 : SIZE_MAX;
if (block_len == SIZE_MAX) {
rhs_type == GGML_TYPE_Q8_0 ? QK8_0 : 0;
if (block_len == 0) {
continue;
}
@@ -1683,43 +1455,26 @@ class extra_buffer_type : ggml::cpu::extra_buffer_type {
bool supports_op(ggml_backend_dev_t, const struct ggml_tensor * op) override {
std::array<ggml_kleidiai_kernels *, GGML_KLEIDIAI_MAX_KERNEL_SLOTS> kernel_chain;
const int slot_total = kleidiai_collect_kernel_chain(op, kernel_chain);
const bool src0_is_kleidiai =
if ((op->op == GGML_OP_MUL_MAT || op->op == GGML_OP_GET_ROWS) &&
(op->src[0]->type == GGML_TYPE_Q4_0 || op->src[0]->type == GGML_TYPE_Q8_0) &&
op->src[0]->buffer &&
(ggml_n_dims(op->src[0]) == 2) &&
op->src[0]->buffer->buft == ggml_backend_cpu_kleidiai_buffer_type() &&
slot_total > 0;
if ((op->op == GGML_OP_MUL_MAT || op->op == GGML_OP_GET_ROWS) &&
(op->src[0]->type == GGML_TYPE_Q4_0 || op->src[0]->type == GGML_TYPE_Q8_0 || op->src[0]->type == GGML_TYPE_F32) &&
src0_is_kleidiai) {
slot_total > 0) {
if (op->src[0]->type == GGML_TYPE_Q4_0 && ctx.kernels_q4 == nullptr) {
return false;
}
if (op->src[0]->type == GGML_TYPE_Q8_0 && ctx.kernels_q8 == nullptr) {
return false;
}
if (op->src[0]->type == GGML_TYPE_F32 && ctx.kernels_f32 == nullptr) {
return false;
}
if (op->src[1]->buffer && !ggml_backend_buft_is_host(op->src[1]->buffer->buft)) {
return false;
}
if (op->src[0]->type == GGML_TYPE_Q4_0 || op->src[0]->type == GGML_TYPE_Q8_0) {
if ((op->src[1]->type == GGML_TYPE_F32 || op->src[1]->type == GGML_TYPE_I32) &&
ggml_ne(op->src[1], 3) == 1) {
return true;
}
return false;
if ((op->src[1]->type == GGML_TYPE_F32 || op->src[1]->type == GGML_TYPE_I32) &&
ggml_ne(op->src[1], 3) == 1) {
return true;
}
if (op->op != GGML_OP_MUL_MAT || op->src[1]->type != GGML_TYPE_F32 || op->type != GGML_TYPE_F32) {
return false;
}
return true;
}
return false;
}
@@ -1728,20 +1483,6 @@ class extra_buffer_type : ggml::cpu::extra_buffer_type {
if (op->src[0]->buffer && op->src[0]->buffer->buft == ggml_backend_cpu_kleidiai_buffer_type()) {
return (ggml::cpu::tensor_traits *) op->src[0]->extra;
} else {
// KleidiAI only has kernels for Q4_0 and Q8_0. For a quantized weight of any
// other type (K-quants, IQ) it declines the op and returns nullptr below, so
// KleidiAI does not accelerate it. Another CPU backend may still take the op,
// and this can run during graph planning, so the message says what KleidiAI
// did rather than what ends up executing. Warn once per process.
if (ggml_is_quantized(op->src[0]->type) &&
op->src[0]->type != GGML_TYPE_Q4_0 && op->src[0]->type != GGML_TYPE_Q8_0) {
static std::atomic<bool> warned(false);
if (!warned.exchange(true)) {
GGML_LOG_WARN("kleidiai: no kernel for tensor type %s, not accelerated by KleidiAI "
"(kernels available for Q4_0 and Q8_0)\n",
ggml_type_name(op->src[0]->type));
}
}
if (op->src[0]->type != GGML_TYPE_F16) {
return nullptr;
}
ggml/src/ggml-cpu/llamafile/sgemm.cpp
+14 -20
View File
@@ -2321,28 +2321,24 @@ class tinyBLAS_Q0_PPC {
}
void matmul(int64_t m, int64_t n) {
int64_t mc = 64;
int64_t nc = 64;
int64_t kc = 64;
int64_t n_chunk = 64;
#if defined(_AIX) || defined(__BIG_ENDIAN__)
mc = 32;
nc = 32;
kc = 32;
n_chunk = 32;
#endif
mnpack(0, m, 0, n);
#else
const int64_t mc = 64;
const int64_t kc = 64;
int64_t nc = 64;
int64_t n_aligned = 0;
if (n % n_chunk == 0) {
if (n % 64 == 0) {
n_aligned = n;
} else if (n == 4) {
n_aligned = 4;
} else if (n < n_chunk) {
} else if (n < 64) {
n_aligned = (n / 8) * 8;
} else {
n_aligned = (n / n_chunk) * n_chunk;
n_aligned = (n / 64) * 64;
}
if (n_aligned > 0) {
if (n_aligned % n_chunk == 0) nc = n_chunk;
if (n_aligned % 64 == 0) nc = 64;
else if (n_aligned == n) nc = n;
else if (n_aligned % 32 == 0) nc = 32;
else if (n_aligned % 24 == 0) nc = 24;
@@ -2358,6 +2354,7 @@ class tinyBLAS_Q0_PPC {
} else {
mnpack(0, m, 0, n);
}
#endif
}
private:
@@ -3198,19 +3195,16 @@ class tinyBLAS_PPC {
}
void matmul(int64_t m, int64_t n) {
int64_t mc = 256;
int64_t nc = 256;
int64_t kc = 256;
#if defined(_AIX) || defined(__BIG_ENDIAN__)
mc = 128;
nc = 128;
kc = 128;
#endif
mnpack(0, m, 0, n);
#else
int64_t mc = 256; int64_t nc = 256; int64_t kc = 256;
if (m % mc == 0 && n % nc == 0 && k % kc == 0) {
matmul_tiled(m, n, mc, nc, kc);
} else {
mnpack(0, m, 0, n);
}
#endif
}
private:
ggml/src/ggml-cpu/ops.cpp
+30 -520
View File
@@ -665,7 +665,6 @@ void ggml_compute_forward_add(
ggml_compute_forward_add_non_quantized(params, dst);
} break;
case GGML_TYPE_Q1_0:
case GGML_TYPE_Q2_0:
case GGML_TYPE_Q4_0:
case GGML_TYPE_Q4_1:
case GGML_TYPE_Q5_0:
@@ -1116,7 +1115,6 @@ void ggml_compute_forward_add1(
}
} break;
case GGML_TYPE_Q1_0:
case GGML_TYPE_Q2_0:
case GGML_TYPE_Q4_0:
case GGML_TYPE_Q4_1:
case GGML_TYPE_Q5_0:
@@ -1247,7 +1245,6 @@ void ggml_compute_forward_acc(
case GGML_TYPE_F16:
case GGML_TYPE_BF16:
case GGML_TYPE_Q1_0:
case GGML_TYPE_Q2_0:
case GGML_TYPE_Q4_0:
case GGML_TYPE_Q4_1:
case GGML_TYPE_Q5_0:
@@ -1916,11 +1913,7 @@ static void ggml_compute_forward_concat_any(
GGML_ASSERT(dim >= 0 && dim < 4);
int64_t o[4] = {0, 0, 0, 0};
if (dim == 0) {
o[dim] = src0->ne[dim]/ggml_blck_size(src0->type);
} else {
o[dim] = src0->ne[dim];
}
o[dim] = src0->ne[dim];
const char * x;
@@ -1928,8 +1921,8 @@ static void ggml_compute_forward_concat_any(
for (int i3 = 0; i3 < ne3; i3++) {
for (int i2 = ith; i2 < ne2; i2 += nth) {
for (int i1 = 0; i1 < ne1; i1++) {
for (int i0 = 0; i0 < ne0/ggml_blck_size(dst->type); i0++) {
if (i0 < ne00/ggml_blck_size(src0->type) && i1 < ne01 && i2 < ne02 && i3 < ne03) {
for (int i0 = 0; i0 < ne0; i0++) {
if (i0 < ne00 && i1 < ne01 && i2 < ne02 && i3 < ne03) {
x = (const char *)src0->data + (i0 )*nb00 + (i1 )*nb01 + (i2 )*nb02 + (i3 )*nb03;
} else {
x = (const char *)src1->data + (i0 - o[0])*nb10 + (i1 - o[1])*nb11 + (i2 - o[2])*nb12 + (i3 - o[3])*nb13;
@@ -2078,14 +2071,6 @@ void ggml_compute_forward_concat(
ggml_tensor * dst) {
const ggml_tensor * src0 = dst->src[0];
const ggml_tensor * src1 = dst->src[1];
if (ggml_is_quantized(src0->type)) {
GGML_ASSERT(ggml_is_contiguous_rows(src0));
GGML_ASSERT(ggml_is_contiguous_rows(src1));
GGML_ASSERT(src0->ne[0] % ggml_blck_size(src0->type) == 0);
GGML_ASSERT(src1->ne[0] % ggml_blck_size(src1->type) == 0);
}
switch (src0->type) {
case GGML_TYPE_F16:
@@ -4449,70 +4434,6 @@ static void ggml_compute_forward_out_prod_q_f32(
}
}
static void ggml_compute_forward_out_prod_f16_f32(
const ggml_compute_params * params,
ggml_tensor * dst) {
const ggml_tensor * src0 = dst->src[0];
const ggml_tensor * src1 = dst->src[1];
GGML_TENSOR_BINARY_OP_LOCALS;
const int ith = params->ith;
const int nth = params->nth;
GGML_ASSERT(src0->type == GGML_TYPE_F16);
GGML_ASSERT(src1->type == GGML_TYPE_F32);
GGML_ASSERT(dst->type == GGML_TYPE_F32);
GGML_ASSERT(ne02 == ne12);
GGML_ASSERT(ne03 == ne13);
GGML_ASSERT(ne2 == ne12);
GGML_ASSERT(ne3 == ne13);
GGML_ASSERT(nb00 == sizeof(ggml_fp16_t));
GGML_ASSERT(nb0 == sizeof(float));
GGML_ASSERT(ne0 == ne00);
GGML_ASSERT(ne1 == ne10);
GGML_ASSERT(ne2 == ne02);
GGML_ASSERT(ne3 == ne03);
if (ith == 0) {
ggml_vec_set_f32(ne0*ne1*ne2*ne3, (float *)dst->data, 0);
}
ggml_barrier(params->threadpool);
const int64_t nr = ne1*ne2*ne3;
const int64_t dr = (nr + nth - 1)/nth;
const int64_t ir0 = dr*ith;
const int64_t ir1 = MIN(ir0 + dr, nr);
float * wdata = (float *) params->wdata + (ne0 + CACHE_LINE_SIZE_F32) * ith;
for (int64_t ir = ir0; ir < ir1; ++ir) {
const int64_t i3 = ir/(ne2*ne1);
const int64_t i2 = (ir - i3*ne2*ne1)/ne1;
const int64_t i1 = (ir - i3*ne2*ne1 - i2*ne1);
const int64_t i02 = i2;
const int64_t i03 = i3;
const int64_t i12 = i2;
const int64_t i13 = i3;
float * d = (float *) ((char *) dst->data + (i1*nb1 + i2*nb2 + i3*nb3));
for (int64_t i01 = 0; i01 < ne01; ++i01) {
const int64_t i11 = i01;
ggml_fp16_t * s0 = (ggml_fp16_t *) ((char *) src0->data + (i01*nb01 + i02*nb02 + i03*nb03));
float * s1 = (float *) ((char *) src1->data + (i1*nb10 + i11*nb11 + i12*nb12 + i13*nb13));
ggml_fp16_to_fp32_row(s0, wdata, ne0);
ggml_vec_mad_f32(ne0, d, wdata, *s1);
}
}
}
void ggml_compute_forward_out_prod(
const ggml_compute_params * params,
ggml_tensor * dst) {
@@ -4521,7 +4442,6 @@ void ggml_compute_forward_out_prod(
switch (src0->type) {
case GGML_TYPE_Q1_0:
case GGML_TYPE_Q2_0:
case GGML_TYPE_Q4_0:
case GGML_TYPE_Q4_1:
case GGML_TYPE_Q5_0:
@@ -4550,8 +4470,9 @@ void ggml_compute_forward_out_prod(
} break;
case GGML_TYPE_F16:
{
ggml_compute_forward_out_prod_f16_f32(params, dst);
} break;
GGML_ABORT("fatal error"); // todo
// ggml_compute_forward_out_prod_f16_f32(params, dst);
}
case GGML_TYPE_F32:
{
ggml_compute_forward_out_prod_f32(params, dst);
@@ -4797,7 +4718,6 @@ void ggml_compute_forward_set(
case GGML_TYPE_F16:
case GGML_TYPE_BF16:
case GGML_TYPE_Q1_0:
case GGML_TYPE_Q2_0:
case GGML_TYPE_Q4_0:
case GGML_TYPE_Q4_1:
case GGML_TYPE_Q5_0:
@@ -5022,7 +4942,6 @@ void ggml_compute_forward_get_rows(
switch (src0->type) {
case GGML_TYPE_Q1_0:
case GGML_TYPE_Q2_0:
case GGML_TYPE_Q4_0:
case GGML_TYPE_Q4_1:
case GGML_TYPE_Q5_0:
@@ -5088,8 +5007,8 @@ void ggml_compute_forward_get_rows(
//}
}
template<typename src_t, typename idx_t>
static void ggml_compute_forward_set_rows_impl(
template<typename idx_t>
static void ggml_compute_forward_set_rows_f32(
const ggml_compute_params * params,
ggml_tensor * dst) {
@@ -5104,7 +5023,7 @@ static void ggml_compute_forward_set_rows_impl(
assert(ne0 == nc);
assert(ne2 == ne02);
assert(ne3 == ne03);
GGML_ASSERT(src0->type == GGML_TYPE_F32 || src0->type == GGML_TYPE_F16);
assert(src0->type == GGML_TYPE_F32);
assert(ne02 % ne11 == 0);
assert(ne03 % ne12 == 0);
@@ -5118,8 +5037,6 @@ static void ggml_compute_forward_set_rows_impl(
const int64_t ir0 = dr*ith;
const int64_t ir1 = std::min(ir0 + dr, nr);
const size_t rs = ggml_row_size(src0->type, nc);
ggml_from_float_t const from_float = ggml_get_type_traits_cpu(dst->type)->from_float;
for (int64_t i03 = 0; i03 < ne03; ++i03) {
@@ -5133,27 +5050,9 @@ static void ggml_compute_forward_set_rows_impl(
GGML_ASSERT(i1 >= 0 && i1 < ne1);
if constexpr (std::is_same_v<src_t, float>) {
from_float(
(const float *) ((char *) src0->data + i*nb01 + i02*nb02 + i03*nb03),
((char *) dst->data + i1*nb1 + i02*nb2 + i03*nb3), nc);
} else if constexpr (std::is_same_v<src_t, ggml_fp16_t>) {
if (dst->type == GGML_TYPE_F16) {
memcpy(
((char *) dst->data + i1*nb1 + i02*nb2 + i03*nb3),
((char *) src0->data + i*nb01 + i02*nb02 + i03*nb03),
rs);
} else {
float * wdata = (float *) params->wdata + (nc + CACHE_LINE_SIZE_F32) * ith;
ggml_fp16_to_fp32_row(
(const ggml_fp16_t *) ((char *) src0->data + i*nb01 + i02*nb02 + i03*nb03),
wdata, nc);
from_float(wdata,
((char *) dst->data + i1*nb1 + i02*nb2 + i03*nb3), nc);
}
} else {
GGML_ABORT("src0->type = %d (%s) not supported", src0->type, ggml_type_name(src0->type));
}
from_float(
(const float *) ((char *) src0->data + i*nb01 + i02*nb02 + i03*nb03),
((char *) dst->data + i1*nb1 + i02*nb2 + i03*nb3), nc);
}
}
}
@@ -5170,19 +5069,9 @@ void ggml_compute_forward_set_rows(
case GGML_TYPE_F32:
{
if (src1->type == GGML_TYPE_I64) {
ggml_compute_forward_set_rows_impl<float, int64_t>(params, dst);
ggml_compute_forward_set_rows_f32<int64_t>(params, dst);
} else if (src1->type == GGML_TYPE_I32) {
ggml_compute_forward_set_rows_impl<float, int32_t>(params, dst);
} else {
GGML_ABORT("src1->type = %d (%s) not supported", src1->type, ggml_type_name(src1->type));
}
} break;
case GGML_TYPE_F16:
{
if (src1->type == GGML_TYPE_I64) {
ggml_compute_forward_set_rows_impl<ggml_fp16_t, int64_t>(params, dst);
} else if (src1->type == GGML_TYPE_I32) {
ggml_compute_forward_set_rows_impl<ggml_fp16_t, int32_t>(params, dst);
ggml_compute_forward_set_rows_f32<int32_t>(params, dst);
} else {
GGML_ABORT("src1->type = %d (%s) not supported", src1->type, ggml_type_name(src1->type));
}
@@ -5779,7 +5668,6 @@ void ggml_compute_forward_clamp(
} break;
case GGML_TYPE_BF16:
case GGML_TYPE_Q1_0:
case GGML_TYPE_Q2_0:
case GGML_TYPE_Q4_0:
case GGML_TYPE_Q4_1:
case GGML_TYPE_Q5_0:
@@ -6430,6 +6318,7 @@ static void ggml_compute_forward_im2col_f16(
const ggml_tensor * src0 = dst->src[0];
const ggml_tensor * src1 = dst->src[1];
GGML_ASSERT(src0->type == GGML_TYPE_F16);
GGML_ASSERT(src1->type == GGML_TYPE_F16 || src1->type == GGML_TYPE_F32);
GGML_ASSERT( dst->type == GGML_TYPE_F16);
@@ -6460,6 +6349,7 @@ static void ggml_compute_forward_im2col_f16(
int ofs0 = is_2D ? nb13 : nb12;
int ofs1 = is_2D ? nb12 : nb11;
GGML_ASSERT(nb00 == sizeof(ggml_fp16_t));
GGML_ASSERT(nb10 == ggml_type_size(src1->type));
// im2col: [N, IC, IH, IW] => [N, OH, OW, IC*KH*KW]
@@ -6532,7 +6422,7 @@ void ggml_compute_forward_im2col_back_f32(
const ggml_tensor * src1 = dst->src[1]; // convolution kernel
GGML_ASSERT(src0->type == GGML_TYPE_F32);
GGML_ASSERT(src1->type == GGML_TYPE_F32 || src1->type == GGML_TYPE_F16);
GGML_ASSERT(src1->type == GGML_TYPE_F32);
GGML_ASSERT( dst->type == GGML_TYPE_F32);
GGML_TENSOR_BINARY_OP_LOCALS;
@@ -6629,6 +6519,7 @@ static void ggml_compute_forward_im2col_3d_f16(
const ggml_tensor * src0 = dst->src[0];
const ggml_tensor * src1 = dst->src[1];
GGML_ASSERT(src0->type == GGML_TYPE_F16);
GGML_ASSERT(src1->type == GGML_TYPE_F32);
GGML_ASSERT( dst->type == GGML_TYPE_F16);
@@ -7364,13 +7255,6 @@ struct ggml_conv_2d_dw_params {
int dilation_y;
};
static inline float ggml_conv_2d_dw_knl_f32(const char * data, int64_t i, ggml_type type) {
if (type == GGML_TYPE_F16) {
return GGML_FP16_TO_FP32(((const ggml_fp16_t *)data)[i]);
}
return ((const float *)data)[i];
}
static void ggml_compute_forward_conv_2d_dw_cwhn(
const ggml_compute_params * params,
const ggml_tensor * src,
@@ -7379,8 +7263,7 @@ static void ggml_compute_forward_conv_2d_dw_cwhn(
const ggml_conv_2d_dw_params & p) {
const int64_t c = p.channels;
const char * knl_data = (const char *)kernel->data;
const ggml_type knl_type = kernel->type;
const float * knl_data = (const float *)kernel->data;
const int64_t rows_total = p.dst_h * p.batch;
const int64_t rows_per_thread = (rows_total + params->nth - 1) / params->nth;
@@ -7388,16 +7271,13 @@ static void ggml_compute_forward_conv_2d_dw_cwhn(
const int64_t row_end = MIN(row_start + rows_per_thread, rows_total);
#ifdef GGML_SIMD
int64_t c_pkg_end = 0;
int64_t pkg_size = GGML_F32_EPR;
if (knl_type == GGML_TYPE_F32) {
#if defined(__ARM_FEATURE_SVE)
pkg_size = svcntw();
const int64_t pkg_size = svcntw();
#else
pkg_size = GGML_F32_EPR;
const int64_t pkg_size = GGML_F32_EPR;
#endif
c_pkg_end = (c / pkg_size) * pkg_size;
}
const int64_t pkg_count = c / pkg_size;
const int64_t c_pkg_end = pkg_count * pkg_size;
#else
const int64_t c_pkg_end = 0;
#endif
@@ -7411,6 +7291,7 @@ static void ggml_compute_forward_conv_2d_dw_cwhn(
const int64_t src_x_base = dst_x * p.stride_x - p.pad_x;
#ifdef GGML_SIMD
// Vectorized loop
for (int64_t c_i = 0; c_i < c_pkg_end; c_i += pkg_size) {
GGML_F32_VEC sum = GGML_F32_VEC_ZERO;
for (int64_t knl_y = 0; knl_y < p.knl_h; ++knl_y) {
@@ -7423,8 +7304,7 @@ static void ggml_compute_forward_conv_2d_dw_cwhn(
if (src_x < 0 || src_x >= p.src_w) {
continue;
}
const float * kp = (const float *)knl_data + (knl_y * p.knl_w + knl_x) * c + c_i;
GGML_F32_VEC k = GGML_F32_VEC_LOAD(kp);
GGML_F32_VEC k = GGML_F32_VEC_LOAD(knl_data + (knl_y * p.knl_w + knl_x) * c + c_i);
GGML_F32_VEC s = GGML_F32_VEC_LOAD(src_data + (src_y * p.src_w + src_x) * c + c_i);
sum = GGML_F32_VEC_FMA(sum, k, s);
}
@@ -7432,6 +7312,7 @@ static void ggml_compute_forward_conv_2d_dw_cwhn(
GGML_F32_VEC_STORE(dst_data + c_i, sum);
}
#endif
// Scalar loop
for (int64_t c_i = c_pkg_end; c_i < c; ++c_i) {
float sum = 0.0f;
for (int64_t knl_y = 0; knl_y < p.knl_h; ++knl_y) {
@@ -7444,7 +7325,7 @@ static void ggml_compute_forward_conv_2d_dw_cwhn(
if (src_x < 0 || src_x >= p.src_w) {
continue;
}
sum += ggml_conv_2d_dw_knl_f32(knl_data, (knl_y * p.knl_w + knl_x) * c + c_i, knl_type)
sum += knl_data[(knl_y * p.knl_w + knl_x) * c + c_i]
* src_data[(src_y * p.src_w + src_x) * c + c_i];
}
}
@@ -7465,11 +7346,9 @@ static void ggml_compute_forward_conv_2d_dw_whcn(
const int64_t per_thread = (n + params->nth - 1) / params->nth;
const int64_t start = params->ith * per_thread;
const int64_t end = MIN(start + per_thread, n);
const char * knl_base = (const char *)kernel->data;
const ggml_type knl_type = kernel->type;
for (int64_t i = start; i < end; ++i) {
const int64_t knl_offset = (i % p.channels) * p.knl_w * p.knl_h;
const float * knl_data = (const float *)kernel->data + (i % p.channels) * p.knl_w * p.knl_h;
const float * src_data = (const float *)src->data + i * p.src_w * p.src_h;
float * dst_data = (float *)dst->data + i * p.dst_w * p.dst_h;
@@ -7487,7 +7366,7 @@ static void ggml_compute_forward_conv_2d_dw_whcn(
if (src_x < 0 || src_x >= p.src_w) {
continue;
}
sum += ggml_conv_2d_dw_knl_f32(knl_base, knl_offset + knl_y * p.knl_w + knl_x, knl_type)
sum += knl_data[knl_y * p.knl_w + knl_x]
* src_data[src_y * p.src_w + src_x];
}
}
@@ -7519,13 +7398,13 @@ void ggml_compute_forward_conv_2d_dw(
p.dilation_x = dst->op_params[4];
p.dilation_y = dst->op_params[5];
GGML_ASSERT(kernel->type == GGML_TYPE_F32 || kernel->type == GGML_TYPE_F16);
GGML_ASSERT(kernel->ne[3] == p.channels);
GGML_ASSERT(dst->ne[3] == p.batch);
if (ggml_is_contiguous(src)) {
ggml_compute_forward_conv_2d_dw_whcn(params, src, kernel, dst, p);
} else if (ggml_is_contiguous_channels(src)) {
// kernel should also have channels most contiguous in memory
GGML_ASSERT(kernel->nb[0] >= kernel->nb[2] && kernel->nb[1] >= kernel->nb[0]);
ggml_compute_forward_conv_2d_dw_cwhn(params, src, kernel, dst, p);
} else {
@@ -10944,291 +10823,6 @@ void ggml_compute_forward_gated_delta_net(
}
}
// ggml_compute_forward_dsv4_hc_comb
static void ggml_dsv4_hc_comb_norm_cols(float * comb, float eps) {
constexpr int64_t hc = 4;
for (int64_t idst = 0; idst < hc; ++idst) {
float sum = eps;
for (int64_t isrc = 0; isrc < hc; ++isrc) {
sum += comb[idst + hc*isrc];
}
const float inv_sum = 1.0f / sum;
for (int64_t isrc = 0; isrc < hc; ++isrc) {
comb[idst + hc*isrc] *= inv_sum;
}
}
}
static void ggml_dsv4_hc_comb_norm_rows(float * comb, float eps) {
constexpr int64_t hc = 4;
for (int64_t isrc = 0; isrc < hc; ++isrc) {
float sum = eps;
for (int64_t idst = 0; idst < hc; ++idst) {
sum += comb[idst + hc*isrc];
}
const float inv_sum = 1.0f / sum;
for (int64_t idst = 0; idst < hc; ++idst) {
comb[idst + hc*isrc] *= inv_sum;
}
}
}
static void ggml_compute_forward_dsv4_hc_comb_f32(
const ggml_compute_params * params,
ggml_tensor * dst) {
const ggml_tensor * mixes = dst->src[0];
const ggml_tensor * scale = dst->src[1];
const ggml_tensor * base = dst->src[2];
GGML_ASSERT(mixes->type == GGML_TYPE_F32);
GGML_ASSERT(scale->type == GGML_TYPE_F32);
GGML_ASSERT(base->type == GGML_TYPE_F32);
GGML_ASSERT(dst->type == GGML_TYPE_F32);
constexpr int64_t hc = 4;
constexpr int64_t comb_offset = 2*hc;
constexpr int64_t hc_mix_dim = (2 + hc)*hc;
const int64_t n_tokens = mixes->ne[1];
GGML_ASSERT(mixes->ne[0] == hc_mix_dim);
GGML_ASSERT(dst->ne[0] == hc);
GGML_ASSERT(dst->ne[1] == hc);
GGML_ASSERT(dst->ne[2] == n_tokens);
GGML_ASSERT(scale->ne[0] >= 3);
GGML_ASSERT(base->ne[0] == hc_mix_dim);
GGML_TENSOR_LOCALS(size_t, nbm, mixes, nb);
GGML_TENSOR_LOCALS(size_t, nbs, scale, nb);
GGML_TENSOR_LOCALS(size_t, nbb, base, nb);
GGML_TENSOR_LOCALS(size_t, nbd, dst, nb);
const float eps = ggml_get_op_params_f32(dst, 0);
const int32_t n_iter = ggml_get_op_params_i32(dst, 1);
GGML_ASSERT(n_iter > 0);
const int ith = params->ith;
const int nth = params->nth;
const int64_t dr = (n_tokens + nth - 1) / nth;
const int64_t it0 = dr * ith;
const int64_t it1 = MIN(it0 + dr, n_tokens);
const float scale_comb = *(const float *) ((const char *) scale->data + 2*nbs0);
for (int64_t it = it0; it < it1; ++it) {
float comb[hc*hc];
for (int64_t isrc = 0; isrc < hc; ++isrc) {
float max = -INFINITY;
for (int64_t idst = 0; idst < hc; ++idst) {
const int64_t idx = idst + hc*isrc;
const float xv = *(const float *) ((const char *) mixes->data + (comb_offset + idx)*nbm0 + it*nbm1);
const float bv = *(const float *) ((const char *) base->data + (comb_offset + idx)*nbb0);
const float v = xv * scale_comb + bv;
comb[idx] = v;
max = MAX(max, v);
}
float sum = 0.0f;
for (int64_t idst = 0; idst < hc; ++idst) {
const int64_t idx = idst + hc*isrc;
const float v = expf(comb[idx] - max);
comb[idx] = v;
sum += v;
}
const float inv_sum = 1.0f / sum;
for (int64_t idst = 0; idst < hc; ++idst) {
const int64_t idx = idst + hc*isrc;
comb[idx] = comb[idx] * inv_sum + eps;
}
}
ggml_dsv4_hc_comb_norm_cols(comb, eps);
for (int32_t i = 1; i < n_iter; ++i) {
ggml_dsv4_hc_comb_norm_rows(comb, eps);
ggml_dsv4_hc_comb_norm_cols(comb, eps);
}
for (int64_t isrc = 0; isrc < hc; ++isrc) {
for (int64_t idst = 0; idst < hc; ++idst) {
const int64_t idx = idst + hc*isrc;
*(float *) ((char *) dst->data + idst*nbd0 + isrc*nbd1 + it*nbd2) = comb[idx];
}
}
}
}
void ggml_compute_forward_dsv4_hc_comb(
const ggml_compute_params * params,
ggml_tensor * dst) {
const ggml_tensor * src0 = dst->src[0];
switch (src0->type) {
case GGML_TYPE_F32:
{
ggml_compute_forward_dsv4_hc_comb_f32(params, dst);
} break;
default:
{
GGML_ABORT("fatal error");
}
}
}
// ggml_compute_forward_dsv4_hc_pre
static void ggml_compute_forward_dsv4_hc_pre_f32(
const ggml_compute_params * params,
ggml_tensor * dst) {
const ggml_tensor * x = dst->src[0];
const ggml_tensor * weights = dst->src[1];
GGML_ASSERT(x->type == GGML_TYPE_F32);
GGML_ASSERT(weights->type == GGML_TYPE_F32);
GGML_ASSERT(dst->type == GGML_TYPE_F32);
const int64_t n_embd = x->ne[0];
const int64_t hc = x->ne[1];
const int64_t n_tokens = x->ne[2];
GGML_ASSERT(dst->ne[0] == n_embd);
GGML_ASSERT(dst->ne[1] == n_tokens);
GGML_ASSERT(weights->ne[0] == hc);
GGML_ASSERT(weights->ne[1] == n_tokens);
GGML_TENSOR_LOCALS(size_t, nbx, x, nb);
GGML_TENSOR_LOCALS(size_t, nbw, weights, nb);
GGML_TENSOR_LOCALS(size_t, nbd, dst, nb);
const int ith = params->ith;
const int nth = params->nth;
const int64_t nr = n_embd * n_tokens;
const int64_t dr = (nr + nth - 1) / nth;
const int64_t ir0 = dr * ith;
const int64_t ir1 = MIN(ir0 + dr, nr);
for (int64_t ir = ir0; ir < ir1; ++ir) {
const int64_t i0 = ir % n_embd;
const int64_t it = ir / n_embd;
float sum = 0.0f;
for (int64_t ih = 0; ih < hc; ++ih) {
const float xv = *(const float *) ((const char *) x->data + i0*nbx0 + ih*nbx1 + it*nbx2);
const float wv = *(const float *) ((const char *) weights->data + ih*nbw0 + it*nbw1);
sum += xv * wv;
}
*(float *) ((char *) dst->data + i0*nbd0 + it*nbd1) = sum;
}
}
void ggml_compute_forward_dsv4_hc_pre(
const ggml_compute_params * params,
ggml_tensor * dst) {
const ggml_tensor * src0 = dst->src[0];
switch (src0->type) {
case GGML_TYPE_F32:
{
ggml_compute_forward_dsv4_hc_pre_f32(params, dst);
} break;
default:
{
GGML_ABORT("fatal error");
}
}
}
// ggml_compute_forward_dsv4_hc_post
static void ggml_compute_forward_dsv4_hc_post_f32(
const ggml_compute_params * params,
ggml_tensor * dst) {
const ggml_tensor * x = dst->src[0];
const ggml_tensor * residual = dst->src[1];
const ggml_tensor * post = dst->src[2];
const ggml_tensor * comb = dst->src[3];
GGML_ASSERT(x->type == GGML_TYPE_F32);
GGML_ASSERT(residual->type == GGML_TYPE_F32);
GGML_ASSERT(post->type == GGML_TYPE_F32);
GGML_ASSERT(comb->type == GGML_TYPE_F32);
GGML_ASSERT(dst->type == GGML_TYPE_F32);
const int64_t n_embd = x->ne[0];
const int64_t n_tokens = x->ne[1];
const int64_t hc = residual->ne[1];
GGML_ASSERT(dst->ne[0] == n_embd);
GGML_ASSERT(dst->ne[1] == hc);
GGML_ASSERT(dst->ne[2] == n_tokens);
GGML_ASSERT(residual->ne[0] == n_embd);
GGML_ASSERT(residual->ne[2] == n_tokens);
GGML_ASSERT(post->ne[0] == hc);
GGML_ASSERT(post->ne[1] == n_tokens);
GGML_ASSERT(comb->ne[0] == hc);
GGML_ASSERT(comb->ne[1] == hc);
GGML_ASSERT(comb->ne[2] == n_tokens);
GGML_TENSOR_LOCALS(size_t, nbx, x, nb);
GGML_TENSOR_LOCALS(size_t, nbr, residual, nb);
GGML_TENSOR_LOCALS(size_t, nbp, post, nb);
GGML_TENSOR_LOCALS(size_t, nbc, comb, nb);
GGML_TENSOR_LOCALS(size_t, nbd, dst, nb);
const int ith = params->ith;
const int nth = params->nth;
const int64_t nr = n_embd * hc * n_tokens;
const int64_t dr = (nr + nth - 1) / nth;
const int64_t ir0 = dr * ith;
const int64_t ir1 = MIN(ir0 + dr, nr);
for (int64_t ir = ir0; ir < ir1; ++ir) {
const int64_t i0 = ir % n_embd;
const int64_t idst = (ir / n_embd) % hc;
const int64_t it = ir / (n_embd * hc);
const float xv = *(const float *) ((const char *) x->data + i0*nbx0 + it*nbx1);
const float pv = *(const float *) ((const char *) post->data + idst*nbp0 + it*nbp1);
float sum = xv * pv;
for (int64_t isrc = 0; isrc < hc; ++isrc) {
const float rv = *(const float *) ((const char *) residual->data + i0*nbr0 + isrc*nbr1 + it*nbr2);
const float cv = *(const float *) ((const char *) comb->data + idst*nbc0 + isrc*nbc1 + it*nbc2);
sum += rv * cv;
}
*(float *) ((char *) dst->data + i0*nbd0 + idst*nbd1 + it*nbd2) = sum;
}
}
void ggml_compute_forward_dsv4_hc_post(
const ggml_compute_params * params,
ggml_tensor * dst) {
const ggml_tensor * src0 = dst->src[0];
switch (src0->type) {
case GGML_TYPE_F32:
{
ggml_compute_forward_dsv4_hc_post_f32(params, dst);
} break;
default:
{
GGML_ABORT("fatal error");
}
}
}
// ggml_compute_forward_rwkv_wkv7
static void ggml_compute_forward_rwkv_wkv7_f32(
@@ -11918,87 +11512,3 @@ void ggml_compute_forward_fwht(const ggml_compute_params * params, ggml_tensor *
}
}
}
// ggml_compute_forward_lightning_indexer
void ggml_compute_forward_lightning_indexer(
const ggml_compute_params * params,
ggml_tensor * dst) {
const ggml_tensor * q = dst->src[0];
const ggml_tensor * k = dst->src[1];
const ggml_tensor * w = dst->src[2]; // weights
const ggml_tensor * m = dst->src[3]; // mask
GGML_ASSERT(dst->type == GGML_TYPE_F32);
GGML_ASSERT( q->type == GGML_TYPE_F32);
GGML_ASSERT( w->type == GGML_TYPE_F32);
GGML_ASSERT( m->type == GGML_TYPE_F16);
GGML_TENSOR_LOCALS(int64_t, neq, q, ne)
GGML_TENSOR_LOCALS(size_t, nbq, q, nb)
GGML_TENSOR_LOCALS(int64_t, nek, k, ne)
GGML_TENSOR_LOCALS(size_t, nbk, k, nb)
GGML_TENSOR_LOCALS(int64_t, new, w, ne)
GGML_TENSOR_LOCALS(size_t, nbw, w, nb)
GGML_TENSOR_LOCALS(int64_t, nem, m, ne)
GGML_TENSOR_LOCALS(size_t, nbm, m, nb)
GGML_TENSOR_LOCALS(int64_t, ne, dst, ne)
GGML_TENSOR_LOCALS(size_t, nb, dst, nb)
GGML_ASSERT( nb0 == ggml_type_size(dst->type));
GGML_ASSERT(nbq0 == ggml_type_size( q->type));
GGML_ASSERT(nbk0 == ggml_type_size( k->type));
GGML_ASSERT(nbw0 == ggml_type_size( w->type));
GGML_ASSERT(nbm0 == ggml_type_size( m->type));
const int n_embd = q->ne[0];
const int n_head = q->ne[1];
const int n_tokens = q->ne[2];
const int n_stream = q->ne[3];
const int n_kv = k->ne[2];
ggml_to_float_t const k_to_float = ggml_get_type_traits(k->type)->to_float;
GGML_ASSERT((k->type == GGML_TYPE_F32 || k_to_float) && "lightning indexer: unsupported K-type");
const int nr = n_kv;
const int ith = params->ith;
const int nth = params->nth;
// (temporary) buffer for K converted to float
float * k_row_f32 = (float *) params->wdata + ith*(1*n_embd + CACHE_LINE_SIZE_F32);
// rows per thread
const int dr = (nr + nth - 1)/nth;
// row range for this thread
const int ir0 = dr*ith;
const int ir1 = MIN(ir0 + dr, nr);
for (int s = 0; s < n_stream; ++s) {
for (int t = 0; t < n_tokens; ++t) {
const float * w_row = (float *) ((char *) w->data + t*nbw1 + s*nbw3);
const ggml_fp16_t * m_row = (ggml_fp16_t *) ((char *) m->data + t*nbm1 + (s%nem3)*nbm3);
float * dst_row = (float *) ((char *) dst->data + t*nb1 + s*nb3 );
for (int ik = ir0; ik < ir1; ++ik) {
char * k_row = (char *) k->data + ik*nbk2 + s*nbk3;
if (k_to_float) {
k_to_float(k_row, k_row_f32, n_embd);
} else {
k_row_f32 = (float *) k_row;
}
float score = 0.0f;
for (int h = 0; h < n_head; ++h) {
// dot product of q and k for head h
float qk = 0.0f;
const float * q_row = (float *) ((char *) q->data + h*nbq1 + t*nbq2 + s*nbq3);
ggml_vec_dot_f32(n_embd, &qk, 0, q_row, 0, k_row_f32, 0, 1);
// ReLU and weights (prescaled)
score += MAX(qk, 0.0f) * w_row[h];
}
// apply mask
dst_row[ik] = score + GGML_CPU_FP16_TO_FP32(m_row[ik]);
}
}
}
}
ggml/src/ggml-cpu/ops.h
-4
View File
@@ -105,10 +105,6 @@ void ggml_compute_forward_rwkv_wkv7(const struct ggml_compute_params * params, s
void ggml_compute_forward_solve_tri(const struct ggml_compute_params * params, struct ggml_tensor * dst);
void ggml_compute_forward_gla(const struct ggml_compute_params * params, struct ggml_tensor * dst);
void ggml_compute_forward_gated_delta_net(const struct ggml_compute_params * params, struct ggml_tensor * dst);
void ggml_compute_forward_lightning_indexer(const struct ggml_compute_params * params, struct ggml_tensor * dst);
void ggml_compute_forward_dsv4_hc_comb(const struct ggml_compute_params * params, struct ggml_tensor * dst);
void ggml_compute_forward_dsv4_hc_pre(const struct ggml_compute_params * params, struct ggml_tensor * dst);
void ggml_compute_forward_dsv4_hc_post(const struct ggml_compute_params * params, struct ggml_tensor * dst);
void ggml_compute_forward_map_custom1(const struct ggml_compute_params * params, struct ggml_tensor * dst);
void ggml_compute_forward_map_custom2(const struct ggml_compute_params * params, struct ggml_tensor * dst);
void ggml_compute_forward_map_custom3(const struct ggml_compute_params * params, struct ggml_tensor * dst);
ggml/src/ggml-cpu/quants.c
-51
View File
@@ -26,10 +26,6 @@ void quantize_row_q1_0(const float * GGML_RESTRICT x, void * GGML_RESTRICT y, in
quantize_row_q1_0_ref(x, y, k);
}
void quantize_row_q2_0(const float * GGML_RESTRICT x, void * GGML_RESTRICT y, int64_t k) {
quantize_row_q2_0_ref(x, y, k);
}
void quantize_row_q4_0(const float * GGML_RESTRICT x, void * GGML_RESTRICT y, int64_t k) {
quantize_row_q4_0_ref(x, y, k);
}
@@ -174,53 +170,6 @@ void ggml_vec_dot_q1_0_q8_0_generic(int n, float * GGML_RESTRICT s, size_t bs, c
*s = sumf;
}
void ggml_vec_dot_q2_0_q8_0_generic(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc) {
const int qk = QK2_0;
const int nb = n / qk;
assert(n % qk == 0);
assert(nrc == 1);
UNUSED(nrc);
UNUSED(bx);
UNUSED(by);
UNUSED(bs);
const block_q2_0 * GGML_RESTRICT x = vx;
const block_q8_0 * GGML_RESTRICT y = vy;
float sumf = 0.0f;
for (int i = 0; i < nb; i++) {
const float d0 = GGML_CPU_FP16_TO_FP32(x[i].d);
float sumi = 0.0f;
// group 64: one Q2_0 block (64 weights) maps to two Q8_0 blocks (2 * 32 = 64)
for (int k = 0; k < 2; k++) {
const block_q8_0 * GGML_RESTRICT yb = &y[i * 2 + k];
const float d1 = GGML_CPU_FP16_TO_FP32(yb->d);
int sumi_block = 0;
const uint8_t * GGML_RESTRICT qs = &x[i].qs[k * 8];
const int8_t * GGML_RESTRICT qy = yb->qs;
for (int b = 0; b < 8; ++b) {
const uint8_t byte = qs[b];
// Extract 4 two-bit values, map {0,1,2,3} -> {-1,0,1,2}
sumi_block += ((int)((byte >> 0) & 3) - 1) * qy[b*4 + 0];
sumi_block += ((int)((byte >> 2) & 3) - 1) * qy[b*4 + 1];
sumi_block += ((int)((byte >> 4) & 3) - 1) * qy[b*4 + 2];
sumi_block += ((int)((byte >> 6) & 3) - 1) * qy[b*4 + 3];
}
sumi += d1 * sumi_block;
}
sumf += d0 * sumi;
}
*s = sumf;
}
void ggml_vec_dot_q4_0_q8_0_generic(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc) {
const int qk = QK8_0;
ggml/src/ggml-cpu/quants.h
-3
View File
@@ -13,7 +13,6 @@ extern "C" {
// Quantization
void quantize_row_q1_0(const float * GGML_RESTRICT x, void * GGML_RESTRICT y, int64_t k);
void quantize_row_q2_0(const float * GGML_RESTRICT x, void * GGML_RESTRICT y, int64_t k);
void quantize_row_q4_0(const float * GGML_RESTRICT x, void * GGML_RESTRICT y, int64_t k);
void quantize_row_q4_1(const float * GGML_RESTRICT x, void * GGML_RESTRICT y, int64_t k);
void quantize_row_q5_0(const float * GGML_RESTRICT x, void * GGML_RESTRICT y, int64_t k);
@@ -39,7 +38,6 @@ void quantize_row_iq4_xs (const float * GGML_RESTRICT x, void * GGML_RESTRICT y,
// Dot product
void ggml_vec_dot_q1_0_q8_0(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
void ggml_vec_dot_q2_0_q8_0(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
void ggml_vec_dot_q4_0_q8_0(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
void ggml_vec_dot_q4_1_q8_1(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
void ggml_vec_dot_q5_0_q8_0(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
@@ -73,7 +71,6 @@ void quantize_row_q8_0_generic(const float * GGML_RESTRICT x, void * GGML_RESTRI
void quantize_row_q8_1_generic(const float * GGML_RESTRICT x, void * GGML_RESTRICT vy, int64_t k);
void quantize_row_q8_K_generic(const float * GGML_RESTRICT x, void * GGML_RESTRICT y, int64_t k);
void ggml_vec_dot_q1_0_q8_0_generic(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
void ggml_vec_dot_q2_0_q8_0_generic(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
void ggml_vec_dot_q4_0_q8_0_generic(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
void ggml_vec_dot_q4_1_q8_1_generic(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
void ggml_vec_dot_q5_0_q8_0_generic(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
ggml/src/ggml-cpu/simd-gemm.h
+1 -1
View File
@@ -78,7 +78,7 @@ static void simd_gemm(
for (int64_t i = 0; i < GEMM_RM; i++) {
float a = C[i * N + jj];
for (int64_t kk = 0; kk < K; kk++) {
a += A[i * K + kk] * B[kk * N + jj];
a += A[i + kk] * B[kk * N + jj];
}
C[i * N + jj] = a;
}
ggml/src/ggml-cpu/simd-mappings.h
+2 -2
View File
@@ -131,8 +131,8 @@ extern float ggml_table_f32_ue4m3[1 << 8];
#define GGML_CPU_E8M0_TO_FP32_HALF(x) GGML_E8M0_TO_FP32_HALF(x)
#endif
// Use lookup table for UE4M3 on x86 and ARM (faster than bit manipulation)
#if defined(__AVX__) || defined(__AVX2__) || defined(__AVX512F__) || defined(__ARM_NEON)
// Use lookup table for UE4M3 on x86 (faster than bit manipulation)
#if defined(__AVX__) || defined(__AVX2__) || defined(__AVX512F__)
#define GGML_CPU_UE4M3_TO_FP32(x) ggml_table_f32_ue4m3[(uint8_t)(x)]
#else
#define GGML_CPU_UE4M3_TO_FP32(x) ggml_ue4m3_to_fp32(x)
ggml/src/ggml-cuda/argsort.cu
+4 -30
View File
@@ -28,20 +28,6 @@ static __global__ void init_offsets(int * offsets, const int ncols, const int nr
#endif // STRIDED_ITERATOR_AVAILABLE
#ifdef GGML_CUDA_USE_CUB
// returns the suggested maximum number of rows to process during one argsort_f32_i32_cuda_cub() call
int argsort_f32_i32_cuda_cub_chunk_nrows(const size_t nb01, const int64_t nrows) {
// perform argsort in chunks up to approximately this size (currently 64MB)
// to avoid excessive temporary buffers memory usage
const int chunk_bytes = 1 << 26;
// calculate how many rows will fit in one chunk (must be at least one)
const int chunk_nrows = std::max((int) (chunk_bytes / nb01), 1);
// limit the resulting amount to total nrows
return std::min((int64_t) chunk_nrows, nrows);
}
void argsort_f32_i32_cuda_cub(ggml_cuda_pool & pool,
const float * x,
int * dst,
@@ -268,23 +254,11 @@ void ggml_cuda_op_argsort(ggml_backend_cuda_context & ctx, ggml_tensor * dst) {
const size_t shared_mem = ncols_pad * sizeof(int);
const size_t max_shared_mem = ggml_cuda_info().devices[ggml_cuda_get_device()].smpb;
// early return if we can use bitonic argsort
if (shared_mem <= max_shared_mem && ncols <= 1024) {
if (shared_mem > max_shared_mem || ncols > 1024) {
ggml_cuda_pool & pool = ctx.pool();
argsort_f32_i32_cuda_cub(pool, src0_d, (int *) dst_d, ncols, nrows, order, stream);
} else {
argsort_f32_i32_cuda_bitonic(src0_d, (int *) dst_d, ncols, nrows, order, stream);
return;
}
const int chunk_nrows = argsort_f32_i32_cuda_cub_chunk_nrows(src0->nb[1], nrows);
ggml_cuda_pool & pool = ctx.pool();
for (int64_t i = 0; i < nrows; i += chunk_nrows) {
int iter_nrows = std::min((int64_t) chunk_nrows, nrows - i);
argsort_f32_i32_cuda_cub(pool, src0_d, (int *) dst_d, ncols, iter_nrows, order, stream);
src0_d += ncols * iter_nrows;
dst_d += ncols * iter_nrows;
}
#else
argsort_f32_i32_cuda_bitonic(src0_d, (int *) dst_d, ncols, nrows, order, stream);
ggml/src/ggml-cuda/argsort.cuh
-1
View File
@@ -3,7 +3,6 @@
void ggml_cuda_op_argsort(ggml_backend_cuda_context & ctx, ggml_tensor * dst);
#ifdef GGML_CUDA_USE_CUB
int argsort_f32_i32_cuda_cub_chunk_nrows(const size_t nb01, const int64_t nrows);
void argsort_f32_i32_cuda_cub(ggml_cuda_pool & pool,
const float * x,
int * dst,
ggml/src/ggml-cuda/common.cuh
+1 -21
View File
@@ -362,15 +362,6 @@ static bool blackwell_mma_available(const int cc) {
ggml_cuda_highest_compiled_arch(cc) < GGML_CUDA_CC_RUBIN;
}
// Checks whether the tensor's base data pointer and higher-dimensional strides are byte-aligned to `alignment` bytes.
static bool ggml_cuda_is_aligned(const ggml_tensor * tensor, const size_t alignment) {
GGML_ASSERT(tensor != nullptr);
return (reinterpret_cast<uintptr_t>(tensor->data) % alignment) == 0 &&
tensor->nb[1] % alignment == 0 &&
tensor->nb[2] % alignment == 0 &&
tensor->nb[3] % alignment == 0;
}
static constexpr __device__ int ggml_cuda_get_physical_warp_size() {
#if defined(GGML_USE_HIP) && (defined(__GFX9__) || defined(__GFX8__))
return 64;
@@ -946,9 +937,6 @@ static __device__ __forceinline__ uint2 fast_div_modulo(uint32_t n, const uint3
typedef void (*dequantize_kernel_t)(const void * vx, const int64_t ib, const int iqs, float2 & v);
template<typename dst_t>
using dequantize_kq_t = void (*)(const void * vx, const int64_t ib, dst_t * y, const int tid);
static __device__ __forceinline__ float get_alibi_slope(
const float max_bias, const uint32_t h, const uint32_t n_head_log2, const float m0, const float m1
) {
@@ -1127,8 +1115,7 @@ struct ggml_cuda_type_traits<GGML_TYPE_IQ3_S> {
//////////////////////
struct ggml_cuda_device_info {
int device_count; // number of (possibly virtual) devices exposed to the rest of ggml
int physical_device_count; // number of physical CUDA devices actually present
int device_count;
struct cuda_device_info {
int cc; // compute capability
@@ -1141,9 +1128,6 @@ struct ggml_cuda_device_info {
size_t total_vram;
int warp_size; // Number of threads in a dispatch
bool supports_cooperative_launch; // whether cooperative launch is supported
int physical_device; // backing physical CUDA device for this (virtual) device
int physical_share_count; // number of (virtual) devices sharing this device's physical GPU
int virtual_index; // index of this (virtual) device among those sharing its physical GPU
};
cuda_device_info devices[GGML_CUDA_MAX_DEVICES] = {};
@@ -1521,16 +1505,12 @@ struct ggml_cuda_mm_fusion_args_host {
const ggml_tensor * x_bias = nullptr;
const ggml_tensor * gate = nullptr;
const ggml_tensor * gate_bias = nullptr;
const ggml_tensor * x_scale = nullptr;
const ggml_tensor * gate_scale = nullptr;
ggml_glu_op glu_op;
};
struct ggml_cuda_mm_fusion_args_device {
const void * x_bias = nullptr;
const void * gate = nullptr;
const void * gate_bias = nullptr;
const void * x_scale = nullptr;
const void * gate_scale = nullptr;
ggml_glu_op glu_op;
};
ggml/src/ggml-cuda/concat.cu
+34 -49
View File
@@ -141,28 +141,28 @@ static __global__ void __launch_bounds__(CUDA_CONCAT_BLOCK_SIZE)
template <typename T>
static void concat_cuda(const ggml_tensor * src0, const ggml_tensor * src1, ggml_tensor * dst, int dim, cudaStream_t stream) {
if (dim != 3 && ggml_is_contiguous_to_3(src0) && ggml_is_contiguous_to_3(src1)) {
if (ggml_is_contiguous(src0) && ggml_is_contiguous(src1)) {
const T * src0_d = (const T *) src0->data;
const T * src1_d = (const T *) src1->data;
T * dst_d = (T *) dst->data;
for (int64_t i3 = 0; i3 < dst->ne[3]; i3++) {
concat_cont_cuda(
src0_d + i3*(src0->nb[3] / sizeof(T)),
src1_d + i3*(src1->nb[3] / sizeof(T)),
dst_d + i3*( dst->nb[3] / sizeof(T)),
ggml_row_size(src0->type, src0->ne[0])/sizeof(T), src0->ne[1], src0->ne[2],
ggml_row_size(dst->type, dst->ne[0])/sizeof(T), dst->ne[1], dst->ne[2], dim, stream);
if (dim != 3) {
for (int64_t i3 = 0; i3 < dst->ne[3]; i3++) {
concat_cont_cuda(
src0_d + i3*(src0->nb[3] / sizeof(T)),
src1_d + i3*(src1->nb[3] / sizeof(T)),
dst_d + i3*( dst->nb[3] / sizeof(T)),
src0->ne[0], src0->ne[1], src0->ne[2],
dst->ne[0], dst->ne[1], dst->ne[2], dim, stream);
}
} else {
const size_t size0 = ggml_nbytes(src0);
const size_t size1 = ggml_nbytes(src1);
CUDA_CHECK(cudaMemcpyAsync((char *) dst->data, src0->data, size0, cudaMemcpyDeviceToDevice, stream));
CUDA_CHECK(cudaMemcpyAsync((char *) dst->data + size0, src1->data, size1, cudaMemcpyDeviceToDevice, stream));
}
} else if (dim == 3 && ggml_is_contiguous(src0) && ggml_is_contiguous(src1)) {
const size_t size0 = ggml_nbytes(src0);
const size_t size1 = ggml_nbytes(src1);
CUDA_CHECK(cudaMemcpyAsync((char *) dst->data, src0->data, size0, cudaMemcpyDeviceToDevice, stream));
CUDA_CHECK(cudaMemcpyAsync((char *) dst->data + size0, src1->data, size1, cudaMemcpyDeviceToDevice, stream));
} else {
GGML_ASSERT(!ggml_is_quantized(src0->type));
dim3 grid_dim(dst->ne[1], dst->ne[2], dst->ne[3]);
auto launch_kernel = [&](auto dim) {
concat_non_cont<T, dim><<<grid_dim, CUDA_CONCAT_BLOCK_SIZE, 0, stream>>>(
@@ -204,39 +204,24 @@ void ggml_cuda_op_concat(ggml_backend_cuda_context & ctx, ggml_tensor * dst) {
GGML_ASSERT(src0->type == src1->type);
GGML_ASSERT(dst->type == src0->type);
GGML_ASSERT(!ggml_is_quantized(src0->type));
GGML_ASSERT(ggml_blck_size(src0->type) == 1);
if (ggml_is_quantized(src0->type)) {
if (dim == 3) {
GGML_ASSERT(ggml_is_contiguous(src0));
GGML_ASSERT(ggml_is_contiguous(src1));
} else {
GGML_ASSERT(ggml_is_contiguous_to_3(src0));
GGML_ASSERT(ggml_is_contiguous_to_3(src1));
}
GGML_ASSERT(src0->ne[0] % ggml_blck_size(src0->type) == 0);
GGML_ASSERT(src1->ne[0] % ggml_blck_size(src1->type) == 0);
// if first 3 dimensions are contiguous and ne[0] is multiple of the block size we can concat both tensors as byte tensors
concat_cuda<uint8_t>(src0, src1, dst, dim, stream);
} else {
GGML_ASSERT(ggml_blck_size(src0->type) == 1);
switch (ggml_type_size(src0->type)) {
case 1:
concat_cuda<uint8_t>(src0, src1, dst, dim, stream);
break;
case 2:
concat_cuda<uint16_t>(src0, src1, dst, dim, stream);
break;
case 4:
concat_cuda<uint32_t>(src0, src1, dst, dim, stream);
break;
case 8:
concat_cuda<uint64_t>(src0, src1, dst, dim, stream);
break;
default:
GGML_ABORT("Unsupported type size: %zu", ggml_type_size(src0->type));
break;
}
switch (ggml_type_size(src0->type)) {
case 1:
concat_cuda<uint8_t>(src0, src1, dst, dim, stream);
break;
case 2:
concat_cuda<uint16_t>(src0, src1, dst, dim, stream);
break;
case 4:
concat_cuda<uint32_t>(src0, src1, dst, dim, stream);
break;
case 8:
concat_cuda<uint64_t>(src0, src1, dst, dim, stream);
break;
default:
GGML_ABORT("Unsupported type size: %zu", ggml_type_size(src0->type));
break;
}
}
ggml/src/ggml-cuda/conv-transpose-1d.cu
+12 -14
View File
@@ -11,32 +11,30 @@ static __global__ void conv_transpose_1d_kernel(
return;
}
int out_t = global_index % dst_ne0;
int out_ch = (global_index / dst_ne0) % dst_ne1;
int plane = global_index / (dst_ne0 * dst_ne1);
int out_index = global_index / dst_ne0;
float accumulator = 0;
for (int c = 0; c < src0_ne2; c++) {
int kernel_offset = src0_ne0 * (out_ch + src0_ne1 * c);
int input_offset = src1_ne0 * (c + src1_ne1 * plane);
int idx = global_index % dst_ne0;
for (int k = 0; k < src0_ne0; k++) {
int input_numer = out_t + p0 - k*d0;
if (input_numer < 0 || input_numer % s0 != 0) {
int kernel_offset = (src0_ne0 * src0_ne1 * c) + (out_index * src0_ne0);
int input_offset = src1_ne0 * c;
for (int i = 0; i < src1_ne0; i++) {
if (!(idx >= i*s0 && idx < i*s0 + src0_ne0)) {
continue;
}
int weight_idx = idx - i*s0;
int input_t = input_numer / s0;
if (input_t >= src1_ne0) {
continue;
}
float kernel_weight = src0[kernel_offset + weight_idx];
float input_value = src1[input_offset+i];
accumulator += src0[kernel_offset + k] * src1[input_offset + input_t];
accumulator += kernel_weight * input_value;
}
}
dst[global_index] = accumulator;
GGML_UNUSED_VARS(src0_ne3, src1_ne2, src1_ne3, dst_ne2, dst_ne3);
GGML_UNUSED_VARS(p0, d0, src0_ne3, src1_ne3, dst_ne3, src1_ne1, dst_ne1, src1_ne2, dst_ne2);
}
static void conv_transpose_1d_f32_f32_cuda(

Some files were not shown because too many files have changed in this diff Show More