Lumpiasty/llama.cpp
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

hexagon: support for multi-NPU devices (IQ9, IQ10) and fully asynchronous backend (#26501)

Browse Source 
* hexagon: use non-host bufs by default and make the backend fully async

* hex-hb: remove optional hostbuf support and fix async copy

* hex-unary: relax supported unary check

* hex-bufs: use same get_alignment for host bufs

* snapdragon: bump android_platform to 34

* hex-rows: super hacky get/set rows for q8_0

* hex-get-rows: fix q8_0

* hex-get-rows: supprot for f16 and cleanup for q8_0

* hex-get-rows: generic macros and specialized thread funcs

* hex-get-rows: add DMA pipeline, vtcm_layout and kernel params

* hex-set-rows: fix q8_0 support, add dma and tracing

* hex-tests: override nmse threshold for HTP of Q8_0 quants

* hex-fa: add support for Q8_0 with inplace dequantizers

* hex-get-rows: simplify type dispatch

* hex-rows: simplify GET/SET_ROWS DMA pipeline

* hex-async: add events, set/get-tensor-async and rest of the async api support

* hex-repack: use slice instead of expert in repack functions

* hex-cpy: update event/async-cpy logging

* hex-set-rows: optimize smaller tensors

* hex-geglu: fix perf regression with larger tensors

* hex-get-rows: add missing header

* hex-set-rows: add missing header

* hex-bufs: ressurect GGML_HEXAGON_HOSTBUF but disable it by default

* hexagon: do not reject ops with non-heaxon buffers

* hex-get-rows: apply >=32 restriction only for q8_0

* hex-res: bump vtcm acquire timeout to 10 seconds

* hex-bufs: add support for cloning buffers between sessions to speed up tensor copies

* hex-async: rework event recording and batch flushing and integrate with meta backend

* hex-bufs: improved handling of repacked tensors

* hex-repack: handle get_tensor_2d offsets

* hex-dev: add support for devices with multiple NPUs

* hex-sync: add support for sync tokens to synchronize npu devices for async splits

* hex-mmap: cleanup mmap calls and add a retry for robustness

* hex-sync: add failsafe if sync wait gets stuck

* hex-sync: use sync_seq to check for completed events

* hex-sync: rotate tokens for extra robustness

* hex-devs: add supprot for legacy device names for now

* hex-bufs: add support for auto-cloning buffers from diff sessions

* hex-fusion: simplify and optimize htp-opnode fusion handling

* hex-sync: override opnode name so that it shows up in the profiles

* hex-trace: update scripts to handle multiple devices

* hex-sync: bump the size of the opbatch queue and number of sync tokens

* hex-cpy-sync: do not explicitly flush opbatches in cpy_tensor_async and add support for cpy-dma

* hex-sync: add graph-flush threshold to avoid single op batches

* hex-sync: add sync_peer so that we can flush peers we depend on during cross-device ops

* hex-bufs: introduce tensor->extra and shadow_bufs for repacking

* hex-l2: flush tiny tensors inline

* hex-sync: use explicit l2flush for sync tokens

* hex-extra: track weight flags via tensor extra

* hex-fence: rename sync to fence

* hex-repack: proper handling of set-tensor-2d in the shadow_buf

* hex-trace: remove obsolete opstage mask that we used for profiling

* hex-env: remove obsolete use_hmx variable

* hexagon: new unified run.py and build.py and updated docs

* snapdragon: update run script to auto-escapt test-backend-op -p argument

* hex-scripts: fix trailing spaces

* hex-scripts: fix flake8 warnings

* snapdragon: cleanup dst lib/bin dirs before copying new build

* hex-ops: add support for allreduce

* hex-ar: improved allreduce with dma pipeline

* hex-ar: align macros

* hex-ar: consistent use of fence_seq

* hex-ar: add AR_SELECT env var to select ALLREDUCE kernel or fallback

* hex-ar: add proper synchronize handling for ALLREDUCE

* hex-opbatch: looks like we now just rely on backend.synchronise to flush the batches, no need to flush them by threshold

* hex-ar: bump block size to improve dma efficiency

* hex-ar: fused ALLREDUCE+ADD

* hex-ar: cleaner fence buffer management

* hex-ar: futher allreduce tweaking to remove race conditions

* hex-ar: add simple solver and remove non-dma kernels

* hex-ar: add row-broadcast to fuse with bias ADD

* hex-fence: pass seq numbers via op_params

* hex-ar: allow for both entry/exit seq for completing entry wait

* hex-ar: align macros

* hex-ar: do not refetch broadcast row

* hex-fusion: move all fusion into opbatch::add_op for consistency with ALLREDUCE and things

* hex-fusion: fix incorrect MUL_MAT reordering

* hex-mm: make fused 2x and 3x matmuls more generic

* hex-fusion: move tensor fusion tagging to graph_compute

* hexagon: make sure to copy tensor->extra by value

* hex-get-rows: fix offset calc with row-chunking

* hex-repack: get_tensor_2d fixes for non-zero offsets

* snapdragon: make profile/trace scripts more robust and donot mix stdout/stderr by default

* hex-devices: use legacy device nameing by default to ease the transition

* hex-devices: hardcode CDSP domain IDs for current devices for now

* hex-optrace: improve multi-NPU timestamp alignment and overall handling of cycle values

* hex-optrace: more robust handling of the fence events
This commit is contained in:
Max Krasnyansky
2026-08-26 18:46:50 -07:00
committed by GitHub
parent 925e117994
commit 192067b72d
44 changed files with 5314 additions and 3139 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/backend/snapdragon/README.md
+103 -115
View File
@@ -2,39 +2,47 @@
## Setup
### Android
The cross-compilation toolchain images are provided by the
[Qualcomm Snapdragon Toolchain registry](https://github.com/snapdragon-toolchain).
These Docker images include the Android NDK, OpenCL SDK, Hexagon SDK, CMake, and the necessary cross-compilers:
The easiest way to build llama.cpp for a Snapdragon-based Android device is using the toolchain Docker image (see github.com/snapdragon-toolchain).
This image includes Android NDK, OpenCL SDK, Hexagon SDK, CMake, etc.
* **Android toolchain**: `ghcr.io/snapdragon-toolchain/arm64-android:v0.7`
* **Linux toolchain**: `ghcr.io/snapdragon-toolchain/arm64-linux:v0.7`
This method works on Linux, macOS, and Windows. macOS and Windows users should install Docker Desktop.
```
~/src/llama.cpp$ docker run -it -u $(id -u):$(id -g) --volume $(pwd):/workspace --platform linux/amd64 ghcr.io/snapdragon-toolchain/arm64-android:v0.7
[d]/> cd /workspace
```
Note: The rest of the **Android** build process assumes that you're running inside the toolchain container.
### Windows On Snapdragon
Native Windows 11 arm64 builds has the following tools dependencies:
- MS Visual Studio 2026 (Community Edition or Pro)
- MSVC arm64 standard and runtime libraries
- UCRT and Driver Kit
- LLVM core libraries and Clang compiler (winget)
- CMake, Git, Python (winget)
- Hexagon SDK Community Edition 6.6 or later (see windows.md)
- OpenCL SDK 2.3 or later (see windows.md)
Note: The rest of the **Windows** build process assumes that you're running natively in Powershell.
Adapt below build commands accordingly.
The unified build utility (`scripts/snapdragon/build.py`) automatically pulls
and orchestrates these containers to perform target compilation.
You only need to ensure that Docker (or Docker Desktop on macOS/Windows) is running on your host machine.
Specific setup, build, and installation details for Linux and Windows on Snapdragon platforms are documented in:
* [Linux on Snapdragon guide](linux.md)
* [Windows on Snapdragon guide](windows.md)
## How to Build
Let's build llama.cpp with CPU, OpenCL, and Hexagon backends via CMake presets:
### Using build.py script (Recommended)
The easiest way to build llama.cpp is by using the `scripts/snapdragon/build.py` script. It automatically copies the CMake presets,
launches the correct compilation Docker container, builds the libraries and tools,
installs them, and optionally pushes them to your ADB device.
Build and deploy for Android target (accepts `android` or `adb` alias):
```
$ ./scripts/snapdragon/build.py --target adb --push
```
Build and deploy for Linux target (accepts `linux` or `lnx` alias):
```
$ ./scripts/snapdragon/build.py --target linux:user@host --push
```
### Manual CMake Build
Alternatively, you can build llama.cpp manually by entering the cross-compilation Docker container and running the CMake commands:
```bash
# Start the cross-compilation container manually:
~/src/llama.cpp$ docker run -it --rm -u $(id -u):$(id -g) --volume $(pwd):/workspace --platform linux/amd64 ghcr.io/snapdragon-toolchain/arm64-android:v0.7
# Inside the container, build the project using presets:
[d]/workspace> cp docs/backend/snapdragon/CMakeUserPresets.json .
[d]/workspace> cmake --preset arm64-android-snapdragon-release -B build-snapdragon
@@ -68,19 +76,19 @@ Preset CMake variables:
To generate an installable "package" simply use cmake --install:
```
[d]/workspace> cmake --install build-snapdragon --prefix pkg-snapdragon/llama.cpp
[d]/workspace> cmake --install build-snapdragon --prefix pkg-android/llama.cpp
-- Install configuration: "Release"
-- Installing: /workspace/pkg-snapdragon/llama.cpp/lib/libggml-cpu.so
-- Installing: /workspace/pkg-snapdragon/llama.cpp/lib/libggml-opencl.so
-- Installing: /workspace/pkg-snapdragon/llama.cpp/lib/libggml-hexagon.so
-- Installing: /workspace/pkg-snapdragon/llama.cpp/lib/libggml-htp-v73.so
-- Installing: /workspace/pkg-snapdragon/llama.cpp/lib/libggml-htp-v75.so
-- Installing: /workspace/pkg-snapdragon/llama.cpp/lib/libggml-htp-v79.so
-- Installing: /workspace/pkg-snapdragon/llama.cpp/lib/libggml-htp-v81.so
-- Installing: /workspace/pkg-snapdragon/llama.cpp/lib/libggml.so
-- Installing: /workspace/pkg-android/llama.cpp/lib/libggml-cpu.so
-- Installing: /workspace/pkg-android/llama.cpp/lib/libggml-opencl.so
-- Installing: /workspace/pkg-android/llama.cpp/lib/libggml-hexagon.so
-- Installing: /workspace/pkg-android/llama.cpp/lib/libggml-htp-v73.so
-- Installing: /workspace/pkg-android/llama.cpp/lib/libggml-htp-v75.so
-- Installing: /workspace/pkg-android/llama.cpp/lib/libggml-htp-v79.so
-- Installing: /workspace/pkg-android/llama.cpp/lib/libggml-htp-v81.so
-- Installing: /workspace/pkg-android/llama.cpp/lib/libggml.so
...
-- Installing: /workspace/pkg-snapdragon/llama.cpp/bin/llama-bench
-- Installing: /workspace/pkg-snapdragon/llama.cpp/bin/llama-cli
-- Installing: /workspace/pkg-android/llama.cpp/bin/llama-bench
-- Installing: /workspace/pkg-android/llama.cpp/bin/llama-cli
...
```
@@ -91,14 +99,14 @@ To generate an installable "package" simply use cmake --install:
For this step, your device needs to be configured for on-device development.
Please see https://developer.android.com/studio/debug/dev-options for details.
Once ADB is enabled, use `adb push` to install `pkg-snapdragon` on the device.
Once ADB is enabled, use `adb push` to install `pkg-android` on the device.
**Note that the toolchain Docker image doesn't have ADB and doesn't set up the ADB bridge. Please use native ADB on the host.**
```
~/src/llama.cpp$ adb push pkg-snapdragon/llama.cpp /data/local/tmp/
pkg-snapdragon/llama.cpp/bin/: 67 files pushed, 0 skipped. 190.2 MB/s (919095042 bytes in 4.607s)
pkg-snapdragon/llama.cpp/include/: 19 files pushed, 0 skipped. 20.5 MB/s (255173 bytes in 0.012s)
pkg-snapdragon/llama.cpp/lib/: 16 files pushed, 0 skipped. 144.4 MB/s (43801382 bytes in 0.289s)
~/src/llama.cpp$ adb push pkg-android/llama.cpp /data/local/tmp/
pkg-android/llama.cpp/bin/: 67 files pushed, 0 skipped. 190.2 MB/s (919095042 bytes in 4.607s)
pkg-android/llama.cpp/include/: 19 files pushed, 0 skipped. 20.5 MB/s (255173 bytes in 0.012s)
pkg-android/llama.cpp/lib/: 16 files pushed, 0 skipped. 144.4 MB/s (43801382 bytes in 0.289s)
102 files pushed, 0 skipped. 186.9 MB/s (963151597 bytes in 4.914s)
```
@@ -115,24 +123,44 @@ Llama-3.2-1B-Instruct-Q4_0.gguf: 1 file pushed, 0 skipped. 38.3 MB/s (773025920
### Windows
All artifacts are already installed in the `pkg-snapdragon` folder.
To run, adapt below instructions to use Powershell scripts in `scripts/snapdragon/windows`.
All artifacts are already installed in the `pkg-wos` folder.
To run, you can use the `scripts/snapdragon/run.py` runner script (see details below).
## How to Run
The easiest way to run llama.cpp cli tools is using provided wrapper scripts that properly set up all required environment variables.
The easiest way to run llama.cpp cli tools is using the provided `scripts/snapdragon/run.py` wrapper script. This script automatically
maps CLI options to environment variables, resolves executable paths, and runs the command locally, via ADB, or remotely via SSH on the
target device.
llama.cpp supports three backends on Snapdragon-based devices: CPU, Adreno GPU (GPUOpenCL), and Hexagon NPU (HTP0-4).
You can select which backend to run the model on using the `D=` variable, which maps to the `--device` option.
llama.cpp supports three backends on Snapdragon-based devices: CPU, Adreno GPU (GPUOpenCL), and Hexagon NPU.
You can select which backend(s) to run the model on using the `--device` option of the tool (or `--devices` option in `run.py`).
Hexagon NPU behaves as a "GPU" device when it comes to `-ngl` and other offload-related options.
Here are some examples of running various llama.cpp tools via ADB.
Here are some examples of running various llama.cpp tools.
Simple question for Llama-3.2-1B
Generating a completion with Gemma on Android (relying on default `HTP0:0` device and default thread count `-t 6`):
```
~/src/llama.cpp$ M=Llama-3.2-1B-Instruct-Q4_0.gguf D=HTP0 ./scripts/snapdragon/adb/run-completion.sh -p "what is the most popular cookie in the world?"
~/src/llama.cpp$ ./scripts/snapdragon/run.py --target adb -- llama-completion -m models/gemma-2-2b-it-Q4_0.gguf -f prompts/sample_prompt_1024.txt --jinja -st
...
ggml-hex: Hexagon backend (experimental) : allocating new registry : ndev 1
ggml-hex: Hexagon Arch version v79
ggml-hex: allocating new session: HTP0:0
...
load_tensors: offloading output layer to GPU
load_tensors: offloaded 27/27 layers to GPU
load_tensors: CPU model buffer size = 300.00 MiB
load_tensors: HTP0:0 model buffer size = 1400.26 MiB
...
llama_perf_context_print: prompt eval time = 320.00 ms / 1024 tokens ( 0.31 ms per token, 3200.00 tokens per second)
llama_perf_context_print: eval time = 2100.00 ms / 100 runs ( 21.00 ms per token, 47.62 tokens per second)
```
Simple question for Llama-3.2-1B:
```
~/src/llama.cpp$ ./scripts/snapdragon/run.py --target android --devices HTP0 -- llama-cli -m Llama-3.2-1B-Instruct-Q4_0.gguf -p "what is the most popular cookie in the world?"
...
ggml-hex: Hexagon backend (experimental) : allocating new registry : ndev 1
ggml-hex: Hexagon Arch version v79
@@ -142,8 +170,7 @@ ggml-hex: new session: HTP0 : session-id 0 domain-id 3 uri file:///libggml-htp-v
load_tensors: offloading output layer to GPU
load_tensors: offloaded 17/17 layers to GPU
load_tensors: CPU model buffer size = 225.49 MiB
load_tensors: HTP0 model buffer size = 0.26 MiB
load_tensors: HTP0-REPACK model buffer size = 504.00 MiB
load_tensors: HTP0 model buffer size = 504.26 MiB
...
I hope this helps you understand the world's most popular cookies! [end of text]
...
@@ -156,60 +183,25 @@ llama_perf_context_print: graphs reused = 473
llama_memory_breakdown_print: | memory breakdown [MiB] | total free self model context compute unaccounted |
llama_memory_breakdown_print: | - HTP0 (Hexagon) | 2048 = 2048 + ( 0 = 0 + 0 + 0) + 0 |
llama_memory_breakdown_print: | - Host | 439 = 225 + 136 + 77 |
llama_memory_breakdown_print: | - HTP0-REPACK | 504 = 504 + 0 + 0 |
```
Summary request for OLMoE-1B-7B. This is a large model that requires two HTP sessions/devices
Op test for MUL_MAT:
```
~/src/llama.cpp$ M=OLMoE-1B-7B-0125-Instruct-Q4_0.gguf NDEV=2 D=HTP0,HTP1 ./scripts/snapdragon/adb/run-completion.sh -f surfing.txt
~/src/llama.cpp$ ./scripts/snapdragon/run.py --target adb --hex-hostbuf 0 --devices HTP0:0 -- test-backend-ops -b HTP0:0 -o MUL_MAT
...
ggml-hex: Hexagon backend (experimental) : allocating new registry : ndev 1
ggml-hex: Hexagon Arch version v81
ggml-hex: allocating new session: HTP0
ggml-hex: allocating new session: HTP1
...
load_tensors: offloading output layer to GPU
load_tensors: offloaded 17/17 layers to GPU
load_tensors: CPU model buffer size = 143.86 MiB
load_tensors: HTP1 model buffer size = 0.23 MiB
load_tensors: HTP1-REPACK model buffer size = 1575.00 MiB
load_tensors: HTP0 model buffer size = 0.28 MiB
load_tensors: HTP0-REPACK model buffer size = 2025.00 MiB
...
llama_context: CPU output buffer size = 0.19 MiB
llama_kv_cache: HTP1 KV buffer size = 238.00 MiB
llama_kv_cache: HTP0 KV buffer size = 306.00 MiB
llama_kv_cache: size = 544.00 MiB ( 8192 cells, 16 layers, 1/1 seqs), K (q8_0): 272.00 MiB, V (q8_0): 272.00 MiB
llama_context: HTP0 compute buffer size = 15.00 MiB
llama_context: HTP1 compute buffer size = 15.00 MiB
llama_context: CPU compute buffer size = 24.56 MiB
...
llama_perf_context_print: prompt eval time = 1730.57 ms / 212 tokens ( 8.16 ms per token, 122.50 tokens per second)
llama_perf_context_print: eval time = 5624.75 ms / 257 runs ( 21.89 ms per token, 45.69 tokens per second)
llama_perf_context_print: total time = 7377.33 ms / 469 tokens
llama_perf_context_print: graphs reused = 255
llama_memory_breakdown_print: | memory breakdown [MiB] | total free self model context compute unaccounted |
llama_memory_breakdown_print: | - HTP0 (Hexagon) | 2048 = 2048 + ( 0 = 0 + 0 + 0) + 0 |
llama_memory_breakdown_print: | - HTP1 (Hexagon) | 2048 = 2048 + ( 0 = 0 + 0 + 0) + 0 |
llama_memory_breakdown_print: | - Host | 742 = 144 + 544 + 54 |
llama_memory_breakdown_print: | - HTP1-REPACK | 1575 = 1575 + 0 + 0 |
llama_memory_breakdown_print: | - HTP0-REPACK | 2025 = 2025 + 0 + 0 |
```
Op test for MUL_MAT
```
~/src/llama.cpp$ HB=0 ./scripts/snapdragon/adb/run-tool.sh test-backend-ops -b HTP0 -o MUL_MAT
...
Backend 2/3: HTP0
Backend 2/3: HTP0:0
Device description: Hexagon
Device memory: 2048 MB (2048 MB free)
MUL_MAT(type_a=q4_0,type_b=f32,m=16,n=1,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0,o=1): OK
MUL_MAT(type_a=q4_0,type_b=f32,m=16,n=2,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0,o=1): OK
MUL_MAT(type_a=q4_0,type_b=f32,m=16,n=3,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0,o=1): OK
```
~/src/llama.cpp-hexagon$ M=Llama-3.2-1B-Instruct-Q4_0.gguf ./scripts/snapdragon/adb/run-bench.sh -p 128 -n 64
Llama benchmark:
```
~/src/llama.cpp$ ./scripts/snapdragon/run.py --target adb --devices HTP0 -- llama-bench -p 128 -n 64 -m Llama-3.2-1B-Instruct-Q4_0.gguf
...
ggml-hex: Hexagon backend (experimental) : allocating new registry : ndev 1
ggml-hex: Hexagon Arch version v79
@@ -219,15 +211,20 @@ ggml-hex: new session: HTP0 : session-id 0 domain-id 3 uri file:///libggml-htp-v
| ---------------| ---------: | -----: | ---------- | --: | ------: | ------: | ---: | ----: | ------------: |
| llama 1B Q4_0 | 729.75 MiB | 1.24 B | HTP | 99 | 4 | 128 | 0 | pp128 | 169.42 ± 1.75 |
| llama 1B Q4_0 | 729.75 MiB | 1.24 B | HTP | 99 | 4 | 128 | 0 | tg64 | 51.54 ± 1.13 |
build: 6a8cf8914 (6733)
```
## Environment variables
- `GGML_HEXAGON_NDEV=1`
Controls the number of devices/sessions to allocate. The default is 1.
Most quantized models under 4B fit into a single session; an 8B model needs two, and a 20B model needs four.
- `GGML_HEXAGON_DEVICES` (default: not set, defaults to HTP0 session)
Controls which NPU devices and sessions to allocate. Can be configured as:
- A single integer `N`: Allocates `N` sessions named `HTP0`, `HTP1`, ..., `HTP<N-1>` (behaves identically to `GGML_HEXAGON_NDEV=N`).
- A comma-separated list of device names in `HTP<physical_idx>:<virtual_idx>` format (or legacy `HTP<idx>` format). For example, `HTP0:0,HTP0:1` creates two virtual
sessions on the first physical NPU (useful for memory limits). `HTP0:0,HTP1:0` allocates one session on each of the two physical NPUs
on a dual-NPU device.
- `GGML_HEXAGON_NDEV` (deprecated)
Replaced by `GGML_HEXAGON_DEVICES`. Controls the number of virtual sessions to allocate on physical NPU `0`.
Allocates sessions named `HTP0`, `HTP1`, etc.
- `GGML_HEXAGON_NHVX=0`
Controls the number of HVX hardware threads to use. The default is all (actual number varies depending on the hardware version).
@@ -255,26 +252,17 @@ build: 6a8cf8914 (6733)
- `2` Extended profile with per-op `usecs`, `cycles` and default PMU counter data
- `0x1,...,0x8` Extended profile with per-op `usecs`, `cycles` and custom PMU counter data
The logging output can be either saved into a file for post-processing or it can be piped directly into the post-processing tool to generate the report.
The logging output can be either saved into a file for post-processing or it can be piped directly into the post-processing tool
to generate the report.
Examples:
`GGML_HEXAGON_PROFILE=1 llama-completion ... |& ./scripts/snapdragon/ggml-hexagon-profile.py -`
- `GGML_HEXAGON_OPSTAGE=0x0`
Allows enabling specific stages of the Op processing pipeline:
- `0x1` Enable Op Queue (i.e., queuing Ops into NPU)
- `0x2` Enable Op Compute (MUL_MAT, etc.)
Examples:
`GGML_HEXAGON_OPSTAGE=0x1 llama-completion ...` - Ops are enqueued to the NPU but dma & compute are disabled
`GGML_HEXAGON_OPSTAGE=0x3 llama-completion ...` - Full queuing and processing of Ops (default)
`GGML_HEXAGON_PROFILE=1 ./scripts/snapdragon/run.py --target adb -- llama-cli ... |& ./scripts/snapdragon/ggml-hexagon-profile.py -`
- `GGML_HEXAGON_OPFILTER=regex`
Allows filtering (disabling) Ops that match the regex pattern:
Examples:
`GGML_HEXAGON_OPFILTER="FLASH_ATTN_EXT" llama-completion ...` - Disable Flash Attention on Hexagon (falls back to CPU or GPU)
`GGML_HEXAGON_OPFILTER="ADD\|SUB" llama-completion ...` - Disable ADD and SUB on Hexagon (fall back to CPU or GPU)
`GGML_HEXAGON_OPFILTER="FLASH_ATTN_EXT" ./scripts/snapdragon/run.py --target adb -- llama-cli ...` - Disable Flash Attention on Hexagon (falls back to CPU or GPU)
`GGML_HEXAGON_OPFILTER="ADD\|SUB" ./scripts/snapdragon/run.py --target adb -- llama-cli ...` - Disable ADD and SUB on Hexagon (fall back to CPU or GPU)