Lumpiasty/llama.cpp
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

hexagon: support for multi-NPU devices (IQ9, IQ10) and fully asynchronous backend (#26501)

Browse Source 
* hexagon: use non-host bufs by default and make the backend fully async

* hex-hb: remove optional hostbuf support and fix async copy

* hex-unary: relax supported unary check

* hex-bufs: use same get_alignment for host bufs

* snapdragon: bump android_platform to 34

* hex-rows: super hacky get/set rows for q8_0

* hex-get-rows: fix q8_0

* hex-get-rows: supprot for f16 and cleanup for q8_0

* hex-get-rows: generic macros and specialized thread funcs

* hex-get-rows: add DMA pipeline, vtcm_layout and kernel params

* hex-set-rows: fix q8_0 support, add dma and tracing

* hex-tests: override nmse threshold for HTP of Q8_0 quants

* hex-fa: add support for Q8_0 with inplace dequantizers

* hex-get-rows: simplify type dispatch

* hex-rows: simplify GET/SET_ROWS DMA pipeline

* hex-async: add events, set/get-tensor-async and rest of the async api support

* hex-repack: use slice instead of expert in repack functions

* hex-cpy: update event/async-cpy logging

* hex-set-rows: optimize smaller tensors

* hex-geglu: fix perf regression with larger tensors

* hex-get-rows: add missing header

* hex-set-rows: add missing header

* hex-bufs: ressurect GGML_HEXAGON_HOSTBUF but disable it by default

* hexagon: do not reject ops with non-heaxon buffers

* hex-get-rows: apply >=32 restriction only for q8_0

* hex-res: bump vtcm acquire timeout to 10 seconds

* hex-bufs: add support for cloning buffers between sessions to speed up tensor copies

* hex-async: rework event recording and batch flushing and integrate with meta backend

* hex-bufs: improved handling of repacked tensors

* hex-repack: handle get_tensor_2d offsets

* hex-dev: add support for devices with multiple NPUs

* hex-sync: add support for sync tokens to synchronize npu devices for async splits

* hex-mmap: cleanup mmap calls and add a retry for robustness

* hex-sync: add failsafe if sync wait gets stuck

* hex-sync: use sync_seq to check for completed events

* hex-sync: rotate tokens for extra robustness

* hex-devs: add supprot for legacy device names for now

* hex-bufs: add support for auto-cloning buffers from diff sessions

* hex-fusion: simplify and optimize htp-opnode fusion handling

* hex-sync: override opnode name so that it shows up in the profiles

* hex-trace: update scripts to handle multiple devices

* hex-sync: bump the size of the opbatch queue and number of sync tokens

* hex-cpy-sync: do not explicitly flush opbatches in cpy_tensor_async and add support for cpy-dma

* hex-sync: add graph-flush threshold to avoid single op batches

* hex-sync: add sync_peer so that we can flush peers we depend on during cross-device ops

* hex-bufs: introduce tensor->extra and shadow_bufs for repacking

* hex-l2: flush tiny tensors inline

* hex-sync: use explicit l2flush for sync tokens

* hex-extra: track weight flags via tensor extra

* hex-fence: rename sync to fence

* hex-repack: proper handling of set-tensor-2d in the shadow_buf

* hex-trace: remove obsolete opstage mask that we used for profiling

* hex-env: remove obsolete use_hmx variable

* hexagon: new unified run.py and build.py and updated docs

* snapdragon: update run script to auto-escapt test-backend-op -p argument

* hex-scripts: fix trailing spaces

* hex-scripts: fix flake8 warnings

* snapdragon: cleanup dst lib/bin dirs before copying new build

* hex-ops: add support for allreduce

* hex-ar: improved allreduce with dma pipeline

* hex-ar: align macros

* hex-ar: consistent use of fence_seq

* hex-ar: add AR_SELECT env var to select ALLREDUCE kernel or fallback

* hex-ar: add proper synchronize handling for ALLREDUCE

* hex-opbatch: looks like we now just rely on backend.synchronise to flush the batches, no need to flush them by threshold

* hex-ar: bump block size to improve dma efficiency

* hex-ar: fused ALLREDUCE+ADD

* hex-ar: cleaner fence buffer management

* hex-ar: futher allreduce tweaking to remove race conditions

* hex-ar: add simple solver and remove non-dma kernels

* hex-ar: add row-broadcast to fuse with bias ADD

* hex-fence: pass seq numbers via op_params

* hex-ar: allow for both entry/exit seq for completing entry wait

* hex-ar: align macros

* hex-ar: do not refetch broadcast row

* hex-fusion: move all fusion into opbatch::add_op for consistency with ALLREDUCE and things

* hex-fusion: fix incorrect MUL_MAT reordering

* hex-mm: make fused 2x and 3x matmuls more generic

* hex-fusion: move tensor fusion tagging to graph_compute

* hexagon: make sure to copy tensor->extra by value

* hex-get-rows: fix offset calc with row-chunking

* hex-repack: get_tensor_2d fixes for non-zero offsets

* snapdragon: make profile/trace scripts more robust and donot mix stdout/stderr by default

* hex-devices: use legacy device nameing by default to ease the transition

* hex-devices: hardcode CDSP domain IDs for current devices for now

* hex-optrace: improve multi-NPU timestamp alignment and overall handling of cycle values

* hex-optrace: more robust handling of the fence events
This commit is contained in:
Max Krasnyansky
2026-08-26 18:46:50 -07:00
committed by GitHub
parent 925e117994
commit 192067b72d
44 changed files with 5314 additions and 3139 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/snapdragon/adb/run-bench.sh
-49
View File
@@ -1,49 +0,0 @@
#!/bin/sh
#
# Basedir on device
basedir=/data/local/tmp/llama.cpp
branch=.
[ "$B" != "" ] && branch=$B
adbserial=
[ "$S" != "" ] && adbserial="-s $S"
adbhost=
[ "$H" != "" ] && adbhost="-H $H"
model="Llama-3.2-3B-Instruct-Q4_0.gguf"
[ "$M" != "" ] && model="$M"
device="HTP0"
[ "$D" != "" ] && device="$D"
verbose=
[ "$V" != "" ] && verbose="GGML_HEXAGON_VERBOSE=$V" cli_opts="$cli_opts -v"
profile=
[ "$PROF" != "" ] && profile="GGML_HEXAGON_PROFILE=$PROF" cli_opts="$cli_opts -v"
opmask=
[ "$OPSTAGE" != "" ] && opmask="GGML_HEXAGON_OPSTAGE=$OPSTAGE"
nhvx=
[ "$NHVX" != "" ] && nhvx="GGML_HEXAGON_NHVX=$NHVX"
ndev=
[ "$NDEV" != "" ] && ndev="GGML_HEXAGON_NDEV=$NDEV"
hb=
[ "$HB" != "" ] && hb="GGML_HEXAGON_HOSTBUF=$HB"
set -x
adb $adbserial $adbhost shell " \
cd $basedir; \
LD_LIBRARY_PATH=$basedir/$branch/lib \
ADSP_LIBRARY_PATH=$basedir/$branch/lib \
$ndev $nhvx $opmask $verbose $profile $hb ./$branch/bin/llama-bench --device $device --load-mode none -m $basedir/../gguf/$model \
--poll 1000 -t 6 --cpu-mask 0xfc --cpu-strict 1 \
--ubatch-size 1024 -fa 1 -ngl 99 $cli_opts $@ \
"
scripts/snapdragon/adb/run-cli.sh
-78
View File
@@ -1,78 +0,0 @@
#!/bin/sh
#
# Basedir on device
basedir=/data/local/tmp/llama.cpp
cli_opts=
branch=.
[ "$B" != "" ] && branch=$B
adbserial=
[ "$S" != "" ] && adbserial="-s $S"
adbhost=
[ "$H" != "" ] && adbhost="-H $H"
model="Llama-3.2-3B-Instruct-Q4_0.gguf"
[ "$M" != "" ] && model="$M"
device="HTP0"
[ "$D" != "" ] && device="$D"
verbose=
[ "$V" != "" ] && verbose="GGML_HEXAGON_VERBOSE=$V" cli_opts="$cli_opts -v"
sched=
[ "$SCHED" != "" ] && sched="GGML_SCHED_DEBUG=2" cli_opts="$cli_opts -v"
profile=
[ "$PROF" != "" ] && profile="GGML_HEXAGON_PROFILE=$PROF" cli_opts="$cli_opts -v"
opmask=
[ "$OPSTAGE" != "" ] && opmask="GGML_HEXAGON_OPSTAGE=$OPSTAGE"
nhvx=
[ "$NHVX" != "" ] && nhvx="GGML_HEXAGON_NHVX=$NHVX"
hmx=
[ "$HMX" != "" ] && hmx="GGML_HEXAGON_USE_HMX=$HMX"
ndev=
[ "$NDEV" != "" ] && ndev="GGML_HEXAGON_NDEV=$NDEV"
hb=
[ "$HB" != "" ] && hb="GGML_HEXAGON_HOSTBUF=$HB"
opbatch=
[ "$OB" != "" ] && opbatch="GGML_HEXAGON_OPBATCH=$OB"
opqueue=
[ "$OQ" != "" ] && opqueue="GGML_HEXAGON_OPQUEUE=$OQ"
opflt=
[ "$OF" != "" ] && opflt="GGML_HEXAGON_OPFILTER=$OF"
vmem=
[ "$VM" != "" ] && opflt="GGML_HEXAGON_VMEM=$VM"
mbuf=
[ "$MB" != "" ] && opflt="GGML_HEXAGON_MBUF=$MB"
vmem=
[ "$VM" != "" ] && vmem="GGML_HEXAGON_VMEM=$VM"
mbuf=
[ "$MB" != "" ] && mbuf="GGML_HEXAGON_MBUF=$MB"
set -x
adb $adbserial $adbhost shell " \
cd $basedir; ulimit -c unlimited; \
LD_LIBRARY_PATH=$basedir/$branch/lib \
ADSP_LIBRARY_PATH=$basedir/$branch/lib \
$verbose $sched $opmask $profile $nhvx $hmx $ndev $hb $opbatch $opqueue $opflt $vmem $mbuf \
./$branch/bin/llama-cli --load-mode none -m $basedir/../gguf/$model \
--poll 1000 -t 6 --cpu-mask 0xfc --cpu-strict 1 \
--ctx-size 8192 --ubatch-size 1024 -fa on \
-ngl 99 --device $device $cli_opts $@ \
"
scripts/snapdragon/adb/run-completion.sh
-86
View File
@@ -1,86 +0,0 @@
#!/bin/sh
#
# Basedir on device
basedir=/data/local/tmp/llama.cpp
cli_opts=
branch=.
[ "$B" != "" ] && branch=$B
adbserial=
[ "$S" != "" ] && adbserial="-s $S"
adbhost=
[ "$H" != "" ] && adbhost="-H $H"
model="Llama-3.2-3B-Instruct-Q4_0.gguf"
[ "$M" != "" ] && model="$M"
device="HTP0"
[ "$D" != "" ] && device="$D"
verbose=
[ "$V" != "" ] && verbose="GGML_HEXAGON_VERBOSE=$V" cli_opts="$cli_opts -v"
sched=
[ "$SCHED" != "" ] && sched="GGML_SCHED_DEBUG=2" cli_opts="$cli_opts -v"
profile=
[ "$PROF" != "" ] && profile="GGML_HEXAGON_PROFILE=$PROF" cli_opts="$cli_opts -v"
opmask=
[ "$OPSTAGE" != "" ] && opmask="GGML_HEXAGON_OPSTAGE=$OPSTAGE"
nhvx=
[ "$NHVX" != "" ] && nhvx="GGML_HEXAGON_NHVX=$NHVX"
hmx=
[ "$HMX" != "" ] && hmx="GGML_HEXAGON_USE_HMX=$HMX"
ndev=
[ "$NDEV" != "" ] && ndev="GGML_HEXAGON_NDEV=$NDEV"
hb=
[ "$HB" != "" ] && hb="GGML_HEXAGON_HOSTBUF=$HB"
opbatch=
[ "$OB" != "" ] && opbatch="GGML_HEXAGON_OPBATCH=$OB"
opqueue=
[ "$OQ" != "" ] && opqueue="GGML_HEXAGON_OPQUEUE=$OQ"
oppoll=
[ "$OP" != "" ] && oppoll="GGML_HEXAGON_OPPOLL=$OP"
opflt=
[ "$OF" != "" ] && opflt="GGML_HEXAGON_OPFILTER=$OF"
opfuse=
[ "$OC" != "" ] && opfuse="GGML_HEXAGON_OPFUSION=$OC"
vmem=
[ "$VM" != "" ] && vmem="GGML_HEXAGON_VMEM=$VM"
mbuf=
[ "$MB" != "" ] && mbuf="GGML_HEXAGON_MBUF=$MB"
mmsel=
[ "$MM" != "" ] && mmsel="GGML_HEXAGON_MM_SELECT=$MM"
fasel=
[ "$FA" != "" ] && fasel="GGML_HEXAGON_FA_SELECT=$FA"
set -x
adb $adbserial $adbhost shell " \
cd $basedir; ulimit -c unlimited; \
LD_LIBRARY_PATH=$basedir/$branch/lib \
ADSP_LIBRARY_PATH=$basedir/$branch/lib \
$verbose $sched $opmask $profile $nhvx $hmx $ndev $hb $opbatch $opqueue $oppoll $opflt $opfuse $vmem $mbuf $mmsel $fasel \
./$branch/bin/llama-completion --load-mode none -m $basedir/../gguf/$model \
--poll 1000 -t 6 --cpu-mask 0xfc --cpu-strict 1 \
--ctx-size 8192 --ubatch-size 1024 -fa on \
-ngl 99 --device $device $cli_opts $@ \
"
scripts/snapdragon/adb/run-mtmd.sh
-71
View File
@@ -1,71 +0,0 @@
#!/bin/sh
#
# Basedir on device
basedir=/data/local/tmp/llama.cpp
cli_opts=
branch=.
[ "$B" != "" ] && branch=$B
adbserial=
[ "$S" != "" ] && adbserial="-s $S"
adbhost=
[ "$H" != "" ] && adbhost="-H $H"
model="gemma-3-4b-it-Q4_0.gguf"
[ "$M" != "" ] && model="$M"
mmproj="mmproj-F16.gguf"
[ "$MMPROJ" != "" ] && mmproj="$MMPROJ"
image=
[ "$IMG" != "" ] && image="$IMG"
device="HTP0"
[ "$D" != "" ] && device="$D"
verbose=
[ "$V" != "" ] && verbose="GGML_HEXAGON_VERBOSE=$V"
experimental="GGML_HEXAGON_EXPERIMENTAL=1"
[ "$E" != "" ] && experimental="GGML_HEXAGON_EXPERIMENTAL=$E"
sched=
[ "$SCHED" != "" ] && sched="GGML_SCHED_DEBUG=2" cli_opts="$cli_opts -v"
profile=
[ "$PROF" != "" ] && profile="GGML_HEXAGON_PROFILE=$PROF"
opmask=
[ "$OPSTAGE" != "" ] && opmask="GGML_HEXAGON_OPSTAGE=$OPSTAGE"
nhvx=
[ "$NHVX" != "" ] && nhvx="GGML_HEXAGON_NHVX=$NHVX"
hmx=
[ "$HMX" != "" ] && hmx="GGML_HEXAGON_USE_HMX=$HMX"
ndev=
[ "$NDEV" != "" ] && ndev="GGML_HEXAGON_NDEV=$NDEV"
# MTMD backend device for vision model (defaults to CPU if not set)
mtmd_backend=
[ "$MTMD_DEVICE" != "" ] && mtmd_backend="MTMD_BACKEND_DEVICE=$MTMD_DEVICE"
set -x
adb $adbserial $adbhost shell " \
cd $basedir; ulimit -c unlimited; \
LD_LIBRARY_PATH=$basedir/$branch/lib \
ADSP_LIBRARY_PATH=$basedir/$branch/lib \
$verbose $experimental $sched $opmask $profile $hmx $nhvx $ndev $mtmd_backend \
./$branch/bin/llama-mtmd-cli --load-mode none -m $basedir/../gguf/$model \
--mmproj $basedir/../gguf/$mmproj \
--image $basedir/../gguf/$image \
--poll 1000 -t 6 --cpu-mask 0xfc --cpu-strict 1 \
--ctx-size 8192 --ubatch-size 1024 -fa on \
-ngl 99 --device $device -v $cli_opts $@ \
"
scripts/snapdragon/adb/run-tool.sh
-72
View File
@@ -1,72 +0,0 @@
#!/bin/sh
#
# Basedir on device
basedir=/data/local/tmp/llama.cpp
cli_opts=
branch=.
[ "$B" != "" ] && branch=$B
adbserial=
[ "$S" != "" ] && adbserial="-s $S"
adbhost=
[ "$H" != "" ] && adbhost="-H $H"
device="HTP0"
[ "$D" != "" ] && device="$D"
verbose=
[ "$V" != "" ] && verbose="GGML_HEXAGON_VERBOSE=$V"
sched=
[ "$SCHED" != "" ] && sched="GGML_SCHED_DEBUG=2" cli_opts="$cli_opts -v"
profile=
[ "$PROF" != "" ] && profile="GGML_HEXAGON_PROFILE=$PROF"
opmask=
[ "$OPSTAGE" != "" ] && opmask="GGML_HEXAGON_OPSTAGE=$OPSTAGE"
nhvx=
[ "$NHVX" != "" ] && nhvx="GGML_HEXAGON_NHVX=$NHVX"
hmx=
[ "$HMX" != "" ] && hmx="GGML_HEXAGON_USE_HMX=$HMX"
ndev=
[ "$NDEV" != "" ] && ndev="GGML_HEXAGON_NDEV=$NDEV"
hb=
[ "$HB" != "" ] && hb="GGML_HEXAGON_HOSTBUF=$HB"
opbatch=
[ "$OB" != "" ] && opbatch="GGML_HEXAGON_OPBATCH=$OB"
opqueue=
[ "$OQ" != "" ] && opqueue="GGML_HEXAGON_OPQUEUE=$OQ"
oppoll=
[ "$OP" != "" ] && oppoll="GGML_HEXAGON_OPPOLL=$OP"
opfuse=
[ "$OC" != "" ] && opfuse="GGML_HEXAGON_OPFUSION=$OC"
mmsel=
[ "$MM" != "" ] && mmsel="GGML_HEXAGON_MM_SELECT=$MM"
fasel=
[ "$FA" != "" ] && fasel="GGML_HEXAGON_FA_SELECT=$FA"
set -x
tool=$1; shift
adb $adbserial $adbhost shell " \
cd $basedir; ulimit -c unlimited; \
LD_LIBRARY_PATH=$basedir/$branch/lib \
ADSP_LIBRARY_PATH=$basedir/$branch/lib \
$verbose $sched $opmask $profile $nhvx $hmx $ndev $hb $opbatch $opqueue $oppoll $opfuse $mmsel $fasel ./$branch/bin/$tool $@ \
"