Commit Graph
523 Commits
Author SHA1 Message Date
Todor BoinovskiandMax Krasnyansky 20a51f24a8 hexagon: add DSV4_HC_PRE and DSV4_HC_POST ops (#30953)
* hexagon: add support for DSV4_HC_PRE

* hexagon: hsv4-hc-pre plain/gated split kernel and l2 prefetch optimization

* hex-dsv4-hc-pre: double-buffered DMA for prefill

* hex-dsv4-hc-pre: refactor DMA block setup

* hex-dsv4-hc-pre: precompute kernel params

* hex-dsv4-hc-pre: support for n_hc != 4

* hexagon: add support for DSV4_HC_POST

* hex-dsv4-hc-pre: refactor gated PRE prefill DMA to dst writeback after each block

* hex-dsv4-hc-pre: remove in-loop vector spills and redundant device-side checks

* hex-run: add support for test-llama-archs tool

* hex-dsv4: refactoring and optimizations

Remove non-dma paths.
Clean up DMA pipelines and dma_addr_t casts.
Add dims collapsing support.

Tested with test-llama-archs -a qwen4ext and test-backend-ops

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-11-03 08:26:18 -08:00
Aman Gupta 74101b3cda ci : update the HIP VGPR spill ignore list for the kv_rows FA kernel argument (#31550)
The kv_rows argument added to the FA kernels in #29510 changes their mangled names, so the known spills were no longer ignored.
2026-11-03 07:54:47 +02:00
Alessandro de Oliveira Faria (A.K.A.CABELO) f6312ad75e vendor : update miniaudio to 0.11.26 (#31459)
[no ci]
2026-10-31 23:37:20 +01:00
Alessandro de Oliveira Faria (A.K.A.CABELO) 1ab23c33b7 vendor : update cpp-httplib to 0.64.0 (#31400) 2026-10-30 11:49:58 +01:00
uvos 78febbad9d HIP/CUDA: enable sparse attn on CDNA (#31259)
* HIP/CUDA: enable sparse attn on CDNA

* Use __ballot for __ballot_sync on old ROCm versions

* Ignore new large head fattn mma variant
2026-10-29 10:14:11 +02:00
Georgi Gerganov 5d7793d6fc sync : ggml 2026-10-28 12:36:51 +02:00
Mengsheng Wu 6c44591049 snapdragon: switch QDC artifact upload and log download to v2 API (#31240)
Signed-off-by: Mengsheng Wu <mengshengwu@outlook.com>
2026-10-27 18:06:41 +01:00
Max Krasnyansky f0fb5984b0 hexagon: generic dims collapse/reshape for unary, binary and matmul ops (#31071)
* hex-tools: support for --compare option in the profile.py

* hex-unary: improve collapse logic to also do splits if nrows is small

* hex-colapse: handle all colapsing at the host and make it generic

* hex-colapse: remove leftover unary helper
2026-10-25 11:11:48 +02:00
Christian Kastner 707616feb5 ci: Fix --strict mode for API check (#30623)
abi-compliance-checker was running in strict mode unconditionally,
rather than honoring the CLI flag.
2026-10-20 13:38:45 +03:00
Max Krasnyansky 8466395353 hexagon: update to snapdragon-toolchain v0.8 and enable hexagon v85 (#30854)
* hex-build: bump toolchain, enable v85

* hex-build: bump toolchain to v0.8 in the CI and build scripts

* hex-build: add a little python script for swiv needed on some iot devices

* hex-build: fix flake8 warnings

* hex-arch: update version cap to v85

* hex-inspect: flexible TOOLS_ROOT

* hex-build: update to snake_case naming

* hex-build: update leftover old toolchain refs

* hex-target: update build and run scripts to support ubuntu with adb transport
2026-10-19 23:49:59 -07:00
7adf160c41 server: add --connect, allow p2p access to server outside LAN (#28277)
* init

* wip

* wire it up

* make it clear that it's experimental

* add props.connect_code

* nits

* fix windows build

* server: do not advertise a connect code without the tunnel

--connect-code on its own had no effect other than being served in
/props, so the Web UI showed a remote access code that no tunnel was
serving. Clear it when --connect is off, and say so once at startup.

The teardown comment claimed that terminate() closes the child stdout.
It does not: the log thread leaves fgets() only once every writer of
that pipe is gone, so a descendant of llama-connect would block the
join forever. Describe the real contract instead.

Also restore the --reasoning-preserve default in the generated docs.

* Update tools/connect/CMakeLists.txt

Co-authored-by: Adrien Gallouët <angt@huggingface.co>

* server: dial llama-connect on the bound port and a reachable host

Pass ctx_http.port so --port 0 and the GCP override work, pick the
dial host from --host (wildcards map to loopback, IPv6 is bracketed)
and refuse --connect when only unix sockets are bound.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
Co-authored-by: Adrien Gallouët <angt@huggingface.co>
2026-10-20 00:23:14 +02:00
Alessandro de Oliveira Faria (A.K.A.CABELO) 7951f601e8 vendor : update cpp-httplib to 0.62.0 (#30785) 2026-10-19 20:23:23 +08:00
Georgi Gerganov 7db0d3d781 pi : turn .pi/gg into a system prompt addendum (#30762)
.pi/gg/SYSTEM.md repeated the full agent ruleset. Move it to .pi/gg/APPEND_SYSTEM.md and keep only the llama.cpp-specific rules, so it can be appended to a main system prompt with --append-system-prompt.

Drop the .pi/SYSTEM.md symlink setup from wc2wt.sh and pr2wt.sh. Worktrees no longer need it now that the folder holds an addendum instead of a full prompt.

Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (RTX 5090 32GB, DDR5 128GB)
2026-10-19 09:11:13 +03:00
Sandro Hanea 83f9389419 ci : added ABI reports as github artifact (#30619) 2026-10-17 11:41:53 +02:00
Max Krasnyansky b797b9ca81 hexagon: tensor repacking rework and support for tiled and flat quantized and fp weights (#30635)
* hex-repack: Remove need_repack hack and streamline the rules -- breaks GET_ROWS tests for K-Quants, will fix in the next round

Add support for GET_ROWS with tiled Q5_K
Add support for raw k-quant tensors

* hex-get-rows: vectorize raw k-quant ukernels

* hex-binary: refactor dma-loop thread functions (no need for macros, etc)

* hex-quant: move/cleanup flat and tiled dequant ukernels

* hex-repack: add support for repacking/tiling F16/F32/BF16 2D weights

* hex-get-rows: remove redundant cast

* hex-repack: proper filtering for BF16 that were leaking through during the tests

* hex-repack: streamling tiling rules (broken MM for now)

* hex-repack: fix mmid after tiling updates

* hex-repack: rename repacked to tiled in all flags/funcs/kernels/vars

* hex-mm: fix valid_rows calc in various kernels

* hex-mm: remove unused hvx_mm_nx_2d_flat

* hex-mm: cleanup and optimize hvx-tiled vec_dot_f16

* hex-mm: another pass at hvx-tiled f16 matmul

* hex-get-rows: optimize small src1

* hex-get-rows: one more pass at recovering get-rows perf regression

* hex-repack: export GGML_HEXAGON_RT_REPACK via run.py

* hex-mm-tiled: fix accuracy regression with hmx-tiled-f16 matmul

* hex-mm: fold tiled_f16 checks into the general flow

* hex-get-rows: remove dead checks for buffer_is_hexagon and tiling during supports_op

* hex-quant: proper tail handling in new dequant ukernels

* hex-get-rows: remove unused code and cleanup support checks

* hex-mm: cleanup ne1 padding/tiling checks

* hex-mm: cleanup fusion checks for tiled tensors

* hex-repack: add a little can_repack helper to reduce duplication

* hex-repack: never repack host buffers

* hex-repack: factor out repack functions into separate .h and .cpp

* hex-tensor: cleanup tensor ifaces for tiled/padded sizes

* hex-repack: update k-quant tiling comments and cleanup #defines

* hex-dcache: factor out l2/dcache dirty tracking and flushing

* hex-tensor: update docs and remove #defs duplication

* hex-tensor: remove unused/incorrect get_data interface

* hex-get-rows: cleanup vtcm layout build

* tests: disable huge MM tests

* hex-dcache: add missing header
2026-10-15 20:30:11 -07:00
Georgi Gerganov a11b6530b5 scripts : add ci-status.sh to list active CI runs (#30586)
Small helper around the GitHub Actions API: lists the workflow runs that are not
finished yet (in_progress, queued, pending, waiting, requested), grouped by status.

Supports selecting runs by status, by branch (server-side, to reduce API traffic)
or by a regex on the branch name, and can cancel the selected runs via --kill.

Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
2026-10-15 16:28:09 +03:00
Georgi Gerganov c0cb53ac4e sync : ggml 2026-10-15 16:20:14 +03:00
Sigbjørn Skjæret 711780ffc7 ci : bump ty to 0.0.88 (#30587)
* bump ty to 0.0.88

* arange -> linspace.tolist() to pass expected type
2026-10-15 14:15:55 +02:00
Adrien Gallouët 3eb101360f vendor : update cpp-httplib to 0.61.0 (#30494)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-10-14 08:28:24 +02:00
Sandro Hanea 10da8c463f unicode : add opt-in NFC normalization for Unigram tokenizers (#30390)
* unicode : add opt-in NFC normalization for Unigram tokenizers

Note: used help from Codex, Sol 6.1 Medium

* Fixing initial bot comments + linter error (using existing out instead of print)

* removed test_nfc_tokenizer
2026-10-14 08:33:24 +03:00
Pranav DhinakarandMax Krasnyansky ec0bfac179 hexagon: add configurable HTP DCVS/HMX power mode (#30423)
* hexagon: add configurable HTP DCVS/HMX power mode

* hexagon: address PR review feedback (drop CLI flag, align QAIRT power modes, comment/log cleanup)

* hexagon: add --hex-power-mode flag to run.py

Hooks GGML_HEXAGON_POWER_MODE into the unified run.py runner, following
the existing --hex-nhvx pattern. The env var must be set before the
process touches the Hexagon backend registry (see comment in
ggml_hexagon_init), so this is exposed as a runner flag rather than a
common/arg.cpp CLI option.

* hexagon: keep HTP awake for DCVS power profiles

* Apply suggestion from @max-krasnyansky

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-10-13 07:47:55 -07:00
8bb3cf0d21 hexagon: optimize CPY/CONCAT — round 3, short-rows edition (#30341)
* hexagon: move the converting CPY in blocks of rows per DMA descriptor so short rows are not DMA-latency bound

* hex-cpy: cleanup cpy cursor handling

* hex-cpy-concat: introduce threaded dma/hvx gather-row helper

Co-authored-by: Ruslan Karymov <67752306+karusrus@users.noreply.github.com>

* hex-cpy: remove leftover rejections for extended buffers

* hex-cpy: relax the gate for fast-path

* hex-cpy: remove scalar divs and vtcm access and use explicit gather_sync

* hex-cpy: add proper support for mdev work splitting to fastpath

* hex-cpy: optimize dense gather-row tasks

* hex-cpy: unroll gather-rows loops

* hex-cpy: add fastpath for f16

* hex-cpy-concat: support for colapsing 3/4D dims

* hex-cpy: double-buffer support

* hex-cpy: factor out gather-rows into more of an op by itself

* hex-cpy-concat: dedup kernel-params

* hex-cpy-concat: step-down nr to fit instead of hard fail

* hex-gather: replace div with a mask

* hex-gather-rows: cleanup dma pipeline and remove the stalls

* hex-gather: specialize compute funcs

* hex-gather: reduce scalar reg.spills and func calls

* hex-gather: do not recompute rows-per-thread on the npu

* hex-build: add support for building/installing just hexagon

* hex-cpy-concat: remove redundant barrier in concat and move cpy barrier under fence

* hex-gather: annotations for dma-out

* hex-mdev: make all key vtcm_acquire sync state atomic (fixes random hangs in mdev setups)

---------

Co-authored-by: Yiwei Shao <yiwei@aizip.ai>
Co-authored-by: Ruslan Karymov <67752306+karusrus@users.noreply.github.com>
2026-10-12 07:39:55 -07:00
Xuan-Son Nguyen 10a60cf303 vendor: apply deep nested json patch from upstream (#30253) 2026-10-10 00:07:44 +02:00
Xuan-Son Nguyen b42b7e6d30 server: use port 9931 by default (#30159)
* server: use port 9931 by default

* revert unrelated changes
2026-10-09 10:40:03 +02:00
Pascal 2c564f43df ci: fix HIP quality check by ignoring the 320/256 FA kernel spill (#30212) 2026-10-09 10:29:22 +02:00
Adrien Gallouët 03aa006acb vendor : update cpp-httplib to 0.60.1 (#30134)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-10-08 10:57:47 +02:00
Max KrasnyanskyandJhen-Jie Hong 9c2e0e491a hexagon: enable alloc_buffer_n (#30126)
* hex-bufs: add support for alloc_buffer_n

* hex-bufs: add support for splitting large tensors into separate buffers

* hex-bufs: update GGML_HEXAGON_MBUF to accept three values dyn,static,total

* hex-bufs: bump dyn. default to 512MB since 128MB causes perf regressions with big MOEs

* hex-run: add --no-embd-offload option to simplify command lines on devices that need it

* Update scripts/snapdragon/run.py

Co-authored-by: Jhen-Jie Hong <iainst0409@gmail.com>

---------

Co-authored-by: Jhen-Jie Hong <iainst0409@gmail.com>
2026-10-07 18:30:06 -07:00
Alessandro de Oliveira Faria (A.K.A.CABELO) b7dafa01e5 vendor : update cpp-httplib to 0.60.0 (#30081) 2026-10-07 11:18:33 +02:00
Daniel Bevenius d7a695ef67 scripts : limit apiabi checks to libllama and libmtmd (#30038)
Refs: https://github.com/ggml-org/llama.cpp/pull/29997#issuecomment-5998572938
2026-10-06 11:42:59 +02:00
Georgi Gerganov c250304960 ci : skip container re-tagging when Require Docker is disabled (#30008)
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
2026-10-05 21:22:08 +03:00
8345f33395 hexagon: matmul and flash-atten scalability updates (#29974)
* hexagon: head-parallel flash_attn partitioning for row-split multicore

In row-split mode each core computes its output row shard of every
MUL_MAT, but flash_attn was previously partitioning by Q tokens
(flat qrow split) instead of by heads. This forced every core to
read the full KV cache (all n_kv_heads), negating the memory
bandwidth benefit of multicore on flash_attn.

Change both HMX and HVX flash_attn kernels to partition by KV heads
when n_kv_heads is divisible by n_cores: core i processes heads
[i*n_kv_heads/N, (i+1)*n_kv_heads/N) exclusively, reading only its
head shard of the KV cache. Falls back to the original token-block
split when n_kv_heads % n_cores != 0 (e.g. Gemma-4 with 2 KV heads
on 4 cores).

Controlled by GGML_HEXAGON_FA_HEAD_SPLIT (default 1 = on).
The flag is packed into bit 1 of the existing is_dst_fp32 kparams
byte to stay within the 128-byte kernel_params blob limit.

Measured gains at 4c row-split (PP t/s, ubatch=1024):
  Qwen3-0.6B:    6977 -> 11026  (+58%)
  llama-3.2-3B:  3717 ->  5522  (+49%)
  Qwen3.5-4B:    2739 ->  2855   (+4%)
  Gemma-4 MoE:   no change (MoE FFN dominates, fallback path)

TG is unchanged (flash_attn is a small fraction of decode time
relative to the matmul+barrier cost per layer).

* hex-fa: cleanup kern_params and head-split selection

* hex-fa: add -fa-head-split option to run.py

* hex-mdev: update matmul solver to account for reduced work in row-split scenarios

* hex-mmid: better work splitting by expers in multi-dev scenarios

* hex-fa: update HMX gating based on the model/n-hvx/ctx-len sweep

* hex-fa: precompute softcap/scale on the host

* hexagon: flatten matmul into 2d to use HMX in multi-sequence

* hex-mm: cleanup kparams and use collapse to 3/4D -> 2D mapping

* hex-mm: fix typo in collapse fallback

* hex-mm: another pass at consistent naming for act tensors

* hex-mm: add support for colapsing dims in fused matmuls

* hex-build: fix WoS build errors

* hex-mm: make sure to enforce dst stride in can_collapse

* hex-fa: add a onliner commit for head-split check

* hex-fa: remove unused local head_split var

* hex-fa: tighten up can_split checks

* hex-mm: update unfused paths to use act instead src1

* hex-mm: make sure to check all dsts for splitting

* hexagon: fix the second weight chunk address in the batched HMX matmul prologue

* hexagon: F16 activation and ragged N in the HMX matmul

* hex-mm: tighten the ragged/split checks in mdev cases

* hex-mm: enable MM fusion for F16 activations

* hex-mm: pass tiled sizes to the solver in fused paths

* hex-mmid: remove scalar divs from expert mapping loops

* hex-mmid: proper cacheline safety enforcement for mdev splits

* hex-mm: improve solver for mdev split scanarios and tail handling

* hex-mm: remove redundant checks

* hex-mm: fix fused HMX MUL_MAT_NX drops the final partial tile for quantized weights

* hex-mm: better handling of ragged shapes (removes scalar memset of vtcm)

---------

Co-authored-by: ebateni <ebateni@qti.qualcomm.com>
Co-authored-by: Jhen-Jie Hong <iainst0409@gmail.com>
Co-authored-by: Yiwei Shao <yiwei@aizip.ai>
2026-10-05 08:55:21 -07:00
Georgi Gerganov d812350493 llama.cpp : bump version to 0.6.0 (#29997)
* llama.cpp : bump version to 0.6.0

* scripts : update summary prompt (#0)
2026-10-05 18:13:51 +03:00
Georgi Gerganov c06f84160a sync : ggml 2026-10-05 17:37:50 +03:00
Georgi Gerganov 994e8f2222 ci : add "Require Docker" flag to make-release workflow (#29989)
Assisted-by: pi:llama.cpp/Qwen3.8-27B
2026-10-05 16:04:47 +03:00
Alessandro de Oliveira Faria (A.K.A.CABELO) eec18f5d32 vendor : update cpp-httplib to 0.59.0 (#29886) 2026-10-03 19:08:47 +02:00
Ehsan BateniandMax Krasnyansky a4d880fd5c Hexagon: optimize ALLREDUCE with support for safe scatter mode (#29757)
* hex-allreduce: add support for safe scatter mode

* hex-allreduce: pare down excessive comments

* hex-allreduce: re-write to remove register spills

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-09-30 12:56:44 -07:00
thelittlefiremanandCarl Philipp Klemm 748d4225b9 ggml-cuda: HIP: optimize packed byte subtraction (__vsubss4 -> __vsub4) (#29478)
* ggml-cuda: HIP: optimize non-saturating packed byte subtraction (`__vsubss4`)

* CI: ignore 1 spilled vgpr in fattn_vec

---------

Co-authored-by: Carl Philipp Klemm <carl@uvos.xyz>
2026-09-29 19:59:19 +02:00
Trivikram Reddy 1c4729414d hex-scripts: show trace events smaller than 100nsec in perfetto (#29614) 2026-09-28 15:02:58 -07:00
uvos f916130d00 ci : ignore more vgpr spills in > 256 DQK fattn kernels (#29571) 2026-09-28 14:52:07 +02:00
uvos 2ebd9ae621 HIP: Enable fattn-mma kernel on cdna for dkq > 256 for large batch sizes (#28907)
* HIP: Enable fattn-mma kernel on cdna for dkq > 256 for large batch sizes

* CI: hip-quality-check: ignore spills for very large mfma mma kernels
2026-09-27 13:06:56 +03:00
Trivikram Reddy 4e7481175c hexagon: find software divide calls using binary inspection tool (#29449)
* hex-scripts: fix table alignment

* hex-scripts: find sw div calls using binary inspection tool
2026-09-25 19:43:38 -07:00
Alessandro de Oliveira Faria (A.K.A.CABELO) 171e8846b4 vendor : update cpp-httplib to 0.58.0 (#29407) 2026-09-26 01:55:18 +02:00
Georgi Gerganov bced4595b8 sync : ggml (#29396)
* ggml : bump version to 0.25.2 (ggml/1642)

* ggml : fix ubsan error in `ggml_graph_nbytes` (ggml/1644)

* ggml : bump version to 0.25.3 (ggml/1645)

* sync : ggml
2026-09-24 22:44:05 +03:00
Georgi Gerganov 013b31c03c scripts : make-release-desc - link previous release in changelog title (#29336)
make-release-desc.sh now emits "Changelog since [vX.Y.Z](<repo>/releases/tag/vX.Y.Z)"
instead of a plain version string, so the release notes link back to the previous release.

The repo URL is derived from the origin remote (SSH or HTTPS); if it cannot be
resolved (local run without origin), the title falls back to plain text.

Assisted-by: pi:llama.cpp/Qwen3.8-27B
2026-09-24 07:46:38 +03:00
Georgi Gerganov 177cd8cc70 sync : ggml 2026-09-23 20:29:29 +03:00
Georgi Gerganov 183d2a04c2 make-release : update summary prompt 2026-09-23 11:47:24 +03:00
Georgi Gerganov 45062d4056 sync : ggml 2026-09-23 11:47:24 +03:00
Max Krasnyansky 58367713a6 hexagon: new HMX-optimized GATED_DELTA_NET (#29199)
* hex-gdn: start putting together HMX support for GDN

* hex-gdn: working hmx but not-pipelined and slow for now

* hex-gdn: re-write vtcm layout handling and prep for pipelining

* hex-gdn: starting to pipeline hmx and dmas

* hex-gdn: add hvx threading for most pipeline stages

* hex-gdb: add detailed trace events

* hex-gdn: vectorize expfs and use aligned hvx reads/writes

* hex-gnd: vectorize the rest of expf

* hex-gdn: optimize tail processing (pad partial chunks)

* hex-gdb: avoid float up/down casts in hot loops

* hex-fa: remove float up/down casts from inner loops

* hex-gdn: do exp() in f16 to improve HVX utilization

* hex-gdn: optimize tiler

* hex-hmx: bump hmx-queue to 128 and dispatch all GDN gemms at once

* hex-gdn: further pipeline improvements

* hex-gdn: optimize gdn prep stage

* hex-gdn: yet more tweaks to optimize GND_SOLVE task and pipeline

* hex-gdn: improve accuracy and optmize gdn-prep further

* hex-gdn: fix rebase conflict

* hex-bufs: revert max_bufsize enforcement, it is enough to just enforce max_vmem

* hex-scripts: improved inspect script to avoid false alarms in reg spill detector

* hex-fa: improve inline softmax with in-reg VKQ32 accum

* hex-fa: minor improvement for dma pipeline in hvx kernel

* hex-fa: reduce ddr reads by 20-30% during token gen

* hex-gdn: proper alignment for hvx vtcm spads
2026-09-21 14:49:52 -07:00
Adrien Gallouët ff0dbb975e vendor : update cpp-httplib to 0.57.1 (#29239)
Signed-off-by: Adrien Gallouët <adrien@gallouet.fr>
2026-09-21 22:50:20 +02:00
Adrien Gallouët 6f41ac59e0 vendor : update cpp-httplib to 0.57.0 (#29214)
Signed-off-by: Adrien Gallouët <adrien@gallouet.fr>
2026-09-21 13:44:43 +02:00