Commit Graph
777 Commits
Author SHA1 Message Date
Eve 186eab1cb8 ci: fix build-cmake-pkg ccache (#31705) 2026-11-05 21:16:56 +00:00
Xuan-Son Nguyen 6023692b1e llama: fix kv shifting, add tests, add per-model build_rope (#31683)
* llama: fix kv shifting and add tests

* llama: add graph build_rope (#31693)

* llama: add graph build_rope

* improve

* migrate new models

* update review skill

* fix ci

* add --kv-shift to CI
2026-11-05 21:10:36 +01:00
Eve 3de52ac388 ci: run self hosted cpu jobs on hf or github (#31439) 2026-11-04 22:34:01 +00:00
Aleksander Grygier 0943ca5472 ui: add Playwright harness for recording demo videos (#31318)
* ui: add Playwright harness for recording demo videos

Scripted walkthroughs under tools/ui/demo are recorded with Playwright and
exported as H.264 mp4 for pull requests (demo/pr) and docs (demo/docs).
Fixture MCP servers and seeded settings keep recordings deterministic, a
drawn cursor and captions make the flow readable, and DEMO_UI_DIR points
the recording at another checkout such as a PR worktree.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: record demo scripts from PR descriptions and a configurable storage

Demo scripts no longer live in the repo. `npm run demo:pr -- <number>`
records the `ts demo` block from a PR description against the PR head,
uploads the mp4, a GIF preview and the script to DEMO_STORAGE, and links
the preview in a `## Demo` section of the PR description.
`npm run demo:docs` records the docs scripts kept in DEMO_STORAGE.
DEMO_STORAGE (hf:// buckets or file://) and the public URL are read
from tools/ui/.env.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: import generated conversations into demo recordings

Demos can now start with existing chats: `test.use({ traces })` imports
llama.app JSONL traces through Settings > Import/Export before the video
starts. Built-in traces are generated deterministically, other names are
downloaded from DEMO_TRACES_DATASET, and `demo:traces` pushes the built-in
ones there. Adds `demo.scroll` for wheel scrolling.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: keep demo captions clear of the chat form and isolate concurrent runs

Captions move to the top of the frame, since at the bottom they covered
the chat form a demo is often about. Each recording run gets its own
Playwright output folder, so a run starting elsewhere no longer clears
another run's video before it is saved.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: record demos at 2x and 60 fps with human-like timing

Recording goes through Chromium's screencast instead of Playwright's
video, which encoded VP8 at about 1 Mbit/s at 1x. Frames are captured at
the device pixel ratio (DEMO_SCALE, default 2) and encoded into a 60 fps
H.264 mp4, starting once the page has settled.

The cursor now glides faster along a slight arc with Fitts-like
durations, settles briefly before each click, and typing and pauses vary
a little. The randomness is seeded per demo, so re-recording gives the
same motion.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: document the demo harness prerequisites

The demo README now lists the tools the harness needs and what each is
for, and walks through creating the Hugging Face bucket and the optional
traces dataset, with the matching .env settings.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: version linked demo URLs so re-recorded previews refresh

GitHub's image proxy caches a preview by URL, so a demo re-recorded at
the same PR head kept showing the old GIF. The Demo section now links
the files with a content hash in the query string.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: label PRs that feature a video demo

Adds a `video demo` label and a workflow that applies it when a PR
description has a demo written by the UI demo pipeline (the managed
demo:start/demo:end markers, or a preview linking to a demo video),
and removes it when that section is gone.

Assisted-by: Claude (Opus 5.5 Medium)

* ci: label video demos with gh instead of github-script

The label workflow now runs the gh CLI in a shell step, like
pr-draft-label.yml, so it needs no action outside the repository's
allowed list. The video link check only counts videos uploaded for the
PR itself, so a description linking other PRs' demos isn't labeled.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: zoom demos in on details with a two-finger pinch

demo.zoom pinches in on a control or a point, holds, and pinches back out
to 1x, so a detail too small to read in the frame can be shown up close.
It drives Chromium's own pinch gesture, so the page's fixed bars stay put,
and works at desktop and phone sizes alike.

The pinch is evened out where Chromium misses: one anchored near the top
does not zoom at all, so it starts lower, and one near the bottom
overshoots its ratio, so a small pinch from the middle corrects it. The
way out reads the real scale until the page is back at 1x, which the next
gesture needs.

The overlay also draws a fingertip per touch, so a pinch on a touch
device shows as two fingers.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: wait out a cold dev server before a demo starts

A fresh checkout's dev server optimizes its dependencies on the first
load and reloads the page, which can take well past an action's timeout,
so a demo failed before it started. The demo now waits up to 90 seconds
for the app; the recording only starts once it is ready.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: record a video per test in a PR demo, shown in a table

A large PR can split its demo into several shorter videos: each test()
in the ts demo block is recorded as its own video (<sha>-1.mp4, ...),
and the Demo section lays their previews out in a table, each column
headed by its test's title.

GIF previews now keep their longer side at DEMO_PREVIEW_WIDTH and shrink
until they are under 5 MB, since GitHub's image proxy refuses anything
larger and shows a broken image instead.

Importing traces works on a phone viewport too, where Settings sits in
the sidebar under the page until the menu key slides it aside.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: fix demo trace imports on desktop and lay landscape demos out two to a row

Importing traces failed on a desktop whose collapsed sidebar sits under
the page: it now expands the sidebar from the page's menu key on a phone
or from the sidebar's logo on a desktop, whichever makes Settings
reachable. Afterwards it forgets the expanded sidebar, so a demo starts
with it collapsed, as on a first visit.

A PR demo with several videos shows up to four portrait (phone) videos
to a row, or two landscape ones, so each preview stays readable.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: only record other people's PR demos at a reviewed head commit

demo:pr runs the PR's code and demo script on this machine, with the
gh and hf logins, so a PR by someone else now needs --trust <sha>
naming the head commit that was read, and is refused again after new
pushes. Recordings also launch Chromium with its sandbox on, which
Playwright turns off by default.

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-03 12:30:49 +01:00
Sigbjørn Skjæret 27b8464e82 ci : add missing ccache-clear to windows behavior check (#31413) 2026-10-30 11:34:25 +01:00
Daniel Bevenius e216f4ec87 ci : extract most apple jobs into daily workflow (#31408)
This commit extracts all apple jobs except for macos-latest-arm64 into a
new daily workflow.

The motivation for this is to reducd the load on the macos runners.
2026-10-30 10:01:28 +01:00
Oliver SimonsandSigbjørn Skjæret 68d20b884c Quant: OCP FP8 E4M3 support (#27512)
* WIP OCP FP8 E4M3 support

* Fix python type-checker

* Materialize input_scales also for FP8 when converting

* Fix conversion of https://huggingface.co/RedHatAI/Qwen3.8-27B-NVFP4

1. Per row FP8 wrongly fell to NVFP4 conversion, as NVFP4 checked on
   dims alone. Also check on dtype for FP8
2. Need to reshape FP8 QKV projections scales in the same way that
   weights are reshaped

* Allow remaining FP8 tensors to be dequantized during conversion

* Allow tensor-strategy for FP8-only compressed-tensor ckpts

* Fix and test gguf-py behavior for FP8_E4M3

* Disallow QKV-fusion for FP8 temporarily

* Fix conversion of fused qkvz & FP8 for Qwen3-Next

* Avoid fusing FP8 for --fuse-gate-up-exps as we may have different ws

* Emit warnings when not fusing FP8 during conversion

* More FP8 conversion fixes

* Add supported for strided copies to/from FP8 in CPU

* remove unneeded scratch buffer from FP8 cpu cpy/dup path

* add explanatory comments and TODOs

* Add dedicated encoder and decoder tests for FP8

* Whitelist ggml_op_get_rows in metal instead of blacklisting

required to fix failing CI, and white-listing should be more
future-proof than adding FP8 to the black-list

https://github.com/ggml-org/llama.cpp/actions/runs/38433916498/job/115422880633?pr=27512

* Fix more conversion bugs

* Defer weight materialization during conversion

* Blacklist fp8 in metal backend instead of white-list

* Adjust indents

* Fix lint + execute tests

* Add fast CPU kernel for F8_E4M3

* Conversion fixes

* Fix remaining gaps

* Enable tied weights with shared scales

* Make mandatory-F32 rules apply to FP8 during conversion

* Fix python type checks

https://github.com/ggml-org/llama.cpp/actions/runs/39765670052/job/119780379595?pr=27512

* Fix kaleidai unit-test failures

https://github.com/ggml-org/llama.cpp/actions/runs/39765670125/job/119780383150

* Relax numeric precision constraints for pre-check

* Leave fast-FP8 CPU on RISC-V for follow-up

We already exclude arm from the fast-path

* Allow output scale tensors to be located in any gguf shard

* Move test_conversion_fp8.py test execution to behavior-check.yaml

* Apply batched suggestions from code review

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Move mixed compressed-tensor logic into dequant_fp8 + add unit-tests for it

* Install all requirements also on Windows for behavior-check

Fixes https://github.com/ggml-org/llama.cpp/actions/runs/40535852590/job/122300758223?pr=27512

* Unmap lazy tensors before clean-up in Windows

Should fix https://github.com/ggml-org/llama.cpp/actions/runs/40539797824/job/122314782459?pr=27512

* LF line endings

https://github.com/ggml-org/llama.cpp/actions/runs/40539798168/job/122314791118?pr=27512

* Extract FP8 tests from llama-arch to its own test-fixture

See https://github.com/ggml-org/llama.cpp/pull/27512#discussion_r4400628221

* Update gguf-py/gguf/scripts/gguf_convert_endian.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-10-29 18:14:37 +01:00
Aaron Teo ff0d396e3d ci(s390x): switch to Ubuntu 26.04 runners (#31317) 2026-10-28 23:32:38 +08:00
Sreeram fae9ac561f ci : use ccache depend mode for ROCm release builds (#31231) 2026-10-27 09:24:00 +01:00
Sigbjørn Skjæret a8276b70cc ci : force python 3.12 on ubuntu 26.04 (#31222) 2026-10-27 09:09:44 +01:00
Sigbjørn Skjæret 4244ad7acd ci : add windows behavior check (#31191) 2026-10-26 23:01:04 +01:00
Georgi Gerganov 48c36110ab ci : drop the macos-latest-x64 job (#31175) 2026-10-26 09:56:19 +02:00
Georgi Gerganov 344be0a600 ci : update CPU job assignments (#31136) 2026-10-25 10:35:11 +02:00
Georgi Gerganov 71255fefc0 ci : pass per-runner results dir to self-hosted jobs on T4 runners (#31100)
The two T4 runners in the ggml-ci group (ggml-6-x86-vulkan-t4,
ggml-8-x86-nvidia-a10) share the same labels and can both pick up the
same jobs, so extend the per-runner results dir from #31046 to them.

Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
2026-10-24 17:17:04 +03:00
Georgi Gerganov 4d32b29aaa ci : exclude test-speculative-rejection-sampling from macos-latest-arm64 (#31088)
Add test-speculative-rejection-sampling to the ctest -E exclusion list of the macos-latest-arm64 CI job.

Assisted-by: dsh:llama.cpp/ggml-org/Flint-GGUF (RTX 5090 32GB, DDR5 128GB)
2026-10-24 14:14:31 +03:00
Georgi Gerganov 6b9ba4f2e8 ci : fix data race between runners sharing the model cache (#31046)
* ci : fix data race between runners sharing the model cache

Two runners per mac mini share HOME, so concurrent jobs corrupt each
other downloads, converted models and venv under the shared MNT cache.

- gg_wget: download in a temp dir, publish with mv, so readers only
  ever see complete files (-N cannot be combined with -O)
- convert, quantize and wikitext unzip now produce under a per-runner
  scratch dir in OUT, published to MNT via hardlink when missing
- gguf-split and quantize tests get a per-runner TMP_DIR
- python venv moved from MNT to OUT
- metal, vulkan and webgpu mac jobs pass a per-runner OUT path

Assisted-by: dsh:llama.cpp/agent-qwen4 (RTX 5090 machine)

* ci : clean up scratch models after tests

Assisted-by: dsh:llama.cpp/agent-qwen4 (RTX 5090 machine)

* ci : fix q4_0 publish when a cached model exists

cp -n skips an existing file with exit code 1, so the ln fallback
aborted the qwen3 job under set -e. Publish through a unique temp
file with an atomic mv: readers never see a partial file and a
stale cached q4_0 gets replaced.

Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (RTX 5090 machine)

* ci : cache-first gg_wget to avoid re-downloads

HF sends no Last-Modified, so wget -N re-downloads the full files on
every run. Skip the download when a non-empty cache file exists: the
HF CI artifacts are static. Downloads still go through a temp dir and
an atomic mv, so concurrent runners never read partial files.

Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (RTX 5090 machine)
2026-10-24 09:21:24 +03:00
Sigbjørn Skjæret f00c9083b2 ci : add behavior check workflow (#31036)
* remove python from tests

* add behavior and backend ctest labels

* add behavior check workflow

[no ci]
2026-10-23 14:53:44 +02:00
Sigbjørn Skjæret b38960f520 ci : remove superfluous webgpu jobs (#30868) 2026-10-20 14:27:55 +02:00
Max Krasnyansky 8466395353 hexagon: update to snapdragon-toolchain v0.8 and enable hexagon v85 (#30854)
* hex-build: bump toolchain, enable v85

* hex-build: bump toolchain to v0.8 in the CI and build scripts

* hex-build: add a little python script for swiv needed on some iot devices

* hex-build: fix flake8 warnings

* hex-arch: update version cap to v85

* hex-inspect: flexible TOOLS_ROOT

* hex-build: update to snake_case naming

* hex-build: update leftover old toolchain refs

* hex-target: update build and run scripts to support ubuntu with adb transport
2026-10-19 23:49:59 -07:00
Mendy Berger 921257f500 ci, docs: add WASI CI and docs (#30754)
* ci, docs: add WASI CI and docs

* accept review feedback
2026-10-20 01:09:55 +02:00
7adf160c41 server: add --connect, allow p2p access to server outside LAN (#28277)
* init

* wip

* wire it up

* make it clear that it's experimental

* add props.connect_code

* nits

* fix windows build

* server: do not advertise a connect code without the tunnel

--connect-code on its own had no effect other than being served in
/props, so the Web UI showed a remote access code that no tunnel was
serving. Clear it when --connect is off, and say so once at startup.

The teardown comment claimed that terminate() closes the child stdout.
It does not: the log thread leaves fgets() only once every writer of
that pipe is gone, so a descendant of llama-connect would block the
join forever. Describe the real contract instead.

Also restore the --reasoning-preserve default in the generated docs.

* Update tools/connect/CMakeLists.txt

Co-authored-by: Adrien Gallouët <angt@huggingface.co>

* server: dial llama-connect on the bound port and a reachable host

Pass ctx_http.port so --port 0 and the GCP override work, pick the
dial host from --host (wildcards map to loopback, IPv6 is bracketed)
and refuse --connect when only unix sockets are bound.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
Co-authored-by: Adrien Gallouët <angt@huggingface.co>
2026-10-20 00:23:14 +02:00
Aaron Teo fda00f18cd tests: add big-endian conversion tests (#30705)
* ci: add quantize to big-endian tests

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* ci: remove incoherent data types from testing

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* tests: add verified hashes

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* tests: rename

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* tests: add result table

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* tests: swap hash and result column, add colours

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* tests: switch to perplexity test

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* tests: update output table

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

---------

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
2026-10-19 10:14:58 +02:00
Sandro Hanea 83f9389419 ci : added ABI reports as github artifact (#30619) 2026-10-17 11:41:53 +02:00
Zack Li 6420cf5dd5 ci: update device matrix for Snapdragon jobs by removing SM8750 (#30620)
* ci: update device matrix for Snapdragon jobs by removing SM8750

* ci: increase timeout for QDC device tests and update job timeout parameters
2026-10-16 16:25:05 -07:00
Neo ZhangandSigbjørn Skjæret 43d57fb8ca [SYCL] Enable ssl in build script and CI (#30456)
* enable OPENSSL=ON as default in build script, update it in CI and dockerfile

* change to LLAMA_BUILD_BORINGSSL on windows

* refactor

* rm Download UI build

* Apply suggestion from @CISC

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Apply suggestion from @CISC

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Apply suggestion from @CISC

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-10-15 21:58:07 -04:00
Sigbjørn Skjæret 711780ffc7 ci : bump ty to 0.0.88 (#30587)
* bump ty to 0.0.88

* arange -> linspace.tolist() to pass expected type
2026-10-15 14:15:55 +02:00
Neo Zhang e36df60470 sycl : replace oneAPI package by DLE+oneDNN package (#30131)
* replace oneAPI package by DLE+oneDNN package

* support to install oneDNN in windows, in release and docker

* for internal test

* remove test code

* Update intel.Dockerfile
2026-10-15 09:26:01 +02:00
Georgi Gerganov a17e94d229 tests : consolidate context state tests into test-llama-context (#30023)
* tests : consolidate the context state tests into test-llama-context

test-save-load-state, test-recurrent-state-rollback and
test-state-restore-fragmented check the same object from three places:
the KV cache and the recurrent state of a llama_context, saved, loaded,
copied, rolled back. Each had its own main, its own loop over the
models, its own copy of the token and the comparison helpers, and a run
of all three loaded the same models several times.

They are one binary now, and the case is the unit of the test:

  tests/llama-context/test-case.h   the interface a case implements
  tests/llama-context/cases-kv.cpp  the cases of the KV cache
  tests/llama-context/cases-rs.cpp  the cases of the recurrent state
  tests/llama-context/test-util.cpp tokens, generation, comparison
  tests/llama-context/test-runner.cpp the models of a case, on threads
  tests/llama-context/test-log.cpp  the log file of a model

A run is a table: a row per model, a column per case, a cell holds the
verdict of that pair. The models load on several threads (-j, --threads,
min(cores, 4) by default), the table is printed when it is complete and
the log of a model that failed is printed after it, in the order of the
rows. A case returns a verdict and the reason for it, and the runner is
the only one who reports - the test does not use tests/testing.h, a run
over 130 models is a matrix of verdicts and not a tree of subtests.

Staging no longer travels through the environment: llama_model_set_attn_rot
replaces the LLAMA_ATTN_ROT_DISABLE variable, which a model loaded in one
thread used to change for the models the other threads were loading, and
llama_memory_get_buffers is added for the cases that copy the buffers of
a memory. common_log_set_stdo keeps the console quiet while the table is
being built.

ggml_aligned_malloc allocates with posix_memalign when the thread
sanitizer is on. On macOS it allocates with the Mach VM, which the
sanitizer does not intercept, so it never sees a page range change hands
and reports the first write into recycled memory as a race - 30 of them
in this test, and every one of them false.

Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next

* cont : run single-threaded by default

* cont : try to build the tests on all platforms

* cont : trim down some body fat

* cont : fix

* cont : fix the fix
2026-10-13 16:21:55 +03:00
Pratyush Kumar 8f574b0ad4 ci : build OpenVINO releases with GGML_NATIVE=OFF and CPU backend variants (#30324) 2026-10-13 10:22:05 +08:00
Pratyush Kumar a49f58dcc3 ci: fix OpenVINO download links (#30401) 2026-10-13 10:20:04 +08:00
Sigbjørn Skjæret 5e1d74043e ci : set run-name for publish release (#30214) 2026-10-09 11:12:24 +02:00
Sigbjørn Skjæret 7049ff0cbe ci : add 1accel label [no ci] (#30016) 2026-10-05 21:52:20 +02:00
Georgi Gerganov c250304960 ci : skip container re-tagging when Require Docker is disabled (#30008)
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
2026-10-05 21:22:08 +03:00
Georgi Gerganov 994e8f2222 ci : add "Require Docker" flag to make-release workflow (#29989)
Assisted-by: pi:llama.cpp/Qwen3.8-27B
2026-10-05 16:04:47 +03:00
Georgi Gerganov 8f9ae20c86 ci : disable failing test on virtual Metal device (#29993) 2026-10-05 15:10:16 +03:00
Sigbjørn Skjæret 2ed93db472 ci : disable unused qemu in docker build (#29984) 2026-10-05 12:33:10 +02:00
Sigbjørn Skjæret e5983d6704 ci : winget urls must be separate strings (#29978) 2026-10-05 09:36:57 +02:00
Sigbjørn Skjæret 4ca6b76f0b ci : fix docker workflow permissions (#29979) 2026-10-05 09:36:03 +02:00
Sigbjørn Skjæret 2e7c58c547 ci : windows llvm build requires ninja multi-config (#29959) 2026-10-04 19:28:24 +02:00
Sigbjørn Skjæret 7f2dd88b0a ci : add windows arm64 vulkan release (#29954)
* add windows vulkan arm64 release

* add link
2026-10-04 18:46:15 +02:00
Sigbjørn Skjæret 46847e6158 ci : set default permissions (#29945) 2026-10-04 16:13:26 +02:00
Sigbjørn Skjæret 0faee50042 ci : pushing tag needs deploy key (#29937) 2026-10-04 09:45:49 +02:00
Sigbjørn Skjæret f98b31c67e ci : improve release flow (#29913)
* improve release flow

* fix copied typo

* fix permissions
2026-10-04 09:31:24 +02:00
b92761a515 ggml-openvino: update to 2026.4.1, optimize performance, expand ops, improve device listing. (#29852)
* ggml-openvino : Qwen3.5 MoE perf (#312)

Squash of ravi9/llama.cpp#312:

- ggml-openvino: add detailed inference profiling (Yu, Zijun)
- ggml-openvino: use remote output tensors by default (Yu, Zijun)
- ggml-openvino: optimize single-sequence recurrent state (Yu, Zijun)
- opt1: remove recurrent reset for single sequence, opt2: direct gdn outputs (break parallel sequence) (Yu, Zijun)
- fix parallel sequences (Yu, Zijun)
- ggml-openvino: simplify graph cache key (ynimmaga)
- enable stateful for qwen35 single sequence (Yu, Zijun)
- Fix after rebasing (Yu, Zijun)
- Add k-requant option q4_asym64 (Yu, Zijun)
- Fix qwen35 llama-bench -p 0 (Yu, Zijun)
- Simplify RESHAPE translation (Yu, Zijun)
- openvino: fuse MoE routing (Yu, Zijun)
- openvino: fuse GDN qk normalization (Yu, Zijun)
- openvino: enable GPU MoE fusion by default (Yu, Zijun)
- ggml-openvino: add cache_only mode to import cached compiled model on disk directly (Yu, Zijun)
- openvino : report the device allocation limit to ggml (Łukasz Ślusarczyk)
- Fix windows build (Yu, Zijun)

Co-authored-by: ynimmaga <ynimmaga@users.noreply.github.com>
Co-authored-by: Łukasz Ślusarczyk <lukasz.slusarczyk@intel.com>

* ggml-openvino: Update doc of compiled model cache

* openvino: implement PRD-compliant device enumeration and memory reporting

* openvino: fix multi-device listing issues from review

- Only the device selected by GGML_OPENVINO_DEVICE reports as GPU; the
  other OpenVINO devices report as IGPU so llama.cpp does not offload to
  them. Initializing a non-selected device logs a warning.
- Name devices OPENVINO<i> again and show the OpenVINO id in the
  description. Raw "CPU" names shadowed the ggml CPU backend.
- Support GPU.N: create the OpenCL queue on OpenVINO's own context for
  the selected device, and replace "GPU"/"NPU" string comparisons with
  ggml_openvino_is_gpu()/ggml_openvino_is_npu().
- An unavailable GGML_OPENVINO_DEVICE is now an error that lists the
  available devices, instead of silently falling back to CPU.
- Memory: cap iGPU/NPU free memory at system available memory, fall back
  to system memory instead of 0/0 when the plugin lacks memory
  properties, and ignore host USM allocations in GPU usage.
- Initialize the device config once under a lock, even if OpenCL setup
  fails.
- Fix supports_op return type for non-selected devices (build error).

* openvino : take USM entry points from the selected device platform

clGetExtensionFunctionAddressForPlatform was called on the first platform
returned by clGetPlatformIDs. The address it returns is only valid for the
platform it was queried on, and the first platform is not always the one that
holds the device OpenVINO selected.

On a host whose first platform comes from another vendor the lookup returns
null, and then every read, write and memset on a GPU buffer fails with
"clEnqueueMemcpyINTEL not available".

Look both entry points up in init(), on the platform of the device OpenVINO
picked, and keep them in the device config next to the command queue.

Assisted-by: Claude Opus 5

* openvino: fuse MoE experts for models with a fused gate_up weight

FuseMoeCompressed only matches models whose gate and up projections are
separate GatherMatmul ops. gemma-4 packs both into one expert weight and
splits the result after the GEMM, so its MoE block stayed unfused and ran
the expert GEMMs as per-token GEMVs.

Add FuseMoeCompressedFusedGateUp, which matches that shape
(one GatherMatmul -> Slice/Slice -> Gelu(ERF) -> Multiply) and folds it into
the same MOECompressed op, using GEMM3_SWIGLU with GEGLU_ERF. The fused
weight, scale and zero point are split into gate/up halves by copying raw
bytes, since a graph Slice would be rewritten to StridedSlice and constant
folded, whose reference evaluator crashes on sub-byte types.

gemma-4 also applies a per-expert output scale to the down projection before
the router weights. MOECompressed takes only one per-expert weight, so that
scale is folded into the routing weights, which is exact.

The op reads the zero point straight off a weight port and needs an integer
Constant there, so the matcher requires one and leaves natively quantized
experts (exact f16 zp) to the unfused path.

gemma-4-26B-A4B on Arc B390, GGML_OPENVINO_REQUANT_KQUANT=q4_asym64_all,
llama-bench -p 512 -n 128 -r 2, against a GGML_OPENVINO_MOE_OP=0 baseline:
pp512 66.16 -> 1608.73 t/s, tg128 25.94 -> 26.46 t/s. Perplexity over 12
chunks is unchanged (1451.3 +/- 177.9 unfused vs 1427.6 +/- 175.1 fused).

No effect without that requant option, on models with separate gate/up
weights, or on CPU. test-backend-ops -b OPENVINO0 is unchanged by this
commit: two MUL_MAT_ID m_v cases fail, the same two on the unmodified base.

* openvino: fix rank-3 axis handling so MoE works under stateful execution

Stateful execution drops the leading size-1 batch dim, so OV tensors are rank
3 while GgmlOvDecoder::get_shape/get_stride still report GGML_MAX_DIMS=4
reversed entries. Several MoE ops derive OV axis indices straight from that
metadata, so they picked the wrong axis. A MoE model with
GGML_OPENVINO_STATEFUL_EXECUTION=1 aborts while building the graph:

  Check 'is_axis_valid(axis, r)' failed at src/core/src/validation_util.cpp:336
  While validating node 'opset11::TopK ... _ffn_moe_probs ...'
  Axis 3 out of the tensor rank range [-3, 2].

Fix idiom throughout: take the axis from the real OV rank, or shift a
metadata-derived axis down by metadata_rank - actual_rank.

  argsort.cpp    the router top-k axis is 2 on rank 3, not 3. This is the
                 abort quoted above.
  add.cpp        the MoE expert-sum bypass collapses the 8-ADD chain into one
                 ReduceSum on hardcoded axis 2, which on rank 3 reduces n_embd
                 instead of the expert axis. Now rank-2, with the following
                 Unsqueeze at rank-3.
  get_rows.cpp   squeezing a hardcoded {0,1} also strips the batch dim
                 whenever it is 1, which is every decode step. Squeeze down to
                 the trailing two dims instead.
  mul_mat_id.cpp pick the reshape dims by actual rank, and skip the trailing
                 Unsqueeze that re-adds the batch dim.
  view.cpp       the expert-plane slice had the Slice axis, dst_ov_axis, the
                 ShapeOf+Gather index and the Reshape target all rank-4.
  utils.cpp      process_view_input_new's "translate_view already resolved
                 this VIEW, skip re-slicing" shortcut required equal ranks. 4
                 vs 3 never matched, so every resolved expert plane got
                 re-sliced. Now compares the common trailing dims. Same axis
                 shift for the Slice in the view-chain walker.

Stateless is unchanged by construction: every edit is gated on the actual
rank, so axis_shift == 0 reproduces the previous code exactly. Checked on
OV-CPU by diffing greedy output against the unmodified base for dense
gemma-4-E2B, granite-1b-a400m and gemma-4-26B-A4B; all identical.

granite-1b-a400m on OV-CPU aborts with the error above before this change;
after it, it generates and is byte-identical to stateless. Dense gemma-4-E2B
is identical stateless vs stateful both before and after. test-backend-ops
-b OPENVINO0 is unchanged: two pre-existing MUL_MAT_ID m_v cases fail, the
same two on the unmodified base.

gemma-4-26B-A4B is a poor correctness vehicle here. On OV it already drifts
into degenerate repetition a few tokens in, in stateless as much as stateful,
and the two modes diverge somewhere inside that degenerate region instead of
matching token for token. Each mode is self-reproducible across runs.

Known limitation: FuseMoeCompressedFusedGateUp does not match the rank-3
graph, so a MoE model run with GGML_OPENVINO_STATEFUL_EXECUTION=1 loses the
prefill fusion while gaining decode. gemma-4-26B-A4B on Arc B390,
GGML_OPENVINO_REQUANT_KQUANT=q4_asym64_all, llama-bench -p 512 -n 128 -r 2:

  unfused (GGML_OPENVINO_MOE_OP=0)  pp512   66.16   tg128  25.94
  fused, stateless (default)        pp512 1608.73   tg128  26.46
  fused, stateful                   pp512   66.18   tg128  29.91

Stateful is opt-in and off by default, and MoE did not run there at all
before this, so nothing that previously worked regresses. Making the pass
match rank 3 is the follow-up.

* OpenVINO Backend: Upgrade graph cache to use node_idx, src_idx, node type

* ggml-openvino : enable more comprehensive conv fusion

* enable conv ops

* Reject kernel size 0 and support IM2COL_3D

* openvino : abort when the GPU remote context cannot be created

init() logged the error and returned, which left the device name a GPU but
remote_context empty. The remote buffer and tensor paths assert only on the
device being a GPU and then dereference that empty optional.

Those paths have no host fallback, and a device that OpenVINO listed should
have a working OpenCL context, so stop instead of continuing. An OpenCL stack
that is broken as a whole is still caught earlier by the device availability
check, which falls back to CPU.

Assisted-by: Claude Opus 5

* openvino : fix build warnings

The single-argument form of the OpenVINO RTTI macros is the intended one, but
their selector macro leaves __VA_ARGS__ empty, which -Wpedantic reports on
every pass and op header. Turn that warning off for this backend only, the
way ggml-cuda and ggml-sycl already do for their own third-party warnings.

Also drop a break and a dead assignment around a GGML_ABORT, which is noreturn.

Assisted-by: Claude Opus 5

* OpenVINO Backend: Support common MTMD ops

* ggml-openvino: give a reshaping view its own ov::Tensor

* ggml-openvino : compute HARDSIGMOID and EXPM1 in f32

HARDSIGMOID used a 1/6 constant in the input type, which is not exact
in bf16, and EXPM1 lost precision for small inputs in f16. Both now
compute in f32 and convert back, except on NPU where the f32 path
gives wrong results.

Fixes the HARDSIGMOID/EXPM1 test-backend-ops failures on GPU.

* ggml-openvino : update device selection and --list-devices

Show the selecting GGML_OPENVINO_DEVICE value and active device in
--list-devices, startup logs, and backend tests.

Clarify OpenVINO selection uses GGML_OPENVINO_DEVICE, not -dev.

* openvino : remove unreachable OpenCL queue checks

A remote buffer exists only on a GPU device, and init() aborts there if the
queue cannot be created, so the queue is never null at these call sites.

Assisted-by: Claude Opus 5

* openvino : update OpenVINO to 2026.4.1 and GPU drivers to 26.35.39758.10

* docs : update OpenVINO validated models and GPU driver version

* ggml-openvino : skip empty views when giving a reshaping view its own tensor

A zero-size view can sit at the end of a GPU USM buffer (Qwen3.5 recurrent cache). Wrapping it as a remote tensor throws "shared USM buffer has smaller size (0)".

Assisted-by: Claude

* ggml-openvino : rebind the cached decoder when llama passes a different graph

llama keeps separate graphs for batches with and without outputs. llama-server splits the prompt into chunks for context checkpoints, so a cached decoder could be reused with a graph built in other memory and bind the previous chunk's input tensors. SWA and recurrent models then lost most of the prompt in llama-cli and llama-server.

Assisted-by: Claude

* docs : update OpenVINO validated models

Smoke test on Lunar Lake (32 GB) with the two fixes above. Re-add the Qwen3.5 and gemma models.

Assisted-by: Claude

---------

Co-authored-by: Yu, Zijun <zijun.yu@intel.com>
Co-authored-by: ynimmaga <ynimmaga@users.noreply.github.com>
Co-authored-by: Łukasz Ślusarczyk <lukasz.slusarczyk@intel.com>
Co-authored-by: haarika-madaka <haarika.madaka@intel.com>
Co-authored-by: Mustafa Cavus <mustafa.cavus@intel.com>
Co-authored-by: Mostafa Faheem <mostafaaafaheem@gmail.com>
2026-10-03 11:59:25 +03:00
Sigbjørn Skjæret 4ebdf2c74a ci : use t4-medium for cuda jobs (#29842)
[no ci]
2026-10-02 17:31:19 +02:00
Aaron Teo 254b177307 ci : fix missing zdnn backend check (#29837)
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
2026-10-02 09:10:45 +02:00
Oliver Simons 7677678503 CUDA: Make CCCL configurable + pin it to 3.4.3 for CI jobs (#29792)
Pinning to >= 3.4.3 is required to enable DeviceTopK, which was affected by
a race condition https://github.com/NVIDIA/cccl/pull/10627.

We will relax this for future CTK versions which will bundle CCCL >
3.4.X (CTK 13.5 will bundle CCCL 3.5.0 for example)
2026-10-01 13:05:29 +02:00
Sigbjørn Skjæret 72db1e02ff ci : add models backend check (#29651)
* add models backend check

* t4-medium for faster build
2026-09-30 09:06:24 +02:00
Aaron Teo cee37ffea0 ci: add zdnn backend build but not test (#29541)
* ci: add zdnn backend build but not test

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* ci: attempt to run a ubuntu 26.04 container

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* ci: set shell to bash

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* ci: clean up comments

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* ggml-zdnn: fix compiler errors

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* vendor: attempt to ignore warnings from vendored files

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

---------

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
2026-09-29 20:45:14 +03:00
R0CKSTARandyeahdongcn 18b74ff68e musa: build the docker image and CI container from the MUSA SDK images (#29624)
* musa: build the docker image and CI container from the MUSA SDK images

Use registry.mthreads.com/mcconline/musa_sdk:5.2.0-{devel,runtime}-ubuntu22.04-s5000
instead of registry.mthreads.com/mcconline/inference/pytorch:2.9.1.post1-py3.10-musa5.2.0-mp31-devel-ubuntu22.04-amd64
for the MUSA docker image and the MUSA CI container, and let the runtime stage use the
runtime image instead of reusing the devel one, which drops the MUSA toolchain from the
published images.

* musa: install the MUSA headers and loader path the SDK images omit

musa_sdk:5.2.0-*-s5000 does not ship the cub and thrust headers that the MUSA
backend builds against, and its runtime image does not register
/usr/local/musa/lib with the dynamic loader.

Install both header packages in the build stage and in the MUSA CI container,
and write the loader path in the runtime stage.

* musa: install libmthreads-compute for the MUSA runtime library

The MUSA SDK images do not install libmthreads-compute, which provides
libmusa.so.1 in /usr/lib/x86_64-linux-gnu, so linking anything against the
MUSA backend fails.

* musa: install libmthreads-compute in the runtime stages

The MUSA runtime image does not install libmthreads-compute, so the published
images would have no libmusa.so.1 at run time.

---------

Co-authored-by: yeahdongcn <yeahdongcn@users.noreply.github.com>
2026-09-29 10:42:20 +02:00