Temporary carry of the open upstream PR
https://github.com/ggml-org/llama.cpp/pull/32740 at head 808fd1c4e,
its four commits (1fa93f902, d29f9d489, ff91e4f84, 808fd1c4e) squashed.
Inkling's per-query bias table is applied inside flash attention
instead of a dense [n_kv, n_tokens, n_head] bias tensor, which cuts
Inkling's compute buffers and long-context prefill time on the RX 580.
Drop this commit when the PR is merged upstream (FORK.md, "Temporary
upstream carries").
One conflict, ggml-vulkan.cpp use_mask_opt: kept the fork's GCN
`HSK >= 256 || HSV >= 256` clause and its comment (F2) and upstream's
`!use_sparse`, added the PR's `!bias &&`.
Original commit trailer: Assisted-by: Codex
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
Add an optional backend interface method free_scratch (with a public
ggml_backend_free_scratch wrapper) that frees transient/scratch device
memory a backend holds outside of any allocated buffer, keeping the
backend usable - the scratch is reallocated lazily on the next compute.
Implement it for the Vulkan backend (ggml_backend_vk_free_scratch): free
the prealloc_x/y/split_k/add_rms_partials and sync_staging device buffers
and reset their sizes, so an idle/cold model does not hold the vision or
matmul compute preallocations in VRAM. All other backends leave the hook
null (no-op).
Assisted-by: Claude
* ggml : add SIGMOID backward pass
Sigmoid is used in Qwen3.5 hybrid models and had no backward pass.
Bump RPC protocol patch version for the new op.
Signed-off-by: gianni-cor <gianfranco.cordella@tether.io>
* replace assert with ggml_assert
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* ggml : keep sigmoid backward checks in release builds
Assisted-by: Codex
* ggml : preserve existing operation IDs when adding SIGMOID_BACK
Assisted-by: Codex
---------
Signed-off-by: gianni-cor <gianfranco.cordella@tether.io>
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* Adds initial scheduler stress tests
* Adds new sleep op to simplify scheduler testing
* Adds testcase addressing https://github.com/ggml-org/llama.cpp/issues/23321
* Adds experimental vulkan GGML_OP_SLEEP
* Improves structure and naming of previous tests. Adds new tests,
including one which detects the nkvo-bug correctly, and validates the
proposed hotfix.
* Cleanup: Removes CUDA specific comments and nvtx-markers
* Fix: forgot to add sleep specific test case
* Fix oversights identified by CI
* Drops default prints so that it behaves like other ctests. Adds -v flag
to toggle this behavior
* Fix: Early exit in CPU-only settings (cause for graviton CI fails)
* Improves stress_test_linked_list by removing overlap with
test_chain_all_backends and test_pair_user_inputs. The latter ones test
either activations or user inputs in isolation and just once, whereas
the stress test tests all combinations multiple rounds
* Exclude ACCEL devices from new tests due to incompatibility
* Fixes vulkan test failure in llvmpipe
* Adds asserts, improves comments and some checks
* Fixes nitpicks in white-space and variable naming
* Adds comments that certain vk drivers could time OP_SLEEP incorrectly
* Drops limitiation to CPU/GPU in test-sleep
* Aligns OP_SLEEP guards in cuda with vk guards (input shape and
contiguity)
* Implements feedback for OP_SLEEP to work in place (no copy)
* Apply non-code suggestions from JohannesGaesslers code review
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* Removes struct backend_supports; enables tests to return skip, instead
of only pass/fail
* Make clang-tidy happy
* Removes sched_backend_caps, improves create_test_scheduler according to
feedback (L149)
* Removes "name" field from struct sched_check in favor of
ggml_set_name/ggml_get_name
* Drops unnecessary shader_clock.comp test and guards
* Replaces sched_ prefixes with helper_ prefix
* Moves test suite to new RAII types for better memory management
* Adds sleep option to test_chain_all_backends
* Adds sleep option to first-run arm of the y-shaped test
* Fix switchcase order for legibility in stress_test_dag
* Simplfies stress_test_linked_list via double for-loop
* Adds pseudo-random input to stress_test_linked_list, and runs all
transitions both with and without CPU input (next to activation copy as
input)
* Fix RPC Proto patch due to GGML_OPS bump
* Adds ggml-cpp.h treatment to test-sleep.cpp
* Adds TODO pointers for subsequent additional tests.
* Apply suggestion to ggml.h
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* Apply suggestion to tests/test-backend-sched.cpp
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* Apply suggestion to tests/test-backend-sched.cpp
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* Fixes node compatibility check to not take buffer locations into account.
This led to skipped tests due to incorrectly rejected ops due to wrong buffer locations in multi-GPU (which is one of the scheduler behaviors under test)
* Apply batched suggestions from gg code review
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* Completes callsite change of gg-suggestion
* Switch test-sleep from std::chrono::steady_clock::now() to ggml_time_us
to avoid mac clock drift and spurious test failures.
This makes the test a bit nonsensical on CPU (same clock checks itself),
but the test still has merit on the other backends where
independent device clocks are used.
---------
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* ggml : prune deprecated API
* cont : bump RPC version
* cont : minor
* ggml : remove outdated documentation from ggml.h
The 172-line comment block at the top of the header has not been updated in a
substantive way since the backend v2 split (Nov 2023) and now documents APIs
that no longer exist:
- `ggml_set_f32()`, `ggml_get_f32_1d()`, `ggml_set_f32_1d()` were removed
- `ggml_conv_1d_1s()`, `ggml_conv_1d_2s()` ops no longer exist
- `ggml_set_param()` no longer takes a context argument
- `ggml_graph_compute_with_ctx()` moved to the CPU backend header
- the optimization API now lives in `ggml-opt.h`
The block also ended with 5 empty `TODO` sections.
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* ggml : remove DIAG_MASK_INF and DIAG_MASK_ZERO (#31946)
* rpc : bump RPC protocol major version
The tensor op enum is sent over the wire, so removing GGML_OP_ADD1,
GGML_OP_DIAG_MASK_INF and GGML_OP_DIAG_MASK_ZERO shifts the value of every
op that follows them and peers built before and after this change would
misinterpret each other's ops.
The handshake compares the major version for exact equality and ignores the
patch version, so bumping the major version is what rejects incompatible
peers instead of silently misbehaving.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
---------
Co-authored-by: losewayy <1134285612@qq.com>
* moe-cache : compute part of the misses on the CPU
--moe-cache-cpu-share F computes a share F of the MoE cache misses on the CPU
from the host experts instead of uploading them. The graph builds the expert
FFN twice: on the device from the cache slots, where the misses left to the CPU
read zero slots, and on the CPU from the host experts, where the experts
computed on the device have id -1. The two results are added before the down
bias.
- ggml-cpu: MUL_MAT_ID skips negative ids and writes zero rows
- sched: the copy callback is called for all the inputs of a split, and before
the host sync when there are no events
- the slot lookup callback reads the inputs of the CPU part together with the
selected experts, so the CPU part does not wait for the device, and its
result is copied back asynchronously
- each token uses distinct zero slots, mul_mat_id does not support an expert
selected twice by a token
- LLAMA_MOE_CACHE_TRACE writes the selected experts of each lookup, for a
simulator of the cache
* scripts : add moe-cache-tune.py to pick the MoE cache arguments
The script wraps a llama-server command: it starts the server at two cache
sizes and three CPU shares, fits the upload and CPU speeds of the machine, and
replays the routing of the runs through a copy of the cache logic to predict
other sizes and shares. It prints the recommended --moe-cache-mib and
--moe-cache-cpu-share.
docs/moe-cache.md describes the MoE cache arguments and the script.
* add am17an as codeowner for MoE cache
* moe-cache : upload all the misses until the cache is full
With a high --moe-cache-cpu-share, few misses are uploaded, so an empty cache
fills slowly and more misses than needed go to the CPU. The CPU share now
starts once all the slots of a cache group hold an expert. moe-cache-tune.py
replays the same rule.
* add backend sync for async transfer queue
* moe-cache : let the scheduler copy the results of the CPU part
* Fix MSVC compilation
* remove sync change
* llama-graph : extract the CPU share condition into can_cpu_share
* llama-graph : make llm_graph_params::moe_cache const
The graph construction no longer updates the MoE cache directly: the CPU
parts of the layers are accumulated in the graph result and applied to
the cache in llama_context::process_ubatch, after the graph inputs are
set and before the graph is computed.
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* scripts : replay the ClassLRU policy in moe-cache-tune.py
* moe-cache : pick CPU misses by CRF instead of random (#31853)
* moe-cache : remove the backend logic of the CPU part from the graph
The CPU part reads the host experts, so the scheduler places it on the
CPU without set_tensor_backend. The cache gives no CPU map for batches
that a device would offload. Graphs with a CPU part look up their own
slot map, so the graph does not register the part with the cache.
ggml-backend : the inputs of the next split that are already computed
are copied to host memory before the current split runs, so a CPU split
does not wait for the device split before it. This replaces the stash
of the cache, and the copy callback is again called for host weights
only.
* llama-graph : return the MoE cache slots and CPU ids in a struct
The scheduler prefetch no longer checks that the next split runs on
another backend: the inputs of a split are never on its own backend,
so the input loop already skips that case.
---------
Co-authored-by: Gaurav Garg <gaugarg@nvidia.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Co-authored-by: Alexey Kopytko <alexey@kopytko.com>
* ggml : support `truncate=false` for YaRN
YaRN calculates correction intervals to switch frequency interpolation
methods, but the setting to not round the boundaries of these intervals
to integers (`truncate=false`) was not supported.
Models that use `truncate=false` include gpt-oss and some PLaMo-3
models; to support them correctly, the GGML operations and CPU backend
are modified to handle calculations without rounding. The default
behavior is `truncate=true`, and existing calls not explicitly
configured will maintain their previous behavior.
* support CUDA backend
* add tests
* move RoPE helpers to internal header
* format
* Add a backend interface for collectives across ranks running in separate processes, implement it in the CUDA backend using NCCL, and use it in the RPC server when available.
* Drop ggml_backend_comm_allreduce_rank_t and reuse ggml_backend_comm_allreduce_tensor for per-rank communicators
* reuse ggml_backend_cuda_comm_allreduce_nccl for per-rank communicators. Remove the helper function backend_comm_proc_address
* Change ggml_backend_comm_get_unique_id_t and ggml_backend_comm_init_rank_t to more generic multi-process backend communicator interfaces
* set maxCTAs in ncclConfig_t to 8 for dgx spark in ggml_backend_cuda_comm_init_rank
* llama : use llm_attn_mask and kv_rows for unified attention
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* cont : optimize `get_seq_cells`
* llama : build the kv_rows lists from per-sequence cell bitmaps
Keep a bitmap over the cells for each sequence (the transpose of `seq`), updated in seq_pos_inc/dec, and build the cell lists by walking its set bits. The lists stay in cell order, cells shared by several sequences go to every list, and nothing has to be rebuilt when cells are removed.
Also leave out of the lists the cells that the SWA hides from every token of the slice (standard and chunked SWA).
* cont : make get_kv_rows_p_min private
* cont : minor metal
* llama : use kv_rows for every ubatch of a unified cache
A ubatch made of equal-length runs with a single sequence each (f.ex. the drafts of several slots) now splits into per-sequence slices.
Any other ubatch (interleaved, uneven or coupled sequences) becomes a single slice over the cells of all its sequences instead of falling back to the dense KQ mask, so the graph keeps the topology of the reserved graph.
* llama : merge get_kv_rows_seq_ids and get_kv_rows_p_min into get_kv_rows_slices
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* WIP OCP FP8 E4M3 support
* Fix python type-checker
* Materialize input_scales also for FP8 when converting
* Fix conversion of https://huggingface.co/RedHatAI/Qwen3.8-27B-NVFP4
1. Per row FP8 wrongly fell to NVFP4 conversion, as NVFP4 checked on
dims alone. Also check on dtype for FP8
2. Need to reshape FP8 QKV projections scales in the same way that
weights are reshaped
* Allow remaining FP8 tensors to be dequantized during conversion
* Allow tensor-strategy for FP8-only compressed-tensor ckpts
* Fix and test gguf-py behavior for FP8_E4M3
* Disallow QKV-fusion for FP8 temporarily
* Fix conversion of fused qkvz & FP8 for Qwen3-Next
* Avoid fusing FP8 for --fuse-gate-up-exps as we may have different ws
* Emit warnings when not fusing FP8 during conversion
* More FP8 conversion fixes
* Add supported for strided copies to/from FP8 in CPU
* remove unneeded scratch buffer from FP8 cpu cpy/dup path
* add explanatory comments and TODOs
* Add dedicated encoder and decoder tests for FP8
* Whitelist ggml_op_get_rows in metal instead of blacklisting
required to fix failing CI, and white-listing should be more
future-proof than adding FP8 to the black-list
https://github.com/ggml-org/llama.cpp/actions/runs/38433916498/job/115422880633?pr=27512
* Fix more conversion bugs
* Defer weight materialization during conversion
* Blacklist fp8 in metal backend instead of white-list
* Adjust indents
* Fix lint + execute tests
* Add fast CPU kernel for F8_E4M3
* Conversion fixes
* Fix remaining gaps
* Enable tied weights with shared scales
* Make mandatory-F32 rules apply to FP8 during conversion
* Fix python type checks
https://github.com/ggml-org/llama.cpp/actions/runs/39765670052/job/119780379595?pr=27512
* Fix kaleidai unit-test failures
https://github.com/ggml-org/llama.cpp/actions/runs/39765670125/job/119780383150
* Relax numeric precision constraints for pre-check
* Leave fast-FP8 CPU on RISC-V for follow-up
We already exclude arm from the fast-path
* Allow output scale tensors to be located in any gguf shard
* Move test_conversion_fp8.py test execution to behavior-check.yaml
* Apply batched suggestions from code review
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Move mixed compressed-tensor logic into dequant_fp8 + add unit-tests for it
* Install all requirements also on Windows for behavior-check
Fixes https://github.com/ggml-org/llama.cpp/actions/runs/40535852590/job/122300758223?pr=27512
* Unmap lazy tensors before clean-up in Windows
Should fix https://github.com/ggml-org/llama.cpp/actions/runs/40539797824/job/122314782459?pr=27512
* LF line endings
https://github.com/ggml-org/llama.cpp/actions/runs/40539798168/job/122314791118?pr=27512
* Extract FP8 tests from llama-arch to its own test-fixture
See https://github.com/ggml-org/llama.cpp/pull/27512#discussion_r4400628221
* Update gguf-py/gguf/scripts/gguf_convert_endian.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* ggml : allow MUL_MAT_ID -1 index to skip the computation
* nits
* update vulkan code
* ggml-webgpu: ensure all threads reach workgroup barriers in mul_mat_id_vec
(cherry picked from commit d8a4b9efb6e2b9abf3a866e847756b7e6bbc6b69)
* fix cuda
* openvino
* exclude CUDA from this PR
* add TODO tag
---------
Co-authored-by: yomaytk <yoshimura.masashi.frbs@gmail.com>
* llama: create new fused op for QSA scoring
* enroll glm5.3-next too
* tests : initialize the remaining tensors in test_kpool_mask
The harness sentinels were left uninitialized, so leftover memory could read as NaN and fail the NaN check.
* ggml: generalize the k-pool mask op into ggml_top_k_mask
The k-pool mask op only covered the pooled QSA selection of qwen4exp
and glm5-next, while the DSA models and the DeepSeek V4 CSA build the
same top-k mask by hand with fill, set_rows and add.
ggml_top_k_mask(mask, top_k, score, blocks, extra) keeps the mask
cells of the live picks and the extra cells, and sets the rest to
-inf. Every row tensor is 4D in the layout of top_k, and the mask is
broadcast to those rows, so a selection can also be made per head.
score, blocks and extra are optional: without score every pick is
live, without blocks a pick is a cell, and extra holds the cells a
row keeps without a pick.
llm_graph_context::build_fused_top_k_mask views the row tensors in
the layout of kq_mask and registers the fused node. The k-pool
models, the DSA attention of build_attn and the DeepSeek V4 CSA mask
use it when the probe enables the op, and keep their current path on
backends without it.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* cuda, hip: GDN chunked kernel
Ported-from: https://github.com/tetherto/qvac-fabric-llm.cpp/pull/257
Source-commit: 53f0027d8070715fe9ad2131f3201bb0001d4eed
Co-authored-by: Aman Gupta <aman.gupta@tether.io>
* fix older builds
* review fixes
* cuda: enable GDN MMA on tested Ampere and Ada GPUs
Enable SM86 and SM89 after RTX 3090 and RTX 4090 correctness and prefill benchmarks. Keep other untested architectures on the existing fallback.
Assisted-by: Codex
* cuda : use refactored swizzle API in GDN MMA
* ggml : GDN only guarantees the final state when n_tokens > K
* cuda : add short GDN MMA phase comments
---------
Co-authored-by: gianni-cor <gianfranco.cordella@tether.io>
Co-authored-by: Aman Gupta <aman.gupta@tether.io>
Add an optional ggml_backend_sched_realloc_callback that user code registers with
ggml_backend_sched_set_realloc_callback to be notified when the buffers reserved for a
graph are not sufficient and have to be reserved again. The callback reports whether the
backend assignments and the graph size changed since the previous scheduling.
llama_context registers the callback when creating the scheduler (wrapped into the new
llama_context::sched_new helper) and treats the reallocations that follow a change of the
backend assignments as expected, counting the rest by graph size - unchanged or changed
from the previously scheduled graph. Both counters are reported on one line in the perf
stats.
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
* ggml: refactor selective expert copying to user code
* tests: enroll two models into selective expert copy test
* tests: use deepseek2 as test model
* improve comment in ggml-backend.h
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* cont: fix whitespace
* cont : better comments
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* ggml : add `alloc_buffer_n` to buffer type interface
Add alloc_buffer_n method to ggml_backend_buffer_type_i
interface, with a public API ggml_backend_buft_alloc_buffer_n.
- Default implementation in ggml-backend.cpp handles multi-buffer
splitting and tensor allocation via ggml_tallocr
- Meta buffer type provides custom implementation that creates
per-device sub-contexts and delegates to simple buffer types
- ggml_backend_alloc_ctx_tensors_from_buft now collects tensors
into a list and delegates to the new API
- Remove temporary ggml_backend_meta_alloc_ctx_tensors_from_buft
- Add NULL alloc_buffer_n to all existing buffer type
interfaces (cpu, metal, openvino, hexagon, webgpu, zdnn, virtgpu, repack)
Assisted-by: llama.cpp:local pi
* cont : fix `cur_buf_size` init after flushing a buffer
* ggml : add TODO tag for shared buffer split logic
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* tests : add alloc_buffer_n coverage
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* cont : fix compile warnings
* tests : add descriptions for alloc_buffer_n tests
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* ggml : address review comments on alloc_buffer_n
- restore GGML_LOG_ERROR on buffer alloc / tensor init failure in the
default impl (name the failing tensor)
- check the malloc result and drop the _impl indirection in
ggml_backend_alloc_ctx_tensors_from_buft
- remove comments that restate the code
- fix the TAG_ALLOC_SHARED_BUFFER_SPLIT typo
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* ggml : add get_alloc_size_n to buffer type interface
- Add ggml_backend_buft_get_alloc_size_n public API
- Add optional get_alloc_size_n callback to ggml_backend_buffer_type_i
- Share tensor->buffer planning between alloc_buffer_n default and get_alloc_size_n default
- Replace unchecked realloc with std::vector in alloc_buffer_n default
- Make ggml_backend_alloc_ctx_tensors_from_buft_size use the new API
- Add test-alloc coverage for get_alloc_size_n
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* cont : report malloc failure
* rpc : hash-cache only weights
ggml_backend_rpc_buffer_set_tensor and ggml_backend_rpc_set_tensor_async
hashed every transfer above HASH_THRESHOLD and let `rpc-server -c` serve it
from its file cache. The cache is meant for weights, but the activations
ggml_backend_sched copies between backends took the same path: with a
two-node split of Qwen3.8-Flash-Next every prefill ubatch above 10 MB was
hashed, written to the worker's cache directory (1.4 TB after a day) and
later served from there. Use the hash path only for tensors in buffers
marked GGML_BACKEND_BUFFER_USAGE_WEIGHTS.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* rpc : save a cache entry only for the tensor that missed the hash check
With the client hashing weights only, the server still wrote every
SET_TENSOR above HASH_THRESHOLD to the cache directory, so the compute
data the scheduler sends kept filling the disk. Remember the hash of the
last SET_TENSOR_HASH that missed and save only the SET_TENSOR that
follows it with that hash - the weight the client is re-sending.
* rpc : signal the cache decision in the SET_TENSOR payload
Replace the server-side `pending_cache` state with a `cache_flag` byte
in the SET_TENSOR message: the client sets it when SET_TENSOR_HASH
reported a miss, the server saves a cache entry only when it is set.
Bump RPC_PROTO_MAJOR_VERSION since the wire format changes.
---------
Co-authored-by: Patrick Hoffmann <patrickhoffmann@MacBook-Pro-14-HOP.local>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
some backends (Metal, SYCL, WebGPU) require additional memory for
fleeting data for certain ops, which is reflected in their
get_alloc_size implementations.
add ggml_backend_op_alloc_size_may_expand() to the backend utils,
listing these ops, and assert in ggml_backend_buft_get_alloc_size
that a backend expanding the alloc size of a compute op only does so
for ops listed in the helper.
use the helper in the RPC backend to decide whether to query the
remote server for the actual alloc size, instead of a hardcoded list.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* add params
* cpu kernel
* metal kernel
* add test backend ops
* gate other backends
* ggml: (cuda) support ggml_rope_set_offset (#27121)
* rm cuda supports_op guard, fix webgpu clang-format
* ggml: support ggml_rope_set_offset on vulkan (#27344)
* ggml: support ggml_rope_set_offset on vulkan
* remove inplace optimization
* Initial changes for Recurrent state rollback for nemotron for cpu and cuda
* Removing CPU RS rollback. Will enable it in subsequent PRs
* addition of test case
* Removing assert and calling runtime API to check if op is supported
* removing extra API and updating the call sites for K
* replace static cuda detection to runtime fused_op api
* address review comments and fallback when SSM rollback not supprted
* Adding changes for supporting RS-rollback in CPU. Also added test-backend-ops for cpu and cuda
* removing memory manipulation as rs rollback is now supported in CPU
* removing the static probe which is not needed now
* correcting the format
* address review comments
* enabling test for all the backends, unsupported backends will fallback to CPU
* Apply suggestions from code review
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* choose different graph based on the result of fused_ssm_op is supported or not and also handled memory->n_rs_seq >1 case incase of op is not supported
* Support K > 1 in ssm_scan for all backends
* Fix CI Issues
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Co-authored-by: Gaurav Garg <gaugarg@nvidia.com>
* llama: add new default load-mode auto which picks mmap unless a non-Metal iGPU is used
* Update ggml/src/ggml-hexagon/ggml-hexagon.cpp
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
* set mmap_support to false on OpenCL backend
* fix order of load modes
* use -1 for auto
* resolve load mode auto earlier to correctly pick gpu host or cpu memory
* add load mode auto to llama-bench
* bump virtgpu api version, regenerate docs
---------
Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com>
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* ggml: add ggml_build_forward_order
ggml_build_forward_expand marks the tensor and all its ancestors for
compute, so using it as a pure ordering hint (keeping q, k and v
together) defeats ggml_build_forward_select: the unselected branch is
forced to run with inputs that were never uploaded. In the mtmd audio
graph this makes GEN_WAV calls execute the GEN_CODE branch with a
stale inp_code0, hitting the get_rows bound assert on CPU.
Add ggml_build_forward_order, which inserts nodes without the compute
flag; the flag is restored when the branch is actually selected.
Switch the q/k/v hints in clip_graph::build_attn to it.
* nit: reduce comments (AGENTS.md)
The current integration treats SME as a single capability (CPU_FEATURE_SME)
with no distinction between SME(v1) and SME2. The kernels dispatched under
CPU_FEATURE_SME use SME2-specific instructions, making dispatch incorrect
on SME(v1)-only hardware.
We introduce build-time and runtime distinction between SME and SME2, and
wire SME(v1) and SME2 kernels based on actual hardware support.