7e960f40fb0bd2eede76406f807d6bc3f7aa339c
169
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
9a080f44b5 |
ggml-openvino: fix OpenVINO CI, minor fixes and optimizations (#32590)
* ggml-openvino: round the GPU KV length up to a power of two
Review notes (follow-ups, not blocking):
- convert_ggml_input_to_ov pads the KQ mask into a std::vector and then
copies it into the ov::Tensor. Writing the padded rows straight into
padded.data() avoids the second allocation and copy (up to 16 MB per
call at a 16K context).
- The padding is skipped when the mask is not contiguous, and the KV
inputs are then sliced to more rows than the mask has columns. Masks
are always contiguous today; a GGML_ASSERT would make that explicit.
- compute_llm_params reads GGML_OPENVINO_STATEFUL_EXECUTION directly;
is_stateful_enabled() is local to ggml-openvino.cpp.
* ggml-openvino: Optimize model recompiles of prefill stage
Optimize the model recompiles of prefill stage when attended size
changed between prompts. This can save model recompile time for
llama-bench and llama-server.
* ggml-openvino: Optimize model recompiles of prefill stage
Optimize the model recompiles of prefill stage when attended size
changed between prompts. This can save model recompile time for
llama-bench and llama-server.
* ggml-openvino: Remove solution C and keep A as default
* ggml-openvino: match the output count before reusing a compiled NPU prefill
The static prefill model has a fixed inp_out_ids shape [1,1,1,output_len].
The new reuse check compared only the attention sizes, so llama-perplexity,
which alternates 1 and 512 outputs per ubatch, reused the model compiled
for 512 outputs on a 1-output ubatch and failed with "got [1,1,1,1]
expecting [1,1,1,512]". Store the output count of the cached prefill
model and require it to match.
* ggml-openvino: Fix LFM2 context create on GPU and DSpark speculative decoding
* ggml-openvino: remove leftovers from merging dev into the LFM2/DSpark fix
- compute_model_outputs exported OUTPUT-flagged nodes twice: dev's block
ran before the new one and still exported result_norm, so the
result_norm exclusion had no effect. Keep only the new block.
- compute_node_dynamic_dims checked is_activation_input twice in a row.
Remove the second copy.
- process_view_input_new created the dynamic Split on split_dim. With
stateful execution the input drops the leading size-1 axis, so use
split_axis, which already accounts for the rank difference.
* ggml-openvino: hash the output flags into the model-cache fingerprint
The fingerprint hashed the ops, names and weights but not the outputs, so
a blob compiled with another output set was imported. With the result_norm
change, an NPU blob written before it gave wrong output, and enabling
llama_set_embeddings_layer_inp() reused a blob without the layer-input
outputs. Hash each node's output flag and bump the cache version to 8.
* ggml-openvino: support f16 SCALE and f16/bf16 TRI
test-backend-ops now runs SCALE on f16 and TRI on f16 and bf16. Both
translators built their constants as f32, and Multiply, Add and Select
need all inputs in one element type, so building the graph threw for
these cases.
SCALE now converts a non-f32 input to f32, applies the scale and bias,
and converts back once, as the CPU backend computes it. TRI creates its
zero constant in the input's element type. f32 graphs are unchanged.
* ggml-openvino: [Qwen3.6 A3B]fix static-shape ids view of the last MoE
layer
The same-stride single-dim Slice in process_view_input_new was followed by a Reshape
whose target baked the static chunk size into the token axis. After the output-ids
GET_ROWS the last layer runs on n_outputs tokens (1 in the prefill graph), so the
Reshape failed shape validation for qwen35moe with FORCE_STATIC or on the NPU.
Skip the Reshape in static graphs when the slice already keeps the rank;
the dynamic graphs are unchanged.
* enable mamba based models on OV CPU-GPU
* revert changes in tests
* docs : add Falcon-H1 to the OpenVINO validated models
Falcon-H1 runs on CPU and GPU since Mamba SSM support was added. Add its
row and the known limitations: multi-sequence batches fail, GPU accuracy
is slightly lower than CPU, and the NPU fails with an element type
mismatch.
* ggml-openvino: simplify the SSM_SCAN build check and clean up
- Drop the SelectiveSSM configure check and its guards: OpenVINO 2026.4.1
always provides the op. Drop the no-op LANGUAGE CXX source property.
- Drop the semicolon after OPENVINO_OP("SelectiveSSM"). GCC 15 reports it
with -Wextra-semi, which fails builds with fatal warnings.
- Add ggml_openvino_is_static() for the repeated NPU or
GGML_OPENVINO_FORCE_STATIC check.
- Share the f16 and f32 KQ mask padding code.
- Drop the redundant inline on in-class static functions.
---------
Co-authored-by: Suhas Somashekar <suhas.somashekar@intel.com>
Co-authored-by: yzhou51 <yaowei.zhou@intel.com>
Co-authored-by: Ananth, Swathi <swathi.ananth@intel.com>
Co-authored-by: Beheshti <nazanin.beheshti@intel.com>
|
||
|
|
08c252e221 |
opencl: allow adjusting KV padding in decomposed FA (#32522)
Assisted-by: Codex |
||
|
+4 |
4577932fab |
ggml-openvino: expand model support, optimize NPU,GPU and memory usage (#32172)
* ggml-openvino: fix SWA mask detection When the SWA cache size matches the full cache size, identify SWA layers by the number of layers using each mask instead of cache size; the larger group is SWA, and a tie leaves the layers unclassified. * ggml-openvino: fix SWA KV cache write indices Give SWA KV cache index tensors unique input names (a _swa suffix, like the masks) to prevent them from being merged with the full-attention cache indices. * ggml-openvino: classify SWA layers only when the mask shows the window Two attention masks do not imply SWA: DeepSeek sparse attention uses two full-attention masks with the same name. When cache extents match, classify the larger mask group as SWA only once its mask has fewer keys per row than the other mask. Keep the classification per KV cache buffer so new sequences reuse the same graph, and include swa_layers when checking graph reuse. * ggml-openvino: classify SWA layers from the first graph Waiting for the window to appear in the mask caused a mid-sequence graph recompile (~9 s for Gemma-4 at -c 1024 after position 512). SWA classification already gives each mask group its own mask, KV write indices, and attention size, which is correct whether or not the mask is a sliding window. Stateful mask rebuild recovers the window from the mask, or the causal mask for full attention. Classify the larger of two mask groups from the first graph and drop the per-KV-buffer bookkeeping. * ggml-openvino: mark SWA fallback as a model-level heuristic Add a TODO above the equal-extent fallback to document that the larger mask group is currently assumed to be the SWA group. Since stateless graphs only require the two groups to remain distinct, this heuristic may only be necessary for stateful execution and could potentially be restricted to that path. * ggml-openvino: derive attention sizes from the KQ mask shape Dynamic CPU/GPU graphs previously passed n_seq_active and attention_size(_swa) as separate inputs. Derive these values directly from the KQ mask shape using ShapeOf + Gather inside the model. This allows the GPU plugin to infer the Q reshape and KV slice shapes from shape inference rather than runtime input values. Q's head size also becomes static, enabling the fused SDPA kernel. GGML_OPENVINO_DISABLE_SHAPE_FROM_MASK=1 restores the previous separate-input behavior. The fused SDPA path also requires two supporting changes: - Clamp the GPU mask to -30000. A key block that is entirely -inf for a row produces NaNs in the fused kernel. - Restate the head count and head size after the KV slice. The plugin loses all slice dimensions when the end index is dynamic, causing SDPA to fall back to the reference kernel. Additional graph and quantization improvements: - Skip the identity reshape of rows in SET_ROWS. - Share a single Split for one-element view slices in dynamic GPU graphs (e.g. Gemma-4 per-layer embeddings), replacing one StridedSlice per layer. - Add GGML_OPENVINO_REQUANT_EMBD=native (off by default) to preserve the file's native quantization for token_embd/output instead of requantizing to Q8_0_C. - Include both environment settings in the graph and model cache keys and document them. * ggml-openvino: only reuse Split outputs for exact chunk-aligned views The dynamic GPU path reused a Split output for any single-element view by deriving the chunk index from offset / stride. This produced incorrect results for views that were not chunk-aligned or whose remaining dimensions did not preserve the source strides, as seen in test-backend-ops CONT with use_view_slice. Restrict Split reuse to cases where the view exactly matches a split chunk, meaning the offset is aligned to the split stride and the remaining non-unit dimensions preserve the original source strides. Gemma-4 per-layer embedding slices continue to use the shared Split path. * ggml-openvino: initialize ModelExtraInputInfo shape fields Initialize the newly added `shape_source` and `shape_axis` fields in the `ModelExtraInputInfo` brace initializer used by `add_extra_inputs()`. Without initializers for these fields, GCC emits `-Wmissing-field-initializers`, causing CI to fail with `LLAMA_FATAL_WARNINGS`. * ggml-openvino: remove GGML_OPENVINO_REQUANT_EMBD The option identifies embedding and output weights by their tensor names, which are defined by core llama.cpp. Remove it from this PR; preserving the file quantization of these weights will be addressed in a separate change that identifies them by their roles in the graph. * ggml-openvino: keep manual GQA attention 4-D The manual GQA attention path reshaped Q, K, and V to 5-D so that query heads sharing a KV head could broadcast against it. The GPU plugin cannot fuse this graph: it materializes the attention scores and copies the transposed K and V caches for every layer. Instead, stack the query heads for each KV head along the sequence axis (Q [B, H, S, D] -> [B, H_kv, factor * S, D]) and tile the mask accordingly. This keeps the graph 4-D from the MatMuls through the Softmax, allowing the GPU plugin to fuse the attention into its SDPA kernel and fold the K and V transposes into the fused operation. Clamp the mask to -30000, matching the SDPA path, because the fused kernel can return NaN when an entire block is masked with -inf. Models where the K or V head size differs from the Q head size now use the SDPA path, since the manual path assumes one head size. * ggml-openvino: use a 4-D stateless KV cache on GPU In stateless GPU mode, bind the entire KV cache as a `[1, ctx, n_heads_kv, head_size]` tensor, with one token per row, instead of binding the first `n_kv` rows as a `[1, 1, ctx, n_heads_kv * head_size]` tensor. Write cache updates with `ScatterUpdate` along the token axis, and read the cache through `VariadicSplit` and `Transpose`. This allows the GPU plugin's `StatelessKVFusion` to replace cache reads and writes with in-place `StatelessKV` nodes, avoiding full-cache copies on every decode step and repeated memory pool allocations as the bound cache size grows. Set `GGML_OPENVINO_DISABLE_KV_4D=1` to retain the previous layout. * make embedding to have dynamic shape for OV GPU * ggml-openvino: make the 4-D stateless KV cache opt-in * add support for Q5_0 data type in OV GPU backend * resolve CI test error * ggml-openvino: fix -fa off and optimize grouped-query attention With -fa off, llama.cpp stores the V cache transposed, with value elements as rows and tokens along each row. The KV slicing code assumes one token per row and binds the wrong data when n_kv < kv_size. Detect transposed caches with GgmlOvDecoder::is_kvcache_token_rows() and skip slicing for them. For grouped-query attention, avoid repeating K and V for each query head. Stack the query heads for each KV head along the row dimension and let SOFT_MAX apply the scale, mask, and softmax so the GPU plugin can fuse the operation with the MatMuls. Read only the current layer's V-cache rows when layers use different cache sizes. Reject -fa off in stateful mode with a clear error and document that it requires stateless mode. * ggml-openvino: fall back to stateless execution when -fa off With `-fa off`, the graph writes the V cache in transposed form, which is not supported by the stateful KV cache. Detect this pattern in `compute_llm_params()` as a `SET_ROWS` into a KV cache through a `[1, n]` reshape. When stateful execution is enabled, log a warning and fall back to stateless execution instead of failing graph conversion. Update the `-fa off` note in `OPENVINO.md` to document this behavior. Co-authored-by: Suhas Somashekar <suhas.somashekar@intel.com> * ggml-openvino: preserve flat KV cache layout when -fa is off With -fa off, the 4-D KV cache layout introduced in #344 breaks MUL_MAT attention for Phi-3.5 mini and Phi-4 mini, causing oneDNN to fail with "could not construct a memory descriptor using strides". Preserve the [1, 1, ctx, n_heads_kv * head_size] layout when the graph writes a transposed V cache. * ggml-openvino: extract Q5_0 weights when requantization is off Q5_0 weights are requantized to Q4_0, but test-backend-ops builds naive graphs, where requantization is skipped. Q5_0 then reached the normal weight extraction, which had no Q5_0 case, so every Q5_0 MUL_MAT and GET_ROWS op test failed with "Unsupported quantized type: q5_0". Extract Q5_0 as symmetric i8 weights (q - 16) with one f16 scale per 32 weights, the same layout as Q8_0. Models are unchanged: they still requantize Q5_0 to Q4_0. * ggml-openvino: Optimize GDN output pre-scale for the NPU static path The NPU GatedDeltaNet kernel writes its output in FP16 and flushes values below 6.1e-5 to zero. * Refine the code as review comments 1. Add envs desc into doc 2. Refine the code with removing duplicated env * ggml-openvino: reuse graph bindings between decode calls Reduce host overhead outside `infer()` from 0.5–1.1 ms to 0.14–0.38 ms per token on Panther Lake. - Reuse KV cache tensors and skip redundant port bindings. - Use persistent USM host tensors for small GPU inputs and outputs. - Defer graph-wide consumer lookup until after shape checks. - Return model and compute params by const reference. - Preserve tensor extras when memory, shape and type are unchanged. * ggml-openvino: Add npuw dump envs for profiling * Add the comments of new npu dump envs into doc * ggml-openvino : ensure all leaves have unique names, remove input_srcs from graph key * openvino: slice NPU prefill KV cache by active length Add GGML_OPENVINO_NPU_KV_SLICE as an opt-in NPU prefill optimization. When enabled, static prefill keeps the graph's attention dimension at the active prompt/KV length instead of expanding it to the full context capacity, and the KV-cache input/output tensors are bound with a shortened context axis over the same backing storage. This reduces the amount of KV cache and attention-mask data imported by the NPU plugin and lets the prefill graph plan smaller attention buffers for early prompt chunks. For example, with a 4096-token context and a 256-token prefill chunk, the first chunk can expose KV/mask dimensions at 256 instead of 4096, avoiding work over padded cache rows. The trade-off is graph stability: NPU static shapes now depend on the aligned active attention length for prefill, so a cached prefill model is reused only when the attention sizes match. Decode keeps a fixed full-context graph to avoid recompiling on every generated token, and SWA/ring-cache cases still fall back to full-size binding when the live prefix is not a simple leading slice. * openvino: use L0 host tensors for NPU buffers Add GGML_OPENVINO_NPU_L0_HOST_TENSORS as an opt-in NPU buffer allocation mode. When enabled, non-remote OpenVINO backend buffers on NPU are allocated through the NPU default context with create_host_tensor(), and ggml tensor data points directly at that OpenVINO-owned Level Zero host allocation. The optimization targets the data path rather than the graph math: KV-cache and compute tensors can be passed back to the NPU plugin as OpenVINO tensors backed by importable Level Zero host memory. That avoids an extra ordinary-host-memory-to-plugin-staging copy on each infer boundary, which is especially useful for prefill where large KV/cache buffers dominate input and output traffic. If Level Zero host allocation fails, the backend logs a warning and falls back to the existing allocation path. The destructor tracks OpenVINO-owned storage so it does not free memory owned by the host tensor. The trade-off is that this is NPU-specific, consumes Level Zero host allocation resources, and improves transfer/import overhead without changing attention compute complexity or tensor shapes. * openvino: fill NPU prefill masks directly Add GGML_OPENVINO_NPU_FAST_MASK as an opt-in NPU static prefill input-packing optimization. When enabled, the prefill attention mask is built directly in the OpenVINO input tensor instead of first materializing a padded std::vector and then copying that whole buffer into the tensor. The new fill_prefill_mask() helper writes each row once: it copies the valid mask prefix, fills padding columns with -inf, handles padded rows, and restores the causal diagonal to zero. This keeps the final mask numerically identical to the existing pad_input() plus set_zero_diagonal() path while removing one intermediate allocation and one chunk_size-by-context_size host copy per prefill chunk. The optimization targets CPU-side staging overhead, not attention math. It composes with GGML_OPENVINO_NPU_KV_SLICE: KV slice reduces the mask dimensions, while FAST_MASK reduces the work needed to construct whatever mask shape remains. The trade-off is a second mask construction path to maintain, with limited benefit for very small prompts or contexts. * openvino: make NPU requant layout configurable Add GGML_OPENVINO_NPU_REQUANT_POLICY as the NPU-side selector for the default weight requantization layout. The existing NPU behavior remains the default: unset or group-128 maps quantized weights to Q4_0_128, while channel-wise selects the experimental Q4_0_C layout. Unknown policy values fail fast during model loading instead of silently compiling a graph with unintended weight packing. The optimization target is NPU weight bandwidth and compressed-weight code generation. Q4_0_128 keeps weights in a 4-bit symmetric layout with one scale per 128 values, reducing both payload bytes and scale metadata versus smaller groups while preserving a more conservative quantization granularity than channel-wise. Q4_0_C uses one scale over the innermost dimension, which can further shrink metadata and simplify dequant broadcast, but may cost accuracy because the quantization block is much larger. Include the selected NPU requant policy in the OpenVINO model cache discriminator. The same cgraph compiled with group-128 and channel-wise has different weight constants and dequant shapes, so reusing a cached decoder across policies would be incorrect. This keeps cache reuse stable within a policy while forcing recompilation when the layout policy changes. Trade-off: policy selection gives a performance/accuracy knob for NPU experiments, but all options still pay the load/compile-time requantization cost. The default remains group-128 for a safer balance of bandwidth, metadata, and accuracy; channel-wise is intentionally documented as experimental. * openvino: centralize NPU plugin config overrides Add GGML_OPENVINO_NPU_CONFIG as the recommended catch-all override for OpenVINO NPU plugin and compiler properties. The value is parsed as comma-separated KEY=VALUE pairs and applied after the built-in defaults and named NPU aliases, so advanced users can override any exposed property without adding a new llama.cpp environment variable for every plugin tuning knob. Keep the common knobs as named aliases for discoverability and compatibility: GGML_OPENVINO_NPU_COMPILER_TYPE maps to NPU_COMPILER_TYPE, GGML_OPENVINO_NPUW_FUNCALL_FOR_ALL maps to NPUW_FUNCALL_FOR_ALL, GGML_OPENVINO_NPUW_UNFOLD_IREQS maps to NPUW_UNFOLD_IREQS, and GGML_OPENVINO_COMPILATION_NUM_THREADS maps to COMPILATION_NUM_THREADS. GGML_OPENVINO_NPU_COMPILE_CONFIG is retained as a legacy alias for NPU_COMPILATION_MODE_PARAMS, while the documentation now recommends expressing it through GGML_OPENVINO_NPU_CONFIG. This keeps the public tuning surface compact while preserving high-signal aliases for settings with known performance or stability implications. NPU_COMPILER_TYPE can select the driver compiler, which currently generates faster prefill kernels on the tested NPU stack. NPUW_FUNCALL_FOR_ALL and NPUW_UNFOLD_IREQS expose NPUW graph lowering trade-offs around dispatch overhead, memory use, and long-context stability. COMPILATION_NUM_THREADS gives a way to cap compiler worker parallelism to reduce peak host memory during large graph compilation. Trade-off: a free-form KEY=VALUE string is less type-safe than one env var per option and invalid plugin properties will only be diagnosed by the OpenVINO stack. Applying it last is intentional: it gives one escape hatch for bisecting driver/compiler behavior and for testing new plugin options without rebuilding llama.cpp, while the named aliases continue to document the most important performance knobs. * openvino: add scatter-elements KV update option Add GGML_OPENVINO_KV_SCATTER_ELEMENTS as an opt-in lowering for non-stateful single-row KV cache writes. When the flag is enabled and the set_rows indices are the simple decode case, translate_set_rows uses ScatterElementsUpdate with broadcasted rank-4 indices instead of ScatterUpdate. The motivation is NPU decode performance. The default ScatterUpdate path updates one KV row but the NPU plugin does not perform that update in place; it effectively copies the full destination tensor to write a small slice. As context length grows, that full-tensor movement dominates the per-token KV update cost. ScatterElementsUpdate expresses the same single-row write at element granularity and is measurably cheaper for this shape. Keep the optimization guarded. Stateful execution keeps its existing concat/update path, and multidimensional indices already use the broader ScatterElementsUpdate lowering. The new flag only changes the common non-stateful decode write, so the default behavior and graph shape remain unchanged unless the user opts in. Trade-off: ScatterElementsUpdate introduces extra ShapeOf, Reshape, and Broadcast nodes to build indices matching the update tensor. That overhead is worthwhile only when it avoids the NPU plugin's full-destination ScatterUpdate copy, so the path is controlled by an environment variable rather than becoming the unconditional lowering. * openvino: gather token embeddings from compressed rows Add NPU token-embedding controls for models whose token_embd.weight would otherwise be widened from Q6_K to FP16. GGML_OPENVINO_TOKEN_EMBD_I8 keeps the embedding table on the normal int8 compressed-weight path, while GGML_OPENVINO_TOKEN_EMBD_I4 requantizes it to the group-128 int4 layout used by the NPU weight path. The performance target is tied embeddings and large vocabulary tables. When token_embd.weight is also consumed by the lm_head matmul, a normal Gather from the dequantized table gives the dequantization subgraph a second consumer and can force the full table to be materialized. The new gather_compressed_rows helper recognizes the dequantization chain emitted for int4/int8 constants, gathers only the selected packed rows from the leaf constants, and rebuilds the convert/subtract/multiply/reshape chain on those rows. This reduces memory pressure and avoids dequantizing or materializing a full embedding table when inference only needs the token rows for the current input. The helper is deliberately conservative: it only rewrites static 2D tables above a size threshold, walks only the known compressed-weight node patterns, limits recursion depth, and falls back to the original Gather when the subgraph is not recognized. Trade-off: int8 and especially int4 token embeddings can change accuracy and may not be faster for small untied embedding tables, because the rewrite adds extra Gather/Reshape nodes and only pays off when it avoids a large full-table dequantization. The default behavior is unchanged; the NPU-specific I8/I4 paths are opt-in through environment variables. * openvino: stream u4 requantization chunks Extend the GGML_OPENVINO_REDUCE_COMPILE_MEM streaming requantization path to Q4_0/u4 targets. The previous streaming path intentionally skipped u4 because Q4_0 packs two weights per byte and the zero-point tensor stores two 4-bit values per byte, so the original quantize_q4_0 helper wrote as if every call started at block zero. Teach quantize_q4_0 to accept a destination block offset and use the absolute block index for weights, scales, and zero-point writes. Streaming requantization emits whole-row chunks in increasing order, and the target block size is required to divide the row width, so no target block straddles a chunk boundary. That keeps packed weight bytes aligned and preserves the even-block assignment before the odd-block zero-point OR for each shared zero-point byte. With that indexing in place, reduce-compile-memory mode can dequantize a chunk of rows into the scratch buffer and immediately quantize it into the final u4 buffers instead of materializing the entire tensor as temporary F32. This extends the peak-memory reduction already used by Q8/F16 targets to the NPU group-128/channel u4 layouts used for compressed weights. Trade-off: the u4 path is more sensitive to block alignment than Q8/F16 because of nibble packing, so streaming remains guarded by the same row-divisibility check and only runs when GGML_OPENVINO_REDUCE_COMPILE_MEM is enabled. When the flag is off, the full-materialization path and quantized output layout remain unchanged. * openvino: fail fast on static recompile after weight release Extend the GGML_OPENVINO_RELEASE_WEIGHTS safety check to the static graph cache-miss path. Once host weight buffers have been released, any later compile would read invalid/zeroed host pages and bake the wrong constants into a new OpenVINO model. The dynamic path already rejected this situation. Static execution can still miss the decoder cache when graph shapes or model parameters change, so it needs the same guard before erasing cached infer requests and rebuilding prefill/decode models. This is a correctness and diagnosability change rather than a throughput optimization. It preserves the memory-saving release-weights mode for stable graph shapes, but turns unsupported recompilation into an explicit abort instead of silently producing a corrupted compiled model. * openvino: add experimental NPU Q4_0 keep option Add GGML_OPENVINO_NPU_KEEP_Q4_0 as an opt-in escape hatch for NPU weight handling. When enabled for native Q4_0 tensors, the NPU requant path returns nullopt and keeps the original Q4_0 extraction instead of regrouping the tensor through the selected NPU requant policy. The motivation is to avoid unnecessary work for weights that are already stored as 4-bit values. The default NPU policy maps quantized weights to layouts such as Q4_0_128, which can reduce scale metadata but requires round-tripping Q4_0 through the requantization path. Keeping Q4_0 native can save that conversion work and preserve the original group-32 packing. Keep the option experimental and disabled by default. Current NPU driver/plugin stacks can compile the resulting group-32 graph, but inference may hang or fail with ZE_RESULT_ERROR_DEVICE_LOST. The default behavior therefore continues to use the safer NPU requant policy; this flag is mainly useful for debugging driver behavior, measuring the true cost of regrouping, and testing future NPU stacks. Trade-off: native Q4_0 avoids regrouping overhead and keeps the original quantization blocks, but it gives up the more regular group-128 layout expected by the default NPU compressed-weight path and may be unstable on current hardware/software combinations. * openvino: unfold NPUW only for decode graph Compile the static prefill and decode models with separate OpenVINO config maps so the decode graph can opt into NPUW_UNFOLD_IREQS without forcing the same lowering on prefill. The build_static_model helper now receives the config map explicitly, and the decode path gets a copy of the base config with NPUW_UNFOLD_IREQS=YES when the user has not already set it. The performance target is NPU token generation. Decode executes a long stream of single-token infers, where folded NPUW function-call dispatch overhead becomes visible in every token. Unfolding the NPUW calls into separate infer requests removes that repeated dispatch cost and measured about +20% token-generation throughput on the tested stack, matching the behavior seen in OpenVINO GenAI. Prefill keeps the original folded config. Prompt processing is a batched workload dominated by larger matmuls, and the folded form is faster there while also using less memory. Splitting the config lets prefill and decode use the lowering that matches their workload shape instead of treating NPUW_UNFOLD_IREQS as a single global compromise. Trade-off: unfolding can increase memory and request-management overhead, so the automatic default is limited to the NPU decode model. If a user explicitly provides NPUW_UNFOLD_IREQS through the named env alias or GGML_OPENVINO_NPU_CONFIG, that value is respected because the decode override is only applied when the property is absent. * openvino: tidy NPU tuning helper ordering Collect the remaining small cleanups from the NPU tuning series. This reorders the NPU KV-slice documentation and environment-variable cache entry next to the other NPU opt-in toggles, adds a short header comment for the centralized NPU helper declarations, and makes the static prefill attention-size assignment in the decoder a little more explicit. There is no intended behavior change in this commit. The purpose is to make the final source layout match the split commits that introduced the NPU tuning knobs, so future readers can find the related flags and helpers in one place without mixing that mechanical cleanup into the functional performance commits. * openvino: normalize NPU compile env overrides Move the default NPU_COMPILER_TYPE=DRIVER setting into the NPU compile_config initializer so the default NPU plugin options are declared in one place. Route GGML_OPENVINO_NPU_COMPILE_CONFIG and GGML_OPENVINO_NPU_COMPILER_TYPE through the same set_compile_option_from_env helper used by the other named NPU compile option aliases. This keeps the legacy NPU_COMPILE_CONFIG mapping to NPU_COMPILATION_MODE_PARAMS intact while making all optional env-to-compile_config overrides follow the same non-empty-value rule. GGML_OPENVINO_NPU_CONFIG is still parsed last, so explicit comma-separated KEY=VALUE overrides continue to have the highest precedence over both defaults and named aliases. * openvino: stream NPU weight requant by default to cut compile peak Default the streaming requantization path on for NPU so the compile-time F32 dequant transient (~1-2 GB for token_embd) is capped without an env flag. Output is byte-identical to full materialization; GGML_OPENVINO_REDUCE_COMPILE_MEM / MEMORY_OPTIMIZE still override (set to 0 to force off). Kept the g_nonov_weight_cache opt-in to avoid holding token_embd resident (which regressed decode steady memory). pp1024_d0: PrefillPeakPrivWS 5974->4664 MiB (-22%), decode steady and pp/tg throughput unchanged. * openvino: add self-contained NPU model cache Add an opt-in embedded-weight cache mode for the static NPU backend. GGML_OPENVINO_SELF_CONTAINED_BLOB=1 exports separate prefill and decode compiled models with ENABLE_WEIGHTLESS=false, uses isolated NPUW weight banks, validates raw GGUF weight manifests, binds imported ports by deterministic frontend order, and serializes the large NPUW compile/export/import operations. Preserve raw model bytes during cold cache construction by requantizing into separately owned OV tensors instead of rewriting the backend model buffer. This keeps later graph fingerprints and manifests stable when llama-bench creates independent prompt and generation contexts. Add strict GGML_OPENVINO_SELF_CONTAINED_MMAP=1 warm-import mode. The backend exposes a non-owning buffer_from_host_ptr wrapper over the GGUF mmap, uses raw tensor allocation sizes, avoids allocating/copying the private OpenVINO model-weight buffer, and stages the few mmap-backed runtime constants into small importable NPU host tensors. Invalid or missing cache entries fail clearly instead of trying to requantize into read-only mapped storage. The normal path and existing weightless MODEL_PTR cache remain unchanged when the new flags are unset. Verified with phi-4-mini: chunk512 pp3072/tg512 cold 1123.10/27.60 t/s and warm mmap 1124.72/27.16 t/s; warm mmap removes roughly 3.9 GiB peak commit in the measured prefill workload. Self-contained blobs consume about 8.34 GiB for the separate pp and tg contexts. * openvino: add lazy import for self-contained mmap cache Add GGML_OPENVINO_SELF_CONTAINED_LAZY_IMPORT to make the low-working-set strict mmap path opt in instead of changing the default warm mmap behavior. When the flag is set with SELF_CONTAINED_BLOB=1 and SELF_CONTAINED_MMAP=1, the static NPU path imports only the current phase blob: prefill imports the prefill blob, decode imports the decode blob later. This avoids the pair-shaped shared compiled-model cache for the lazy path and caches only the current phase infer request. The regular warm mmap path remains eager and continues to prepare both prefill and decode compiled models. Non-mmap self-contained builds remain serial, and the regular non-self-contained path keeps the existing parallel build behavior. Also register the new environment variable, document the memory/latency tradeoff, and add profiling-only import timing around model-cache imports. Validated with ReleaseOV build: cmake --build build/ReleaseOV --target llama-bench llama-simple --config Release. Lazy warm mmap run measured PrefillPeakWS around 5759 MiB versus the eager warm mmap path around 7897 MiB. * openvino: release GGUF mmap resident pages before blob import Add GGML_OPENVINO_SELF_CONTAINED_RELEASE_MMAP_PAGES as an explicit opt-in for self-contained compiled-model cache hits. Before importing a valid self-contained blob, recursively walk the graph to find unique external GGUF mmap-backed weight buffers and drop their resident pages while preserving the mapping address and tensor pointers. On Linux use page-aligned madvise(MADV_DONTNEED). On Windows, where there is no reliable per-range mapped-file equivalent, call VirtualUnlock for the identified range and trim the current process working set with SetProcessWorkingSetSize(-1, -1). Pages required later can fault back on demand. Keep the behavior disabled by default and document the physical-memory versus commit-memory tradeoff. Profiling reports the submitted GGUF range and model import duration. Validated with ReleaseOV llama-bench/llama-simple build and pp3072+tg512 warm mmap lazy import. With page release enabled, PrefillPeakWS dropped from about 5758 MiB to 3053 MiB while throughput remained comparable; PeakCommit stayed about 3331 MiB. * openvino: add GGML_OPENVINO_NPU_SHARED_BANK to share prefill/decode weights bank When set, prefill and decode self-contained blobs use one shared NPUW_WEIGHTS_BANK name (ggml-sc-shared-<pfp>-<dfp>) instead of separate per-phase banks. Since both decoders are built from the same model_weights map, NPUW dedups them into a single in-process bank; on warm import the identical (uid,tensor) pairs are content-verified and reused (patched NPUW weights_bank). Cuts resident compiled-weight footprint ~2GB (pp3072 warm: In-Use 7954->5871 MiB, WS 7891->5765) with prefill throughput unchanged. * openvino: drop SELF_CONTAINED_LAZY_IMPORT and RELEASE_MMAP_PAGES The shared weights bank (GGML_OPENVINO_NPU_SHARED_BANK) supersedes both: it deduplicates the prefill/decode compiled-weight copy directly, so lazy per-phase import and GGUF-page release no longer reduce steady memory. Remove the two env vars, the self-contained lazy-import branches (defaulting to the eager path that imports both blobs), the release_external_weight_pages helper, and the now-unused ggml_openvino_buffer_is_external / _release_external_pages buffer APIs. * openvino: demand-page large NPU compute buffer in strict warm-mmap mode In strict self-contained warm-mmap mode weights come via buffer_from_host_ptr, so every alloc_buffer is graph scratch/KV. The large prefill compute buffer is never touched on the host by the on-device fused OV model, yet the NPU L0 host tensor (create_host_tensor + memset) makes all ~2.4 GiB resident. For buffers >= 1 GiB in this mode, reserve demand-zero pages (no L0, no memset) so they stay non-resident; KV/small buffers keep the L0 host tensor for zero-copy import. pp3072 warm mmap + shared bank: prefill peak WS 5765 -> 3314 MiB (-43%), In-Use delta ~5940 -> ~3232 MiB; prefill 1256.7 -> 1252.9 t/s and decode 28.27 -> 27.96 t/s (both flat). Commit charge unchanged (demand-paged). * perf improvement success * openvino: size static NPU sliding-window masks to the window cache The static graph built the sliding-window KQ mask at the full context length, while the windowed layers bind their (smaller) window KV cache. With gemma-4 at -c 4096 -ub 1024 the window cache is 1536 cells, so SDPA shape inference failed ([1,1,1024,4096] mask vs [1,8,1536,256] K/V) and every prompt errored out. With GGML_OPENVINO_NPU_KV_SLICE the prefill mask was meant to use attention_size_swa, but the check matched "swa" in the ggml tensor name, and llama.cpp names both masks "attn_inp_kq_mask"; the windowed layers got the full-attention sliced length instead ([1,1,1024,256] vs [1,8,1024,256]). Identify the sliding-window mask by tensor identity (is_swa_mask) and size it to ctx_per_seq_swa, both in the graph input shape and in the runtime mask padding. * ggml-openvino: create the model cache dir under a Windows drive root make_dirs() mkdir'ed every path prefix, including the drive root "D:", which fails with EACCES (not EEXIST) on Windows. Any absolute cache path therefore logged "could not create model cache dir ... caching disabled" and silently ran without GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR. Skip the drive root. * ggml-openvino: split device-specific passes and NPU config out of shared code Mechanical refactor, no behavior change (step 1 of the NPU/MoE architecture plan): * openvino/pipelines/: TranslateSession::apply_transformations() now registers common passes, then the device's passes, then the execution-mode passes (stateful KV, static shapes), selected by a per-graph GraphTraits. The GPU MoE fusions and the GPU eltwise-rank workaround move to pipelines/gpu.cpp; CPU and NPU pipelines are empty placeholders for device lowerings. Pass order is unchanged. * npu/npu_config.*: the NPU (NPUW) compile config built at backend init, the NPUW strip for cacheless graphs and the per-graph static prefill/decode variants (HFA prefill, decode NPUW_UNFOLD_IREQS) move out of ggml-openvino-extra.cpp and utils.cpp. * npu/npu_quant_policy.*: the NPU requantization rules (requant policy, token_embd Q6_K, keep-Q4_0) move out of ggml_openvino_get_requant_type(). Verified identical output before/after on OV-CPU, GPU and NPU (gemma-4 E4B). * ggml-openvino: one KV-slice switch for dynamic and static graphs KV-cache slicing (bind the KV cache, and size the KQ masks, at the attended length instead of the full context) existed twice: on by default for dynamic graphs (GGML_OPENVINO_DISABLE_KV_SLICE to turn it off) and opt-in for static prefill (GGML_OPENVINO_NPU_KV_SLICE). The static mode is not NPU-specific; it applies to any static graph, including CPU/GPU with GGML_OPENVINO_FORCE_STATIC. Describe it as one mechanism with two modes behind GGML_OPENVINO_KV_SLICE (unset = each mode's default, 0 = off, 1 = on) and two predicates, ggml_openvino_{dynamic,static}_kv_slice_enabled(), used by the KV binding, the static graph shapes, the mask padding and the cache keys. The two old variables remain as deprecated aliases. * ggml-openvino: initialize all ModelExtraInputInfo fields for the HFA inputs dev_backend_openvino added shape_source/shape_axis to ModelExtraInputInfo; the HFA past-KV and mask initializers still listed four fields, which GCC rejects under -Werror=missing-field-initializers. * ggml-openvino: make new NPU defaults opt-in and fix static model conversion - Convert the static prefill and decode models one after the other, both before either is compiled. The decoders share the weight nodes; converting on two threads crashed intermittently inside libopenvino, and converting decode after the prefill compile gave wrong output on Bonsai-8B. - Compile prefill and decode in parallel only with GGML_OPENVINO_NPU_PARALLEL_COMPILE=1, since it raises the host memory peak. - Unfold NPUW function calls on the decode graph only with GGML_OPENVINO_NPU_DECODE_UNFOLD=1. The unfold default gave wrong output on gemma-4-E4B and slowed decode on several small models. - Stream NPU weight requantization only with GGML_OPENVINO_REDUCE_COMPILE_MEM or GGML_OPENVINO_MEMORY_OPTIMIZE, as on the other devices. - Fix a -Werror=missing-field-initializers build error in utils.cpp. - Declare ggml_openvino_get_npu_requant_type() only in npu/npu_quant_policy.h. - Docs: document the new switches, deprecate stateful execution, show stateless-only results in the validated models table, and update the validation system. * ggml-openvino: honour the -1 skip id in MUL_MAT_ID Upstream made an id of -1 in MUL_MAT_ID skip the slot and zero its row of the result. The translator passed -1 straight to GatherMatmul or Gather, so 169 of the 743 MUL_MAT_ID op tests on CPU (all skip=1) failed. A skipped slot now looks up a valid expert, and the result is multiplied by a 0/1 mask of the kept slots. For the packed MXFP4 path (Gather and MatMul), clamping the id to expert 0 is enough. GatherMatmul computes a wrong row when a token names the same expert twice, so the general path gives each skipped slot an expert that no other slot of the token uses. TopK indices are never -1, and model graphs take the ids from the router's TopK. Those graphs skip the handling and stay unchanged, so the GPU MoE fusion still matches them. * ggml-openvino: Gather rows before dequantizing a quantized embedding table 1. Optimize gather to fetch row before dequant 2. add NPU token_embd quantization option * ggml-openvino: select the NPU token_embd type with GGML_OPENVINO_NPU_TOKEN_EMBD_QUANT GET_ROWS had two rewrites that gather the rows of a quantized table before dequantizing them: gather_compressed_rows, which runs on every device, and gather_before_dequant, enabled on NPU by GGML_OPENVINO_GATHER_BEFORE_DEQUANT. Both match the int4 and int8 chains built by make_int4_weights() and make_int8_weights(). Keep gather_compressed_rows and drop the second rewrite and its switch. The NPU type of a Q6_K token_embd.weight was set by two booleans, GGML_OPENVINO_TOKEN_EMBD_I8 and GGML_OPENVINO_TOKEN_EMBD_I4, and by GGML_OPENVINO_NPU_TOKEN_EMBD_QUANT. Keep only the string variable, with the values f16 (default), q8_0_c, q4_0_128 and q4_0_c, so the int8 and group-128 int4 settings stay available. An unknown value aborts, as for GGML_OPENVINO_NPU_REQUANT_POLICY, instead of silently using f16. The variable replaces the two booleans in the NPU compiled-model cache key. * align PAD pads to input rank in stateful mode * Gate the changes under is_stateful check to address review comment * ggml-openvino: lift Q tensors for non-RoPE layers to rank 4 in stateful mode * ggml-openvino: unify KV slicing under GGML_OPENVINO_KV_SLICE, on by default Replace ggml_openvino_dynamic_kv_slice_enabled() and ggml_openvino_static_kv_slice_enabled() with a single ggml_openvino_kv_slice_enabled(). KV slicing binds the KV cache, and sizes the KQ masks, at the attended length instead of the full context. GGML_OPENVINO_KV_SLICE now defaults to 1 on CPU, GPU and NPU; set it to 0 to disable. Dynamic graphs (CPU/GPU) bind a view over the cache without recompiling. Static graphs (NPU or GGML_OPENVINO_FORCE_STATIC) compile prefill at the aligned attended length and recompile when it changes; decode keeps its full-context graph. Remove the deprecated aliases GGML_OPENVINO_DISABLE_KV_SLICE and GGML_OPENVINO_NPU_KV_SLICE from the env var list, the model-cache settings, docs/backend/OPENVINO.md and the backend README. Behavior change: static KV slicing is now on by default on NPU, so prefill compiles one graph per distinct n_kv (multiples of 256). * ggml-openvino: keep stateful execution with -fa off Keep the V cache token-major in stateful mode instead of falling back to stateless execution. SET_ROWS appends to the state, and PERMUTE reads it back in the layout attention needs, so MUL_MAT does not transpose it. Dropping a reshape in grouped attention lets OpenVINO fuse it into SDPA. For padded V caches, slice after the transpose so the cache is read in place, without a copy that is recompiled for every KV length. This removes the stateless fallback for -fa off. * ggml-openvino: Add the stateful check to make the gate safer * ggml-openvino: support gemma-2 soft-capped attention Gemma-2 caps its attention scores as softcap * tanh(scores / softcap) before the softmax. FLASH_ATTN_EXT now applies the cap on the manual attention path, as SDPA has no soft-capping, and the decoder looks through the SCALE, TANH, SCALE chain of -fa off when it matches attention. One helper finds the K cache of an attention for both the sliding-window pre-pass and the main pass. The pre-pass copy did not see through the chain, so the sliding-window layers used the full-attention sizes once the context exceeded n_swa + n_ubatch. Soft-capped attention keeps the flat KV cache layout: with the 4-D layout the GPU plugin fails in its oneDNN path. * ggml-openvino: enable NPU Level Zero host tensors by default Back NPU buffers with Level Zero host memory unless GGML_OPENVINO_NPU_L0_HOST_TENSORS=0. The plugin imports these buffers without a host-to-device copy each infer; on phi-4-mini Q4_0 (pp3072+tg512) decode goes from 7.2 to 10.6-10.8 t/s. Allocation failures still fall back to plain host memory. * ggml-openvino: enable NPU fast prefill mask by default GGML_OPENVINO_NPU_FAST_MASK now defaults to 1; set it to 0 to fall back to pad_input() + set_zero_diagonal(). The fast path builds each static prefill mask chunk directly in the OpenVINO input tensor (row memcpy of the valid prefix, -inf padding, zero diagonal), avoiding a temporary chunk_size x context_size vector and a full-tensor copy per chunk. The result is bit-identical to the fallback path. * ggml-openvino: add opt-in exact zero points for asymmetric weights Asymmetric quantized weights (Q4_K, Q4_1, Q5_1, and non-requantized Q5_K) represent each group as `w = d * q - m`. The backend currently converts these to u4/u8 weights with an f16 scale and an integer zero point `round(m / d)`, which can shift all weights in a group by up to half a quantization step. Add `GGML_OPENVINO_EXACT_ZERO_POINT=1` to preserve the zero point in f16 instead, using the same path already used for MoE expert weights. This is disabled by default because, with current OpenVINO releases, an f16 zero point moves many GPU fully-connected layers off oneDNN, resulting in slower prompt processing. * ggml-openvino: rename npu/ to device/ and openvino/pipelines/ to openvino/pass_registry/ npu/ holds device-specific backend behaviour (NPU compile config and requant policy), and pipelines/ only registers passes per device; the new names describe what the folders contain. No functional change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: document NPU self-contained model cache and shared weights bank Add GGML_OPENVINO_SELF_CONTAINED_BLOB, GGML_OPENVINO_SELF_CONTAINED_MMAP and GGML_OPENVINO_NPU_SHARED_BANK to the OpenVINO environment variable table, and a note describing the export/import workflow and the requirement to keep NPU settings identical between runs. * ggml-openvino: remove experimental GGML_OPENVINO_NPU_KEEP_Q4_0 Keeping native Q4_0 (group-32) on NPU compiles but hangs the device at inference (ZE_RESULT_ERROR_DEVICE_LOST), so the option was never usable. NPU Q4_0 weights now always go through GGML_OPENVINO_NPU_REQUANT_POLICY. Drop the env var, the requant bypass, its self-contained blob fingerprint term and the docs entry. * ggml-openvino: Update PERMUTE and CONCAT dynamic dim axis selection, Keep fp32 precision for sum-of-squares All changes of https://github.com/ravi9/llama.cpp/pull/358 as one commit. The PR branch had dev_backend_openvino merged into it, so its commits could not be replayed one by one when dev_backend_openvino was rebased onto upstream master. Commits of the PR: - Update fix to just maintain fp32 precision instead of modifying the graph - Limit the fp32 precision spreading in the graph by fencing it with Convert. Prevents perf degrade. - Update OPENVINO.md - Update flash_attn_ext.cpp - Update utils.cpp * enable flash_attn with sink input * ggml-openvino: fix FLASH_ATTN_EXT input detection and sink scale translate_flash_attn_ext read get_input_names()[3] and [4] on every FLASH_ATTN_EXT node, also on nodes with only 3 or 4 inputs. The out-of-bounds read crashed test-thread-safety and test-speculative-rejection-sampling. The inputs are q, k, v, then the optional mask and sinks, and supports_op rejects sinks without a mask, so the input count tells which inputs are present. With sinks, the scale was replaced with 1.0. llama.cpp passes the attention scale in op_params[0] and adds the sinks separately, so keep op_params[0]. The "scale" input branch is removed: ggml's src[4] is only ever the sinks. Only SDPA applies the sinks. The manual GQA path and the NPU HFA prefill path drop them, so attention with sinks now always takes the SDPA path. SDPA has no logit soft-capping, so supports_op rejects sinks together with soft-capping. * ggml-openvino: accept flash attention on layers without RoPE On layers without RoPE (SmolLM3: every 4th layer), Q reaches FLASH_ATTN_EXT through PERMUTE -> VIEW -> RESHAPE. The support check rejected this pattern, causing these layers to fall back to CPU and the split graph to fail with -fa on/auto. * ggml-openvino: keep only weight-divisor DIVs on CPU The GPU DIV guard for per-channel scale vectors (`MPT`'s `act_scales`) currently checks only the divisor's shape. During single-token Qwen2-MoE decode, the shared-expert gate `silu(x) / x` has a `[1,1,1,1]` divisor that matches this shape, causing its DIVs to be kept on the CPU. As a result, every decode step incurred CPU execution and Qwen1.5-MoE decoded at only ~2 tok/s. Apply the guard only when the divisor is a model weight, so runtime tensors such as the Qwen2-MoE gate divisor can remain on the GPU. * ggml-openvino: Qwen MTP and multi-seq optimization with new OV ops All changes of https://github.com/ravi9/llama.cpp/pull/370 as one commit. The PR branch had dev_backend_openvino merged into it, so its commits could not be replayed one by one when dev_backend_openvino was rebased onto upstream master. Commits of the PR: - openvino : support Qwen MTP decoding - openvino: Optimize TopK - openvino: remove single-slot recurrent cache shortcuts - openvino: fuse generic recurrent caches and remove shape waits - openvino: include dense Q8 weights in all INT4 requantization - openvino: use two GDN outputs for snapshots - docs: explain native OpenVINO recurrent cache contracts - docs: clarify Qwen3.5 purpose of native OpenVINO cache ops - llama: skip unallocated hybrid graph inputs - quantize: preserve Qwen hybrid layer metadata when pruning - openvino: emit fused recurrent computation and rotate device caches - openvino: limit KV slicing to attention caches and share compiled blobs - tools: add natural-text server benchmark and OpenVINO profile summary - docs: record fused-cache branch split and UD validation - server-bench: preserve native token counts and save request timings - openvino: document MTP regression and host USM alias recovery - openvino: record large-context allocation diagnosis and default-batch validation - server-bench: count buffered native tokens and reject stream errors - openvino: document UD split and unified server benchmarks - openvino: record Vulkan long-context baseline and GPU-fit controls - openvino: finalize UD benchmark report and MTP comparison - openvino: use stable in-place recurrent caches and optional separate convolution - openvino: accept matching empty axes in DIV repeat translation - openvino: remove the obsolete cache rotation environment setting - openvino: validate imported KV results in their four-dimensional layout - docs: record the bounded in-place KV and convolution benchmark rerun - docs: complete default-batch OV unified-KV 32K benchmarks - openvino: remove the separate recurrent convolution path - openvino: handle zero-logits batches and changing sampler sequences - docs: compare default and large Qwen3.6 benchmark batches - docs: use the historical Vulkan default-batch reference consistently - server-bench: use an integer histogram bin count - benchmarks: summarize current OV unified MMLU timings - docs: record cache-only unified KV MMLU results - openvino: preserve the TopK index port in GET_ROWS - openvino: reuse the MTP sampling model across active slots - openvino: expose selection flags and offsets in graph dumps - docs: consolidate Qwen OpenVINO defaults and performance evidence - docs: retain continuous-request MTP benchmark results - docs: compare stable MTP with the non-MTP serving baseline - openvino: isolate GDN snapshots and skip naive cache fusion - ggml: document Qwen3.6 GenAI benchmarks - ggml-openvino: keep PR changes inside the backend folder * ggml-openvino: drop the extra semicolons after OPENVINO_OP OPENVINO_OP expands to member function definitions that end in a closing brace, so a semicolon after it is an empty declaration inside the class. GCC 15.2 reports it with -Wextra-semi, which ggml enables for GCC 8.1 and newer, so a build with LLAMA_FATAL_WARNINGS=ON (the default in ci/run.sh) fails in gated_delta_net.hpp. The CI compilers do not report it. recurrent_cache_update.hpp has the same pattern twice. It is compiled only when the OpenVINO package provides the recurrent cache-update ops. * ggml-openvino: make q8_0_c the default NPU token_embd requant type Default of `GGML_OPENVINO_NPU_TOKEN_EMBD_QUANT` changed from `f16` to `q8_0_c`. Applies only to NPU and only to a Q6_K `token_embd.weight`; other devices and quantization types are unchanged. * ggml-openvino: do not treat the recurrent state gather as out_ids The output_len scan now runs for every model and takes the first GET_ROWS whose index is a graph input. On Qwen3.5 that is the recurrent state gather of build_rs, which indexes the inp->s_copy leaf directly, so output_len became 1. In static (NPU) mode the prompt model then returned one logits row per chunk, and llama-perplexity read stale rows (PPL about 240000 on Qwen3.5-2B). The same predicate also gave the s_copy input the out_ids shape and data in static mode. is_output_idx now skips GET_ROWS that read a recurrent state cache. The old predicate stays as is_get_rows_input_idx for the dynamic-dim bookkeeping, so the s_copy leaf keeps its dynamic length there. * ggml-openvino: prefer the size-preserving dim when a RESHAPE repeats a stride The RESHAPE dynamic-dim rule matches the source's dynamic stride against the result's strides. A size-1 dim repeats the next dim's stride: Gemma 4 E2B has one KV head, so [head_size, n_tokens] -> [head_size, 1, n_tokens] has the same stride on dims 1 and 2. The first match picked the size-1 head dim, so the K/V views lost their dynamic token dim and became static reshapes with the first graph's token count. Every later graph with another token count failed. Prefer the stride match that keeps the dynamic size, and fall back to the first stride match for reshapes that change it. * ggml-openvino: correct the RESHAPE dynamic-dim comment for size-1 dims The comment above the RESHAPE stride match in compute_node_dynamic_dims() still said the match is unambiguous because result strides are strictly monotone. A size-1 dim has the same stride as the dim after it ([head_dim, T] -> [head_dim, 1, T] with one KV head), so it is not. Explain the size-1 case in the block comment, drop the duplicate comment inside the loop, and name the source dim's size once. * examples: add llama-chat-replay Replays JSONL prompts like OpenVINO GenAI chat.py (-pf, -i chat history, greedy, -o, -pc metrics). The first prompt is prefilled plus one decode step during load (Warmup time) and the memory is cleared, so compile/import is counted in Load time as in GenAI. * ggml-openvino: make the NPU lazy host threshold configurable, default 2 GiB With a large context the KV cache (~1 GiB at ctx 8K) crossed the 1 GiB threshold, lost its Level Zero host tensor in strict warm-mmap mode and made decode copy the KV every token. Default to 2 GiB so only the larger prefill compute buffer stays demand-zero, and allow overriding it with GGML_OPENVINO_LAZY_HOST_THRESHOLD_MB. * ggml-openvino: fix NPU HFA prefill with ordinal-suffixed KV input names KV cache inputs are now named with a graph-ordinal suffix (cache_k_l0#l12), but flash_attn_ext looks the HFA past input up as ggml_hfa_past_cache_k_l<layer>, so GGML_OPENVINO_NPU_HFA_PREFILL failed with 'ggml_hfa_past_cache_k_l0 not found in tensor map'. Name the past inputs after the bare cache name and resolve the source cache by that name when binding. * ggml-openvino: keep the NPU prefill model on cache hits after decode With KV slicing, decode calls overwrite the shared decoder's attention size with the full context, so the next prefill compared that against its sliced size, missed the cache and rebuilt the decoder and both infer requests (plus a cold first chunk). Remember the attention sizes the prefill model was built with and compare against those. * ggml-openvino: bind NPU HFA past KV zero-copy in the cache layout The HFA prefill past K/V inputs were [1, heads, past, head_size] copies of the cache, relaid out per chunk on the host and allocated per decoder (256 MiB on phi-4-mini at 3k). Declare them in the ggml cache layout [1, past, heads, head_size], bind the first past rows of the cache directly and transpose in the graph; NPUW HFA reads the tiles from the BSHD past (needs the matching NPUW HFA support). HFA input ABI v4. * ggml-openvino: size the NPU decode KV window by GGML_OPENVINO_NPU_DECODE_BUCKET Static decode attended the whole context, so every token read and wrote the full KV cache. With a bucket base set, decode attends the smallest base*2^k rows covering the next position (clamped to ctx): the KV inputs and masks are sliced to the window, the compiled pair's decode graph is built at the window the prefill reaches, and other windows are built or imported from the self-contained cache on first use with the pair's settings and bank. With self-contained blobs, keep the token_embd weight node for the later window models so they reuse the pair's bank entry instead of embedding a second vocab copy. The static build/convert lambdas become functions shared with the window path. * ggml-openvino: allocate the NPU KV cache on demand with GGML_OPENVINO_NPU_KV_ON_DEMAND The NPU KV buffer was committed for the whole context up front. With the option set, the KV buffer is swapped for demand-zero memory at its first KV tensor and every KV tensor gets its own storage in a registry, grown in base*2^k rows (base = decode bucket or 1024) as the static graphs reach further. Host accessors, KV inputs and outputs, sliced KV views and the HFA past bind that storage; when it moves, idle requests bound to the old storage are recreated so it is released. With decode buckets and KV slicing the storage also takes the layout GenAI uses on NPU: K is [heads][rows][head_size] and V is [heads][head_size][rows], each bound whole to prefill and decode. SET_ROWS scatters new K rows on axis 2 and V rows on axis 3, the attention permute only cuts the window, the HFA past is the storage itself, and attention multiplies the probabilities by V^T with transpose_b instead of SDPA. A chunk-sized scatter on the innermost axis is slow on NPU, so the HFA prefill returns each chunk's V^T rows and the host copies them into the cache. Host reads, writes and fills go through a layout-aware copy, storage growth re-packs each head, and the layout is part of the model cache fingerprint. * ggml-openvino: NPU weightless blob cache with one shared weights file GGML_OPENVINO_NPU_WEIGHTLESS_BANK=1 (with GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR) exports weightless prefill/decode NPUW blobs plus one <prefill>-<decode>.weights.bin. Weight Constants are laid out once by data pointer and their WeightlessCacheAttribute is retagged with the real offsets. Warm import uses ov::weights_path, and NPUW dedups both models into one bank through the LazyTensor hashes, so the OpenVINO weights_bank duplicate-uid patches are not needed. Decode KV-window models resolve the shared weights through the pair's file, or get their own file when the pair was imported. Also drop the old blob and manifest before publishing a re-export: std::rename does not replace an existing file on Windows. Based on anikulk/llama.cpp 4ab270aa9. Co-authored-by: Anisha Kulkarni <anisha.dattatraya.kulkarni@intel.com> * ggml-openvino: prebuild all NPU decode KV windows with GGML_OPENVINO_NPU_DECODE_PREBUILD Decode windows were built lazily when the sequence first grew into them, which stalled generation for a compile without a cache and made a strict-mmap import fail for windows the export run never reached. GGML_OPENVINO_NPU_DECODE_PREBUILD replaces GGML_OPENVINO_NPU_DECODE_BUCKET: 0 (default) keeps decode over the whole context; 1 sizes decode by 1024*2^k KV windows and the call that builds or imports the prefill/decode pair also builds or imports the decode graphs of every other window up to the context, from the pair's graph and compute params. Requests are still created on first use. Window building is split out of the request lookup so both paths share it, and window ports are checked against the pair's decode ports. * Revert "examples: add llama-chat-replay" This reverts commit 4eded2c52c2df9fa1251673a523f8c7cded20d7c. * ggml-openvino: initialize the new compiled-graph and build-args fields ov_compiled_graph and static_build_args gained members, but three brace initializations still listed only the old ones. GCC reports -Wmissing-field-initializers there, and CI builds with warnings as errors. Set the fields by name, and pass {} for weights_path, which the window build sets afterwards. * ggml-openvino: fix gemma-4 when only the output rows reach the KV-shared layers With one sequence, gemma-4 now drops the rows that are not output before its first KV-shared layer. Those layers take their positions through GET_ROWS(inp_pos, out_ids), and the backend mishandled three things: - The positions are reshaped from [n] to [1, n]. That matched RESHAPE case 1, which marks the wrong dim dynamic. Use case 3, which keeps the source dim as dim 1. - ROPE shared one sin/cos table per frequency setting, so the KV-shared layers reused the table built from the full inp_pos. Add the position input to the cache key. - compute_llm_params took input_len from the last attention op. That is now a KV-shared layer with only the output rows, so the static (NPU) path padded prefill to the wrong length. Take the largest query length. Stateful execution still fails on the per-layer embedding view and is not covered here. * ggml-openvino: keep the inp_pos layout for gemma-4's gathered positions A batch can have no output rows, for example a prompt chunk that llama-cli processes before its last tokens. gemma-4 then reshapes an empty position gather for the ROPEs of its KV-shared layers. The size checks of the RESHAPE cases cannot place a dim of size 0, so the positions got the wrong layout and the ROPE multiply failed on CPU and GPU. A RESHAPE that feeds the ROPE positions now always takes the inp_pos layout [1, 1, 1, n]. * ggml-openvino: enable stateless GPU execution for MLA models The per-head MLA weights (`attn_k_b` / `attn_v_b`) are 3D quantized tensors. Their `MUL_MAT` operations were falling back to the CPU, splitting the graph at every layer and causing GPU execution to fail. Accept these tensors in the GPU support check and address the following issues exposed by enabling GPU execution: * Quantization: Use exact zero points only for MoE expert weights. U4 weights with F16 zero points fail in batched GPU MatMul. * Dynamic VIEW dimensions: Select the correct dimension when a size-1 dimension shares its stride (e.g. `k_pe`), and match KV cache views by stride. * KV cache V view: Translate the V view of the K cache as a reshape while preserving the dynamic cell axis. * Flash Attention: Accept the MLA Q (`CONCAT`) pattern and V views derived from another view. Limitations: The GPU plugin fails on batched compressed weights with static shapes. Keep these unsupported in op tests and `GGML_OPENVINO_FORCE_STATIC`. * openvino : use writable recurrent cache inputs Assisted-by: Codex * ggml-openvino: fix decode crash for Qwen3.5 4B When the dynamic size is 1, the code now keeps the innermost match. When the size is greater than 1, it still uses the MLA rule, so the MLA k_pe fix still works for multi-token graphs. * ggml-openvino: keep the GDN state in the OpenVINO layout Adds an opt-in switch, GGML_OPENVINO_GDN_NATIVE_STATE. When it's on, the GATED_DELTA_NET recurrent state stays in the layout the OpenVINO GatedDeltaNet op uses, [B, H_v, key_dim, value_dim], instead of the ggml layout [B, H_v, value_dim, key_dim]. That removes the two state transposes the translator adds on every call. * ggml-openvino: support Ling-mini-2.0 group-routed MoE Ling-mini-2.0 selects experts by group, but the group-selection graph used ops that were not supported by the backend. This left 370 nodes on the CPU and reduced decode to ~1.5 tok/s. * Translate the batched 4D `GET_ROWS` gather used for group scores. * Support `SET_ROWS` for compute tensors, not just KV caches. * Preserve the dynamic token dimension in `FILL`. * Skip the `DIV` repeat when dimensions already match. * Use the tracked token dimension for `RESHAPE` / `GET_ROWS` op cases, so a graph compiled during one-row warmup also works for ubatches with more output rows. * ggml-openvino: disable output staging The backend stages small outputs (up to 4 MB) in a USM host buffer and binds larger outputs directly to the ggml buffer. When the number of outputs changes between calls to the same graph, as in llama-perplexity, an output port can switch from USM host memory to plain host memory. The OpenVINO GPU plugin mishandles this switch for some dynamic outputs, causing out-of-bounds writes on subsequent calls. This results in `CL_OUT_OF_RESOURCES`. For example, `llama-perplexity -c 2048 -ub 512` fails on the fourth ubatch with Gemma-2, Gemma-4, Phi-3.5, Phi-4-mini, and Qwen3.5. Disable output staging until the plugin issue is fixed, so each output is always bound the same way. Input staging remains enabled. * ggml-openvino: match attention masks by identity Gemma-4 prunes non-output rows before its KV-shared layers. These layers select the corresponding mask rows with GET_ROWS(mask, out_ids). Mask parameters were named by their consumers, but GET_ROWS is not recognized as a mask consumer. As a result, the sliding-window layers selected their rows from the full-attention mask instead of the sliding-window mask. This either caused the mask add to fail broadcasting beyond the sliding-window cache (d1024 and above with -ub 512) or allowed attention beyond the sliding window at shorter contexts. Track graph-input attention masks by identity and assign them the same parameter name for every consumer. The SWA pre-pass now keeps each layer's own attention mask, as the KV-shared layers read the same cache with mask rows derived from it. * ggml-openvino: preserve input mask names and window sizes KV-shared layers appear at the end of the graph, and their derived masks are not graph inputs. Processing these masks overwrote the names and sliding-window sizes recorded from the original input masks with empty values. Preserve the names and window sizes of graph-input masks so the attention dimensions continue to be derived from the original mask shapes. * ggml-openvino: accept empty tensors in stride checks In Gemma-4 batches without output rows, the attention tensors of KV-shared layers are empty. Their strides beyond the empty dimension are zero, causing has_strides_on_element_grid() to reject them. This sent SOFT_MAX, the KQV MUL_MAT, and CONT to the CPU with -fa off. It also caused the per-layer input crossing the graph split to be bound with the wrong layout. Accept empty tensors in the stride check. Their contents are never read, so the remaining strides do not need to satisfy the element grid constraints. * ggml-openvino: classify transposed V caches using v_trans With -fa off, the V cache is transposed. When the attended length is the whole cache, as with a prompt that fits in one ubatch, its V view is contiguous and was mistaken for the K cache layout. The K layout path does not keep only the rows of the layer's own V size. Gemma-4 allocates 512 V rows in every layer, the size of its widest layer, and its sliding-window layers use the first 256. These layers then attended with all 512 rows, and prompts fitting in a single ubatch failed with a broadcast error. Use v_trans to identify the V cache layout instead of relying on contiguity. * ggml-openvino: use per-layer Split in stateful execution Stateful execution was excluded from the per-layer Split path used for Gemma-4's per-layer embedding slices. The generic re-slicing path therefore cut the token axis instead of the layer axis. Allow stateful execution to use the per-layer Split and adjust its axis by the rank dropped during stateful graph construction. * ggml-openvino: centralize stateful KQ mask lookup Stateful execution rebuilds the KQ masks from positions. The existing replacement logic applied to every mask input of FLASH_ATTN_EXT and SOFT_MAX, including masks derived through GET_ROWS. For Gemma-4's KV-shared layers, this replaced the row-selected mask with the full rebuilt mask, losing the output-row selection. As a derived mask's name does not name the sliding window, these layers also got the full-attention mask instead of the sliding-window one. Add get_kq_mask_input() to replace only the actual mask parameters. Use it for FLASH_ATTN_EXT, SOFT_MAX, and GET_ROWS, so the derived masks select their rows from the rebuilt masks, and the lookup is no longer duplicated. * ggml-openvino: Removed duplicated if clause. * fix flash_attn with sink conversion to OV SDPA * ggml-openvino: drop the unused sink-without-mask path in FLASH_ATTN_EXT supports_op already rejects FLASH_ATTN_EXT with sinks but no mask, so the translator never sees a 4-input node whose fourth input is the sink; the name check could only reach the assert after it. Restore the input-count detection, fetch the sink where it is used, and format the sink reshape. * ggml-openvino: fix dimensions bug in stateful recurrent-state remainder writeback * ggml-openvino: move scalar scales before weight MatMuls on GPU and NPU GPU and NPU run the graph in f16. Granite multiplies the attention and FFN outputs by residual_scale right after their output projections. For some tokens the ffn_down output is far above the f16 maximum (65504), so it becomes inf and every later layer turns NaN: granite-4.0-350m and granite-4.0-1b answer with repeated tokens on GPU, and llama-perplexity reports NaN. Add MoveScaleBeforeMatMul. It rewrites Multiply(MatMul(x, W), s), with a scalar constant |s| < 1 and a constant W, as MatMul(Multiply(x, s), W). The result is the same up to rounding and the MatMul output stays in range. Register it for GPU and NPU; CPU runs in f32. * ggml-openvino: translate convolution rollback snapshots Assisted-by: Codex * ggml-openvino: run GDN rollback snapshots in OpenVINO on CPU Speculative decoding that rolls back recurrent state (MTP, EAGLE3, DFlash) builds GATED_DELTA_NET with K = n_rs_seq + 1 state snapshots and copies them into the recurrent cache. supports_op accepted the snapshots only on GPU, so on CPU the GDN ran in ggml while the copy, whose destination is the OpenVINO cache buffer, ran in OpenVINO. The copy read the packed ggml GDN output wrongly and stored wrong states without an error: test-llama-context rollback failed and MTP accepted no drafts. translate_gdn_snapshots builds plain GatedDeltaNet ops on token prefixes and does not depend on the device. Accept the snapshots on every device with dynamic shapes and keep rejecting them only for static execution (NPU and GGML_OPENVINO_FORCE_STATIC). Update the two doc sentences that said the snapshots need the GPU. * ggml-openvino: abort when GDN snapshots would run in ggml The snapshot copy writes into the recurrent cache in the OpenVINO buffer, so it always runs in OpenVINO. When OpenVINO rejects the GATED_DELTA_NET that produces the snapshots (per-key gating as in Ling-3.0, a permuted v, static execution), the GDN runs in ggml and the OpenVINO side reads its packed output wrongly. Depending on the shapes, this silently corrupts the cache or fails inference with a reshape error. Abort when the graph is scheduled, with the reason. * ggml-openvino: share one weights file and NPUW bank across all NPU prefill/decode pairs With GGML_OPENVINO_NPU_WEIGHTLESS_BANK, every prefill/decode pair (one per KV window, e.g. llama-server's startup probe and a long prompt) wrote its own weights file and used its own NPUW bank, so import kept the weights resident once per pair (+2.1 GiB for phi-4-mini). Pairs now use a model-level <wfp>-<wfp>.weights.bin and bank; the weights file records a layout/content hash and is reused when identical, with a per-pair file and bank as fallback. Old caches are invalidated. * ggml-openvino: export and preload NPU prefill blobs for every KV window With head-major KV the prefill also runs at a KV window picked by prompt length, so an import-only run failed for prompts whose window was never exported. After a pair is exported, also export the prefill model of every other 1024*2^k window (weights resolved through the pair's shared weights file), and on import preload them into a process-wide cache of imported blobs that pair and decode-window builds reuse. Cache the source fingerprint of weights outside OpenVINO buffers, which made each model fingerprint cost ~9 ms. A window switch now costs ~60 ms instead of ~250 ms. Rename GGML_OPENVINO_NPU_DECODE_PREBUILD to GGML_OPENVINO_NPU_BLOB_PREBUILD since it now covers prefill as well as decode windows. * ggml-openvino: drop invalid prefill blobs and clear blob caches on backend free The prefill preload imported each KV window's cached blob without a try, so one blob that failed to import failed the graph compute, and the blob stayed in the cache for every later run. Remove the blob and its manifest and log a warning, as the pair import does; the window is built when a prompt first needs it. The imported-blob map and the fingerprints of weights outside OpenVINO buffers lived until the process exited. A process that loads models one after another kept every compiled model resident, and a later model whose weight landed at a freed weight's address reused that weight's fingerprint. Clear both with the shared compiled-model cache when the last backend is freed. Also apply clang-format to the changed lines. * ggml-openvino: enable NPU KV windows by default With GGML_OPENVINO_NPU_BLOB_PREBUILD unset, every NPU decode step ran a static graph that attends the whole context, so decode was slow for any sequence much shorter than -c. On a Lunar Lake NPU, Llama-3.2-1B decodes at 0.79 t/s with -c 4096 and 0.20 t/s with -c 16384, against 11.1 and 12.0 t/s with KV windows. Each window adds one decode graph to compile at load: with -c 4096, load goes from 11.1 s to 14.6 s for Llama-3.2-1B and from 61.2 s to 81.6 s for Llama-3.1-8B, whose decode goes from 0.20 to 0.82 t/s. Default GGML_OPENVINO_NPU_BLOB_PREBUILD to 1; 0 keeps the full-context decode graph. Update the env table and the Boolean-flag note, which said an unset flag is always disabled. * ggml-openvino: add POOL_1D and allow F32 EXP on GPU * skip empty tensors * ggml-openvino: detect the gated LFM2 conv input as a recurrent sequence A reshape whose output reaches SSM_CONV through concat(conv_state, transpose(x)) gets op_case 6, which keeps the active sequence count apart from the token count, as the conv state does. The LFM2 and LFM3 short convolutions gate x first, so the transposed tensor is mul(view(x), view(x)) and the check missed it. The reshape got op_case 5 instead, and: - in stateful mode, which drops the leading dim for op_case 5, TRANSPOSE got a rank-3 input with a 4-axis order and graph conversion failed; - with more than one sequence, the concat with the conv state failed. Accept the gated input too. The output for a single stateless sequence does not change. * ggml-openvino: always stream weight requantization Requantizing Q6_K/Q5_K weights to 8-bit (or F16 for some tensors) previously converted the entire tensor to f32 first. For large output or embedding matrices, this required 2–2.5 GB of temporary memory. The chunked conversion path already processes 256 rows at a time and produces identical output. It was previously enabled by `GGML_OPENVINO_REDUCE_COMPILE_MEM` or `GGML_OPENVINO_MEMORY_OPTIMIZE`; use it by default whenever the target block size divides a row. Q4_1_64, which has no chunked quantizer, still converts the whole tensor. The variable retains its other effects. * ggml-openvino: allocate host buffers as untouched OS memory Host buffers (weights, KV cache and compute) were allocated with `ggml_aligned_malloc` and explicitly zeroed with `memset`. This made the entire weight buffer resident before any tensors were loaded. Allocate host buffers using `mmap(MAP_ANONYMOUS)` on Linux and `VirtualAlloc` on Windows instead. Both provide zero-initialized memory without touching the pages, allowing the OS to bring them into memory as tensors are written. This preserves the zero-initialization required by ggml-org/llama.cpp#20857 while avoiding unnecessary memory residency during model loading. Remove the cache-only allocation branches, the corresponding `ggml_aligned_free` path from the destructor and the `data_owned_by_ov` member it used, since they are no longer needed. * ggml-openvino: release GGUF file pages once a weight is converted Once `set_tensor` converts a weight into the backend buffer, its source pages in the mapped GGUF file are no longer needed for that buffer. Release these clean pages using `madvise(MADV_DONTNEED)` on Linux and `VirtualUnlock` on Windows. The mapping itself remains intact, and pages can be loaded again from the page cache or file if needed. Only release whole pages within tensor ranges belonging to read-only file mappings. Skip buffers that wrap the mapping itself. Keep locked pages resident. On Linux, `MADV_DONTNEED` fails on mlocked memory. On Windows, check page lock status with `QueryWorkingSetEx` and skip ranges containing locked pages, since `VirtualUnlock` would otherwise unlock them. * ggml-openvino: key weight constants by name for the NPU model cache The NPU compiled-model cache (GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR without the weightless bank) imports a blob with the freshly converted model as ov::hint::model, and NPUW finds each constant by its WeightlessCacheAttribute bin_offset. The keys came from a per-model counter, and a constant that already had a key kept it. Weight nodes are shared between models: between the NPU prefill and decode models, and across the process through the tensor extras. So a weight kept the key from whichever model converted it first. The decode model for a KV window is built both as the decode half of the prefill/decode pair and as a standalone window model, under one fingerprint, and the two builds got different keys. Importing the blob of the other build failed with "Couldn't find Constant in cache!", and the decode model was recompiled and exported again on every run. With GGML_OPENVINO_NPU_BLOB_PREBUILD on by default, this happens whenever a prompt reaches a larger KV window. Key weight constants by a hash of the weight name and their position in the weight's subgraph, above the range of the counter. Other constants are created per model and keep the counter. * ggml-openvino: drop CACHE_MODE=OPTIMIZE_SIZE from the NPU plugin cache With GGML_OPENVINO_CACHE_DIR set, the NPU compile config added CACHE_MODE=OPTIMIZE_SIZE next to NPUW_CACHE_DIR. With it, the NPU plugin compiles the NPUW submodels with weights separation, which fails for them: with NPUW folding, "No WeightlessCacheAttribute has been found in any of the model's Constant nodes"; without folding, a driver compile error. So every NPU run with GGML_OPENVINO_CACHE_DIR failed on its first compile. Keep the default cache mode. NPUW_CACHE_DIR still caches the compiled submodels. * ggml-openvino: load each self-contained NPU blob once per process A self-contained blob (GGML_OPENVINO_SELF_CONTAINED_BLOB) embeds its NPUW weights bank, and NPUW cannot load a blob into a bank that already holds those weights: Bank::read_and_add_tensor asserts. Two cases did this on every run that imported the cache: - Pairs that differ only in the prefill model share the decode model and its bank. A long prompt builds such pairs, and each one imported the decode blob again. The import failed, so the decode model was compiled and exported again for every pair, until the run ran out of memory. - The decode model of each KV window uses the bank of the pair's decode model, so the import of a window blob failed and stopped the run. Keep each self-contained model in the process once it is imported or compiled, and reuse it for every pair with the same fingerprint. Turn the KV windows off for self-contained blobs; the weightless bank keeps them. * ggml-openvino: update the Docker image GPU drivers to 26.40.40196.8 The OpenVINO image pins the Intel GPU drivers at compute-runtime 26.35.39758.10. With that driver, gemma-4-E4B hangs on the Lunar Lake GPU. With 26.40.40196.8 it runs. Take the GPU driver pins from the compute-runtime 26.40.40196.8 release: IGC v2.42.6, compute-runtime 26.40.40196.8 and gmmlib 22.10.2. * ggml-openvino: fix long NPU prompts failing with KV slicing With GGML_OPENVINO_KV_SLICE, on by default, a static prefill is compiled at the attended KV length that llama reports for the ubatch. That length grows with every ubatch of a long prompt, and each new length compiled another prefill/decode pair, with GGML_OPENVINO_NPU_BLOB_PREBUILD (on by default) also its decode KV windows. The shared compiled cache keeps every pair until the backend is freed. Each loaded prefill model keeps NPU memory for its inferences. On a Lunar Lake NPU, a prefill model loaded next to two or three others cannot get it: DRM_IOCTL_IVPU_BO_CREATE fails with "Buffer size is too big", and pfnAppendGraphExecute returns ZE_RESULT_ERROR_UNINITIALIZED. With -c 4096 and the default -ub 512, a 2200-token prompt failed on smollm2-1.7B, Phi-4-mini, Ministral-3b and gemma-3-4b. - Round the attended length of a sliced static prefill up to a power of two, clamped to the context. A growing sequence now compiles a prefill model when its attended length doubles, not at every ubatch. The added rows are masked, so the output does not change. - When only the prefill attention size changes, the new pair takes over the decode models (base and KV windows) of the entry's current pair, because they do not depend on that size, and the current pair is dropped before the new prefill model compiles. A pair that another backend may use stays cached. Self-contained pairs are exported as one blob pair, so they keep their own decode models. - The taken-over KV window that covers the next decode position becomes the base decode model, as in a pair built at that length, so the first decode step does not have to create an infer request for it. * ggml-openvino: pre-scale the GDN q input by default on the NPU The NPU GatedDeltaNet kernel writes its output in FP16 and flushes values below 6.1e-5 to zero, and the RMS norm after it amplifies the loss. GGML_OPENVINO_GDN_Q_SCALE lifts the output above that floor, but it was off by default. On a Panther Lake NPU with the compiler bundled in OpenVINO, Qwen3.5-2B scores a perplexity of 14.77 without it and 13.00 with it (12.93 for the same weights on CPU), and its greedy chat no longer loops while thinking. - Enable the pre-scale by default with the factor 64. GGML_OPENVINO_GDN_Q_SCALE=0 turns it off. - Scale the eps of the norm only when the GDN that feeds it is pre-scaled. A norm that reads the GDN output directly got the scaled eps on every device, while q is scaled only on the NPU. With the variable set, Qwen3.5 gave wrong output on CPU and GPU. - Add the factor to the model cache fingerprint of NPU graphs with a GATED_DELTA_NET, so a cached model is not reused across settings. - docs: update the GGML_OPENVINO_GDN_Q_SCALE row and note. * openvino-ggml: Decompose the GDN for LNL 1. Automatically decompose LNL GDN prefill into chunked MatMuls; retain the native op for decode. 2. Isolate lowering selection, fix positive-gate handling, and add overflow guards. 3. Update cache identity and add chunk-boundary regression cases. * ggml-openvino: fix GDN decomposition in decode and on long prompts Two problems remained with the GATED_DELTA_NET decomposition on a Lunar Lake NPU. The decomposition covered only multi-token (prefill) graphs, and decode kept the GatedDeltaNet op. Under Linux the decode kernel loses the device as well: after a successful prefill, the first generated token failed with ZE_RESULT_ERROR_DEVICE_LOST in llama-completion, llama-bench and llama-server. The chunk masks were new constants in every converted model. The NPUW weights bank is shared by all compiled models (NPUW_WEIGHTS_BANK=shared) and identifies a constant by its data pointer, also after the model that owned it is gone. When a longer sequence needs a second prefill model, its masks can get the addresses of freed masks of the first one and are then read as those. The Qwen3.5-2B perplexity over 512 tokens changed between 12.99 and 56.94 from run to run, and a 1900-token prompt gave garbage. - Lower a single-token GATED_DELTA_NET to one recurrence step when the decomposition is on: the chunk formulas for a chunk of one token, where P = 1 and the decay is 1. - Keep one instance of each chunk mask for the whole process. - test-backend-ops: add a single-token case with two sequences, repeated heads and a positive gate. - docs: GGML_OPENVINO_GDN_DECOMPOSE covers single-token graphs. * openvino-ggml: Enhance the safe env parsing * ggml-openvino: keep the GDN q pre-scaling on when its variable is unset ggml_openvino_gdn_prescale_enabled() returned false when GGML_OPENVINO_GDN_Q_SCALE is unset, which switched the default-on pre-scaling off. Qwen3.5-2B on a Panther Lake NPU went from a perplexity of 13.00 back to 14.77. - Treat an unset variable as enabled. Zero, negative and non-numeric values still turn the pre-scaling off. - Comments: ggml_openvino_getenv_int() no longer goes through std::atoi. - docs: a boolean flag with a non-integer value is no longer always disabled; a flag whose default is 1 stays enabled. * ggml-openvino: use the PLUGIN compiler for GDN graphs on the NPU The NPU driver compiler, the default, gives wrong output for graphs with GATED_DELTA_NET (Qwen3.5). Qwen3.5-2B on a Panther Lake NPU answers with unrelated text and has a perplexity of 16.77, against 13.00 with the compiler bundled with OpenVINO. Other models stay on the driver compiler, which builds faster prefill kernels. - Compile static graphs with GATED_DELTA_NET with NPU_COMPILER_TYPE=PLUGIN, unless GGML_OPENVINO_NPU_COMPILER_TYPE or NPU_COMPILER_TYPE in GGML_OPENVINO_NPU_CONFIG sets the compiler type. - Add the choice to the model cache key of these graphs, so blobs from the driver compiler are not imported. - docs: describe it in the GGML_OPENVINO_NPU_COMPILER_TYPE row. * ggml-openvino: fix gemma-4 E2B on the Lunar Lake NPU at small contexts The NPU 4000 compiler computes ScaledDotProductAttention wrongly when K has one KV head and is square, that is when the attended rows equal the head size. gemma-4 E2B has one KV head, 512-wide full-attention heads and 256-wide sliding-window heads, so a context of 512 tokens or less, and a prefill that attends 256 or 512 rows, gave wrong output. For that shape on NPU 4000, build the attention from MatMul, Softmax and MatMul, as the transposed-V path already does. Other shapes and devices keep the op. * ggml-openvino: support multiple GPUs with -sm layer * ggml-openvino: check GGML_OPENVINO_* variables and reorganize the docs The backend ignored a misspelled GGML_OPENVINO_* name and a value that was not valid, and the docs listed every variable in one table. Code: - One table holds the known variables and their types. A value that is not valid stops the backend with a message that names the variable and the accepted values. An unknown GGML_OPENVINO_* name gives a warning. - With log verbosity 4, one line shows the variables in effect. - GGML_OPENVINO_GPU_CONFIG and GGML_OPENVINO_CPU_CONFIG pass plugin properties to the GPU and the CPU, as GGML_OPENVINO_NPU_CONFIG does for the NPU. If the plugin rejects a property, the error names the variable. - GGML_OPENVINO_STATEFUL_EXECUTION gives a deprecation warning. - The key of the compiled model cache covers the two new variables, GGML_OPENVINO_KV_SCATTER_ELEMENTS and GGML_OPENVINO_DISABLE_ELTWISE_RANK_ALIGN. - GGML_OPENVINO_ENABLE_CACHE, which no code read, is no longer registered. Docs: - docs/backend/OPENVINO.md: Quick Start, collapsible blocks, runtime configuration in groups (basic, plugin properties, advanced tuning, compiled model cache, deprecated), updated validated models and known limitations, corrected stale entries. - dev_doc/OpenVINO-ENV-VARS.md: the variables for development and the NPU properties that the backend sets. * docs : add LFM2.5 and Ling-mini-2.0 to the OpenVINO validated models * ggml-openvino: rebuild converted weights in get_tensor set_tensor converts a quantized weight in place into the extracted or requantized layout, so get_tensor returned bytes that are not in the ggml format. A reader such as the CPU reference in test-backend-ops then computed on garbage and produced NaN (MUL_MAT cases with weights=1). get_tensor now rebuilds the ggml bytes for such a weight: it evaluates the weight node on the host and quantizes the values back to the tensor type. The result is close to the original data, not bit-exact. Other reads are unchanged, and the inference path does not call get_tensor on weights. * ggml-openvino: drop the GDN test cases from test-backend-ops Restore tests/test-backend-ops.cpp to the upstream version. * docs : improve the OpenVINO backend docs * docs : restore the OpenVINO intro and the development note * ggml-openvino: fix slow performance of MoE models with requantize and workaround to use fused MOE Compressed * ggml-openvino: document the fused GPU MoE op limits Converting K-quant MoE experts to Q4_1_64 on GPU lets the plugin use its fused MoE op, which runs in f16 only. granitemoe models give wrong output with it, and an f32 inference precision hint fails to compile. Document GGML_OPENVINO_MOE_OP=0 as the way to turn the op off, and that llama-perplexity with MoE models needs -b equal to -c. Also split the long TODO comment over two lines. * ggml-openvino : FILL for in-place clear of recurrent cache, sticky NaN fix * restore this part * ggml-openvino: bump the model-cache version for the recurrent-cache reset change Recurrent-cache resets now clear slots with Select instead of Multiply, so a NaN in a cleared slot no longer survives. Compiled models cached before this change keep the old graph, so bump the fingerprint version to rebuild them. Also note how SCALE and FILL store their parameters. * ggml-openvino: read rope_factors_short/long as weight constants create_weight_nodes() registers the ROPE frequency factors as "rope_freqs.weight", but set_input_output() and compute_model_inputs() kept the tensor's own name. For models whose factors are named rope_factors_short or rope_factors_long (Phi-3.5, Phi-4, MiniCPM with LongRoPE) the factors became a Parameter fed from the host weight buffer. That buffer is empty in cache-only mode and after the host weights are released, so these models gave garbage output with GGML_OPENVINO_COMPILED_MODEL_CACHE_ONLY, GGML_OPENVINO_RELEASE_WEIGHTS and GGML_OPENVINO_MEMORY_OPTIMIZE. Use the same name in all three places, so ROPE always reads the weight Constant. The graph changes, so bump the model-cache version to 7. * ggml-openvino: import cached models from a memory-mapped tensor A stream import reads the whole compiled blob into a heap buffer while the plugin builds its own copy of the weights. On CPU this held the blob twice at the peak. Import from ov::read_tensor_data() instead, which maps the file. This lowers the CPU cache-hit and cache-only peak memory; GPU memory is unchanged. * ggml-openvino: do not write CPU model-cache blobs for MoE models The CPU plugin exports its MoE op (GatherMatmulCompressed) but cannot import it again with OpenVINO 2026.4 and 2026.5 ("unsupported opset"). Every later run then failed the import, compiled again and rewrote the blob. Skip the blob for such models on CPU and print a warning. In cache-only mode no blob exists for them, so the generic advice to run once without cache-only cannot help. Print an error that says cache-only does not work for MoE models on CPU instead. * ggml-openvino: fix GGML_OPENVINO_DEBUG_INPUT on GPU GPU inputs can be device (USM) tensors, whose data() throws, so every GPU compute failed with GGML_OPENVINO_DEBUG_INPUT=1. Print a host copy of remote tensors instead. * docs : add OpenVINO memory optimization tips and MiniCPM-V-4.5 Add a "Tips for Memory Optimization" section to OpenVINO-ENV-VARS.md with examples for the spill directory, the compiled-model cache, host weight release on GPU, CPU MoE compile memory and the NPU weightless cache, and link it from OPENVINO.md. Add MiniCPM-V-4.5 to the validated models. * ggml-openvino: run MoE models with F16 and BF16 weights on GPU On GPU, supports_op rejected MUL_MAT_ID with non-quantized expert weights and GET_ROWS with BF16 data, so the scheduler ran those ops on the CPU. For F16 MoE models the CPU MUL_MAT_ID then read the strided top-k view as expert ids and crashed. For BF16 models the token embedding came from the CPU and reached the OpenVINO graph as a static-shaped input, which failed once the token count changed. Both ops translate correctly on GPU, so remove the two rejections. * ggml-openvino: fix BERT embedding models on CPU and GPU BERT models (bge-m3, nomic-embed-text-v1.5) failed on every device: - A VIEW of a weight with a GGML_OP_NONE source was marked as having a dynamic token dimension, a rule meant for KV cache views. The token type row view of BERT was then never sliced, and the whole two-row table was added to the activations. Skip the rule for model weights. - A row range view of a 2D weight now becomes a Slice of the weight rows, as GET_ROWS already does. - Attention without a KV cache pads K and V up to the mask width, which depends on the token count. PAD baked the amount from the first graph, so later token counts gave mismatched shapes. Compute the end padding from the mask width at run time. The NPU still fails for these models. * docs : add bge-m3 and nomic-embed-text-v1.5 to the OpenVINO validated models --------- Co-authored-by: Sabnis, Omkar <omkar.sabnis@intel.com> Co-authored-by: Beheshti <nazanin.beheshti@intel.com> Co-authored-by: Mustafa Cavus <mustafa.cavus@intel.com> Co-authored-by: Suhas Somashekar <suhas.somashekar@intel.com> Co-authored-by: yzhou51 <yaowei.zhou@intel.com> Co-authored-by: Mostafa Faheem <mostafaaafaheem@gmail.com> Co-authored-by: zhaixuejun1993 <xuejun.zhai@intel.com> Co-authored-by: Anisha Kulkarni <anisha.dattatraya.kulkarni@intel.com> Co-authored-by: Ananth, Swathi <swathi.ananth@intel.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Yu, Zijun <zijun.yu@intel.com> Co-authored-by: Proshunin, Nikolay <nikolay.proshunin@intel.com> Co-authored-by: Alexey Kopytko <alexey@kopytko.com> |
||
|
|
3a7660371d |
ui: move the web UI to ui/apps/chat and call it Llama Chat (#32032)
* ui: move tools/ui to ui/apps/chat The chat app moves to its own root folder, ui/apps/chat, as the first step of splitting the UI into one library (ui/lib) and the apps built on it (ui/apps). The code is unchanged; every path that pointed at tools/ui now points at ui/apps/chat: CMake adds it from tools/, and the Dockerfiles, the Nix package, the workflows, the labeler, CODEOWNERS, .editorconfig, the git hooks, the dev and demo scripts and the docs follow it. Assisted-by: Claude (Opus 5.5 Medium) * ui: name Llama Chat's package @ggml-org/llama-chat It was still called llama-ui, the name of the library that now sits beside it in ui/lib (@ggml-org/llama-ui). The package stays private; the app's display name and the llama-ui.zip build artifact are unchanged. Assisted-by: Claude (Opus 5.5 Medium) * ui: call the built-in web UI Llama Chat in docs, help text and comments The app in tools/ui is Llama Chat; Llama UI is the component library it is built from. Prose that said "Web UI", "web UI" or "WebUI" for the app now names it: the READMEs, SECURITY.md, the --ui help text, the CMake option description, one server log line, code comments and the Storybook intro. Names that others depend on are unchanged: the deprecated --webui flag aliases, the LLAMA_*_WEBUI CMake and environment names, the webui_settings field, the storage migration keys and the Nix attributes. Assisted-by: Claude (Opus 5.5 Medium) * ui: name tools/ui as a former path of Llama Chat in the code-review skill Changes under the old tools/ui path, as well as tools/server/webui, still get the Llama Chat checklist. Assisted-by: Claude (Opus 5.5 Medium) * ui: name Llama Chat's path in the security policy's exclusions Llama Chat no longer lives under tools/server/, so the exclusion listed under it now names ui/apps/chat. What is covered stays the same. Assisted-by: Claude (Opus 5.5 Medium) |
||
|
|
61dbf49228 |
Hexagon: add support for IQ2_XXS and NVFP4 (#31954)
* ggml-hexagon: IQ2_XXS and NVFP4 tiled repack layouts * ggml-hexagon: IQ2_XXS HVX matmul kernel * ggml-hexagon: NVFP4 HVX matmul kernel * ggml-hexagon: NVFP4 per-tensor scale * ggml-hexagon: HMX paths for IQ2_XXS and NVFP4 * ggml-hexagon: supports_op coverage for NVFP4 and IQ2_XXS * fix HMX IQ2_XXS dequant scale underflow in fp16 * hexagon: IQ2_XXS 2-gather tile decode with compact grid, small-weight tests * IQ2_XXS vpopcount sign parity and gather pipelining * pipeline IQ2_XXS HMX dequant gathers and add large_w tests * hex-iq2-xxs: rebase and cleanup Remove unused helper functions. Add lut_size to the overall vtcm required size. Trim down (way way down) the new tests. * hex-iq2-xxs: actually trim down the tests --------- Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com> |
||
|
|
2a4daf0e08 |
hexagon: add Q5_0 and Q5_1 support (#32094)
* hexagon: add q5_0 and q5_1 support * hex-mm: dedicated q5_0 kernels |
||
|
|
18c5a95e2b |
sycl: Add device-side MoE scheduler (#30156)
* sycl: Add device-side MoE scheduler
Assisted-by: Codex
* Add width-heuristic override bit and Q2_K and Q3_K types
Assisted-by: Codex
* tests: cover skipped MUL_MAT_ID slots on the device MoE schedule
Assisted-by: Claude Opus 5.5
* sycl: keep graphs on for lazily reordered MUL_MAT and reject BN=64 combos on the device MoE schedule
The graph check disabled SYCL graphs until every MUL_MAT weight with a
reorder extra was reordered, which never happens for some (dense MXFP4,
the fused GLU plain path), so graphs stayed off and the check reran on
every compute. Upstream already reorders those during recording with
the async memory extension, so only MUL_MAT_ID weights are reordered
ahead of recording now.
ggml_sycl_grouped_dequant_gemm_device_supported() accepted any XMX
combo, but the device schedule only runs with BN == GGML_SYCL_FG_BN, so
a 32x64 combo (GGML_SYCL_XMX_GATHER_SHAPES=4 or 8) made the graph check
pass and the compute fall back to the host sync path while recording.
Assisted-by: Claude Opus 5.5
* sycl: keep the ggml_sycl_mul_mat_id_needs_sync name for the graph check
Assisted-by: Claude Opus 5.5
* fix: stop erroneously reordering host-offloaded tensors
With -ncmoe, prompt processing offloads the CPU-resident expert matmuls
to the GPU: the scheduler uploads the experts into a copy in the compute
buffer before every run. init_tensor gave that copy a reorder record, so
the MUL_MAT_ID gather path reordered every offloaded expert tensor on
every ubatch, and the next upload overwrote the result.
Qwen3-30B-A3B Q4_K_M, 1x B60, llama-perplexity -c 4096 -ub 2048
-ncmoe 48, 8 chunks, on master
|
||
|
|
d5bbcd1f31 |
ci: update oneAPI 2026.2, oneDNN 2026.1 and Level Zero SDK 1.34.0 (#31540)
Co-authored-by: Neo Zhang <zhang.jianyu@outlook.com> |
||
|
|
818ac6b078 | sycl: refuse device buffers that would exceed VRAM (#31285) | ||
|
|
e9a2a44fe9 |
hexagon: n_hvx and n_threads cleanup (#31857)
* hex-nhvx: apply nhvx updates to the latest master * hex-nhvx: set N_HVX and N_THREADS based on the arch to avoid overallocation * hex-build: add support for enabling HTP_DEBUG via build.py * hex-trace: move tracing state/helpers into new htp-trace.h header * hex-dcache: cleanup dcache #defines * hex-hnvx: minimize the diff by reflowing the macros * hex-ci: fix python type/linter warnings * hex-nhvx: consitent logging |
||
|
|
67107ac34f |
hexagon: Implement set_tensor_n and improve repacking/tiling robustness (#31744)
* hex-repack: move repacking/tiling into graph-optimize * hex-tiling: simplify tiling rules and remove obsolete helpers * hex-ops: use a hash for de-duplicating tensor descriptors * hex-bufs: improve tensor hash * hex-ops: accept empty (zero-elements) ops * hex-bufs: enable hostbuf by default and reject tensors that do not fit into max_vmem * hex-bufs: improve error message and update readme * hex-bufs: make sure to sync flush opbatch in all sync set_tensor paths |
||
|
|
458fcfc340 |
docs: CANN backend supports BF16 on 910B, document GGML_CANN_NO_PINNED (#31249)
The note under 'Run the inference' still said that the CANN backend only supports FP16/Q4_0/Q8_0 models, while the DataType table in the same file, and ggml-cann.cpp (BF16 is enabled for MUL_MAT, GET_ROWS, CPY and CONT for every non-310P device), say otherwise. Spell out the per-device support and mark BF16 as unsupported on 310P instead of leaving the cell empty. GGML_CANN_NO_PINNED is read by ggml-cann.cpp when allocating host buffers and when advertising the host_buffer device capability, but it was not documented, so the escape hatch for a failing pinned host allocation was not discoverable. Signed-off-by: li-lizhe <147392333@qq.com> |
||
|
|
f7c01c5226 |
[SYCL] add DPAS flash attention kernel for quantized KV cache (#31125)
* sycl : add DPAS flash attention kernel for quantized KV cache Decode and small batches (1 to 31 query tokens) now run flash attention straight on q4_0 and q8_0 KV blocks, instead of scalar dequant or converting the whole cache to f16 per call. Q.K uses s8 x s4 and s8 x s8 DPAS on XMX, V.P stays f16. dpas.s4.s8 is inline vISA since joint_matrix has no s4. Covers head sizes 64 to 512, GQA up to 32 and attention sinks. Prompts stay on the existing kernels. GGML_SYCL_FA_DPAS=0 turns it off. B580, Llama 3.1 8B Q4_K_M, q4_0 KV, tg128: 31.3 to 54.2 t/s at 16K context. FLASH_ATTN_EXT 4582/4584, same 2 failures as master. Assisted-by: Claude * sycl : 192 head dropped, Wildcat Lake added to docs Assisted-by: Claude * sycl : stricter checks from review Assisted-by: Claude * [SYCL] : rename XMX macro, add GGML_SYCL_SUPPORT_XMX option, document DPAS Assisted-by: Claude |
||
|
|
d930fe3c26 |
opencl: refine mul mat id -1 index (#31170)
Assisted-by: Codex |
||
|
|
7aca0d2ea8 |
hexagon: add GGML_HEXAGON_VTCM_SIZE (#31221)
* hexagon: add GGML_HEXAGON_VTCM_MB and GGML_HEXAGON_NTHREADS caps Change-Id: Idffe1613c8db6f96aa842d43d1b2efaed45462a8 * hexagon: rename VTCM cap to GGML_HEXAGON_VTCM_SIZE, drop NTHREADS cap Change-Id: If4c5b9ce9fd6b725c3561421c2295d40599d012d * hexagon: make GGML_HEXAGON_VTCM_SIZE an override, accept K/M/G suffixes Change-Id: Ib8eea84c22f24e6d986b37f215e504b046478c2c |
||
|
|
7377bada8c |
sycl: Add graph record and replay (#28725)
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com> |
||
|
|
d81d0d8fb9 |
Multi-column on Intel XMX matrix engines refactored (#29864)
* Multi-column on Intel XMX matrix engines refactored Multi-column (2 to 80 activation columns) mul_mat_vec_q for reordered K-quant and Q8_0 weights on the Intel XMX matrix engines (int8 DPAS, ESIMD), for the SYCL backend. It targets speculative decoding verification (2 to 8 tokens per step) and short prompt batches (9 to 80 tokens). Three types also use it for a single column (Q2_K, Q3_K, Q8_0). * Fixed regression bug real bug found: the first version produced NaN whenever the row count wasn't a multiple of 4 * Removed GGML_TYPE_Q8_0 activation * rewritten xmx_mul_mat 64-byte alignment and 16-byte pitch check on the activations * Implemented fused gate/up kernel One launch computes up * act(gate) for both FFN weights, reading the activations once and applying SWIGLU or GEGLU in the epilogue. It covers up to 16 columns and works for all six XMX types. * Removed the temporary XMX_NOGLU environment switch * Q2_K min-sum optimization Q2_K weight is d·scl·q − dmin·mc, where scl (the 4-bit scale) and mc (the 4-bit min code) apply to each half of a 32-weight group * Added dispatch check and GRF sweep tg128 on Q4_K_M goes from 26.67 to 28.23 tok/s on the B70 (+5.9%). Q6_K is unchanged. MUL_MAT passes 1727/1727 and MUL_MAT_VEC_FUSION 1361/1361. Perplexity at batch 1 is 5.2064 with XMX and 5.2058 without. * Renamed variables * Added 2 environment variables GGML_SYCL_XMX_MAX_COLS (default 80, any value 0 or higher) and GGML_SYCL_XMX_GLU_MAX_COLS (default 16, range 0 to 16): the largest batch for the fused gate/up kernel. It is clamped to 16 because that is the most the fused kernel is compiled for, so the variable can only lower it. * Update esimd.hpp 1. All five quant structs were repeating the same boilerplate, so I moved it into shared pieces. 2. A new helper xmx_term does one DPAS, converts it to float and scales it per row. The four branches are now one or two lines each, and the shared Cb += z * d8r is written once. The refactor passes all three op tests. MUL_MAT_VEC_FUSION is 1369/1369, MUL_MAT is 2281/2281 (the count includes your uncommitted f32 tests) and MUL_MAT_ID is 937/937. |
||
|
|
522f2b6eed |
ggml-zdnn: introduce flash attention op (#30512)
* ggml-zdnn: add scratch buffer pool Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * ggml-zdnn: implement fattn, ditch scratch slots in favour of scratch buffer pool Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * docs: add note about fattn Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> --------- Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> |
||
|
|
8466395353 |
hexagon: update to snapdragon-toolchain v0.8 and enable hexagon v85 (#30854)
* hex-build: bump toolchain, enable v85 * hex-build: bump toolchain to v0.8 in the CI and build scripts * hex-build: add a little python script for swiv needed on some iot devices * hex-build: fix flake8 warnings * hex-arch: update version cap to v85 * hex-inspect: flexible TOOLS_ROOT * hex-build: update to snake_case naming * hex-build: update leftover old toolchain refs * hex-target: update build and run scripts to support ubuntu with adb transport |
||
|
|
797de63b2f |
hexagon: add support for Q1_0 and Q2_0 (#30122)
* hexagon: add support for q1_0 quant * hexagon: enable Q1_0 weights in the 2D HMX MUL_MAT path * hex-q1: cleanup gates, generalize tiling and remove in-loop divs * hex-tiling: document tiled formats in developer.md and update htp-tensor.h * hex-q2: add support for Q2_0 * hex-q2: remove stale asserts and speacial cases * hex-mm: fix MUL_MAT_NX assumption about square tiling, Q1/Q2 have non-square tiles * hex-q1/2: minor optimizations (unrolls/un-unrolls/...) * hex-q1: optimize Q1_0 layout to reduce unpack/vrmpy overhead * hex-q1/q2_0: further optimize vec_dots --------- Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com> |
||
|
|
b797b9ca81 |
hexagon: tensor repacking rework and support for tiled and flat quantized and fp weights (#30635)
* hex-repack: Remove need_repack hack and streamline the rules -- breaks GET_ROWS tests for K-Quants, will fix in the next round Add support for GET_ROWS with tiled Q5_K Add support for raw k-quant tensors * hex-get-rows: vectorize raw k-quant ukernels * hex-binary: refactor dma-loop thread functions (no need for macros, etc) * hex-quant: move/cleanup flat and tiled dequant ukernels * hex-repack: add support for repacking/tiling F16/F32/BF16 2D weights * hex-get-rows: remove redundant cast * hex-repack: proper filtering for BF16 that were leaking through during the tests * hex-repack: streamling tiling rules (broken MM for now) * hex-repack: fix mmid after tiling updates * hex-repack: rename repacked to tiled in all flags/funcs/kernels/vars * hex-mm: fix valid_rows calc in various kernels * hex-mm: remove unused hvx_mm_nx_2d_flat * hex-mm: cleanup and optimize hvx-tiled vec_dot_f16 * hex-mm: another pass at hvx-tiled f16 matmul * hex-get-rows: optimize small src1 * hex-get-rows: one more pass at recovering get-rows perf regression * hex-repack: export GGML_HEXAGON_RT_REPACK via run.py * hex-mm-tiled: fix accuracy regression with hmx-tiled-f16 matmul * hex-mm: fold tiled_f16 checks into the general flow * hex-get-rows: remove dead checks for buffer_is_hexagon and tiling during supports_op * hex-quant: proper tail handling in new dequant ukernels * hex-get-rows: remove unused code and cleanup support checks * hex-mm: cleanup ne1 padding/tiling checks * hex-mm: cleanup fusion checks for tiled tensors * hex-repack: add a little can_repack helper to reduce duplication * hex-repack: never repack host buffers * hex-repack: factor out repack functions into separate .h and .cpp * hex-tensor: cleanup tensor ifaces for tiled/padded sizes * hex-repack: update k-quant tiling comments and cleanup #defines * hex-dcache: factor out l2/dcache dirty tracking and flushing * hex-tensor: update docs and remove #defs duplication * hex-tensor: remove unused/incorrect get_data interface * hex-get-rows: cleanup vtcm layout build * tests: disable huge MM tests * hex-dcache: add missing header |
||
|
|
c114bd4c58 |
docs: update IBM zDNN backend docs (#30527)
* docs: update zdnn lib install + add notes on running models Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * docs: change title Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * docs: add note about minimum version Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> --------- Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> |
||
|
|
318a471a62 |
docs : clarify installation recommendation for Intel oneAPI Toolkit (#30407)
Added that the oneapi toolkit includes the sycl dependencies. This makes it clear for first time users, that the toolkit includes all necessary dependencies, so that the user is guided to install the toolkit if they do not already have the dependencies separately |
||
|
|
091b6a7fc5 |
Enable OpenSSL in FP32/FP16 SYCL builds to restore HF HTTPS downloads (#30330)
Co-authored-by: Raffaele Di Donna <235519540+raffaeleedidonna@users.noreply.github.com> |
||
|
|
ec0bfac179 |
hexagon: add configurable HTP DCVS/HMX power mode (#30423)
* hexagon: add configurable HTP DCVS/HMX power mode * hexagon: address PR review feedback (drop CLI flag, align QAIRT power modes, comment/log cleanup) * hexagon: add --hex-power-mode flag to run.py Hooks GGML_HEXAGON_POWER_MODE into the unified run.py runner, following the existing --hex-nhvx pattern. The env var must be set before the process touches the Hexagon backend registry (see comment in ggml_hexagon_init), so this is exposed as a runner flag rather than a common/arg.cpp CLI option. * hexagon: keep HTP awake for DCVS power profiles * Apply suggestion from @max-krasnyansky --------- Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com> |
||
|
|
af50f2f248 |
ggml-zdnn : introduce Q8_0 mat mul (#29836)
* ggml-zdnn: initial prototype of q8_0 matmul Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * docs: update zdnn docs Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> --------- Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> |
||
|
|
b42b7e6d30 |
server: use port 9931 by default (#30159)
* server: use port 9931 by default * revert unrelated changes |
||
|
|
000bee54a5 |
sycl: stage bulk uploads (model loading) through a pinned ring buffer (#29608)
Co-authored-by: cwriter <cwriter@localhost> |
||
|
|
847f447c31 |
sycl: add grouped MoE XMX GEMM (#29245)
Co-authored-by: cwriter <cwriter@localhost> |
||
|
|
c479922ac5 |
hexagon: CPY/CONCAT/CONT/DUP overhaul to use DMA/HVX for all cases (#30067)
* hex-cpy: replace more paths with dma and simplify l2flush * hex-cpy: use DMA in all sametype paths * hex-cpy: rewrite the rest of the copy paths (diff type) to use dma * hex-concat: use dma for multi-dev path which also removes the need for l2-line alignment * hex-concat: proper support for mdev splitting * hex-concat: cleanup ctx and kern params usage * hex-cpy: cleanup contex and remove left-over non-dma checks * hex-cpy: clean dma_cpy naming * hex-cpy: proper kernel params and kernel selection * hex-build: resolve left-over rebase conflicts * hex-cpy: update dev guide to clarify 128 byte alignment requirement * hex-concat: make sure we go through mdev barrier * hex-concat: make sure to flush dma-queue * hex-cpy/concat: cleanup kparams and vtcm layout handling * hex-cpy: remove dead check for contig (routed to diff kernel) and update comments * hex-dev: update developer guide based on latest changes * hex-cpy: safe skip of noop copies * hex-dup: route DUP to CPY |
||
|
|
b9a5a00b86 |
ggml-openvino: fix CI tests; fix GPU regressions. (#30037)
* ggml-openvino: skip unselected graph branches and support DUP Upstream #29622 adds a mixed token/embd branch to every input embedding graph through ggml_build_forward_select(). Its nodes are not flagged for compute, but the backend translated them anyway, and the DUP in that branch was unsupported, so the scheduler split the graph and passed the embeddings across the split with a fixed token count. The first single-token decode then failed (test-thread-safety on CPU and GPU). Build the OV model from the compute nodes only, and translate a same-type contiguous DUP like CONT so the graph stays on one backend. * ggml-openvino: make inp_scale_rows token dim dynamic #29622 also moves the per-token embedding scale (gemma3, gemma3n, gemma4) into a new [1, n_tokens] input. Give it a dynamic token dim and pad it per chunk on the static (NPU) path. * ggml-openvino: skip GPU MUL_MAT op tests with unbound Q4_1/Q4_K weights Op tests build Q4_1/Q4_K weights as u4 with an f16 zero point. The GPU plugin fails to compile that form for some row counts with "clFinish, error code: -5 CL_OUT_OF_RESOURCES", which aborts test-backend-ops on the MUL_MAT cases added in #29869 (e.g. m=1000, n=2, k=1024). Model weights use a u4 zero point and are not affected. Report these cases as unsupported on GPU until the plugin is fixed. Op tests check support before allocating, so the check matches unbound weights only; model loading probes with a dummy buffer and keeps its weights on the GPU. * ggml-openvino: create FILL in the output type translate_fill always built an f32 constant, so an f16 FILL produced f32 data and the copy back overran the f16 output buffer. Use the output type for the constant. * ggml-openvino: reject CONCAT with a quantized type Quantized inputs are dequantized when translated, so the backend cannot write a quantized CONCAT output. Report it as unsupported, as for CPY to a quantized type. * ggml-openvino: handle the single recurrent state gather of build_rs #29856 changed build_rs to gather all recurrent states with one GET_ROWS on the s_copy leaf and take the ubatch and extra states as views of it. The stateful path matched only the previous form, a GET_ROWS per view of s_copy, so Qwen3.5 failed with stateful execution on CPU and GPU ("is_axis_valid(axis, r)" in a Concat). For a single-slot cache, treat the GET_ROWS on the s_copy leaf as the active-state gather, keep the rank-4 layout of reshapes that read a view of it, and map the copy of the empty extra-state view to the single-slot remainder writeback. Do not warn about the dynamic dim of empty views. * openvino: align eltwise operand ranks to work around a GPU-plugin defect * openvino: match the MoE fusion on the rank-3 stateful graph * ggml-openvino: do not unsqueeze an RMS norm output in AlignEltwiseOperandRanks The pass unsqueezes the lower-rank operand of an Add/Multiply/Subtract whose operand ranks differ. In gemma-3 the lower-rank operand of the post-attention residual add is the norm output, and unsqueezing it makes the GPU plugin compute the layer wrongly: gemma-3 returns empty answers on GPU with stateful execution. Skip the rewrite when the lower-rank operand is an RMS norm output. * docs : update OpenVINO validated models --------- Co-authored-by: Mustafa Cavus <mustafa.cavus@intel.com> |
||
|
|
b92761a515 |
ggml-openvino: update to 2026.4.1, optimize performance, expand ops, improve device listing. (#29852)
* ggml-openvino : Qwen3.5 MoE perf (#312) Squash of ravi9/llama.cpp#312: - ggml-openvino: add detailed inference profiling (Yu, Zijun) - ggml-openvino: use remote output tensors by default (Yu, Zijun) - ggml-openvino: optimize single-sequence recurrent state (Yu, Zijun) - opt1: remove recurrent reset for single sequence, opt2: direct gdn outputs (break parallel sequence) (Yu, Zijun) - fix parallel sequences (Yu, Zijun) - ggml-openvino: simplify graph cache key (ynimmaga) - enable stateful for qwen35 single sequence (Yu, Zijun) - Fix after rebasing (Yu, Zijun) - Add k-requant option q4_asym64 (Yu, Zijun) - Fix qwen35 llama-bench -p 0 (Yu, Zijun) - Simplify RESHAPE translation (Yu, Zijun) - openvino: fuse MoE routing (Yu, Zijun) - openvino: fuse GDN qk normalization (Yu, Zijun) - openvino: enable GPU MoE fusion by default (Yu, Zijun) - ggml-openvino: add cache_only mode to import cached compiled model on disk directly (Yu, Zijun) - openvino : report the device allocation limit to ggml (Łukasz Ślusarczyk) - Fix windows build (Yu, Zijun) Co-authored-by: ynimmaga <ynimmaga@users.noreply.github.com> Co-authored-by: Łukasz Ślusarczyk <lukasz.slusarczyk@intel.com> * ggml-openvino: Update doc of compiled model cache * openvino: implement PRD-compliant device enumeration and memory reporting * openvino: fix multi-device listing issues from review - Only the device selected by GGML_OPENVINO_DEVICE reports as GPU; the other OpenVINO devices report as IGPU so llama.cpp does not offload to them. Initializing a non-selected device logs a warning. - Name devices OPENVINO<i> again and show the OpenVINO id in the description. Raw "CPU" names shadowed the ggml CPU backend. - Support GPU.N: create the OpenCL queue on OpenVINO's own context for the selected device, and replace "GPU"/"NPU" string comparisons with ggml_openvino_is_gpu()/ggml_openvino_is_npu(). - An unavailable GGML_OPENVINO_DEVICE is now an error that lists the available devices, instead of silently falling back to CPU. - Memory: cap iGPU/NPU free memory at system available memory, fall back to system memory instead of 0/0 when the plugin lacks memory properties, and ignore host USM allocations in GPU usage. - Initialize the device config once under a lock, even if OpenCL setup fails. - Fix supports_op return type for non-selected devices (build error). * openvino : take USM entry points from the selected device platform clGetExtensionFunctionAddressForPlatform was called on the first platform returned by clGetPlatformIDs. The address it returns is only valid for the platform it was queried on, and the first platform is not always the one that holds the device OpenVINO selected. On a host whose first platform comes from another vendor the lookup returns null, and then every read, write and memset on a GPU buffer fails with "clEnqueueMemcpyINTEL not available". Look both entry points up in init(), on the platform of the device OpenVINO picked, and keep them in the device config next to the command queue. Assisted-by: Claude Opus 5 * openvino: fuse MoE experts for models with a fused gate_up weight FuseMoeCompressed only matches models whose gate and up projections are separate GatherMatmul ops. gemma-4 packs both into one expert weight and splits the result after the GEMM, so its MoE block stayed unfused and ran the expert GEMMs as per-token GEMVs. Add FuseMoeCompressedFusedGateUp, which matches that shape (one GatherMatmul -> Slice/Slice -> Gelu(ERF) -> Multiply) and folds it into the same MOECompressed op, using GEMM3_SWIGLU with GEGLU_ERF. The fused weight, scale and zero point are split into gate/up halves by copying raw bytes, since a graph Slice would be rewritten to StridedSlice and constant folded, whose reference evaluator crashes on sub-byte types. gemma-4 also applies a per-expert output scale to the down projection before the router weights. MOECompressed takes only one per-expert weight, so that scale is folded into the routing weights, which is exact. The op reads the zero point straight off a weight port and needs an integer Constant there, so the matcher requires one and leaves natively quantized experts (exact f16 zp) to the unfused path. gemma-4-26B-A4B on Arc B390, GGML_OPENVINO_REQUANT_KQUANT=q4_asym64_all, llama-bench -p 512 -n 128 -r 2, against a GGML_OPENVINO_MOE_OP=0 baseline: pp512 66.16 -> 1608.73 t/s, tg128 25.94 -> 26.46 t/s. Perplexity over 12 chunks is unchanged (1451.3 +/- 177.9 unfused vs 1427.6 +/- 175.1 fused). No effect without that requant option, on models with separate gate/up weights, or on CPU. test-backend-ops -b OPENVINO0 is unchanged by this commit: two MUL_MAT_ID m_v cases fail, the same two on the unmodified base. * openvino: fix rank-3 axis handling so MoE works under stateful execution Stateful execution drops the leading size-1 batch dim, so OV tensors are rank 3 while GgmlOvDecoder::get_shape/get_stride still report GGML_MAX_DIMS=4 reversed entries. Several MoE ops derive OV axis indices straight from that metadata, so they picked the wrong axis. A MoE model with GGML_OPENVINO_STATEFUL_EXECUTION=1 aborts while building the graph: Check 'is_axis_valid(axis, r)' failed at src/core/src/validation_util.cpp:336 While validating node 'opset11::TopK ... _ffn_moe_probs ...' Axis 3 out of the tensor rank range [-3, 2]. Fix idiom throughout: take the axis from the real OV rank, or shift a metadata-derived axis down by metadata_rank - actual_rank. argsort.cpp the router top-k axis is 2 on rank 3, not 3. This is the abort quoted above. add.cpp the MoE expert-sum bypass collapses the 8-ADD chain into one ReduceSum on hardcoded axis 2, which on rank 3 reduces n_embd instead of the expert axis. Now rank-2, with the following Unsqueeze at rank-3. get_rows.cpp squeezing a hardcoded {0,1} also strips the batch dim whenever it is 1, which is every decode step. Squeeze down to the trailing two dims instead. mul_mat_id.cpp pick the reshape dims by actual rank, and skip the trailing Unsqueeze that re-adds the batch dim. view.cpp the expert-plane slice had the Slice axis, dst_ov_axis, the ShapeOf+Gather index and the Reshape target all rank-4. utils.cpp process_view_input_new's "translate_view already resolved this VIEW, skip re-slicing" shortcut required equal ranks. 4 vs 3 never matched, so every resolved expert plane got re-sliced. Now compares the common trailing dims. Same axis shift for the Slice in the view-chain walker. Stateless is unchanged by construction: every edit is gated on the actual rank, so axis_shift == 0 reproduces the previous code exactly. Checked on OV-CPU by diffing greedy output against the unmodified base for dense gemma-4-E2B, granite-1b-a400m and gemma-4-26B-A4B; all identical. granite-1b-a400m on OV-CPU aborts with the error above before this change; after it, it generates and is byte-identical to stateless. Dense gemma-4-E2B is identical stateless vs stateful both before and after. test-backend-ops -b OPENVINO0 is unchanged: two pre-existing MUL_MAT_ID m_v cases fail, the same two on the unmodified base. gemma-4-26B-A4B is a poor correctness vehicle here. On OV it already drifts into degenerate repetition a few tokens in, in stateless as much as stateful, and the two modes diverge somewhere inside that degenerate region instead of matching token for token. Each mode is self-reproducible across runs. Known limitation: FuseMoeCompressedFusedGateUp does not match the rank-3 graph, so a MoE model run with GGML_OPENVINO_STATEFUL_EXECUTION=1 loses the prefill fusion while gaining decode. gemma-4-26B-A4B on Arc B390, GGML_OPENVINO_REQUANT_KQUANT=q4_asym64_all, llama-bench -p 512 -n 128 -r 2: unfused (GGML_OPENVINO_MOE_OP=0) pp512 66.16 tg128 25.94 fused, stateless (default) pp512 1608.73 tg128 26.46 fused, stateful pp512 66.18 tg128 29.91 Stateful is opt-in and off by default, and MoE did not run there at all before this, so nothing that previously worked regresses. Making the pass match rank 3 is the follow-up. * OpenVINO Backend: Upgrade graph cache to use node_idx, src_idx, node type * ggml-openvino : enable more comprehensive conv fusion * enable conv ops * Reject kernel size 0 and support IM2COL_3D * openvino : abort when the GPU remote context cannot be created init() logged the error and returned, which left the device name a GPU but remote_context empty. The remote buffer and tensor paths assert only on the device being a GPU and then dereference that empty optional. Those paths have no host fallback, and a device that OpenVINO listed should have a working OpenCL context, so stop instead of continuing. An OpenCL stack that is broken as a whole is still caught earlier by the device availability check, which falls back to CPU. Assisted-by: Claude Opus 5 * openvino : fix build warnings The single-argument form of the OpenVINO RTTI macros is the intended one, but their selector macro leaves __VA_ARGS__ empty, which -Wpedantic reports on every pass and op header. Turn that warning off for this backend only, the way ggml-cuda and ggml-sycl already do for their own third-party warnings. Also drop a break and a dead assignment around a GGML_ABORT, which is noreturn. Assisted-by: Claude Opus 5 * OpenVINO Backend: Support common MTMD ops * ggml-openvino: give a reshaping view its own ov::Tensor * ggml-openvino : compute HARDSIGMOID and EXPM1 in f32 HARDSIGMOID used a 1/6 constant in the input type, which is not exact in bf16, and EXPM1 lost precision for small inputs in f16. Both now compute in f32 and convert back, except on NPU where the f32 path gives wrong results. Fixes the HARDSIGMOID/EXPM1 test-backend-ops failures on GPU. * ggml-openvino : update device selection and --list-devices Show the selecting GGML_OPENVINO_DEVICE value and active device in --list-devices, startup logs, and backend tests. Clarify OpenVINO selection uses GGML_OPENVINO_DEVICE, not -dev. * openvino : remove unreachable OpenCL queue checks A remote buffer exists only on a GPU device, and init() aborts there if the queue cannot be created, so the queue is never null at these call sites. Assisted-by: Claude Opus 5 * openvino : update OpenVINO to 2026.4.1 and GPU drivers to 26.35.39758.10 * docs : update OpenVINO validated models and GPU driver version * ggml-openvino : skip empty views when giving a reshaping view its own tensor A zero-size view can sit at the end of a GPU USM buffer (Qwen3.5 recurrent cache). Wrapping it as a remote tensor throws "shared USM buffer has smaller size (0)". Assisted-by: Claude * ggml-openvino : rebind the cached decoder when llama passes a different graph llama keeps separate graphs for batches with and without outputs. llama-server splits the prompt into chunks for context checkpoints, so a cached decoder could be reused with a graph built in other memory and bind the previous chunk's input tensors. SWA and recurrent models then lost most of the prompt in llama-cli and llama-server. Assisted-by: Claude * docs : update OpenVINO validated models Smoke test on Lunar Lake (32 GB) with the two fixes above. Re-add the Qwen3.5 and gemma models. Assisted-by: Claude --------- Co-authored-by: Yu, Zijun <zijun.yu@intel.com> Co-authored-by: ynimmaga <ynimmaga@users.noreply.github.com> Co-authored-by: Łukasz Ślusarczyk <lukasz.slusarczyk@intel.com> Co-authored-by: haarika-madaka <haarika.madaka@intel.com> Co-authored-by: Mustafa Cavus <mustafa.cavus@intel.com> Co-authored-by: Mostafa Faheem <mostafaaafaheem@gmail.com> |
||
|
|
392ded6546 |
SYCL: Q8_0 DMMV ESIMD and MMVQ wide load (#29186)
* Adding wide-load mmvq for Q8_0 and esimd dmmv for q8_0 Assisted-by: Codex * remove guard for q8_0 * remove docs * Simplify by committing to clean code without fallback * Add feature flag as requested Assisted-by: Claude Opus 5 --------- Co-authored-by: cwriter <cwriter@localhost> |
||
|
|
b0aca3c653 |
BLAS : Document AOCL-BLAS build and label the device AOCL-BLAS (#29640)
* BLAS : Document AOCL-BLAS build and label the device AOCL-BLAS * AOCL-Blas : Add an AOCL-BLAS Quick Start and drop the fixed version path * AOCL-BLAS doc : Note on ZenDNN |
||
|
|
fc07d781e6 |
ci : update the oneAPI toolkit to 2026.1 (#29273)
* ci : update the oneAPI toolkit to 2026.1 oneDNN is removed from Intel Deep Learning Essentials in 2026.0, so staying on the deep-learning-essentials path would silently lose oneDNN support when the toolkit version is updated. Switch both the Ubuntu and Windows CI jobs to the new unified Intel oneAPI Toolkit installer, which still includes oneDNN (until 2027.0) and keeps the component IDs unchanged for the Windows install script. Measured with the same code (b10899) built with oneAPI 2026.1 vs the 2025.3-based release build on Arc B570: prompt processing 1331 vs 434 t/s (3.1x), token generation 50.1 vs 45.3-48.0 t/s. Assisted-by: GLM (z-ai/glm-5.3-flash) * docs : update the SYCL backend build requirements for oneAPI 2026.1 With the 2026.0 release the Base toolkit and the HPC toolkit are combined into the oneAPI Toolkit, and oneDNN is removed from the Deep Learning Essentials package. Update the install instructions, the verified release table and the news section accordingly. Assisted-by: GLM (z-ai/glm-5.3-flash) * ci : update the release workflow for oneAPI 2026.1 and Level Zero SDK 1.33.1 Align the release package build with the CI build update: - oneAPI toolkit 2025.3.3 -> 2026.1 (the unified oneAPI Toolkit) - Level Zero SDK 1.28.2 -> 1.33.1, and the Debian package names (level-zero/level-zero-devel -> libze1/libze-dev) - The Windows DLL copy list for the 2026.1 runtime: sycl9.dll and the .6/.3 MKL library versions Assisted-by: GLM (z-ai/glm-5.3-flash) * ci : remove the removed .spv fallback files from the Windows DLL copy list oneAPI 2026.1 no longer ships libsycl-fallback-bfloat16.spv and libsycl-native-bfloat16.spv (the OpenCL fallback mechanism changed), so the copy step failed with exit 1. Assisted-by: GLM (z-ai/glm-5.3-flash) * devops : update the oneAPI toolkit image in the Intel Dockerfile Assisted-by: GLM (z-ai/glm-5.3-flash) --------- Co-authored-by: Asahi-Prv <Asahi-Prv@users.noreply.github.com> |
||
|
|
4e7481175c |
hexagon: find software divide calls using binary inspection tool (#29449)
* hex-scripts: fix table alignment * hex-scripts: find sw div calls using binary inspection tool |
||
|
|
cd74ef6274 |
[SYCL] support sparse FA (#28796)
* fix conflict * fix format issue * rm unused code |
||
|
|
0c3626ec06 |
hexagon: overhaul of buffer and DMA handling to support 64bit mappings + improvements (#29197)
* hex-dma64: enable support extended buffer mappings and 64bit dma hex-dma64: expand binary ops to support more DMA scenarios hex-dma64: add binary-ops.h hex-dma64: add --hex-dma64 to run.py and fix minor issues hex-dma64: update SSM_CONV to use dma with proper support for 64bit hex-ops: remove obsolete gate for % 128 in binary ops hex-l2: dont check weight tensors against dirty ranges hex-dma64: most binary ops now support dma hex-dma: use dma_addr_t instead of plain uint64_t to avoid overhead on older targets hex-dma: update all dma users to use dma_data (instead of pointers) hex-dma64: simplify lazy buffer mapping and clonning hex-fusion: factor out try_fuse_common that checks for dma64 buffers hex-bufs: minor cleanup for mmaping logic hex-bufs: simplify buffer clonning hex-ssm-conv: tighten gating checks and check vtcm size in kparams hex-binary: fix incorred mod/wrap in scalar ops hex-binary: make sure to call precompute kparams in support checks hex-dma64: update addr handling in mm,concat,binary hex-dma64: fixing up leftover of dma_addr_t conversion hex-binary: redo the kernel selection again and fix regressions in MOEs hex-binary: specialize per-type/per-op hex-binary: vtcm-layout and per-src dma-queue hex-dma64: update dma_push to transparently handle 64bit/extended * hex-cpy: fix improper rebase with the fixes for cont. tensors * hex-dma-cpy: update CPY to use safe dma rows/size limits * hex-mmap: bump number of mmaps to 64 to allow avoid eviction in larger models * hex-dma: add support for the secondary ring as a fallback for too-large transactions * hex-rope: fix freq_factors access with 64bit dma * hex-dma: audit all ops for proper use/gards for 64bit addresses * hex-dma64: uninline glu-compute funcs to avoid register pressure due to 64bit addr math * hex-dma64: refactor binary ops to separate dma loops * hex-devel: add inspect script to help with dbg and analysis * hex-dma: refactor dma-pipelines in unary-ops * hex-dma: rewrite softmax to use dma * hex-dma: rewrite GDN dma loops and improve HVX register usage * hex-gdn: fuse GDN+CPY * hex-mm: factor out HVX solver * hex-mm: remove hvx-flat kernels, the chunked version now handles vtcm limits much better * hex-buffs: reject huge buffer allocations that we cannot memory map * hex-inspect: add logic to look for float promo calls * hex-mm: reduce HVX register spills in HVX prompt kernels * hex-bufs: do not double count buffers from tensors in the same op * hex-roll: fix merge conflict * hex-dma: reroute all matmul ddr kernels to new chunked dma/vtcm kernels * hex-dev: update developer docs to include inspection for register spils and float promos * hex-ops: forgot to add new headers * hex-softmax: fix gpt-oss dims * hex-dma64: cleanup dma_addr_t casts * hex-dma64: add support for dma/vtcm for flash-atten with sinks * hex-mm-add: fix MUL_MAT+ADD fusion with bias.weights in extended bufs * hex-add-id: add support for dma for src1 (exp. table) * hex-dma: imrpove v73 fallback paths * hex-bufs: do not drop extended mappings during va defrag * hex-scripts: fix flake8 warnings * hex-docs: fix editor-config warnings * hex-inspect: fix warnings from ty |
||
|
|
ebbb185227 |
openvino : Update OpenVINO to 2026.4;fix clangd,MSVC warnings; (#29009)
* Update to openvino-2026.4 * Update OV docs * ggml-openvino : fix clangd and MSVC warnings * fix int to ptr cast, more internal linkage enforcement, and avoiding duplicate switch case --------- Co-authored-by: Mostafa Faheem <mostafaaafaheem@gmail.com> |
||
|
|
77d554b26d |
OpenVINO: optimize stateful decode and GPU MoE inference (#28638)
* exclude GPU/NPU failing POOL_2D case
* Fix pool case
* ggml-openvino: fix stateful decode for Gemma-4 per-layer-type head sizes
* ggml-openvino: fix MSVC narrowing error in permute
* ggml-openvino: classify sliding-window layers structurally on interleaved-SWA models
* ggml-openvino: add GGML_OPENVINO_REQUANT_KQUANT to select a 4-bit requant target
* ggml-openvino: add GGML_OPENVINO_SPILL_DIR to spill weight buffers to disk
* Stateful Performance: Added pass::KVStateSeqAxis to change KV layout
* ggml-openvino: fix stateful decode past the sliding-window size
Assisted-by: Claude Sonnet
* ggml-openvino: refuse stateful decode that cannot resume from the KV state
The stateful path seeds its KV state from ggml's cache when the decode position
is ahead of what the state holds. That only works when ggml's cache is a plain
prefix, where cell i holds position i. A sliding-window layer keeps just the last
n_swa positions and drops the rest, so past the window cell i no longer holds
position i and the seeded state is wrong.
Slicing the state to the decode position also had no bounds check, so a position
past the end surfaced as a bare ov::Exception from the ROI constructor
(llama_decode ret = -3, with no reason given at default verbosity).
Refuse both cases with a clear message instead, and refuse on the compile path
too, where a new model starts with an empty state and so can only serve a
sequence from its beginning. Reproducible with llama-bench -d, which restores a
saved sequence state rather than recomputing the depth prefill.
Assisted-by: Claude Opus 5
* ggml-openvino: use the per-layer KV head count for the stateful KV state
The stateful path reinterprets ggml's KV buffer [1, 1, seq, n_heads_kv * head_size]
as [1, seq, n_heads_kv, head_size]. The head size is already taken from the
tensor's own combined dim, because gemma-4 varies it per layer type, but the head
count still came from a model-level scalar that compute_llm_params() overwrites
per attention node, so it ended up holding whatever the last layer said.
gemma-4 varies the head count per layer too: 12B has 8 x 256 sliding layers and
1 x 512 full layers, 31B has 16 x 256 and 4 x 512. So 40 of 12B's 48 layers were
split as 1 x 2048 instead of 8 x 256, and attention read the state with the wrong
head split - both models decoded garbage on CPU and GPU. E2B is unaffected, its
head count is 1 everywhere.
Record the count per layer instead and look it up by the cache_k_l<N> leaf name.
Key it by layer, not by layer type: the sliding/full classification comes from
cache extents, which tie at a small -c, while the head count does not.
The stateful state trim now derives its sequence axis per state for the same
reason, since pass::KVStateSeqAxis matches per state on the head count.
Assisted-by: Claude Opus 5
* ggml-openvino: apply the KV state relayout to any KV head count
pass::KVStateSeqAxis was limited to states with a single KV head, where moving
the sequence axis from dim 1 to dim 2 is a pure metadata change. The limit was
also based on a measurement showing no gain for a multi-head model, but that was
taken at depth 0, which is the one depth where this change does nothing.
With several heads the pass does more than move metadata: it drops the reader
side transpose of the whole accumulated state, which the graph otherwise redoes
every token at a cost that grows with the context length, and replaces it with a
transpose of the single new row. Measured on GPU, tg128, alternating arms:
gemma-4-12B 6.27 -> 9.11 t/s at depth 8192 (stateless is 7.69, so stateful now
wins at depth instead of losing), Llama-3.2-1B 47.8 -> 59.6 t/s. Both are within
noise at depth 0, which is why the earlier check saw nothing.
The state refill needs the rows copied rather than reinterpreted now: ggml stores
[seq][n_heads_kv * head_size], and a relayout state with several heads is a
different element order. Without that, a refill would seed wrong data - it is
reachable today through llama-bench -d.
Assisted-by: Claude Opus 5
* ggml-openvino : support ggml_rope_set_offset and simplify op support gating
* add more cpy cases
* reject BF16 cpy on NPU
* Remove mul_mat_id fallback, gate large mul_mat_id only for mxfp4
* ggml-openvino: fuse the MoE expert block into MOECompressed on GPU
* ggml-openvino: skip GPU MUL_MAT_ID for unbound expert tensors
* ggml-openvino: requantize grouped 8-bit MoE experts on GPU
* Enable special strided CPY for conv state writeback
* openvino: support cacheless encoder models on NPU
Packed QKV views used by mmBERT were rejected by the ROPE support check. This split Q/K RoPE onto CPU, prevented cacheless attention detection, and sent fragmented encoder graphs through the decoder-oriented NPUW path.
Accept packed QKV RoPE views, detect cacheless attention from its mask, and run these models as a single full-sequence prefill without NPUW or a decode graph. Also provide static mask, output index, and mean-pooling shapes and inputs.
* openvino: optimize norm and RoPE translation
Replace the decomposed mean/variance normalization graph with an opset6 MVN operation. This preserves the GGML epsilon placement while allowing OpenVINO plugins to compile normalization as one operation with fewer intermediate tensors.
Cache RoPE sine and cosine outputs in the graph-wide tensor map. Build the cache key from all RoPE parameters and the optional frequency-factor input so compatible Q/K and layer nodes share one subgraph without mixing different RoPE configurations.
Expose NodeContext::put_shared() to publish translator-created outputs for graph-level reuse.
* ggml-openvino : simplify op translators and enable IMROPE/NEOX RoPE fusion
* remove unnecessary include and clean up PAD
* fix mulmat bug
* use ov::as_type_ptr instead of std::dynamic_pointer_cast
* ggml-openvino: fix mixed-dtype ADD/SWIGLU_CLAMP, gate unsupported ROPE/SOFTPLUS cases
- translate_add: upcast mismatched operand types (e.g. f16/f32 in fused
ADD_ADD) to f32, add, then cast once to the output type. opset1::Add
requires matching input types and downcasting first lost precision.
- translate_glu_swiglu_clamp: same fix, f16 Swish/Clamp rounding was
drifting past the test tolerance.
- supports_op: reject ROPE with ne[3] > 1 (multi-sequence) since the
cos/sin tables only cover one sequence, and SOFTPLUS on GPU since the
OpenVINO GPU kernel overflows to inf for large inputs (CPU is fine).
- ci/run.sh: serialize test-backend-ops on OpenVINO GPU; running two
workers concurrently crashes the GPU plugin (CL_OUT_OF_RESOURCES).
* openvino: share compiled models with per-context inference state; fix thread-safety
* ggml-openvino: gate MoE expert-sum ReduceSum shortcut past 8 experts
The ReduceSum shortcut for the MoE expert-plane-sum ADD chain drifts past
the 1e-7 test tolerance for >8 experts (f32 accumulation order vs CPU
reference), intermittently, like the existing Q4_K/Q5_K NMSE case.
Expose is_moe_expert_sum_add() so supports_op can gate on expert count
and fall back to CPU for just that reduction op.
* ggml-openvino: gate degenerate m=1,n=1 MUL_MAT on GPU
CI hit ERR=1.8e-3 (> 5e-4 tolerance) for a scalar-output f32 dot product
(m=1,n=1,k=2048); didn't reproduce locally in 8 tries, so likely an
internal fp16 accumulation path the GPU plugin picks for this tiny
shape. m=1 output dim doesn't occur in real model weights, so gate it.
* ggml-openvino: make SoftPlus decomposition opt-in native
Assisted-by: Codex
---------
Co-authored-by: Mostafa Faheem <mostafaaafaheem@gmail.com>
Co-authored-by: Mustafa Cavus <mustafa.cavus@intel.com>
Co-authored-by: zhaixuejun1993 <xuejun.zhai@intel.com>
Co-authored-by: ravi9 <ravi.panchumarthy@intel.com>
|
||
|
|
b6b003d2cb |
sycl : Fix get mem error (#28227)
* fix for unsupport zes API * optimize the code * adjust the log level * rm unused head files * Update docs/backend/SYCL.md Co-authored-by: Titaniumtown <titaniumtown@proton.me> * fix the error to detect level zero SDK/dev package, stop build after detect the error * update the message * fix the build error when missed to install level zero dev package * rm GGML_SYCL_DEV_DEBUG, mv read env vars in all entry functions --------- Co-authored-by: Neo Zhang Jianyu <jianyu.zhang@intel.com> Co-authored-by: Titaniumtown <titaniumtown@proton.me> Co-authored-by: Neo Zhang <NA> |
||
|
|
eafe15a5e3 |
hexagon: support for multi-device model split (aka row-split) (#28589)
* hex-row-split: add support for multi-device row spliting Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com> * hex-mdev: add work splitting to fused kernels * hex-mdev: use mdev_ prefix for all multi-device state * hex-mdev: make device configuration more expressive to support device groups * hex-mdev: fix mdev session init * hex-mdev: fused nx (2x,3x) matmuls must update row counts for each w/o * hex-mdev: fix MUL_MAT work partitioning bugs introduced by mdev * hex-cont: fix crashes with new tests due to wrong striding * hex-mdev: move fences after l2flushes * hex-cont: fix work splitting for mnpu -- align chunks to cachelines * hex-mdev: fix CPY tests with multi-dev * hex-mmid: fix work partitioning with mnpu * hex-mm: fix test failures with mdev * hex-binary: fix work partitioning for mdev * hex-argsort: fix mdev partitioning * hex-mdev: fix work partitioning and general updates for all simple ops * hex-fa: fix mdev work splitting issues * hex-mdev: fixing more failing ops test * hex-mdev: update the rest of the ops * hex-mdev: refactor all mdev splitting logic to be contained within if (mdev_count > 1) {...} * hex-mdev: fix macros * hex-mdev: simplify session flush logic * hex-sync: fix recursion in session flush * hex-mdev: factor out fence buffer and allocator * hex-fence: make fence allocation more robust with reserved slots for mdev * hex-mdev: keep all mdev state in htp_mdev_group * hex-mdev: further cleanup mdev group handling at the host * hex-mdev: update group idx in the opbatch before serializing * hex-batch: remove separate op_pending and use batch_req/rsp_seq * hex-async: workaround another missing tensor_init in ggml-meta * hex-fence: cleanup and robustify fences and error handling in multi-device scenarios * hex-ar: improve ALLREDUCE error handling * hex-async: robust error handling for op_cpy_fence * hex-async: use seq0 from allreduce context to allocate fence_seq * hex-mdev: fix remaining issues with fence and barrier clearing in CPY_FENCE * hex-misc: realign macros and fix misplaces trace events * hex-misc: align macros * hex-mdev: fix unclone buffer re-entrancy * hex-glu: fix mdev partitioning logic * hex-mdev: make buffer uncloning/cleanup work with tensor-split scenarios * hex-mdev: tighten up the can_split check in act-ops * hex-mdev: factor out common bits of the partitioning logic * hex-mm: minor realignment of the macros * hex-bufs: fix incorrectly placed assert for MAX_BUFS * hex-pad: tighten up gating checks for PAD * hex-kparams: make sure all kernels properly use kparams->n_threads * hex-docs: update user and developer docs with new features and detailed guide for ops development * hex-scripts: update run script to properly parse dev groups * hex-misc: formatting * hex-sess: minor cleanup for session init * hex-ar: fix vtcm size calc in allreduce kparams * hex-scripts: fix flake8 warnings * hex-rope: update ROPE to support mdev work split * hex-ops: remove redunant checks and minor reformat * hex-dev-guide: update dev-guide to avoid redundant null checks * hex-async: improve event_wait, event_sync and fence implementations * hex-async: remove synchronous flush from event_sync * hex-async: symplify fence recovery protocol and make sync more robust * hex-async: futher simplify error recovery for fences * hex-err: return status instead of just -1 * hex-async: print all seq nums in hex * hex-async: make sure fences flush dirty ranges * hex-async: add dirty ranges merging to reduce fence flushes * hex-async: properly sync before freeing the event * hex-async: make sure fence owner session is not overriden * hex-async: more fence write order more robust * hex-async: make sure not to fuse ALLREDUCE+ADD if their dsts overlap * hex-fusion: cleanup redundant checks --------- Co-authored-by: Alexander Lu <alexlu@qti.qualcomm.com> |
||
|
|
cd8cdf397d |
sycl: attribute device allocations by site (GGML_SYCL_MEMTRACE) (#27631)
define two new environment variables to better understand how much memory is being allocated, and when. This has been invaluable in inproving the --fit algorithm, and is likely to be useful when debugging other memory-related issues. `-lv 4` will be required to enable the following: GGML_SYCL_MEMTRACE=1 will show per-site memory usage, updated whenever it increases by more than 64MiB. GGML_SYCL_MEMTRACE=2 will show every allocation and deallocation. To change the default 64MiB threshold for reporting memory usage increases, use GGML_SYCL_MEMTRACE_STEP. A sample log line: [SYCL-MEMTRACE] device memory query (dev): total 59493 MiB, free 4494, in use 54998; allocated 0 (buffers 0 + scratch 0), peak 0 MiB |
||
|
|
6703d7894c |
sycl: fuse rms_norm+mul+add and add+add residual chains (#27610)
Fuse RMS_NORM+MUL+ADD and ADD+ADD under GGML_SYCL_ENABLE_FUSION. ADD+ADD uses the same binbcast indexing and type matrix as standalone add() (f32, f16, f16/f32, i32, i16, bf16, including broadcast and non-contiguous). Unsupported combinations fall back to two add() launches. |
||
|
|
5eec3ad017 | sycl : support limit max alloc memory within 2GB for host-pinned memory (#27559) | ||
|
|
a32af33de2 |
sycl : Enhance to get the free memory of Intel GPU (#27968)
* enhance get mem info by l0 an SYCL API * remove debug code, format the code * update SYCL.md for GGML_SYCL_GET_MEM_API |
||
|
|
d7bd3bfcad |
snapdragon: python SDK setup (Windows) (#27903)
* port setup-build.ps1 to setup_sdk.py, to facilitate installation of Hexagon and OpenCL SDKs on Windows * rename setup_sdk.py -> setup-sdk.py * flake8 fix: print() -> logger.info() --------- Co-authored-by: Kristopher Urquhart <kurquhar@qti.qualcom.com> |
||
|
|
511f9c1379 |
OpenVINO: Update OV to 2026.3.1, whisper.cpp support, Qwen3.5 on NPU, and new ops (#27843)
* OpenVINO Backend: Fuse IM2COL + MatMul convolution into OpenVINO convolution * ci:ggml-ov: Skip recurrent state rollback tests * ci:ggml-ov: Skip recurrent state rollback tests * Update OPENVINO.md * ggml-openvino : add env-var gated op support debugging * Fix ggml_rope_set_offset case * OpenVINO backend: Support Whisper.cpp * Fix code style * openvino : enable qwen35 on NPU Static shapes: - get_graph_input_shape() left the s_copy / s_copy-leaf inputs dynamic ([1,1,1,-1]) even in static mode, which propagated a dynamic slot dim through GET_ROWS into the conv/GDN state, the state reshapes and the GDN output. - With -np 1 the s_copy defrag remainder gathers zero rows; short-circuit that CPY to the untouched cache instead of emitting a degenerate Slice/Concat, and skip binding its zero-byte ggml tensor as an output (the dynamic path already did the latter, the static path wrote the full cache over a 0-byte buffer). Token-count independence: - In static mode the compiled model's token count is the prefill chunk size or 1, not the captured cgraph's. Offsets derived from the captured count were therefore wrong. Anchor the GDN state slice at the end of the packed [attn | state] output and drop the rs_src_begin runtime inputs, and make VIEWs over the GDN output / conv_input pass through so the consumer does the slicing. - CONT could not identify its token axis when the graph was captured with a single token (every trailing dim has the same stride and size 1) and baked the captured shape into the prefill model. Chunked prefill: - The last chunk is padded with fabricated tokens. Attention masks them, but the recurrent path folded them into cache_r/cache_s permanently. Add a chunk_valid_len runtime input, use it to zero g and beta for padded steps (making the recurrence an exact identity) and to end the conv snapshot window at the last valid token, and disable the recurrent-cache reset after the first chunk so earlier chunks are not wiped. - get_is_prefill() and the chunk loop bound read inp_pos->ne[0] directly, but IMROPE stacks 4 position planes, so every decode step was run through the padded prefill model and the loop ran extra out-of-bounds chunks. cache_rs_reset_idx/len now stay runtime Parameters in static mode, since can_reuse_statically() does not invalidate the cached model on ComputeParams changes. Add GGML_OPENVINO_FORCE_STATIC to exercise the static path on CPU. * Update to OpenVINO 2026.3.1 * ggml-openvino: forward NPU compilation mode parameters Add GGML_OPENVINO_NPU_COMPILE_CONFIG to the backend's cached environment so callers can configure the NPU compiler without using the generic property escape hatch. When the value is non-empty, pass it to OpenVINO as NPU_COMPILATION_MODE_PARAMS. This enables settings such as optimization-level=3 for NPU compilation while preserving the existing behavior when the variable is unset and leaving CPU and GPU configuration unchanged. Document the variable, its NPU-only scope, and the optimization-level=3 example in the OpenVINO backend runtime configuration table. * ggml-openvino : support RELU, POOL_2D, QUICK_GEGLU, and ROLL ops * reorder op table * exclude GPU/NPU failing POOL_2D case * move op type detection to compute_op_case * Relax rope supported cases * Fix pool case * Update openvino doc, gpu driver in ov docker * openvino: remove unused static remote context branch * openvino: parallelize static model build * Apply editorconfig --------- Co-authored-by: Mostafa Faheem <mostafaaafaheem@gmail.com> Co-authored-by: Ravi Panchumarthy <ravi.panchumarthy@intel.com> Co-authored-by: zhaixuejun1993 <xuejun.zhai@intel.com> |
||
|
|
192067b72d |
hexagon: support for multi-NPU devices (IQ9, IQ10) and fully asynchronous backend (#26501)
* hexagon: use non-host bufs by default and make the backend fully async * hex-hb: remove optional hostbuf support and fix async copy * hex-unary: relax supported unary check * hex-bufs: use same get_alignment for host bufs * snapdragon: bump android_platform to 34 * hex-rows: super hacky get/set rows for q8_0 * hex-get-rows: fix q8_0 * hex-get-rows: supprot for f16 and cleanup for q8_0 * hex-get-rows: generic macros and specialized thread funcs * hex-get-rows: add DMA pipeline, vtcm_layout and kernel params * hex-set-rows: fix q8_0 support, add dma and tracing * hex-tests: override nmse threshold for HTP of Q8_0 quants * hex-fa: add support for Q8_0 with inplace dequantizers * hex-get-rows: simplify type dispatch * hex-rows: simplify GET/SET_ROWS DMA pipeline * hex-async: add events, set/get-tensor-async and rest of the async api support * hex-repack: use slice instead of expert in repack functions * hex-cpy: update event/async-cpy logging * hex-set-rows: optimize smaller tensors * hex-geglu: fix perf regression with larger tensors * hex-get-rows: add missing header * hex-set-rows: add missing header * hex-bufs: ressurect GGML_HEXAGON_HOSTBUF but disable it by default * hexagon: do not reject ops with non-heaxon buffers * hex-get-rows: apply >=32 restriction only for q8_0 * hex-res: bump vtcm acquire timeout to 10 seconds * hex-bufs: add support for cloning buffers between sessions to speed up tensor copies * hex-async: rework event recording and batch flushing and integrate with meta backend * hex-bufs: improved handling of repacked tensors * hex-repack: handle get_tensor_2d offsets * hex-dev: add support for devices with multiple NPUs * hex-sync: add support for sync tokens to synchronize npu devices for async splits * hex-mmap: cleanup mmap calls and add a retry for robustness * hex-sync: add failsafe if sync wait gets stuck * hex-sync: use sync_seq to check for completed events * hex-sync: rotate tokens for extra robustness * hex-devs: add supprot for legacy device names for now * hex-bufs: add support for auto-cloning buffers from diff sessions * hex-fusion: simplify and optimize htp-opnode fusion handling * hex-sync: override opnode name so that it shows up in the profiles * hex-trace: update scripts to handle multiple devices * hex-sync: bump the size of the opbatch queue and number of sync tokens * hex-cpy-sync: do not explicitly flush opbatches in cpy_tensor_async and add support for cpy-dma * hex-sync: add graph-flush threshold to avoid single op batches * hex-sync: add sync_peer so that we can flush peers we depend on during cross-device ops * hex-bufs: introduce tensor->extra and shadow_bufs for repacking * hex-l2: flush tiny tensors inline * hex-sync: use explicit l2flush for sync tokens * hex-extra: track weight flags via tensor extra * hex-fence: rename sync to fence * hex-repack: proper handling of set-tensor-2d in the shadow_buf * hex-trace: remove obsolete opstage mask that we used for profiling * hex-env: remove obsolete use_hmx variable * hexagon: new unified run.py and build.py and updated docs * snapdragon: update run script to auto-escapt test-backend-op -p argument * hex-scripts: fix trailing spaces * hex-scripts: fix flake8 warnings * snapdragon: cleanup dst lib/bin dirs before copying new build * hex-ops: add support for allreduce * hex-ar: improved allreduce with dma pipeline * hex-ar: align macros * hex-ar: consistent use of fence_seq * hex-ar: add AR_SELECT env var to select ALLREDUCE kernel or fallback * hex-ar: add proper synchronize handling for ALLREDUCE * hex-opbatch: looks like we now just rely on backend.synchronise to flush the batches, no need to flush them by threshold * hex-ar: bump block size to improve dma efficiency * hex-ar: fused ALLREDUCE+ADD * hex-ar: cleaner fence buffer management * hex-ar: futher allreduce tweaking to remove race conditions * hex-ar: add simple solver and remove non-dma kernels * hex-ar: add row-broadcast to fuse with bias ADD * hex-fence: pass seq numbers via op_params * hex-ar: allow for both entry/exit seq for completing entry wait * hex-ar: align macros * hex-ar: do not refetch broadcast row * hex-fusion: move all fusion into opbatch::add_op for consistency with ALLREDUCE and things * hex-fusion: fix incorrect MUL_MAT reordering * hex-mm: make fused 2x and 3x matmuls more generic * hex-fusion: move tensor fusion tagging to graph_compute * hexagon: make sure to copy tensor->extra by value * hex-get-rows: fix offset calc with row-chunking * hex-repack: get_tensor_2d fixes for non-zero offsets * snapdragon: make profile/trace scripts more robust and donot mix stdout/stderr by default * hex-devices: use legacy device nameing by default to ease the transition * hex-devices: hardcode CDSP domain IDs for current devices for now * hex-optrace: improve multi-NPU timestamp alignment and overall handling of cycle values * hex-optrace: more robust handling of the fence events |
||
|
|
a298422da7 | docs: fix typos in ET.md (#27457) |