Lumpiasty/llama.cpp
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

OpenVINO: Qwen3.5, memory optimization, and test-recurrent-state-rollback (#26952)

Browse Source 
* OpenVINO backend: 1) enable gpt-oss moe on OV bk; 2) enable mxfp4 support

* OpenVINO backend: disable TOPK_MOE op test

* OpenVINO Backend: Add op FILL support

* OpenVINO backend: enable set rows with multi dims

* fix the name missmatch in setrow + view

* OpenVINO backend: enable op GGML_UNARY_OP_SIGMOID

* OpenVINO Backend: enable SQR & SQRT

* OpenVINO backend: 1) ensure unique node names for OpenVINO; 2) add org_src to recorde the src ggml tensor for OpenVINO dynamic shape infer

* OpenVINO backend: enable fallback for openVINO to CPU backend

* OpenVINO backend: fix accurace issue in gemma3n arch test

* fix mpt failed case

* OpenVINO backend: clean nodeinfo

* OpenVINO Backend: enable zero-size copy for view

* add concat ssm_conv in compute_dynamic_dim

enable qwen35

Fix after rebase

remove logging

* OpenVINO backend: disable EXP with FP32, which failed in op test. Root reason: the backend test initializes unary op inputs over a wide range, [-150, 150]. For FP32, exp(x) overflows around x ~= 88.7, so this test can randomly generate values right in or beyond the overflow region

* OpenVINO backend: fix CPY op test failed issue

* OpenVINO backend: fix GATED_DELTA_NET op test failed issue

* handle in-place op, handle qwen35 dynamic clearing of cache in cgraph

* handle qwen35 dynamic clearing of cache correctly

* Enable qwen35 dense multi seq

* Fix qwen35 9b gqa

* Fix after rebase

* Disable SOLVE_TRI

* openvino: fix NEOX RoPE accuracy on GPU stateful (mixed-rank Multiply)

In stateful mode the NEOX RoPE branch fed rank-3 data ([S, n_heads,
head_size]) into the Multiply against the rank-4 cos/sin tables
([1, S, 1, n_dims/2]). That mixed-rank broadcast is miscomputed by the
OpenVINO GPU plugin, corrupting the rotated Q/K and producing garbage
output (e.g. Phi-3-mini). Lift the data to rank-4 before the split/
Multiply so the operands are equal-rank, matching what the TYPE_NORMAL
branch already does. CPU and stateless paths are unaffected.

Phi-3-mini-Q4_K_M, wiki.test perplexity, GPU stateful:
  before: PPL = 27120.43
  after:  PPL = 6.2263   (CPU reference: 6.2251)

* OpenVINO backend: 1) remove the unique name in llama.cpp; 2) add new ov name in ov bk; 3) fix issue in arch test & op test with latest code update

* OpenVINO Backenb: remove changes in llama.cpp

* Doc change (use x64 Native Tools Command Prompt for VS)

* Cleaner sentence

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* OpenVINO Backend: cache key upgrade includes all src name

* OpenVINO Backend: enable llama arch test on ci

* OpenVINO Backend: move parameter node creating from decoder into translate

* OpenVINO Backend: create extra input ov node move from decoder to translate

* fix for op regression due to is_model_splitted

* openvino: fix CPY writeback for recurrent state rollback

Detect the rollback conv/gdn state writeback CPY nodes structurally
instead of by tensor name, since the rollback path in
build_conv_state does not call cb() and left the nodes unnamed. Add
per-node runtime offsets (rs_slot_begin_*, rs_src_begin_*) so the
cached IR handles any kv head, sequence count and snapshot slot for
both the conv state and the GDN state writeback.

Assisted-by: GitHub Copilot

* qwen35 moe

* optimize MoE expert aggregation with ReduceSum

* Skip GET_ROWS inaccurate test

* openvino: fallback dynamic MUL_MAT_ID shapes

* OpenVINO Backend: fix error in arch test model mpt

* fix error caused by cpy in arch test model kimi-linear

* OpenVINO Backend: fix error in arch test model minimax-m3

* openvino: fix GPU mul_mat_id op tests

* ggml-openvino: add GGML_OPENVINO_RELEASE_WEIGHTS to reclaim host weight RSS on GPU

The OpenVINO weight Constants are zero-copy views into host buffers
allocated by the backend (ggml_aligned_malloc, anonymous memory). On GPU
the plugin holds its own device copy after compile_model, so these host
pages are dead weight for inference. For a 1B Q4_K_M model this leaves
~850 MB of host RSS resident that the GPU path never reads again.

Add an opt-in GGML_OPENVINO_RELEASE_WEIGHTS mode that madvise(MADV_DONTNEED)s
the registered host weight buffers once the model is compiled, dropping
their resident pages while keeping the mappings valid (ggml still owns the
lifetime; tensors still point in). Measured steady-state RSS drops from
~1555 MB to ~710 MB on Llama-3.2-1B-Q4_K_M (Arc iGPU) with unchanged
throughput and correct output.

The GPU backend uses a single dynamic-shape model for both prefill and
decode, so a graph is compiled once and reused; the only event that forces
a recompile is clear_caches() on backend teardown. The change therefore:
  - releases on the first cache-hit (model compiled, plugin has its copy);
  - pins the compiled-model cache across backend teardown so a later
    context reuses it instead of recompiling against the dropped pages;
  - fails loud (GGML_ABORT) on a cache-miss recompile or on a second model
    load, both of which would otherwise read zeroed weights or silently
    reuse the wrong compiled graph.

Scope/limitations (all fail loud, never silently wrong): GPU only (the CPU
plugin reads the host Constants at inference time), one model per process,
and stable graph shapes. This reduces steady-state RSS, not the transient
compile-time peak. All changes are confined to the OpenVINO backend.

* ggml-openvino: stream weight requantization to cut the compile-time RSS peak

requantize_to_buffers() dequantized the entire tensor to a temporary
std::vector<float> of n_elements before requantizing. For token_embd.weight
(128256 x 2048) that transient is ~1 GB (1B model) / ~2 GB (8B), and it is
the single largest contributor to the OpenVINO compile-time memory peak --
it also fires twice for token_embd (once at load, once at graph build,
because token_embd is loaded via a CPU/mmap buffer and not cached as an OV
weight extra).

Stream the dequant instead: process a fixed window of complete rows
(CHUNK_ROWS=256) into a small scratch buffer and quantize/convert each chunk
straight into the output buffers. The transient F32 footprint is now
CHUNK_ROWS*ne0 floats regardless of tensor size.

quantize_q8_0/q8_1 gain an optional block_offset arg (default 0) so a chunk
writes its weights/scales/zp at the correct block. Streaming is applied to
the Q8_0_C / Q8_1_C / F16 targets (the large requant cases); the u4 (Q4_0)
path keeps the whole-array call because it packs two weights per byte with
running zp ORs, and a fallback handles any future target whose block size
does not divide a row.

Measured peak RSS (cold compile, GPU): 1B 2868 -> 1809 MB (-1.06 GB);
8B 11618 -> 9608 MB (-2.0 GB). Output verified unchanged
("capital of France is Paris"); throughput unchanged. Unlike
GGML_OPENVINO_RELEASE_WEIGHTS this reduces the transient peak, not just
steady-state, and needs no env flag. All changes confined to the OpenVINO
backend.

* ggml-openvino: avoid redundant token_embd requantization at compile

token_embd.weight is referenced twice in the graph path: as the GET_ROWS
embedding (a CPU/mmap-buffer tensor) it was re-extracted/re-requantized on
every weight-node build, and is_model_splitted() built a full (naive) set of
weight nodes just to test name membership — each requant is a ~1-2 GB F32
dequant of the 262M-element embedding.

Two changes:
- Add collect_weight_names(): a name-only collector for topology checks.
  is_model_splitted() now uses it instead of create_weight_nodes(cgraph,
  true), so the splitted-check no longer triggers any weight extraction.
- Memoize weight nodes built from non-OpenVINO buffers in a process-lifetime
  cache keyed by tensor->data. These tensors have no OV buffer context to own
  a cached extra, so without this they were rebuilt on every (re)compile;
  prefill and decode graphs now share one build (verified: 2nd graph hits the
  cache instead of re-requantizing).

Peak RSS is unchanged (the streaming-requant commit already removed the F32
transient); this removes redundant compile-time work. Output verified
unchanged ("capital of France is Paris"). Confined to the OpenVINO backend.

* ggml-openvino: gate compile-memory optimizations behind GGML_OPENVINO_REDUCE_COMPILE_MEM

The streaming requantization and the non-OpenVINO-buffer weight-node cache
(plus the name-only is_model_splitted path that pairs with it) are now opt-in
via GGML_OPENVINO_REDUCE_COMPILE_MEM. When unset, requantize_to_buffers()
fully materializes the F32 buffer and weights are rebuilt per compile exactly
as before; when set, the streaming path and the cross-compile weight cache
are used.

Default off keeps behavior identical to upstream unless explicitly enabled.
Verified: flag off -> peak RSS 2800 MB (original), flag on -> 1810 MB; output
"capital of France is Paris" in both modes. (GGML_OPENVINO_RELEASE_WEIGHTS,
added earlier, remains a separate opt-in for the steady-state release.)

* ggml-openvino: add frontend model cache (GGML_OPENVINO_MODEL_CACHE_DIR)

The plugin-level ov::cache_dir caches the compiled blob keyed by the OV
model, but producing that model still runs the full frontend every time:
weight requantization (incl. the large token_embd F32 transient) and the
ggml->OV graph conversion. This adds an opt-in frontend cache keyed off a
fingerprint computed directly from the ggml cgraph, so a hit imports a
previously exported CompiledModel and skips requant + convert + compile
entirely.

Key (model-cache.{h,cpp}) = 64-bit FNV-1a of: graph topology (n_nodes + per
node op/name), a sampled per-weight fingerprint (name/shape/type + bounded
head+tail byte sample), and blob-affecting config (device, flash-attn, rope
params, REDUCE_COMPILE_MEM/stateful flags, OpenVINO version). A sidecar
manifest stores every weight's fingerprint and is re-verified on load, so a
sampled-hash collision cannot cause a wrong-model hit (verified: two
different quantizations of the same model produce distinct cache entries).

Flow (dynamic single-model path only; split models defer to ov::cache_dir):
on a verified hit, core.import_model() restores the CompiledModel and a
lightweight decoder is built with a names-only weight map (membership is all
the decoder needs for I/O mapping; weights live in the imported model). On a
miss, compile as usual then export the blob (atomic temp+rename, manifest
written first). The frontend cache supersedes ov::cache_dir, so CACHE_DIR/
CACHE_MODE are stripped from the config used for the cached compile and the
import — a blob compiled with cache_dir set cannot be re-imported.

Measured 8B Q4_K_M (GPU): full requant+convert+compile 15.3s -> import 6.3s
(~2.4x faster compile phase). Output verified unchanged on cold and warm,
standalone and combined with REDUCE_COMPILE_MEM + RELEASE_WEIGHTS. Default
off; confined to the OpenVINO backend.

* ggml-openvino: harden frontend model cache correctness

The frontend model cache imports a previously exported CompiledModel keyed by a fingerprint of the ggml graph, weights, and blob-affecting config. The original key covered device, stateful execution, REDUCE_COMPILE_MEM, RoPE params, OpenVINO version, topology, and sampled weights, but missed runtime/frontend toggles that can change the lowered graph or the I/O binding contract. That made it possible to reuse a blob produced under a different OpenVINO backend configuration.

Add a small extra-config helper for the dynamic model-cache path and fold in the effective values of GGML_OPENVINO_DISABLE_KV_SLICE and GGML_OPENVINO_MANUAL_GQA_ATTN. MANUAL_GQA_ATTN is keyed by the behavior that actually takes effect: an explicit env value wins, otherwise GPU defaults to enabled and other devices default to disabled. This matches flash_attn_ext lowering and avoids unnecessary cache splits for equivalent configurations while separating genuinely different attention graphs.

DISABLE_KV_SLICE is also included because it changes the KV-cache tensor shape/output binding strategy used around imported models. Even when weights and graph topology are identical, switching this flag should not inherit a CompiledModel cache entry created for a different binding mode.

Also make cache artifact publication cleaner: write manifest.tmp and blob.tmp, publish the blob first, and publish the manifest last. Cache hits already require both blob and a verified manifest, so making the manifest the final visible artifact avoids leaving an apparently complete manifest for a failed or interrupted blob export. Temporary files are removed on the handled failure paths.

While touching this path, fix the indentation of the non-imported compile branch so the cache miss flow is easier to review. Behavior is otherwise unchanged: verified hits still import, misses still create weights, convert, compile, export, and create the infer request normally.

* ggml-openvino: add memory optimization umbrella switch

Add GGML_OPENVINO_MEMORY_OPTIMIZE as a single opt-in switch for the OpenVINO backend memory-saving paths. The existing fine-grained GGML_OPENVINO_REDUCE_COMPILE_MEM and GGML_OPENVINO_RELEASE_WEIGHTS variables remain supported and explicitly override the umbrella switch when set, so users can still bisect or disable one side of the optimization independently.

Centralize the policy in ggml_openvino_reduce_compile_mem_enabled() and ggml_openvino_release_weights_enabled(device). The umbrella switch enables compile-memory reductions everywhere REDUCE_COMPILE_MEM is used today: streaming requantization, non-OV weight-node caching, split-model weight-name collection, and the frontend model-cache fingerprint. On GPU it also enables host weight-buffer release unless GGML_OPENVINO_RELEASE_WEIGHTS is explicitly set.

Keep host weight release GPU-only because it relies on the plugin holding its own device copy after compile_model. Update the fail-fast diagnostic and comments to mention GGML_OPENVINO_MEMORY_OPTIMIZE, so users who enable the umbrella switch get accurate guidance if a later cache-miss recompile would read released host weight pages.

* ggml-openvino: rename compiled model cache env

Rename the frontend export/import cache environment variable from GGML_OPENVINO_MODEL_CACHE_DIR to GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR. The cache stores blobs produced by ov::CompiledModel::export_model() and restores them with core.import_model(), so the new name distinguishes it from GGML_OPENVINO_CACHE_DIR, which configures OpenVINO plugin-level ov::cache_dir.

Update the registered env var, the cache-directory lookup, and comments around the frontend compiled-model cache. The old GGML_OPENVINO_MODEL_CACHE_DIR name is removed rather than kept as a fallback so there is a single spelling for the new option.

* docs: document OpenVINO memory optimization env vars

Add runtime configuration entries for the newly recognized OpenVINO environment variables.

Document GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR as the frontend compiled-model cache used to export and import compiled blobs for matching single-graph models.

Document GGML_OPENVINO_MEMORY_OPTIMIZE as the umbrella switch, including how GGML_OPENVINO_REDUCE_COMPILE_MEM and the GPU-only GGML_OPENVINO_RELEASE_WEIGHTS override or inherit from it.

* ggml-openvino: fix Qwen3VL crash and deepstack correctness bug

1. GGML_OP_PAD was missing from compute_node_dynamic_dims(), causing a crash
on decode for models that pad the token embedding (n_embd -> n_embd_inp).
PAD never reorders/merges dims, so it keeps the same dynamic dim index as
its source.

2. process_view_input_new() chained VIEW inputs through src[0] (the
immediate op-graph parent) using offsets treated as relative to that
parent. But ggml_tensor::view_offs is always absolute from the true root
allocation (ggml collapses VIEW-of-VIEW chains internally). For the
per-layer deepstack view ("embd (view)", whose src[0] is "embd" - itself
an already-narrowed, zero-offset VIEW of the padded root, with the SAME
ggml shape as the deepstack view but a different absolute offset), this
caused an out-of-bounds re-slice that silently fell back to returning the
wrong (already-resolved sibling) tensor. In practice every deepstack ADD
ended up adding the real base token embedding into the residual stream
instead of zero, corrupting generation ("Hello my name is 1000000..."
instead of coherent text). Fixed by detecting this pattern (same shape as
the immediate src, different absolute offset) and re-slicing directly
from the untouched root tensor using the innermost view's absolute
offset.

Also adds a GGML_OPENVINO_DEBUG_NODE=<name1>,<name2>,... env var that attaches
extra debug Result nodes for arbitrary intermediate tensors, without binding
them to any ggml buffer (avoiding the risk of reading a ggml buffer that has
since been overwritten by a later in-place op). This was instrumental in
diagnosing bug #2 above and is left in as a general-purpose debugging aid.

* ggml-openvino: fix IMROPE inp_pos padding for NPU static shapes

IMROPE's inp_pos tensor packs 4 stacked t/h/w/e position planes into
ne[0] = 4*n_tokens instead of one value per token. On NPU's static-shape
path, inp_pos was padded/shaped as if it held a single plane, which
interleaved padding across the 4 planes and desynced later reshapes
from the rest of the (chunk_size-wide) graph.

- add GgmlOvDecoder::get_inp_pos_n_planes() to detect IMROPE's 4-plane layout
- get_graph_input_shape(): size inp_pos as n_planes * chunk_size (prefill)
  or n_planes (decode) instead of assuming 1 value per token
- get_ov_input_tensor_static_prefill(): pad each plane to chunk_size
  independently instead of one flat block
- get_ov_input_tensor_static_decode(): copy n_planes contiguous values
  instead of asserting/copying a single scalar

* disable test-llama-archs tests.

* openvino: gate fallback with env var

* Revert changes in test-llama-archs

* Apply editor config

* reject CPY with quantized destination as unsupported

---------

Co-authored-by: Xuejun <Xuejun.Zhai@intel.com>
Co-authored-by: Mustafa Cavus <mustafa.cavus@intel.com>
Co-authored-by: virajwad <84867530+virajwad@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
Co-authored-by: Mustafa Cavus <mustafacavus@intel.com>
Co-authored-by: Ravi Panchumarthy <ravi.panchumarthy@intel.com>
This commit is contained in:
Zijun Yu
2026-08-13 20:30:48 +03:00
committed by GitHub
co-authored by Xuejun Mustafa Cavus virajwad Copilot Autofix powered by AI suryasidd Mustafa Cavus Ravi Panchumarthy
parent a97123e497
commit aee56b3abf
42 changed files with 3480 additions and 557 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ggml/src/ggml-openvino/ggml-decoder.cpp
+423 -73
View File
@@ -16,6 +16,7 @@
#include <iomanip>
#include <map>
#include <memory>
#include <mutex>
#include <openvino/core/dimension.hpp>
#include <openvino/core/except.hpp>
#include <openvino/core/node.hpp>
@@ -25,12 +26,13 @@
#include <openvino/core/type/float16.hpp>
#include <openvino/op/constant.hpp>
#include <openvino/op/convert.hpp>
#include <openvino/op/parameter.hpp>
#include <openvino/runtime/tensor.hpp>
#include <ostream>
#include <set>
#include <stdexcept>
#include <string>
#include <cstring>
#include <unordered_map>
#include <vector>
GgmlOvDecoder::GgmlOvDecoder(ggml_cgraph * cgraph,
@@ -98,27 +100,119 @@ GgmlOvDecoder::GgmlOvDecoder(ggml_cgraph * cgraph, std::map<std::string, std::sh
}
}
namespace {
bool is_inplace_op(const ggml_tensor * node) {
return node->op == GGML_OP_SET_ROWS || node->op == GGML_OP_CPY || (node->op == GGML_OP_SCALE && node->view_src);
}
bool is_same_shape(const ggml_tensor * a, const ggml_tensor * b) {
for (int i = 0; i < GGML_MAX_DIMS; i++) {
if (a->ne[i] != b->ne[i]) {
return false;
}
}
return true;
}
bool is_conv_states_all_tensor(const ggml_tensor * tensor) {
return tensor != nullptr && strncmp(tensor->name, "conv_states_all", strlen("conv_states_all")) == 0;
}
// CPY writing the tail of conv_input (the concat of the previous conv state and the new tokens)
// back into a slot block of the recurrent state cache. Detected structurally because the rollback
// variant (cparams.n_rs_seq > 0) emits one such CPY per snapshot slot without naming them.
bool is_conv_state_writeback(const ggml_tensor * node) {
return node->op == GGML_OP_CPY && node->view_src != nullptr && GgmlOvDecoder::is_kvcache(node->view_src, nullptr) &&
node->src[0] != nullptr && node->src[0]->op == GGML_OP_VIEW && node->src[0]->src[0] != nullptr &&
node->src[0]->src[0]->op == GGML_OP_CONCAT && node->src[1] != nullptr && node->src[1]->op == GGML_OP_VIEW &&
node->src[1]->view_src == node->view_src;
}
// MoE expert aggregation (build_moe_ffn in llama-graph.cpp): each expert plane is
// `ggml_view_2d(experts, n_embd, n_tokens, experts->nb[2], i*experts->nb[1])` and the planes
// are summed with a chain of ADDs: moe_out = ((view_0 + view_1) + view_2) + ... + view_{n-1}.
// Detected structurally by walking the ADD chain and checking every leaf is a same-shape,
// same-stride VIEW of one common base tensor, indexed by a distinct expert-plane offset, and
// that the chain covers every plane of that base (leaf count == base->ne[1]). Only the
// outermost ADD of the chain satisfies this (inner ADDs see fewer leaves than base->ne[1]).
bool is_moe_expert_sum_add(const ggml_tensor * node) {
std::vector<const ggml_tensor *> leaves;
const ggml_tensor * cur = node;
while (cur->op == GGML_OP_ADD) {
if (cur->src[0] == nullptr || cur->src[1] == nullptr) {
return false;
}
leaves.push_back(cur->src[1]);
cur = cur->src[0];
}
leaves.push_back(cur);
const ggml_tensor * base = nullptr;
std::set<int64_t> plane_indices;
for (const ggml_tensor * leaf : leaves) {
if (leaf->op != GGML_OP_VIEW || leaf->src[0] == nullptr) {
return false;
}
const ggml_tensor * leaf_base = leaf->src[0];
if (base == nullptr) {
base = leaf_base;
} else if (leaf_base != base) {
return false;
}
if (leaf->ne[0] != base->ne[0] || leaf->ne[1] != base->ne[2] || leaf->ne[2] != 1 || leaf->ne[3] != 1 ||
leaf->nb[1] != base->nb[2]) {
return false;
}
if (base->nb[1] == 0 || leaf->view_offs % base->nb[1] != 0) {
return false;
}
int64_t plane = static_cast<int64_t>(leaf->view_offs / base->nb[1]);
if (plane < 0 || plane >= base->ne[1] || !plane_indices.insert(plane).second) {
return false;
}
}
return base != nullptr && base->ne[1] > 1 && plane_indices.size() == static_cast<size_t>(base->ne[1]);
}
} // namespace
static std::string get_tensor_ov_name(const ggml_cgraph * cgraph, const ggml_tensor * tensor) {
if (tensor == nullptr) {
return "";
}
const size_t hash_pos = ggml_hash_find(&cgraph->visited_hash_set, tensor);
if (((tensor->flags & GGML_TENSOR_FLAG_COMPUTE) || GgmlOvDecoder::is_kvcache(tensor, nullptr)) &&
hash_pos != GGML_HASHSET_FULL && ggml_bitset_get(cgraph->visited_hash_set.used, hash_pos)) {
return std::string(tensor->name) + "#" + std::to_string(hash_pos);
}
return tensor->name;
}
static std::string get_tensor_graph_input_ov_name(const GgmlOvDecoder * decoder,
const ggml_cgraph * cgraph,
const ggml_tensor * tensor,
const ggml_tensor * op) {
if (GgmlOvDecoder::is_inp_pos(tensor, op)) {
return "inp_pos";
}
if (GgmlOvDecoder::is_inp_emb(tensor, op)) {
return "embd";
}
if (decoder->is_stateful() && GgmlOvDecoder::is_inp_mask(tensor, op)) {
return std::string(tensor->name).find("swa") == std::string::npos ? "self_kq_mask" : "self_kq_mask_swa";
}
return get_tensor_ov_name(cgraph, tensor);
}
void GgmlOvDecoder::set_input_output() {
for (int node_n = 0; node_n < m_cgraph->n_nodes; node_n++) {
auto node = m_cgraph->nodes[node_n];
auto * node = m_cgraph->nodes[node_n];
NodeInfo current_node_info;
auto node_name = std::string(node->name);
auto node_output_name = node_name;
auto * node_output = node;
if (node->op == GGML_OP_SET_ROWS) {
// SET_ROWS updates the tensor in place. For later ov op that uses the
// the view_src of SET_ROWS, we need to make sure they get the updated tensor
// by putting the view_src name in the tensor_map in
// <openvino>/src/frontends/ggml/src/translate_session.cpp
node_output_name = std::string(node->view_src->name);
node_output = node->view_src;
}
auto node_name = get_tensor_ov_name(m_cgraph, node);
current_node_info.node = node;
current_node_info.node_name = node_name;
current_node_info.node_output = node_output;
current_node_info.node_output_name = node_output_name;
current_node_info.node_op_case = 0;
current_node_info.data_addr = node->data;
@@ -127,9 +221,9 @@ void GgmlOvDecoder::set_input_output() {
if (src == nullptr) {
continue;
}
auto src_name = std::string(src->name);
auto src_name = get_tensor_ov_name(m_cgraph, src);
if (src->flags & GGML_TENSOR_FLAG_INPUT) {
src_name = get_graph_input_ov_name(src, node);
src_name = get_tensor_graph_input_ov_name(this, m_cgraph, src, node);
}
current_node_info.node_inputs[src_name] = src;
current_node_info.node_inputs_names.push_back(src_name);
@@ -140,9 +234,9 @@ void GgmlOvDecoder::set_input_output() {
auto current = src;
while (current != nullptr) {
auto current_name = std::string(current->name);
auto current_name = get_tensor_ov_name(m_cgraph, current);
if (current->flags & GGML_TENSOR_FLAG_INPUT) {
current_name = get_graph_input_ov_name(current, node);
current_name = get_tensor_graph_input_ov_name(this, m_cgraph, current, node);
}
view_chain.emplace_back(current_name, current);
// If current src is also a VIEW, continue traversing
@@ -166,6 +260,7 @@ int GgmlOvDecoder::compute_op_case(const ggml_tensor * node) const {
int op_case = 0;
switch (node->op) {
case GGML_OP_RESHAPE: {
auto name = std::string(node->name);
auto * src = node->src[0];
if (src->op == GGML_OP_RESHAPE && src->src[0]->ne[0] == node->ne[0] && src->src[0]->ne[1] == node->ne[1]) {
op_case = 4;
@@ -178,11 +273,12 @@ int GgmlOvDecoder::compute_op_case(const ggml_tensor * node) const {
}
} else if (src->ne[0] * src->ne[1] * src->ne[2] == node->ne[1]) {
op_case = 3;
} else if (src->ne[1] * src->ne[2] == node->ne[1]) {
op_case = 6;
}
if (op_case == 0 && ggml_nelements(node) == ggml_nelements(src)) {
} else if (name.find("linear_attn_qkv_mixed") == 0 || name.find("alpha") == 0) {
op_case = 6;
} else if (name.find("linear_attn_out") == 0) {
op_case = 7;
} else if (name.find("state_predelta") == 0) {
op_case = 8;
}
break;
}
@@ -232,7 +328,14 @@ int GgmlOvDecoder::compute_op_case(const ggml_tensor * node) const {
}
case GGML_OP_GET_ROWS: {
if (node->src[1]->op == GGML_OP_VIEW) {
op_case = 2;
// GET_ROWS gathering recurrent state cache rows via the inp->s_copy index list:
// src[0] is a reshape of cache_r/cache_s, src[1] is a view of the s_copy leaf.
// op_case 3: main view (active sequences, view offset 0)
// op_case 4: extra view (defrag remainder, nonzero view offset)
if (node->src[0]->op == GGML_OP_RESHAPE && node->src[0]->src[0] != nullptr &&
is_kvcache(node->src[0]->src[0], nullptr)) {
op_case = node->src[1]->view_offs == 0 ? 1 : 2;
}
}
break;
}
@@ -260,7 +363,7 @@ int GgmlOvDecoder::compute_op_case(const ggml_tensor * node) const {
// throw std::runtime_error("Unsupported VIEW case");
}
op_case = 0;
if (m_model_is_splitted && m_model_inputs.find(std::string(src->name)) != m_model_inputs.end()) {
if (m_model_is_splitted && m_model_inputs.find(get_tensor_ov_name(m_cgraph, src)) != m_model_inputs.end()) {
op_case = 0;
}
}
@@ -295,6 +398,56 @@ int GgmlOvDecoder::compute_op_case(const ggml_tensor * node) const {
}
break;
}
case GGML_OP_RMS_NORM: {
if (node->src[0]->op == GGML_OP_VIEW) {
if (is_same_shape(node->src[0]->src[0], node->src[0])) {
op_case = 1;
} else if (node->src[0]->src[0]->op == GGML_OP_GATED_DELTA_NET) {
op_case = 2;
}
}
break;
}
case GGML_OP_CPY: {
if (node->src[0]->op == GGML_OP_VIEW) {
if (node->src[0]->src[0]->op == GGML_OP_GATED_DELTA_NET) {
op_case = 1;
} else if (is_conv_state_writeback(node)) {
op_case = 2;
break;
} else if (is_conv_states_all_tensor(node->view_src) && node->src[1] != nullptr &&
node->src[1]->op == GGML_OP_VIEW && node->src[1]->view_src == node->view_src) {
op_case = 4;
break;
}
} else if (node->src[0]->op == GGML_OP_GET_ROWS && node->src[1] != nullptr &&
node->src[1]->op == GGML_OP_VIEW && node->src[1]->view_src != nullptr &&
is_kvcache(node->src[1]->view_src, nullptr)) {
// s_copy defrag remainder writeback: gathered extra state rows copied back into the cache
op_case = 3;
}
break;
}
case GGML_OP_ADD: {
if (is_moe_expert_sum_add(node)) {
// Outermost ADD of a MoE expert-plane sum chain: translated as a single
// ReduceSum over the base tensor instead of N-1 chained Adds over N Slices.
op_case = 1;
}
break;
}
case GGML_OP_SCALE: {
if (node->view_src && node->buffer->usage == GGML_BACKEND_BUFFER_USAGE_ANY) {
op_case = 1;
}
break;
}
case GGML_OP_L2_NORM: {
if (std::string(node->name).find("predelta") != std::string::npos) {
op_case = 1;
}
break;
}
default:
break;
}
@@ -476,6 +629,43 @@ std::pair<ModelParams, ComputeParams> GgmlOvDecoder::compute_llm_params(ggml_cgr
model_params.mixed_rope_params = true;
}
}
if (node->op == GGML_OP_GATED_DELTA_NET) {
model_params.state_size = node->src[0]->ne[0];
}
if (node->op == GGML_OP_SCALE && node->view_src != nullptr && is_kvcache(node->view_src, nullptr)) {
compute_params.cache_rs_reset_len = ggml_nelements(node) / node->view_src->ne[0];
compute_params.cache_rs_reset_idx = node->src[0]->view_offs / node->view_src->ne[0];
}
// Capture the destination slot block of every recurrent state cache writeback, plus the
// conv_input window the conv state writeback copies. The active sequences occupy a
// contiguous slot block [begin, begin + n_seqs) of the cache; the block and the window move
// with the batch, so they are fed to the cached model as runtime inputs.
if (node->op == GGML_OP_CPY && node->view_src != nullptr && is_kvcache(node->view_src, nullptr) &&
node->src[1] != nullptr && node->src[1]->op == GGML_OP_VIEW && node->src[1]->view_src == node->view_src) {
const bool is_conv = is_conv_state_writeback(node);
const bool is_gdn = node->src[0]->op == GGML_OP_VIEW && node->src[0]->src[0] != nullptr &&
node->src[0]->src[0]->op == GGML_OP_GATED_DELTA_NET;
const bool is_extra = node->src[0]->op == GGML_OP_GET_ROWS;
const ggml_tensor * dest_view = node->src[1];
const ggml_tensor * cache = node->view_src;
const size_t row_bytes = cache->ne[0] * ggml_type_size(cache->type);
if (row_bytes > 0 && (is_conv || is_gdn || is_extra)) {
ComputeParams::RsWriteback writeback;
writeback.slot_begin = (int) (dest_view->view_offs / row_bytes);
if (is_conv) {
// conv_input column the copied window starts at
writeback.src_begin = (int) (node->src[0]->view_offs / node->src[0]->view_src->nb[0]);
} else if (is_gdn) {
// first row of the state part of the gated-delta-net output
writeback.src_begin = (int) (node->src[0]->view_offs / node->src[0]->view_src->nb[1]);
}
compute_params.rs_writebacks[get_tensor_ov_name(cgraph, node)] = writeback;
}
if (is_conv || is_gdn) {
compute_params.s_copy_active_slot_len = (int) dest_view->ne[1];
}
}
}
auto * output_tensor = cgraph->nodes[cgraph->n_nodes - 1];
compute_params.output_len = output_tensor->ne[1];
@@ -505,6 +695,10 @@ ov::PartialShape GgmlOvDecoder::get_graph_input_shape(const ggml_tensor * op,
if (is_inp_tok(input, op) || is_inp_pos(input, op)) {
// tokens or positions
int len = m_is_static ? (m_is_prefill ? m_prefill_chunk_size : 1) : -1;
if (m_is_static && is_inp_pos(input, op)) {
// IMROPE stacks n_planes (t/h/w/e) position planes back to back
len *= get_inp_pos_n_planes(op);
}
input_shape = ov::PartialShape{1, 1, 1, len};
} else if (is_output_idx(input, op)) {
@@ -543,6 +737,9 @@ ov::PartialShape GgmlOvDecoder::get_graph_input_shape(const ggml_tensor * op,
int len = m_is_static ? (m_is_prefill ? m_prefill_chunk_size : 1) : -1;
input_shape = ov::PartialShape{1, 1, 1, len};
} else if (is_inp_s_copy(input, op) || is_s_copy_leaf(input)) {
input_shape = ov::PartialShape{1, 1, 1, -1};
} else {
input_shape = ov::PartialShape{get_shape(input)};
}
@@ -558,6 +755,35 @@ ov::PartialShape GgmlOvDecoder::get_graph_input_shape(const ggml_tensor * op,
return input_shape;
}
bool GgmlOvDecoder::is_s_copy_leaf(const ggml_tensor * tensor) const {
if (tensor == nullptr || tensor->op != GGML_OP_NONE || m_cgraph == nullptr) {
return false;
}
for (int i = 0; i < m_cgraph->n_nodes; i++) {
const ggml_tensor * node = m_cgraph->nodes[i];
if (node->op != GGML_OP_GET_ROWS || node->src[0] == nullptr || node->src[1] == nullptr) {
continue;
}
// The index list may reach the s_copy leaf through one or more VIEWs.
const ggml_tensor * idx = node->src[1];
while (idx != nullptr && idx->op == GGML_OP_VIEW) {
idx = idx->src[0];
}
if (idx != tensor) {
continue;
}
// The gathered data must be a recurrent state cache (cache_r/cache_s).
const ggml_tensor * data = node->src[0];
while (data != nullptr && (data->op == GGML_OP_VIEW || data->op == GGML_OP_RESHAPE)) {
data = data->src[0];
}
if (data != nullptr && is_kvcache(data, nullptr)) {
return true;
}
}
return false;
}
void GgmlOvDecoder::add_extra_inputs() {
// Extra inputs:
// 1. `attention_size`, used in FLASH_ATTN where the shape of the matmul's are 256 aligned,
@@ -565,21 +791,7 @@ void GgmlOvDecoder::add_extra_inputs() {
// 2. `n_seq_active` and `seq_active_start`, used in FLASH_ATTN_EXT to indicate the active sequences in the batch
auto create_1d_input = [this](const std::string & name, int64_t value) {
if (m_is_static) {
auto constant =
std::make_shared<ov::op::v0::Constant>(ov::element::i64, ov::Shape{1}, std::vector<int64_t>{value});
constant->set_friendly_name(name);
m_model_extra_inputs[name] = constant;
} else {
auto param_node = std::make_shared<ov::op::v0::Parameter>(ov::element::i64, ov::Shape{1});
param_node->set_friendly_name(name);
param_node->output(0).get_tensor().set_names({name});
m_model_extra_inputs[name] = param_node;
auto tensor = std::make_shared<ov::Tensor>(ov::element::i64, ov::Shape{1});
*tensor->data<int64_t>() = value;
m_model_extra_input_values[name] = tensor;
}
m_model_extra_inputs[name] = {ov::element::i64, ov::Shape{1}, value, !m_is_static};
};
if (m_compute_params.attention_size != -1) {
@@ -595,6 +807,20 @@ void GgmlOvDecoder::add_extra_inputs() {
create_1d_input("token_len_per_seq", m_compute_params.token_len_per_seq);
}
// create_1d_input("token_len", m_compute_params.token_len_per_seq * m_compute_params.n_seq_active);
if (m_compute_params.cache_rs_reset_idx != -1) {
create_1d_input("cache_rs_reset_idx", m_compute_params.cache_rs_reset_idx);
create_1d_input("cache_rs_reset_len", m_compute_params.cache_rs_reset_len);
}
if (m_compute_params.s_copy_active_slot_len != -1) {
create_1d_input("s_copy_active_slot_len", m_compute_params.s_copy_active_slot_len);
}
for (const auto & [node_name, writeback] : m_compute_params.rs_writebacks) {
create_1d_input("rs_slot_begin_" + node_name, writeback.slot_begin);
create_1d_input("rs_src_begin_" + node_name, writeback.src_begin);
}
}
bool GgmlOvDecoder::node_is_used_as_src(const int node_idx) {
@@ -617,14 +843,11 @@ void GgmlOvDecoder::compute_model_inputs() {
ggml_tensor * node = m_cgraph->nodes[i];
// the node op is NONE means this node maybe as input of later nodes, we should add it to model inputs for this node.
if (node->op == GGML_OP_NONE && node_is_used_as_src(i)) {
std::string node_name(node->name);
std::string node_name = get_tensor_ov_name(m_cgraph, node);
if (m_model_weights.find(node_name) == m_model_weights.end()) {
m_inputs[node_name] = node;
auto param_node = std::make_shared<ov::op::v0::Parameter>(
get_ov_type(node), get_graph_input_shape(node, nullptr, m_node_dynamic_dims[node]));
param_node->set_friendly_name(node_name);
param_node->output(0).get_tensor().set_names({node_name});
m_model_inputs[node_name] = param_node;
m_model_inputs[node_name] = {get_ov_type(node),
get_graph_input_shape(node, nullptr, m_node_dynamic_dims[node])};
}
continue;
}
@@ -633,9 +856,9 @@ void GgmlOvDecoder::compute_model_inputs() {
if (src == nullptr) {
continue;
}
std::string src_name = std::string(src->name);
std::string src_name = get_tensor_ov_name(m_cgraph, src);
if (src->flags & GGML_TENSOR_FLAG_INPUT) {
src_name = get_graph_input_ov_name(src, node);
src_name = get_tensor_graph_input_ov_name(this, m_cgraph, src, node);
}
if (m_model_weights.find(src_name) != m_model_weights.end()) {
continue;
@@ -668,14 +891,11 @@ void GgmlOvDecoder::compute_model_inputs() {
// Resolve nested VIEW nodes by following src[0] until the first non-VIEW tensor.
while (src->op == GGML_OP_VIEW && src->src[0] != nullptr) {
src = src->src[0];
src_name = std::string(src->name);
src_name = get_tensor_ov_name(m_cgraph, src);
}
m_inputs[src_name] = src;
ov::PartialShape param_shape = get_graph_input_shape(node, src, m_node_dynamic_dims[src]);
auto param_node = std::make_shared<ov::op::v0::Parameter>(get_ov_type(src), param_shape);
param_node->set_friendly_name(src_name);
param_node->output(0).get_tensor().set_names({src_name});
m_model_inputs[src_name] = param_node;
m_model_inputs[src_name] = {get_ov_type(src),
get_graph_input_shape(node, src, m_node_dynamic_dims[src])};
}
}
}
@@ -691,8 +911,8 @@ void GgmlOvDecoder::compute_model_outputs() {
}
auto cur_node_use_count = m_cgraph->use_counts[ggml_hash_find(&m_cgraph->visited_hash_set, cur_node)];
if (cur_node_use_count == 0) {
// The output of SET_ROWS is the view_src tensor, which is updated in place. We should use the view_src name as the output name to make sure it can be correctly matched with the later ops that use the view_src.
if (cur_node != nullptr && cur_node->op == GGML_OP_SET_ROWS) {
// The output of in-place ops is the view_src tensor, which is updated in place. We should use the view_src name as the output name to make sure it can be correctly matched with the later ops that use the view_src.
if (cur_node != nullptr && ::is_inplace_op(cur_node) && ggml_nbytes(cur_node) > 0) {
cur_node = cur_node->view_src;
}
} else {
@@ -710,9 +930,9 @@ void GgmlOvDecoder::compute_model_outputs() {
}
}
if (cur_node != nullptr) {
std::string node_output_name(cur_node->name);
m_model_outputs[node_output_name] = cur_node;
m_model_output_names.push_back(node_output_name);
std::string cur_node_name = get_tensor_ov_name(m_cgraph, cur_node);
m_model_outputs[cur_node_name] = cur_node;
m_model_output_names.insert(cur_node_name);
}
}
}
@@ -740,7 +960,7 @@ const ggml_tensor * GgmlOvDecoder::get_tensor_from_name(const std::string & name
if (src == nullptr) {
break;
}
if (std::string(src->name) == name) {
if (get_tensor_ov_name(m_cgraph, src) == name) {
return src;
}
}
@@ -756,6 +976,16 @@ std::map<std::string, std::string> GgmlOvDecoder::get_kv_param_res_names() const
return kv_param_res_names;
}
// MUL_MAT_ID's src[0] is the [k, m, n_expert] expert-weight tensor. It is always a constant per-expert
// weight table -- never a computed activation -- regardless of whether the backend happened to mark its
// buffer as GGML_BACKEND_BUFFER_USAGE_WEIGHTS (test-backend-ops, for example, never sets that usage
// flag, unlike real inference). Without this, non-quantized (F16/F32/BF16) expert weights would fall
// through the check below as "not a weight", get decoded as a Parameter/activation instead of a
// Constant, and crash GatherMatmul's "only constant weights are supported" check.
static bool is_mul_mat_id_expert_weight(const ggml_tensor * node, int src_index) {
return node->op == GGML_OP_MUL_MAT_ID && src_index == 0;
}
std::map<std::string, std::shared_ptr<ov::Node>> GgmlOvDecoder::create_weight_nodes(ggml_cgraph * cgraph, bool naive) {
std::map<std::string, std::shared_ptr<ov::Node>> model_weights;
auto * nodes = cgraph->nodes;
@@ -768,13 +998,14 @@ std::map<std::string, std::shared_ptr<ov::Node>> GgmlOvDecoder::create_weight_no
continue;
}
std::string src_name(src->name);
std::string src_name = get_tensor_ov_name(cgraph, src);
if (is_rope_freqs_weight(src, node)) {
src_name = "rope_freqs.weight";
}
if (!src->view_src) {
ggml_backend_buffer * buffer = src->buffer;
if (buffer->usage == GGML_BACKEND_BUFFER_USAGE_WEIGHTS || ggml_is_quantized(src->type)) {
if (buffer->usage == GGML_BACKEND_BUFFER_USAGE_WEIGHTS || ggml_is_quantized(src->type) ||
is_mul_mat_id_expert_weight(node, i)) {
if (model_weights.find(src_name) == model_weights.end()) {
auto weight_node = create_weight_node(src, naive);
weight_node->set_friendly_name(src_name);
@@ -787,6 +1018,42 @@ std::map<std::string, std::shared_ptr<ov::Node>> GgmlOvDecoder::create_weight_no
return model_weights;
}
// Process-lifetime cache for weight nodes built from NON-OpenVINO buffers (e.g. the
// token_embd.weight copy that lives in a CPU/mmap buffer and feeds GET_ROWS). Such
// tensors have no OV buffer context to own a cached extra, so without this they are
// re-extracted/re-requantized on every (re)compile — for token_embd that is a ~1-2 GB
// F32 dequant each time. Keyed by tensor->data, which is stable for the process and
// uniquely identifies the immutable weight bytes. OV-buffer weights keep using the
// per-tensor extra cache and never reach here.
static std::mutex g_nonov_weight_cache_mutex;
static std::unordered_map<const void *, std::shared_ptr<ov::Node>> g_nonov_weight_cache;
std::set<std::string> GgmlOvDecoder::collect_weight_names(ggml_cgraph * cgraph) {
// Mirrors the name-selection logic of create_weight_nodes() but builds no nodes,
// so topology checks don't trigger weight extraction/requantization.
std::set<std::string> names;
for (int node_i = 0; node_i < cgraph->n_nodes; node_i++) {
auto * node = cgraph->nodes[node_i];
for (int i = 0; i < GGML_MAX_SRC; i++) {
auto * src = node->src[i];
if (src == nullptr) {
continue;
}
std::string src_name(src->name);
if (is_rope_freqs_weight(src, node)) {
src_name = "rope_freqs.weight";
}
if (!src->view_src) {
ggml_backend_buffer * buffer = src->buffer;
if (buffer->usage == GGML_BACKEND_BUFFER_USAGE_WEIGHTS || ggml_is_quantized(src->type)) {
names.insert(src_name);
}
}
}
}
return names;
}
std::shared_ptr<ov::Node> GgmlOvDecoder::create_weight_node(ggml_tensor * tensor, bool naive) {
const bool is_ov_buffer = ggml_backend_buffer_is_openvino(tensor->buffer);
@@ -826,6 +1093,21 @@ std::shared_ptr<ov::Node> GgmlOvDecoder::create_weight_node(ggml_tensor * tensor
return weight_node;
}
// Non-OV-buffer weights (CPU/mmap, e.g. the GET_ROWS token_embd copy) have no buffer
// context to cache an extra in, so memoize them here keyed by their (stable) data
// pointer to avoid re-extracting on every recompile. Opt-in via
// GGML_OPENVINO_REDUCE_COMPILE_MEM or GGML_OPENVINO_MEMORY_OPTIMIZE. Skip
// for `naive` (test/naive path) since use_bias changes the produced node.
const bool cacheable_nonov = ggml_openvino_reduce_compile_mem_enabled() && !is_ov_buffer &&
!naive && tensor->data != nullptr;
if (cacheable_nonov) {
std::lock_guard<std::mutex> lock(g_nonov_weight_cache_mutex);
auto it = g_nonov_weight_cache.find(tensor->data);
if (it != g_nonov_weight_cache.end()) {
return it->second;
}
}
// There are three cases where we need to create a new weight node:
// 1. weights are in openvino_host_buffer. Weight loading to host buffer will not trigger backend_buffer_set_tensor
// 2. weights are in cpu/cpu_mapped buffer. On token_embd.weight goes to case 1 or 2, depending on whether mmap or direct_io is used
@@ -834,7 +1116,7 @@ std::shared_ptr<ov::Node> GgmlOvDecoder::create_weight_node(ggml_tensor * tensor
// GGML_LOG_DEBUG("%s: creating new weight node for %s\n", __func__, tensor->name);
static const std::set<ggml_type> weight_types = {GGML_TYPE_F32, GGML_TYPE_F16, GGML_TYPE_BF16, GGML_TYPE_Q8_0,
GGML_TYPE_Q4_0, GGML_TYPE_Q4_1, GGML_TYPE_Q5_1, GGML_TYPE_Q4_K,
GGML_TYPE_Q5_K, GGML_TYPE_Q6_K};
GGML_TYPE_Q5_K, GGML_TYPE_Q6_K, GGML_TYPE_MXFP4};
if (weight_types.find(tensor->type) == weight_types.end()) {
throw std::runtime_error("Unexpected weight tensor type: " + std::string(tensor->name) + " with type " +
ggml_type_name(tensor->type));
@@ -863,6 +1145,12 @@ std::shared_ptr<ov::Node> GgmlOvDecoder::create_weight_node(ggml_tensor * tensor
ov_weight.weight_node->set_friendly_name(tensor->name);
if (!is_ov_buffer) {
if (cacheable_nonov) {
std::lock_guard<std::mutex> lock(g_nonov_weight_cache_mutex);
// Another thread may have inserted concurrently; keep the first.
auto [it, inserted] = g_nonov_weight_cache.emplace(tensor->data, ov_weight.weight_node);
return it->second;
}
return ov_weight.weight_node;
}
@@ -1178,7 +1466,7 @@ std::string GgmlOvDecoder::get_view_input_name(int node_idx, const std::string &
auto it = m_node_info_list[node_idx].node_inputs_views.find(name);
if (it != m_node_info_list[node_idx].node_inputs_views.end()) {
if (view_index < it->second.size()) {
return it->second[view_index].second->name;
return it->second[view_index].first;
}
}
return "";
@@ -1190,7 +1478,7 @@ std::string GgmlOvDecoder::get_view_input_src_name(int node_idx, const std::stri
if (view_index < it->second.size()) {
auto * view_tensor = it->second[view_index].second;
if (view_tensor && view_tensor->src[0]) {
return view_tensor->src[0]->name;
return get_tensor_ov_name(m_cgraph, view_tensor->src[0]);
}
}
}
@@ -1214,7 +1502,7 @@ std::vector<std::string> GgmlOvDecoder::get_input_names(int node_idx) const {
}
ov::PartialShape GgmlOvDecoder::get_output_shape(int node_idx) const {
auto * ggml_tensor = m_node_info_list[node_idx].node_output;
auto * ggml_tensor = m_node_info_list[node_idx].node;
return ov::PartialShape(get_shape(ggml_tensor));
}
@@ -1228,7 +1516,28 @@ std::vector<size_t> GgmlOvDecoder::get_output_stride(int node_idx) const {
}
std::vector<std::string> GgmlOvDecoder::get_output_names(int node_idx) const {
return {m_node_info_list[node_idx].node_output_name};
return {m_node_info_list[node_idx].node_name};
}
std::string GgmlOvDecoder::get_inplace_op_src(int node_idx) const {
auto * node = m_node_info_list[node_idx].node;
if (!::is_inplace_op(node) || node->view_src == nullptr || ggml_nbytes(node) == 0) {
return "";
}
const int op_case = m_node_info_list[node_idx].node_op_case;
if (node->op == GGML_OP_CPY && (op_case == 1 || op_case == 2 || op_case == 3) &&
m_compute_params.s_copy_active_slot_len == -1) {
return "";
}
return get_tensor_ov_name(m_cgraph, node->view_src);
}
bool GgmlOvDecoder::is_view_like_alias_of(int node_idx, const std::string & view_src_name) const {
auto * node = m_node_info_list[node_idx].node;
if (node->view_src == nullptr || get_tensor_ov_name(m_cgraph, node->view_src) != view_src_name) {
return false;
}
return node->op == GGML_OP_RESHAPE || node->op == GGML_OP_VIEW;
}
const std::string & GgmlOvDecoder::get_op_name() const {
@@ -1404,14 +1713,18 @@ void GgmlOvDecoder::compute_node_dynamic_dims() {
}
if (m_node_dynamic_dims[node] != -1 && dynamic_dim_value != node->ne[m_node_dynamic_dims[node]]) {
m_node_dynamic_dims[node] = -1;
// std::cout << "Warning: Dynamic dim value mismatch for node: " << node->name
// << " and its src[0]: " << node->src[0]->name << std::endl;
GGML_LOG_WARN("ggml-openvino: dynamic dim value mismatch for VIEW node '%s', src[0]: '%s'\n",
node->name, node->src[0]->name);
}
}
break;
}
case GGML_OP_TRANSPOSE:
case GGML_OP_RESHAPE: {
if (is_same_shape(node->src[0], node)) {
m_node_dynamic_dims[node] = m_node_dynamic_dims[node->src[0]];
break;
}
// RESHAPE requires src[0] to be contiguous, so both src and result
// have standard compact strides: nb[i] = type_size * prod(ne[0..i-1]).
// Match src->nb[dynamic_dim] against result->nb[i] to find the output
@@ -1429,7 +1742,7 @@ void GgmlOvDecoder::compute_node_dynamic_dims() {
}
}
if (m_node_dynamic_dims[node] == -1) {
// std::cout << "Cannot determine dynamic dim for RESHAPE node: " << node->name << std::endl;
GGML_LOG_WARN("ggml-openvino: cannot determine dynamic dim for RESHAPE node '%s'\n", node->name);
}
}
break;
@@ -1480,15 +1793,29 @@ void GgmlOvDecoder::compute_node_dynamic_dims() {
}
if (matched_dim_count != 1) {
m_node_dynamic_dims[node] = -1;
// std::cout << "Warning: Cannot determine dynamic dim for CONT node: " << node->name
// << " and its src[0]: " << node->src[0]->name << std::endl;
GGML_LOG_WARN("ggml-openvino: cannot determine dynamic dim for CONT node '%s', src[0]: '%s'\n",
node->name, node->src[0]->name);
}
}
}
break;
case GGML_OP_CONCAT:
for (int i = 0; i < GGML_MAX_DIMS; i++) {
if (node->src[0]->ne[i] != node->ne[i]) {
m_node_dynamic_dims[node] = i;
break;
}
}
break;
case GGML_OP_SSM_CONV:
case GGML_OP_GATED_DELTA_NET:
m_node_dynamic_dims[node] = 1;
break;
case GGML_OP_RMS_NORM:
case GGML_OP_L2_NORM:
case GGML_OP_NORM:
case GGML_OP_ADD:
case GGML_OP_SUB:
case GGML_OP_GLU:
case GGML_OP_ROPE:
case GGML_OP_SCALE:
@@ -1496,9 +1823,31 @@ void GgmlOvDecoder::compute_node_dynamic_dims() {
case GGML_OP_ARGSORT:
case GGML_OP_ADD_ID:
case GGML_OP_UNARY:
case GGML_OP_CUMSUM:
case GGML_OP_FILL:
case GGML_OP_SET:
case GGML_OP_DIAG:
case GGML_OP_TRI:
case GGML_OP_REPEAT:
// Shape-preserving elementwise ops: the dynamic dim is unchanged from src[0].
// DIV/CLAMP are used in the MoE routing-weight normalization
// (sum_rows -> clamp -> div). If they are left untracked here the dynamic
// (token) dim is lost there, the captured prefill token count gets baked into
// the downstream reshapes, and every decoder layer after layer 0 turns static
// (which then triggers the GPU in-place-concat KV-cache corruption).
case GGML_OP_DIV:
case GGML_OP_CLAMP:
case GGML_OP_PAD:
m_node_dynamic_dims[node] = m_node_dynamic_dims[node->src[0]];
break;
case GGML_OP_SUM_ROWS:
// SUM_ROWS reduces ggml axis 0 to size 1 and preserves all other axes, so the
// dynamic dim is preserved unless it was axis 0 (then it is summed away).
m_node_dynamic_dims[node] =
(m_node_dynamic_dims[node->src[0]] == 0) ? -1 : m_node_dynamic_dims[node->src[0]];
break;
case GGML_OP_MUL_MAT_ID:
case GGML_OP_SOLVE_TRI:
m_node_dynamic_dims[node] = m_node_dynamic_dims[node->src[1]];
break;
case GGML_OP_CPY:
@@ -1534,7 +1883,8 @@ void GgmlOvDecoder::compute_node_dynamic_dims() {
break;
}
default:
// std::cout << "Doesn't handle node name: " << node->name << " op: " << ggml_op_name(node->op) << std::endl;
GGML_LOG_DEBUG("ggml-openvino: compute_node_dynamic_dims: unhandled op %s for node '%s'\n",
ggml_op_name(node->op), node->name);
break;
}
};