Lumpiasty/llama.cpp
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

openvino: OV 2026.2, context-shift, Q5_1 support, gemma4 dense/embedding, and -fa off (#24503)

Browse Source 
* Add interface is_model_splitted() to check the c-graph is splited or not

* Infer and propagate dynamic-dimension indices for all tensors in the GGML graph in api compute_model_outputs()

* Only do this for fallback sub graph

* Move dynamic dims compute in graph missmatch

* ggml-openvino: fix tensor data handling for PERMUTE/VIEW ops in split models

* ggml-openvino:add comments

* ggml-openvino: override VIEW op_case to 0 for split model inputs

* openvino backend: Handle unsupported VIEW shape-mismatch in OpenVINO backend

* Enable additional mul_mat tests and add tensor data saving function (#81)

* ggml-openvino: fix CONT/TRANSPOSE mapping and improve dynamic-dimension handling

* OpenVINO: add NORM/TANH support and rework SOFT_MAX translation

* ggml-openvino: extend VIEW handling

* Enable -fa off (#118)

* Enable --context-shift

* Fix llm param compute error for normal softmax not the softmax in attention

* OpenVINO backend: fix error for attention size compute in llm param

* use tensor->extra in infer_request i/o

* OpenVINO backend: refacter the compute_llm_params() func add get_attention_pattern_case to easy extand

* OpenVINO backend: clean unused code

* 1to1 match op update (#146)

* added translate_1to1_match_1_input function and updated gelu and tanh translations

* Remove unused translation function calls

---------

Co-authored-by: Mustafa Cavus <mustafacavus@intel.com>

* initial gemma4 support

* removed hardcoded names for kv cache slicing

* OpenVINO backend: Add new attention pattern for llm parameters compute

* flash attn Q shape static conversion

* Remove slice in permute translation when n_seq is 1

* return optional in extract_layer_from_name

* OpenVINO backend: refactor VIEW related operation (#148)

* OpenVINO backend: refactor VIEW related operation

* Enable VIEW handling in following ops

* OpenVINO backend does not support GGML_OP_NORM & GGML_OP_L2_NORM with VIEW input accuracy issue from OpenVINO

* OpenVINO backend: Add ops l2_norm & pad

* OpenVINO backend does not support CPY with non-contiguous data or mismatched types

* add op SSM_CONV GATED_DELTA_NET

* OpenVINO backend: fix error for bf16 in OV gpu plugin

* reverted static Q input shape for attention layer

* OpenVINO backend: remove hardcode name inp_tokens, which ignore some leaf case

* Disable remote tensor due to bug in ov gpu

* Disable n_token > 1 GATED_DELTA_NET on gpu

* OpenVINO backend: fix the view op dynamic handling issue in gemma4 & enable view + get_row

* OpenVINO backend: clean code

* OpenVINO backend: enable view + norm/rms_norm

* OpenVINO backend: concat op

* OpenVINO backend: argsort op

* OpenVINO backend: enable unary + view & GGML_UNARY_OP_SOFTPLUS

* Fix issue for test-backend-ops in TOPK_MOE, which compare VIEW ops result, VIEW node in OpenVINO no need compare, the whole graph result is correct

* OpenVINO backend: enable sum_rows

* OpenVINO backend: enable clamp

* OpenVINO backend: enable DIV

* OpenVINO backend: enable GGML_OP_MUL_MAT_ID

* OpenVINO backend: disable MUL_MAT_ID_FUSION case with large mem needed

* OpenVINO backend: Disable GGML_OP_ARGSORT, cause test_backend-ops failed

* OpenVINO backend: fix issue in mul_mat_id

* OpenVINO backend: Disable DIV with broadcast on GPU

* OpenVINO backend: update DIV

* use ov internal op GatedDeltaNet

* OpenVINO backend: enable llama erch test qwen3next

* OpenVINO backend: enable RMS_NORM + VIEW & remove op_case 2 for rope

* OpenVINO backend: fix error

* suggested changes, need review

* suggested changes, need review

* OpenVINO backend: clean unused code & fix build warning

* OpenVINO backend: enable minicpm3 for arch test

* Disable GDN op (#177)

* disable gated_delta_net

* update stateful_kv_size correctly in mismatch case

* OpenVINO backend: enable arch test for qwen3vl

* OpenVINO backend: enable cohere2 for arch test

* OpenVINO backend: enable t5 for arch test

* OpenVINO backend: enable jamba for arch test

* OpenVINO backend: remove warning for tmp

* OpenVINO backend: enable kimi-linear for arch test

* Remove unused

* Fix gpt-oss accuracy issue

* OpenVINO backend: enable arctic for arch test

* OpenVINO backend: enable grok for arch test

* Gemma4 initial npu support (#179)

* Initiall gemma4 npu support

* temp. fix for gemma4 accuracy bug on npu

* Remove hardcoded names for npu-fold handling

* revert static n tokens for cont translation as it is not needed

* removed unused variable

* ggml-openvino: add GGML_OPENVINO_ENABLE_CACHE env var to control decoder cache. Add environment variable GGML_OPENVINO_ENABLE_CACHE (default: YES). When set to NO, the decoder_cache is bypassed and models are rebuilt from the cgraph on every inference call in both dynamic and static compute paths. This is useful for debugging and verifying correctness without caching interference.

* Revert "Gemma4 initial npu support (#179)"

This reverts commit 0d29a9c4a52dc2c8aa52990f1a3854cfb01768ad.

* OpenVINO backend: disable debug log print

* Update TBB discovery. Delegated to OpenVINOs own config.

* OpenVINO backend: GGML_OPENVINO_ENABLE_CACHE YES -> 1

* OpenVINO backend: fallback FLASH_ATTN_EXT in gemma3n to CPU backend

* Add raw ov infer profiling metric

* Add OV raw infer time metric to static compute path

Co-authored-by: virajwad <84867530+virajwad@users.noreply.github.com>

* Modify precision of static profiling

* update to OV 2026.2, add OV windows CI

* fix editorconfig-checks

* Initiall gemma4 npu support

* temp. fix for gemma4 accuracy bug on npu

* Remove hardcoded names for npu-fold handling

* revert static n tokens for cont translation as it is not needed

* removed unused variable

* test-llama-archs fix

* Fix gemma4 flash_attn fallback

* support im2col

* fix code style

* disable add_rope_sin_cos optimization

* stateless boradcast and rope optimizations

* Enable manual gqa attn by default for stateless gpu

* manual gqa: fixed static batch

* gemma4 llama-bench ctx update fix

* Update OV win CI

* stateful rope fusion temp. fix

* OpenVINO backend: Conslolidate supported ops

* Exclude unsupported GGML_OP_SUB cases

* Exclude unsupported TOPK_MOE cases

* OpenVINO Backend: MUL_MAT enhancements

* Update OV CI

* support f16 mask input for npu

* Make GGML_OPENVINO_* env vars usage uniform

Standardize all GGML_OPENVINO_* env flags:
positive integers >0 to enable. Unset, empty, =0, or non-numeric values to disable.
This fixes cases where text values or empty strings enabled features.

* OpenVINO backend: Enhance envvar handling

* more cleanup

* move ggml_openvino_env_flag to appropriate place

* OpenVINO backend: add REPEAT translator, Q5_1 weights, and GLU view-input fix

* ggml-openvino: fix -Werror=cast-qual in extract_q5_1_data

* Update openvino.Dockerfile

Use BuildKit cache mounts for faster Docker rebuilds.
Use apt instead of dpkg, remove unused .ddeb downloads, add DLLAMA_BUILD_TESTS=OFF.

* ggml-openvino: centralize env var access via *getenv_str/getenv_int helpers

Replace getenv and legacy flags with _str and _int helpers.Minor cleanup, doc updates.

* OpenVINO backend: Enable GGML_OP_ADD_ID

* Uptade openvino backend clamg-format

* clang-format

* Update OPENVINO.md (#211)

* OpenVINO backend: fix accuracy issue for op CONCAT with i64 precision

* Remove strict concurrency for gpu-openvino-low-perf

* Update openvino CI keynames; add ccache-clear

* Apply suggestions from code review

Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>

* Fix formatting

---------

Co-authored-by: Xuejun Zhai <Xuejun.Zhai@intel.com>
Co-authored-by: Mustafa Cavus <mustafa.cavus@intel.com>
Co-authored-by: Mustafa Cavus <mustafacavus@intel.com>
Co-authored-by: Xuejun <XuejunZhai@intel.com>
Co-authored-by: Wang Yang <yang4.wang@intel.com>
Co-authored-by: Ravi Panchumarthy <ravi.panchumarthy@intel.com>
Co-authored-by: virajwad <84867530+virajwad@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: Mostafa Faheem <mostafaaafaheem@gmail.com>
Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>
This commit is contained in:
Zijun Yu
2026-06-17 09:11:21 +03:00
committed by GitHub
co-authored by Sigbjørn Skjæret Xuejun Zhai Mustafa Cavus Mustafa Cavus Xuejun Wang Yang Ravi Panchumarthy virajwad copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Mostafa Faheem
parent 58728bdbf0
commit 890f1a27ed
62 changed files with 5280 additions and 909 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ggml/src/ggml-openvino/openvino/utils.cpp
+548 -3
View File
@@ -17,6 +17,7 @@
#include <openvino/op/reshape.hpp>
#include <openvino/op/shape_of.hpp>
#include <openvino/op/sin.hpp>
#include <openvino/op/split.hpp>
#include <openvino/op/squeeze.hpp>
#include <openvino/op/subtract.hpp>
#include <openvino/op/transpose.hpp>
@@ -123,7 +124,8 @@ std::pair<ov::Output<Node>, ov::Output<Node>> make_sin_cos(int32_t * rope_params
bool imrope,
bool stateful) {
if (stateful) {
inp_pos = std::make_shared<ov::op::v0::Squeeze>(inp_pos, ov::op::v0::Constant::create(ov::element::i64, {1}, {0}));
inp_pos =
std::make_shared<ov::op::v0::Squeeze>(inp_pos, ov::op::v0::Constant::create(ov::element::i64, {1}, {0}));
inp_pos = std::make_shared<ov::op::v0::Convert>(inp_pos, ov::element::f32);
auto pos_perm =
std::make_shared<ov::op::v0::Constant>(ov::element::i64, ov::Shape{3}, std::vector<int64_t>{2, 1, 0});
@@ -212,8 +214,9 @@ std::pair<ov::Output<Node>, ov::Output<Node>> make_sin_cos(int32_t * rope_params
}
auto one_minus_ramp = std::make_shared<ov::op::v1::Subtract>(one, ramp_mix);
theta = std::make_shared<ov::op::v1::Add>(std::make_shared<ov::op::v1::Multiply>(theta_interp, one_minus_ramp),
std::make_shared<ov::op::v1::Multiply>(theta_extrap, ramp_mix));
theta =
std::make_shared<ov::op::v1::Add>(std::make_shared<ov::op::v1::Multiply>(theta_interp, one_minus_ramp),
std::make_shared<ov::op::v1::Multiply>(theta_extrap, ramp_mix));
mscale *= (1.0f + 0.1f * std::log(1.0f / freq_scale));
}
}
@@ -252,6 +255,548 @@ ov::Output<ov::Node> process_view_input(const NodeContext & context, int input_i
return sliced;
}
ov::Output<ov::Node> process_view_input_new(const NodeContext & context, int input_index) {
auto input = context.get_input(input_index);
// Check if this input has view inputs
size_t view_input_size = context.get_view_input_size(input_index);
if (view_input_size == 0) {
// No view inputs, return the input as is
return input;
}
// If translate_view already resolved this VIEW (produced a Slice), the input
// will already have the expected shape — skip re-slicing.
auto expected_ov_shape = context.get_view_input_ov_shape(input_index, 0);
auto actual_shape = input.get_partial_shape();
if (expected_ov_shape.rank().is_static() && actual_shape.rank().is_static() &&
expected_ov_shape.rank() == actual_shape.rank()) {
bool shapes_match = true;
for (int64_t i = 0; i < expected_ov_shape.rank().get_length(); ++i) {
if (!expected_ov_shape[i].is_static() || !actual_shape[i].is_static()) {
shapes_match = false;
break;
}
if (expected_ov_shape[i] != actual_shape[i]) {
shapes_match = false;
break;
}
}
if (shapes_match) {
return input;
}
}
// In static mode, use Split instead of Slice for single-dimension reductions.
// This ensures NPUW's FOLD doesn't parametrize per-layer slice indices (which
// would introduce dynamic shapes). A shared Split node sits outside the repeated
// subgraph boundary; each layer receives one of its output ports.
if (context.is_static() && view_input_size == 1) {
auto view_stride_v = context.get_view_input_stride(input_index, 0);
auto view_src_stride_v = context.get_view_input_src_stride(input_index, 0);
auto view_ggml_shape = context.get_view_input_ggml_shape(input_index, 0);
auto view_src_ggml_shape = context.get_view_input_src_ggml_shape(input_index, 0);
auto view_offset = context.get_view_input_offset(input_index, 0);
auto view_src_offset = context.get_view_input_src_offset(input_index, 0);
size_t ndims = view_ggml_shape.size();
std::vector<int> diff_dims;
if (view_src_ggml_shape.size() == ndims) {
for (size_t i = 0; i < ndims; ++i) {
if (view_ggml_shape[i] != view_src_ggml_shape[i]) {
diff_dims.push_back(static_cast<int>(i));
}
}
}
if (diff_dims.size() == 1) {
int split_dim = diff_dims[0];
int64_t num_splits = static_cast<int64_t>(view_src_ggml_shape[split_dim]);
int64_t chunk_size = static_cast<int64_t>(view_ggml_shape[split_dim]);
// Only apply when slicing exactly 1 element from a multi-element dimension
if (chunk_size == 1 && num_splits > 1) {
// Check suffix strides match (dimensions after split_dim)
bool suffix_ok = view_stride_v.size() == view_src_stride_v.size();
if (suffix_ok) {
for (size_t i = static_cast<size_t>(split_dim) + 1; i < ndims; ++i) {
if (view_stride_v[i] != view_src_stride_v[i]) {
suffix_ok = false;
break;
}
}
}
if (suffix_ok && view_src_stride_v[split_dim] > 0) {
size_t relative_offset = view_offset >= view_src_offset ? view_offset - view_src_offset : 0;
int64_t split_index = static_cast<int64_t>(relative_offset / view_src_stride_v[split_dim]);
if (split_index >= 0 && split_index < num_splits) {
auto src_node = input.get_node_shared_ptr();
std::string rt_key = "split_dim_" + std::to_string(split_dim);
auto & rt_info = src_node->get_rt_info();
if (rt_info.find(rt_key) == rt_info.end()) {
auto axis_const =
ov::op::v0::Constant::create(ov::element::i64, {}, {static_cast<int64_t>(split_dim)});
auto split_node =
std::make_shared<ov::op::v1::Split>(input, axis_const, static_cast<size_t>(num_splits));
split_node->set_friendly_name(src_node->get_friendly_name() + "_split");
rt_info[rt_key] = split_node;
}
auto split_node = rt_info[rt_key].as<std::shared_ptr<ov::op::v1::Split>>();
return split_node->output(static_cast<size_t>(split_index));
}
}
}
}
}
// Lambda function to process a single view operation
auto process_single_view =
[](ov::Output<ov::Node> current, size_t view_offset, const std::vector<size_t> & view_stride,
const ov::Shape & view_ggml_shape, const ov::PartialShape & view_ov_shape, const std::string & view_name,
size_t view_src_offset, const std::vector<size_t> & view_src_stride, const ov::Shape & view_src_ggml_shape,
const ov::PartialShape & view_src_ov_shape, const std::string & view_src_name) -> ov::Output<ov::Node> {
auto build_reshape_pattern = [](const ov::PartialShape & target_ov_shape,
const ov::Shape & target_ggml_shape) -> std::vector<int64_t> {
const size_t ndims = target_ggml_shape.size();
std::vector<int64_t> reshape_pattern(ndims);
size_t dynamic_dims = 0;
if (target_ov_shape.rank().is_static() &&
target_ov_shape.rank().get_length() == static_cast<int64_t>(ndims)) {
for (size_t i = 0; i < ndims; ++i) {
if (target_ov_shape[i].is_static()) {
reshape_pattern[i] = target_ov_shape[i].get_length();
} else {
reshape_pattern[i] = -1;
++dynamic_dims;
}
}
} else {
dynamic_dims = 2;
}
if (dynamic_dims > 1) {
for (size_t i = 0; i < ndims; ++i) {
reshape_pattern[i] = static_cast<int64_t>(target_ggml_shape[i]);
}
}
return reshape_pattern;
};
auto build_prefix_tail_reshape_pattern = [](const ov::PartialShape & target_ov_shape,
const ov::Shape & target_ggml_shape, size_t prefix_dims,
int64_t tail_dim) -> std::vector<int64_t> {
std::vector<int64_t> reshape_pattern(prefix_dims + 1);
size_t dynamic_dims = 0;
if (target_ov_shape.rank().is_static() &&
target_ov_shape.rank().get_length() == static_cast<int64_t>(target_ggml_shape.size())) {
for (size_t i = 0; i < prefix_dims; ++i) {
if (target_ov_shape[i].is_static()) {
reshape_pattern[i] = target_ov_shape[i].get_length();
} else {
reshape_pattern[i] = -1;
++dynamic_dims;
}
}
} else {
dynamic_dims = 2;
}
if (dynamic_dims > 1) {
for (size_t i = 0; i < prefix_dims; ++i) {
reshape_pattern[i] = static_cast<int64_t>(target_ggml_shape[i]);
}
}
reshape_pattern[prefix_dims] = tail_dim;
return reshape_pattern;
};
bool same_stride = view_stride.size() == view_src_stride.size();
if (same_stride) {
for (size_t i = 0; i < view_stride.size(); ++i) {
if (view_stride[i] != view_src_stride[i]) {
same_stride = false;
break;
}
}
}
bool same_ggml_shape = view_ggml_shape.size() == view_src_ggml_shape.size();
if (same_ggml_shape) {
for (size_t i = 0; i < view_ggml_shape.size(); ++i) {
if (view_ggml_shape[i] != view_src_ggml_shape[i]) {
same_ggml_shape = false;
break;
}
}
}
if (same_stride && same_ggml_shape) {
return current;
}
if (same_stride) {
const size_t relative_offset = view_offset >= view_src_offset ? view_offset - view_src_offset : 0;
const size_t ndims = view_stride.size();
std::vector<int> diff_dims;
if (view_ggml_shape.size() == ndims && view_src_ggml_shape.size() == ndims) {
for (size_t i = 0; i < ndims; ++i) {
if (view_ggml_shape[i] != view_src_ggml_shape[i]) {
diff_dims.push_back(static_cast<int>(i));
}
}
}
if (diff_dims.size() == 1) {
const int slice_dim = diff_dims[0];
const int64_t dim_size = static_cast<int64_t>(view_src_ggml_shape[slice_dim]);
if (view_stride[slice_dim] > 0 && relative_offset % view_stride[slice_dim] == 0) {
const int64_t begin_val = static_cast<int64_t>((relative_offset / view_stride[slice_dim]) %
static_cast<size_t>(dim_size));
const int64_t end_val = begin_val + static_cast<int64_t>(view_ggml_shape[slice_dim]);
if (begin_val >= 0 && end_val <= dim_size) {
auto sliced = std::make_shared<ov::op::v8::Slice>(
current, ov::op::v0::Constant::create(ov::element::i64, {1}, {begin_val}),
ov::op::v0::Constant::create(ov::element::i64, {1}, {end_val}),
ov::op::v0::Constant::create(ov::element::i64, {1}, {1}),
ov::op::v0::Constant::create(ov::element::i64, {1}, {slice_dim}));
if (view_ov_shape.is_static()) {
auto reshaped = std::make_shared<ov::op::v1::Reshape>(
sliced,
ov::op::v0::Constant::create(ov::element::i64, {ndims}, view_ov_shape.to_shape()),
false);
reshaped->set_friendly_name(view_name);
return reshaped;
}
sliced->set_friendly_name(view_name);
return sliced;
}
}
int64_t tail_src_elems = 1;
int64_t tail_dst_elems = 1;
for (size_t i = slice_dim; i < ndims; ++i) {
tail_src_elems *= static_cast<int64_t>(view_src_ggml_shape[i]);
tail_dst_elems *= static_cast<int64_t>(view_ggml_shape[i]);
}
const size_t elem_stride = view_stride[ndims - 1];
int64_t tail_begin = 0;
if (elem_stride > 0) {
tail_begin =
static_cast<int64_t>((relative_offset / elem_stride) % static_cast<size_t>(tail_src_elems));
}
const int64_t tail_end = tail_begin + tail_dst_elems;
if (tail_begin >= 0 && tail_end <= tail_src_elems) {
std::vector<int64_t> flat_shape;
for (int i = 0; i < slice_dim; ++i) {
flat_shape.push_back(static_cast<int64_t>(view_src_ggml_shape[i]));
}
flat_shape.push_back(tail_src_elems);
const size_t flat_ndims = flat_shape.size();
auto flat = std::make_shared<ov::op::v1::Reshape>(
current, ov::op::v0::Constant::create(ov::element::i64, {flat_ndims}, flat_shape), false);
auto sliced = std::make_shared<ov::op::v8::Slice>(
flat, ov::op::v0::Constant::create(ov::element::i64, {1}, {tail_begin}),
ov::op::v0::Constant::create(ov::element::i64, {1}, {tail_end}),
ov::op::v0::Constant::create(ov::element::i64, {1}, {1}),
ov::op::v0::Constant::create(ov::element::i64, {1}, {slice_dim}));
if (view_ov_shape.is_static()) {
auto reshaped = std::make_shared<ov::op::v1::Reshape>(
sliced, ov::op::v0::Constant::create(ov::element::i64, {ndims}, view_ov_shape.to_shape()),
false);
reshaped->set_friendly_name(view_name);
return reshaped;
}
sliced->set_friendly_name(view_name);
return sliced;
}
}
std::vector<int64_t> begin(ndims, 0);
std::vector<int64_t> end(ndims, 0);
std::vector<int64_t> step(ndims, 1);
std::vector<int64_t> axes(ndims, 0);
size_t remaining_offset = relative_offset;
for (size_t i = 0; i < ndims; ++i) {
axes[i] = static_cast<int64_t>(i);
if (view_stride[i] > 0) {
begin[i] = static_cast<int64_t>(remaining_offset / view_stride[i]);
remaining_offset %= view_stride[i];
}
end[i] = begin[i] + static_cast<int64_t>(view_ggml_shape[i]);
}
bool in_bounds = view_src_ggml_shape.size() == ndims && view_ggml_shape.size() == ndims;
if (in_bounds) {
for (size_t i = 0; i < ndims; ++i) {
if (end[i] > static_cast<int64_t>(view_src_ggml_shape[i])) {
in_bounds = false;
break;
}
}
}
if (in_bounds && remaining_offset == 0) {
auto sliced = std::make_shared<ov::op::v8::Slice>(
current, ov::op::v0::Constant::create(ov::element::i64, {ndims}, begin),
ov::op::v0::Constant::create(ov::element::i64, {ndims}, end),
ov::op::v0::Constant::create(ov::element::i64, {ndims}, step),
ov::op::v0::Constant::create(ov::element::i64, {ndims}, axes));
sliced->set_friendly_name(view_name);
return sliced;
}
} else {
bool same_rank = view_stride.size() == view_src_stride.size() &&
view_ggml_shape.size() == view_src_ggml_shape.size() &&
view_stride.size() == view_ggml_shape.size();
const size_t relative_offset = view_offset >= view_src_offset ? view_offset - view_src_offset : 0;
if (same_rank) {
const size_t ndims = view_ggml_shape.size();
std::vector<int> diff_dims;
for (size_t i = 0; i < ndims; ++i) {
if (view_ggml_shape[i] != view_src_ggml_shape[i]) {
diff_dims.push_back(static_cast<int>(i));
}
}
if (diff_dims.size() == 1) {
const size_t slice_dim = static_cast<size_t>(diff_dims[0]);
bool suffix_stride_match = true;
for (size_t i = slice_dim + 1; i < ndims; ++i) {
if (view_stride[i] != view_src_stride[i]) {
suffix_stride_match = false;
break;
}
}
if (suffix_stride_match && view_src_stride[slice_dim] > 0 &&
relative_offset % view_src_stride[slice_dim] == 0) {
const int64_t begin_val = static_cast<int64_t>(relative_offset / view_src_stride[slice_dim]);
const int64_t end_val = begin_val + static_cast<int64_t>(view_ggml_shape[slice_dim]);
const int64_t dim_size = static_cast<int64_t>(view_src_ggml_shape[slice_dim]);
if (begin_val >= 0 && end_val <= dim_size) {
auto sliced = std::make_shared<ov::op::v8::Slice>(
current, ov::op::v0::Constant::create(ov::element::i64, {1}, {begin_val}),
ov::op::v0::Constant::create(ov::element::i64, {1}, {end_val}),
ov::op::v0::Constant::create(ov::element::i64, {1}, {1}),
ov::op::v0::Constant::create(ov::element::i64, {1}, {static_cast<int64_t>(slice_dim)}));
sliced->set_friendly_name(view_name);
return sliced;
}
}
}
}
size_t view_elems = 1;
size_t src_elems = 1;
if (same_rank) {
for (size_t i = 0; i < view_ggml_shape.size(); ++i) {
view_elems *= view_ggml_shape[i];
src_elems *= view_src_ggml_shape[i];
}
}
bool same_num_elements = same_rank && view_elems == src_elems;
if (same_rank && relative_offset == 0 && same_num_elements) {
auto reshape_pattern = build_reshape_pattern(view_ov_shape, view_ggml_shape);
auto reshaped = std::make_shared<ov::op::v1::Reshape>(
current, ov::op::v0::Constant::create(ov::element::i64, {reshape_pattern.size()}, reshape_pattern),
false);
reshaped->set_friendly_name(view_name);
return reshaped;
}
if (same_rank) {
const size_t ndims = view_ggml_shape.size();
// Match views that can be expressed as a regular strided slice over the
// already reconstructed source tensor, e.g. offset on one axis plus step > 1
// on another axis.
bool is_regular_slice = view_src_ggml_shape.size() == ndims;
std::vector<int64_t> begin(ndims, 0);
std::vector<int64_t> end(ndims, 0);
std::vector<int64_t> step(ndims, 1);
std::vector<int64_t> axes(ndims, 0);
size_t remaining_offset = relative_offset;
if (is_regular_slice) {
for (size_t i = 0; i < ndims; ++i) {
axes[i] = static_cast<int64_t>(i);
if (view_src_stride[i] == 0 || view_stride[i] == 0 ||
view_stride[i] % view_src_stride[i] != 0) {
is_regular_slice = false;
break;
}
step[i] = static_cast<int64_t>(view_stride[i] / view_src_stride[i]);
if (step[i] <= 0) {
is_regular_slice = false;
break;
}
begin[i] = static_cast<int64_t>(remaining_offset / view_src_stride[i]);
remaining_offset %= view_src_stride[i];
if (view_ggml_shape[i] == 0) {
end[i] = begin[i];
continue;
}
end[i] = begin[i] + step[i] * static_cast<int64_t>(view_ggml_shape[i] - 1) + 1;
if (begin[i] < 0 || end[i] > static_cast<int64_t>(view_src_ggml_shape[i])) {
is_regular_slice = false;
break;
}
}
}
if (is_regular_slice && remaining_offset == 0) {
auto sliced = std::make_shared<ov::op::v8::Slice>(
current, ov::op::v0::Constant::create(ov::element::i64, {ndims}, begin),
ov::op::v0::Constant::create(ov::element::i64, {ndims}, end),
ov::op::v0::Constant::create(ov::element::i64, {ndims}, step),
ov::op::v0::Constant::create(ov::element::i64, {ndims}, axes));
sliced->set_friendly_name(view_name);
return sliced;
}
const size_t elem_stride = view_src_stride.back();
const bool aligned_offset = elem_stride > 0 && relative_offset % elem_stride == 0;
if (aligned_offset) {
size_t suffix_start = 0;
size_t expected_stride = elem_stride;
for (int i = static_cast<int>(ndims) - 1; i >= 0; --i) {
if (view_stride[i] != expected_stride) {
suffix_start = static_cast<size_t>(i + 1);
break;
}
expected_stride *= view_ggml_shape[i];
}
size_t prefix_elems = 1;
size_t suffix_elems = 1;
for (size_t i = 0; i < suffix_start; ++i) {
prefix_elems *= view_ggml_shape[i];
}
for (size_t i = suffix_start; i < ndims; ++i) {
suffix_elems *= view_ggml_shape[i];
}
if (prefix_elems > 0 && src_elems % prefix_elems == 0) {
const size_t src_tail_elems = src_elems / prefix_elems;
const int64_t tail_begin = static_cast<int64_t>(relative_offset / elem_stride);
const int64_t tail_end = tail_begin + static_cast<int64_t>(suffix_elems);
if (tail_begin >= 0 && tail_end <= static_cast<int64_t>(src_tail_elems)) {
auto prefix_tail_pattern = build_prefix_tail_reshape_pattern(
view_ov_shape, view_ggml_shape, suffix_start, static_cast<int64_t>(src_tail_elems));
auto prefix_tail = std::make_shared<ov::op::v1::Reshape>(
current,
ov::op::v0::Constant::create(ov::element::i64, {prefix_tail_pattern.size()},
prefix_tail_pattern),
false);
ov::Output<ov::Node> selected = prefix_tail;
if (tail_begin != 0 || tail_end != static_cast<int64_t>(src_tail_elems)) {
selected = std::make_shared<ov::op::v8::Slice>(
prefix_tail, ov::op::v0::Constant::create(ov::element::i64, {1}, {tail_begin}),
ov::op::v0::Constant::create(ov::element::i64, {1}, {tail_end}),
ov::op::v0::Constant::create(ov::element::i64, {1}, {1}),
ov::op::v0::Constant::create(ov::element::i64, {1},
{static_cast<int64_t>(suffix_start)}));
}
auto reshape_pattern = build_reshape_pattern(view_ov_shape, view_ggml_shape);
auto reshaped = std::make_shared<ov::op::v1::Reshape>(
selected,
ov::op::v0::Constant::create(ov::element::i64, {reshape_pattern.size()},
reshape_pattern),
false);
reshaped->set_friendly_name(view_name);
return reshaped;
}
}
}
}
return current;
}
(void) view_name;
(void) view_src_ov_shape;
(void) view_src_name;
return current;
};
// Process views from the base tensor (last) to the current view (first)
// Start with the base tensor
ov::Output<ov::Node> current = input;
// Process each view in reverse order (from base to current)
for (int view_idx = view_input_size - 1; view_idx >= 0; view_idx--) {
auto view_offset = context.get_view_input_offset(input_index, view_idx);
auto view_stride = context.get_view_input_stride(input_index, view_idx);
auto view_ggml_shape = context.get_view_input_ggml_shape(input_index, view_idx);
auto view_ov_shape = context.get_view_input_ov_shape(input_index, view_idx);
auto view_name = context.get_view_input_name(input_index, view_idx);
// print view info
// std::cout << "View " << view_idx << ": name = " << view_name << ", offset = " << view_offset << ", stride = ["
// << view_stride[0] << "," << view_stride[1] << "," << view_stride[2] << "," << view_stride[3]
// << "], ggml shape = [" << view_ggml_shape[0] << "," << view_ggml_shape[1] << ","
// << view_ggml_shape[2] << "," << view_ggml_shape[3] << "], ov shape = " << view_ov_shape << std::endl;
auto view_src_offset = context.get_view_input_src_offset(input_index, view_idx);
auto view_src_stride = context.get_view_input_src_stride(input_index, view_idx);
auto view_src_ggml_shape = context.get_view_input_src_ggml_shape(input_index, view_idx);
auto view_src_ov_shape = context.get_view_input_src_ov_shape(input_index, view_idx);
auto view_src_name = context.get_view_input_src_name(input_index, view_idx);
// print source view info
// std::cout << "View " << view_idx << ": source name = " << view_src_name
// << ", source offset = " << view_src_offset << ", source stride = [" << view_src_stride[0] << ","
// << view_src_stride[1] << "," << view_src_stride[2] << "," << view_src_stride[3]
// << "], source ggml shape = [" << view_src_ggml_shape[0] << "," << view_src_ggml_shape[1] << ","
// << view_src_ggml_shape[2] << "," << view_src_ggml_shape[3]
// << "], source ov shape = " << view_src_ov_shape << std::endl;
current = process_single_view(current, view_offset, view_stride, view_ggml_shape, view_ov_shape, view_name,
view_src_offset, view_src_stride, view_src_ggml_shape, view_src_ov_shape,
view_src_name);
}
return current;
}
} // namespace ggml
} // namespace frontend
} // namespace ov