Lumpiasty/llama.cpp
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

OpenVINO: Update OV to 2026.3.1, whisper.cpp support, Qwen3.5 on NPU, and new ops (#27843)

Browse Source 
* OpenVINO Backend: Fuse IM2COL + MatMul convolution into OpenVINO convolution

* ci:ggml-ov: Skip recurrent state rollback tests

* ci:ggml-ov: Skip recurrent state rollback tests

* Update OPENVINO.md

* ggml-openvino : add env-var gated op support debugging

* Fix ggml_rope_set_offset case

* OpenVINO backend: Support Whisper.cpp

* Fix code style

* openvino : enable qwen35 on NPU

Static shapes:
- get_graph_input_shape() left the s_copy / s_copy-leaf inputs dynamic
  ([1,1,1,-1]) even in static mode, which propagated a dynamic slot dim through
  GET_ROWS into the conv/GDN state, the state reshapes and the GDN output.
- With -np 1 the s_copy defrag remainder gathers zero rows; short-circuit that
  CPY to the untouched cache instead of emitting a degenerate Slice/Concat, and
  skip binding its zero-byte ggml tensor as an output (the dynamic path already
  did the latter, the static path wrote the full cache over a 0-byte buffer).

Token-count independence:
- In static mode the compiled model's token count is the prefill chunk size or
  1, not the captured cgraph's. Offsets derived from the captured count were
  therefore wrong. Anchor the GDN state slice at the end of the packed
  [attn | state] output and drop the rs_src_begin runtime inputs, and make
  VIEWs over the GDN output / conv_input pass through so the consumer does the
  slicing.
- CONT could not identify its token axis when the graph was captured with a
  single token (every trailing dim has the same stride and size 1) and baked
  the captured shape into the prefill model.

Chunked prefill:
- The last chunk is padded with fabricated tokens. Attention masks them, but
  the recurrent path folded them into cache_r/cache_s permanently. Add a
  chunk_valid_len runtime input, use it to zero g and beta for padded steps
  (making the recurrence an exact identity) and to end the conv snapshot window
  at the last valid token, and disable the recurrent-cache reset after the
  first chunk so earlier chunks are not wiped.
- get_is_prefill() and the chunk loop bound read inp_pos->ne[0] directly, but
  IMROPE stacks 4 position planes, so every decode step was run through the
  padded prefill model and the loop ran extra out-of-bounds chunks.

cache_rs_reset_idx/len now stay runtime Parameters in static mode, since
can_reuse_statically() does not invalidate the cached model on ComputeParams
changes. Add GGML_OPENVINO_FORCE_STATIC to exercise the static path on CPU.

* Update to OpenVINO 2026.3.1

* ggml-openvino: forward NPU compilation mode parameters

Add GGML_OPENVINO_NPU_COMPILE_CONFIG to the backend's cached environment so callers can configure the NPU compiler without using the generic property escape hatch.

When the value is non-empty, pass it to OpenVINO as NPU_COMPILATION_MODE_PARAMS. This enables settings such as optimization-level=3 for NPU compilation while preserving the existing behavior when the variable is unset and leaving CPU and GPU configuration unchanged.

Document the variable, its NPU-only scope, and the optimization-level=3 example in the OpenVINO backend runtime configuration table.

* ggml-openvino : support RELU, POOL_2D, QUICK_GEGLU, and ROLL ops

* reorder op table

* exclude GPU/NPU failing POOL_2D case

* move op type detection to compute_op_case

* Relax rope supported cases

* Fix pool case

* Update openvino doc, gpu driver in ov docker

* openvino: remove unused static remote context branch

* openvino: parallelize static model build

* Apply editorconfig

---------

Co-authored-by: Mostafa Faheem <mostafaaafaheem@gmail.com>
Co-authored-by: Ravi Panchumarthy <ravi.panchumarthy@intel.com>
Co-authored-by: zhaixuejun1993 <xuejun.zhai@intel.com>
This commit is contained in:
Zijun Yu
2026-08-28 14:42:07 +03:00
committed by GitHub
co-authored by Mostafa Faheem Ravi Panchumarthy zhaixuejun1993
parent b19cbe925b
commit 511f9c1379
27 changed files with 1119 additions and 264 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ggml/src/ggml-openvino/ggml-openvino.cpp
+134 -97
View File
@@ -908,11 +908,27 @@ static bool has_non_contiguous_view_input(const ggml_tensor * op) {
}
static bool is_supported_flash_attn_pattern(const ggml_tensor * op) {
// pattern of q,k,v should be q->op==PERMUTE, q->src[0]->op==VIEW, q->src[0]->src[0]->view_src==nullptr
// Each Q/K/V input must follow one of:
// PERMUTE -> VIEW -> base (view_src==nullptr) (llama KV-cache path)
// PERMUTE -> RESHAPE -> base (view_src==nullptr) (whisper Q)
// VIEW -> base (view_src==nullptr) (whisper K/V from kv_pad)
for (int i = 0; i < 3; i++) {
const ggml_tensor * src = op->src[i];
if (src->op != GGML_OP_PERMUTE || src->src[0] == nullptr || src->src[0]->op != GGML_OP_VIEW ||
src->src[0]->src[0] == nullptr || src->src[0]->src[0]->view_src != nullptr) {
if (src->op == GGML_OP_PERMUTE) {
if (src->src[0] == nullptr) {
return false;
}
if (src->src[0]->op != GGML_OP_VIEW && src->src[0]->op != GGML_OP_RESHAPE) {
return false;
}
if (src->src[0]->src[0] == nullptr || src->src[0]->src[0]->view_src != nullptr) {
return false;
}
} else if (src->op == GGML_OP_VIEW) {
if (src->src[0] == nullptr || src->src[0]->view_src != nullptr) {
return false;
}
} else {
return false;
}
}
@@ -1030,18 +1046,29 @@ static bool is_msa_block_mask_expansion(const ggml_tensor * op) {
return tensor_name_starts_with(src, "msa_block_mask");
}
static bool is_op_unsupported_case(const ggml_tensor * op) {
namespace {
struct ggml_openvino_op_support {
bool is_supported = true;
std::string reason;
operator bool() const {
return is_supported;
}
};
} // namespace
static ggml_openvino_op_support is_op_supported_case(const ggml_tensor * op) {
if (is_msa_block_mask_expansion(op)) {
return true;
return {false, "MSA block mask expansion is not supported"};
}
switch (op->op) {
case GGML_OP_CONCAT: {
if (op->type == GGML_TYPE_I64) {
return true;
return {false, "CONCAT with I64 type is not supported"};
}
if (ggml_openvino_get_device_name() == "GPU" && op->type == GGML_TYPE_BF16 && has_view_op_input(op)) {
return true;
return {false, "CONCAT with BF16 type and VIEW input is not supported on GPU"};
}
break;
}
@@ -1052,24 +1079,21 @@ static bool is_op_unsupported_case(const ggml_tensor * op) {
// OpenVINO SET translation currently supports dst layouts that match src0 strides.
if (op->src[0] == nullptr || nb1 != op->src[0]->nb[1] || nb2 != op->src[0]->nb[2] || nb3 != op->src[0]->nb[3]) {
// std::cout << "Unsupported SET op with dst nb1=" << nb1 << ", nb2=" << nb2 << ", nb3=" << nb3
// << " that does not match src0 strides nb[1]="
// << (op->src[0] != nullptr ? std::to_string(op->src[0]->nb[1]) : "null")
// << ", nb[2]=" << (op->src[0] != nullptr ? std::to_string(op->src[0]->nb[2]) : "null")
// << ", nb[3]=" << (op->src[0] != nullptr ? std::to_string(op->src[0]->nb[3]) : "null")
// << std::endl;
return true;
return {false, "SET op with dst nb1=" + std::to_string(nb1) + ", nb2=" + std::to_string(nb2) + ", nb3=" + std::to_string(nb3) +
" that does not match src0 strides nb[1]=" + (op->src[0] != nullptr ? std::to_string(op->src[0]->nb[1]) : "null") +
", nb[2]=" + (op->src[0] != nullptr ? std::to_string(op->src[0]->nb[2]) : "null") +
", nb[3]=" + (op->src[0] != nullptr ? std::to_string(op->src[0]->nb[3]) : "null")};
}
break;
}
case GGML_OP_GET_ROWS:
case GGML_OP_SET_ROWS: {
if (op->ne[3] != 1) {
return true;
return {false, "GET_ROWS/SET_ROWS with ne[3] != 1 (ne[3]=" + std::to_string(op->ne[3]) + ") is not supported"};
}
if (op->op == GGML_OP_GET_ROWS && ggml_openvino_get_device_name() == "GPU" &&
op->src[0]->type == GGML_TYPE_BF16) {
return true;
return {false, "GET_ROWS with BF16 src0 is not supported on GPU"};
}
if (op->ne[0] == 256 && (op->src[0]->type == GGML_TYPE_Q4_K || op->src[0]->type == GGML_TYPE_Q5_K ||
op->src[0]->type == GGML_TYPE_Q4_1 || op->src[0]->type == GGML_TYPE_Q5_1)) {
@@ -1078,14 +1102,14 @@ static bool is_op_unsupported_case(const ggml_tensor * op) {
// make_int8_weights/make_int4_weights: dequant is done in f16, not f32, to keep the
// Convert/Subtract/Multiply chain fusable into GatherMatmulCompressed/FullyConnectedCompressed
// for the shared non-test code paths).
return true;
return {false, "GET_ROWS/SET_ROWS with ne[0] == 256 and type " + std::string(ggml_type_name(op->src[0]->type)) +
" rejected due to f16-arithmetic dequant rounding errors that intermittently exceed 1e-7 NMSE threshold"};
}
break;
}
case GGML_OP_RESHAPE: {
if (strncmp(op->name, "ffn_norm_exps", sizeof("ffn_norm_exps") - 1) == 0) {
return true;
return {false, "RESHAPE for ffn_norm_exps is not supported"};
}
break;
}
@@ -1093,11 +1117,13 @@ static bool is_op_unsupported_case(const ggml_tensor * op) {
case GGML_OP_MUL:
case GGML_OP_SUB: {
if (op->src[1]->op == GGML_OP_PERMUTE) {
return true;
return {false, "ADD/MUL/SUB with PERMUTE src1 is not supported"};
}
for (int i = 0; i < 4; i++) {
if (op->src[0]->ne[i] != op->src[1]->ne[i] && (op->src[0]->ne[i] != 1 && op->src[1]->ne[i] != 1)) {
return true;
return {false, "ADD/MUL/SUB with incompatible broadcast shapes: src0->ne[" + std::to_string(i) + "]=" +
std::to_string(op->src[0]->ne[i]) + ", src1->ne[" + std::to_string(i) + "]=" +
std::to_string(op->src[1]->ne[i])};
}
}
break;
@@ -1106,7 +1132,7 @@ static bool is_op_unsupported_case(const ggml_tensor * op) {
// Keep support aligned with the CPU backend implementation, which only handles f32 inputs/output and i32 ids.
if (op->type != GGML_TYPE_F32 || op->src[0]->type != GGML_TYPE_F32 || op->src[1]->type != GGML_TYPE_F32 ||
op->src[2]->type != GGML_TYPE_I32) {
return true;
return {false, "ADD_ID only supports F32 inputs/output and I32 ids"};
}
break;
}
@@ -1116,14 +1142,27 @@ static bool is_op_unsupported_case(const ggml_tensor * op) {
// until the fused GPU kernel is reliable. (falied case llama-arch-test mpt)
if (ggml_openvino_get_device_name() == "GPU" && op->src[1]->ne[0] == op->ne[0] &&
op->src[1]->ne[1] == 1 && op->src[1]->ne[2] == 1 && op->src[1]->ne[3] == 1) {
return true;
return {false, "DIV per-channel scale broadcast is not supported on GPU"};
}
break;
}
case GGML_OP_POOL_2D: {
const auto& name = ggml_openvino_get_device_name();
if (name == "GPU") {
const int32_t * params = op->op_params;
const int k0 = params[1];
const int k1 = params[2];
const int p0 = params[5];
const int p1 = params[6];
if ((p0 > 0 || p1 > 0) && (k0 < 3 || k1 < 3)) {
return {false, "POOL_2D with padding and kernel size < 3 is not supported on " + name};
}
}
break;
}
case GGML_OP_SUM_ROWS: {
// if the input is PERMUTE skip
if (op->src[0]->op == GGML_OP_PERMUTE) {
return true;
return {false, "SUM_ROWS with PERMUTE input is not supported"};
}
break;
}
@@ -1140,54 +1179,51 @@ static bool is_op_unsupported_case(const ggml_tensor * op) {
// accuracy drift in the OpenVINO path. Restrict by scale=1.0 to avoid
// affecting non-gemma3n models such as Llama-3.2.
if (fabsf(scale - 1.0f) < 1e-6f && is_gemma3n_flash_attn_pattern(op)) {
return true;
return {false, "FLASH_ATTN_EXT gemma3n pattern on GPU is not supported"};
}
if (op->src[4] != nullptr) {
// GGML_LOG_WARN("OpenVINO backend does not support FLASH_ATTN_EXT with sinks\n");
return true;
return {false, "FLASH_ATTN_EXT with sinks is not supported"};
}
if (!is_supported_flash_attn_pattern(op)) {
return true;
return {false, "FLASH_ATTN_EXT unsupported attention pattern"};
}
if (max_bias > 0) {
// GGML_LOG_WARN("OpenVINO backend does not support FLASH_ATTN_EXT with max_bias > 0\n");
return true;
return {false, "FLASH_ATTN_EXT with max_bias > 0 (max_bias=" + std::to_string(max_bias) + ") is not supported"};
}
if (logit_softcap != 0) {
// GGML_LOG_WARN("OpenVINO backend does not support FLASH_ATTN_EXT with logit_softcap != 0\n");
return true;
return {false, "FLASH_ATTN_EXT with logit_softcap != 0 (logit_softcap=" + std::to_string(logit_softcap) + ") is not supported"};
}
break;
}
case GGML_OP_PERMUTE: {
if (op->type == GGML_TYPE_BF16) {
// err msg: [GPU] Could not find a suitable kernel for transpose
// GGML_LOG_WARN("OpenVINO backend does not support PERMUTE with BF16 type\n");
return true;
if (op->type == GGML_TYPE_BF16 && ggml_openvino_get_device_name() == "GPU") {
return {false, "PERMUTE with BF16 type is not supported on GPU"};
}
break;
}
case GGML_OP_CPY: {
if (op->src[0]->type == GGML_TYPE_BF16 || op->src[1]->type == GGML_TYPE_BF16) {
// GGML_LOG_WARN("OpenVINO backend does not support CPY with non-contiguous data or bf16 types\n");
return true;
return {false, "CPY with BF16 src type is not supported"};
}
// CPY to a quantized destination (e.g. f32 -> q4_0) is numerically unstable with OpenVINO backend.
if (ggml_is_quantized(op->type)) {
return true;
return {false, "CPY to quantized destination (e.g. f32 -> q4_0) is numerically unstable"};
}
if (ggml_nelements(op->src[0]) != ggml_nelements(op->src[1])) {
return true;
return {false, "CPY with mismatched element counts is not supported: src0=" + std::to_string(ggml_nelements(op->src[0])) +
" != src1=" + std::to_string(ggml_nelements(op->src[1]))};
}
// op test case with non-contiguous src or dst
if ((op->ne[0] == 3 && op->ne[1] == 4 && op->ne[2] == 3 && op->ne[3] == 2) ||
(op->ne[0] == 1 && op->ne[1] == 4 && op->ne[2] == 3 && op->ne[3] == 2) ||
(op->ne[0] == 2 && op->ne[1] == 4 && op->ne[2] == 3 && op->ne[3] == 2)) {
return true;
return {false, "CPY with non-contiguous shape [" + std::to_string(op->ne[0]) + ", " +
std::to_string(op->ne[1]) + ", " + std::to_string(op->ne[2]) + ", " +
std::to_string(op->ne[3]) + "] is not supported"};
}
if (!cpy_output_view_is_supported(op)) {
return true;
return {false, "CPY with non-contiguous output view is not supported"};
}
break;
}
@@ -1196,13 +1232,14 @@ static bool is_op_unsupported_case(const ggml_tensor * op) {
ggml_is_quantized(op->src[0]->type) && strcmp(op->src[0]->name, "a") == 0 &&
strcmp(op->src[1]->name, "b") == 0 && op->src[0]->ne[1] == 1 && op->src[1]->ne[1] == 64 &&
op->src[0]->ne[0] == 256 && op->src[1]->ne[0] == 256) {
return true;
return {false, "MUL_MAT quantized benchmark test case on GPU is not supported"};
}
if (op->src[0]->ne[3] != op->src[1]->ne[3] && op->src[0]->ne[3] != 1 && op->src[1]->ne[3] != 1) {
return true;
return {false, "MUL_MAT with incompatible broadcast on ne[3]: src0->ne[3]=" + std::to_string(op->src[0]->ne[3]) +
", src1->ne[3]=" + std::to_string(op->src[1]->ne[3])};
}
if (op->src[0]->op == GGML_OP_VIEW && op->src[1]->op == GGML_OP_VIEW) {
return true;
return {false, "MUL_MAT with both inputs as VIEW is not supported"};
}
break;
}
@@ -1210,16 +1247,17 @@ static bool is_op_unsupported_case(const ggml_tensor * op) {
// Single-expert (or empty) MUL_MAT_ID is a degenerate shape that stresses GatherMatmul edge
// cases and never occurs in real MoE; let it fall back to CPU.
if (op->src[0] != nullptr && op->src[0]->ne[2] <= 1) {
return true;
return {false, "MUL_MAT_ID with single-expert or empty ne[2] <= 1 (ne[2]=" +
std::to_string(op->src[0]->ne[2]) + ") is not supported"};
}
if (ggml_openvino_get_device_name() == "GPU" && op->src[0] != nullptr && op->src[0]->type == GGML_TYPE_BF16) {
return true;
return {false, "MUL_MAT_ID with BF16 weights on GPU is not supported"};
}
// GPU MUL_MAT_ID uses a Gather+MatMul fallback because the GPU plugin rejects internal
// GatherMatmul for these test shapes. Skip cases that would materialize a large selected
// expert-weight temporary.
if (ggml_openvino_get_device_name() == "GPU" && mul_mat_id_requires_large_tmp(op)) {
return true;
return {false, "MUL_MAT_ID requires large temporary on GPU"};
}
break;
}
@@ -1229,51 +1267,46 @@ static bool is_op_unsupported_case(const ggml_tensor * op) {
const int mode = op_params[2];
if (op_params[15] != 0) {
// FIXME: support ggml_rope_set_offset
return true;
return {false, "ggml_rope_set_offset is not supported"};
}
if (mode != GGML_ROPE_TYPE_NORMAL && mode != GGML_ROPE_TYPE_NEOX && mode != GGML_ROPE_TYPE_IMROPE) {
// GGML_LOG_WARN("OpenVINO backend does not support ROPE with mode %d\n", mode);
return true;
return {false, "ROPE with mode " + std::to_string(mode) + " is not supported"};
}
const int64_t head_dim = op->src[0]->ne[0];
const int64_t rope_dims = n_dims == 0 ? head_dim : n_dims;
if (rope_dims <= 0 || rope_dims > head_dim || (rope_dims % 2) != 0) {
// GGML_LOG_WARN("OpenVINO backend does not support ROPE with n_dims %d and src[0]->ne[0] %ld\n", n_dims,
// op->src[0]->ne[0]);
return true;
return {false, "ROPE with n_dims=" + std::to_string(n_dims) + ", head_dim=" + std::to_string(head_dim) + " is not supported"};
}
if (op->type != GGML_TYPE_F32 && op->type != GGML_TYPE_F16) {
// GGML_LOG_WARN("OpenVINO backend does not support ROPE with type %s\n", ggml_type_name(op->type));
return true;
return {false, "ROPE with type " + std::string(ggml_type_name(op->type)) + " is not supported"};
}
if (op->src[0]->op == GGML_OP_VIEW) {
if (op->src[0]->view_src->ne[1] != op->src[0]->ne[2]) {
// GGML_LOG_WARN(
// "OpenVINO backend does not support ROPE with src[0]->view_src->ne[1] %ld != src[0]->ne[2] "
// "%ld\n",
// op->src[0]->view_src->ne[1], op->src[0]->ne[2]);
return true;
const struct ggml_tensor * view = op->src[0];
const struct ggml_tensor * view_src = view->view_src;
if (view_src->ne[1] != view->ne[1] || view_src->ne[2] != view->ne[2] || view_src->ne[3] != view->ne[3]) {
return {false, "ROPE with view_src->ne [" + std::to_string(view_src->ne[1]) + ", " +
std::to_string(view_src->ne[2]) + ", " + std::to_string(view_src->ne[3]) +
"] != view->ne [" + std::to_string(view->ne[1]) + ", " +
std::to_string(view->ne[2]) + ", " + std::to_string(view->ne[3]) +
"] is not supported"};
}
}
if (mode == GGML_ROPE_TYPE_IMROPE &&
(op->src[2] != 0 || ((const float *) op_params)[6] != 1 || ((const float *) op_params)[7] != 0 ||
((const float *) op_params)[8] != 1)) {
// GGML_LOG_WARN("OpenVINO backend does not support IMROPE with freq_factors, freq_scale, ext_factor, and attn_factor\n");
return true;
return {false, "IMROPE with freq_factors, freq_scale, ext_factor, and attn_factor is not supported"};
}
break;
}
case GGML_OP_TRANSPOSE: {
// if the type is bf16, will return true
if (op->type == GGML_TYPE_BF16) {
// GGML_LOG_WARN("OpenVINO backend does not support CONT with BF16 type\n");
return true;
return {false, "TRANSPOSE with BF16 type is not supported"};
}
break;
}
case GGML_OP_REPEAT: {
if (ggml_openvino_get_device_name() == "GPU" && op->type == GGML_TYPE_BF16) {
return true;
return {false, "REPEAT with BF16 type is not supported on GPU"};
}
break;
}
@@ -1285,15 +1318,15 @@ static bool is_op_unsupported_case(const ggml_tensor * op) {
// return true;
// }
if (op->src[2]->op == GGML_OP_PERMUTE) {
return true;
return {false, "GATED_DELTA_NET with PERMUTE src2 is not supported"};
}
// kda (per-key-dimension gating) not supported by fused GatedDeltaNet op
if (op->src[3]->ne[0] != 1) {
return true;
return {false, "GATED_DELTA_NET with kda (per-key-dimension gating) is not supported"};
}
// K > 1 (multiple state snapshots) not supported by fused op
if (((const int32_t *) op->op_params)[0] > 1) {
return true;
return {false, "GATED_DELTA_NET with K > 1 (multiple state snapshots) is not supported"};
}
break;
}
@@ -1307,17 +1340,17 @@ static bool is_op_unsupported_case(const ggml_tensor * op) {
// Skip TOPK_MOE fused tests until it is fully supported.
// The argsort_top_k VIEW wrapping ARGSORT is named "selected_experts" in test_topk_moe.
if (strcmp(op->name, "selected_experts") == 0) {
return true;
return {false, "VIEW for selected_experts (argsort_top_k) is not supported"};
}
break;
}
default:
break;
}
return false;
return {true, ""};
}
static bool ggml_backend_openvino_device_supports_op(ggml_backend_dev_t dev, const ggml_tensor * op) {
static ggml_openvino_op_support ggml_backend_openvino_device_supports_op_impl(ggml_backend_dev_t dev, const ggml_tensor * op) {
GGML_ASSERT(dev->reg != nullptr);
static std::unordered_set<ggml_type> supported_types{
@@ -1367,48 +1400,41 @@ static bool ggml_backend_openvino_device_supports_op(ggml_backend_dev_t dev, con
case GGML_OP_UNARY: {
auto supported = supported_unary_ops.find(ggml_get_unary_op(op)) != supported_unary_ops.end();
if (!supported) {
// GGML_LOG_WARN("OpenVINO backend does not support unary op %s\n", ggml_unary_op_name(ggml_get_unary_op(op)));
return false;
return {false, "unary op " + std::string(ggml_unary_op_name(ggml_get_unary_op(op))) + " has no op translator"};
}
if (ggml_get_unary_op(op) == GGML_UNARY_OP_EXP && op->type == GGML_TYPE_F32) {
return false;
return {false, "UNARY_EXP with F32 type is not supported"};
}
break;
}
case GGML_OP_GLU: {
auto supported = supported_glu_ops.find(ggml_get_glu_op(op)) != supported_glu_ops.end();
if (!supported) {
// GGML_LOG_WARN("OpenVINO backend does not support GLU op %s\n", ggml_glu_op_name(ggml_get_glu_op(op)));
return false;
return {false, "GLU op " + std::string(ggml_glu_op_name(ggml_get_glu_op(op))) + " has no op translator"};
}
// if (has_view_op_input(op)) {
// // GGML_LOG_WARN("OpenVINO backend does not support unary op %s with view input\n",
// // ggml_glu_op_name(ggml_get_glu_op(op)));
// return false;
// return {false, "GLU op " + std::string(ggml_glu_op_name(ggml_get_glu_op(op))) + " with view input is not supported"};
// }
if (op->src[1] == nullptr && op->src[0]->ne[0] % 2 != 0) {
// triggers bug in ov gpu
return false;
return {false, "GLU op with odd src0 ne[0] and null src1 is not supported"};
}
break;
}
default: {
auto supported = supported_ops.find(op->op) != supported_ops.end();
if (!supported) {
// GGML_LOG_WARN("OpenVINO backend does not support op %s\n", ggml_op_name(op->op));
return false;
return {false, "op " + std::string(ggml_op_name(op->op)) + " has no op translator"};
}
static std::set<ggml_op> ops_not_support_view_input{};
if (ops_not_support_view_input.find(op->op) != ops_not_support_view_input.end() && has_view_op_input(op)) {
// GGML_LOG_WARN("OpenVINO backend does not support op %s with view input\n", ggml_op_name(op->op));
return false;
return {false, "op " + std::string(ggml_op_name(op->op)) + " with VIEW input is not supported"};
}
}
}
if (supported_types.find(op->type) == supported_types.end()) {
// GGML_LOG_WARN("OpenVINO backend does not support tensor type %s\n", ggml_type_name(op->type));
return false;
return {false, "tensor type " + std::string(ggml_type_name(op->type)) + " is not supported"};
}
for (int i = 0; i < GGML_MAX_SRC; i++) {
auto * src = op->src[i];
@@ -1416,21 +1442,32 @@ static bool ggml_backend_openvino_device_supports_op(ggml_backend_dev_t dev, con
break;
}
if (supported_types.find(src->type) == supported_types.end()) {
// GGML_LOG_WARN("OpenVINO backend does not support tensor type %s\n", ggml_type_name(src->type));
return false;
return {false, "src[" + std::to_string(i) + "] type " + std::string(ggml_type_name(src->type)) + " is not supported"};
}
const bool is_supported_3d_moe_expert =
op->op == GGML_OP_MUL_MAT_ID && i == 0 && (src->type == GGML_TYPE_MXFP4 || src->ne[3] == 1);
if (ggml_is_quantized(src->type) && src->ne[2] != 1 && !is_supported_3d_moe_expert) {
// GGML_LOG_WARN("OpenVINO backend does not support 3D quantized tensors\n");
return false;
return {false, "3D quantized tensor for src[" + std::to_string(i) + "] is not supported"};
}
}
if (is_op_unsupported_case(op)) {
return false;
auto op_support_case = is_op_supported_case(op);
if (!op_support_case.is_supported) {
return op_support_case;
}
return true;
return {true, ""};
}
static bool ggml_backend_openvino_device_supports_op(ggml_backend_dev_t dev, const ggml_tensor * op) {
auto res = ggml_backend_openvino_device_supports_op_impl(dev, op);
if (!res.is_supported) {
static const bool log_unsupported = ggml_openvino_getenv_int("GGML_OPENVINO_LOG_UNSUPPORTED_OPS") != 0;
if (log_unsupported) {
GGML_LOG_WARN("OpenVINO op unsupported: op '%s' (%s), type %s: %s\n",
op->name, ggml_op_name(op->op), ggml_type_name(op->type), res.reason.c_str());
}
}
return res.is_supported;
}
static bool ggml_backend_openvino_device_supports_buft(ggml_backend_dev_t dev, ggml_backend_buffer_type_t buft) {