ggml : add OpenVINO backend (#15307)
* Update build doc * Add cgraph tensor output name to OV op name * Update openvino build instructions * Add initial NPU support * draft NPU support version 2: prefill + kvcache * NPU support version 2: prefill + kvcache * Change due to ggml cgraph changes, not correct yet * Change due to ggml cgraph changes, llama-3.2 CPU work * Add AMD64 to CMakeLists * Change due to ggml cgraph changes, all device work * Refactor: clean, fix warning * Update clang-format * Statful transformation for CPU GPU * Add SwiGLU * Fuse to SDPA * Replace Concat with Broadcast in MulMat for GQA * Pull out indices creation for kv cache update * Refactor: remove past_token_len from extra_inputs * Fix Phi3 SwiGLU and SoftMax * Pull out sin cos from rope * Reduce memory: free ov weights node after graph conversion * Fix CPY due to cgraph change * Added OpenVINO CI/CD. Updated docs * Fix llama-cli * Fix Phi3 ROPE; Add test-backend-ops * Fix NPU * Fix llama-bench; Clang-format * Fix llama-perplexity * temp. changes for mark decomp * matmul in fp32 * mulmat input conversion fix * mulmat type conversion update * add mark decomp pass * Revert changes in fuse_to_sdpa * Update build.md * Fix test-backend-ops * Skip test-thread-safety; Run ctest only in ci/run.sh * Use CiD for NPU * Optimize tensor conversion, improve TTFT * Support op SET_ROWS * Fix NPU * Remove CPY * Fix test-backend-ops * Minor updates for raising PR * Perf: RMS fused to OV internal RMS op * Fix after rebasing - Layout of cache k and cache v are unified: [seq, n_head, head_size] - Add CPY and FLASH_ATTN_EXT, flash attn is not used yet - Skip test-backend-ops due to flash attn test crash - Add mutex around graph conversion to avoid test-thread-safety fali in the future - Update NPU config - Update GPU config to disable SDPA opt to make phi-3 run * Change openvino device_type to GPU; Enable flash_attn * Update supports_buft and supports_op for quantized models * Add quant weight conversion functions from genai gguf reader * Quant models run with accuracy issue * Fix accuracy: disable cpu_repack * Fix CI; Disable test-backend-ops * Fix Q4_1 * Fix test-backend-ops: Treat quantized tensors as weights * Add NPU Q4_0 support * NPU perf: eliminate zp * Dequantize q4_1 q4_k q6_k for NPU * Add custom quant type: q8_1_c, q4_0_128 * Set m_is_static=false as default in decoder * Simpilfy translation of get_rows * Fix after rebasing * Improve debug util; Eliminate nop ReshapeReshape * STYLE: make get_types_to_requant a function * Support BF16 model * Fix NPU compile * WA for npu 1st token acc issue * Apply EliminateZP only for npu * Add GeGLU * Fix Hunyuan * Support iSWA * Fix NPU accuracy * Fix ROPE accuracy when freq_scale != 1 * Minor: not add attention_size_swa for non-swa model * Minor refactor * Add Q5_K to support phi-3-q4_k_m * Requantize Q6_K (gs16) to gs32 on GPU * Fix after rebasing * Always apply Eliminate_ZP to fix GPU compile issue on some platforms * kvcachefusion support * env variable GGML_OPENVINO_DISABLE_SDPA_OPTIMIZATION added * Fix for Phi3 * Fix llama-cli (need to run with --no-warmup) * Fix add_sliced_mask; Revert mulmat, softmax; Remove input attention_size, iSWA model not working * fix after rebasing * Fix llama-3-8b and phi3-mini q4_0 NPU * Update to OV-2025.3 and CMakeLists.txt * Add OV CI cache * Apply CISC review and update CI to OV2025.3 * Update CI to run OV dep install before build * Update OV dockerfile to use OV2025.3 and update build docs * Style: use switch in supports_ops * Style: middle ptr and ref align, omit optional struct keyword * NPU Unify PD (#14) * Stateless. Fix llama-cli llama-server * Simplify broadcast op in attention * Replace get_output_tensor+memcpy with set_output_tensor * NPU unify PD. Unify dynamic and static dims * Clean placeholders in ggml-openvino.cpp * NPU unify PD (handled internally) * change graph to 4d, support multi sequences * Fix llama-bench * Fix NPU * Update ggml-decoder.cpp Hitting error while compiling on windows: error C3861: 'unsetenv': identifier not found Reason: unsetenv() is a POSIX function; it doesn’t exist on Windows. Visual Studio (MSVC) won’t recognize it. Proposed fix: Use _putenv_s() (Windows equivalent) This is supported by MSVC and achieves the same effect: it removes the environment variable from the process environment. This keeps cross-platform compatibility. * Update ggml-decoder.cpp * Update ggml-decoder.cpp * Update ggml-decoder.cpp * Update ggml-decoder.cpp * Update ggml-decoder.cpp * Remove the second decoder for node. Moving the function into the model decoder * Fix error for naive * NPU prefill chunking * NPU fix llama-bench * fallback naive run with accuracy issue * NPU support llma-perplexity -b 512 --no-warmup * Refactor: split ov_graph_compute for dynamic and static * remove unused API GgmlOvDecoder::get_output_stride(const std::string & name) * minor update due to ov 2025.4 * remove unused API GgmlOvDecoder::get_output_names() * remove unused API get_output_shape(const std::string & name) * Modified API GgmlOvDecoder::get_output_type(const std::string & name) * Removed API GgmlOvDecoder::get_output_op_params(const std::string & name) * Removed API get_output_ggml_tensor(const std::string & name) * Removed API m_outputs * Removed m_output_names * Removed API GgmlOvDecoder::get_input_names() * Removed API GgmlOvDecoder::get_input_stride(const std::string& name) * Removed API get_input_type * Removed API get_input_type * Removed API GgmlOvDecoder::get_input_shape(const std::string & name) * Removed API GgmlOvDecoder::get_input_op_params(const std::string & name) * Fix error for decoder cache * Reuse cached decoder * GPU remove Q6_K requantization * NPU fix wrong model output shape * NPU fix q4 perf regression * Remove unused variable nodes * Fix decoder can_reuse for llama-bench * Update build.md for Windows * backend buffer: allocate on host * Use shared_buffer for GPU NPU; Refactor * Add ov_backend_host_buffer; Use cached remote context * Put kvcache on GPU * Use ggml_aligned_malloc * only use remote tensor for kvcache * only use remote tensor for kvcache for GPU * FIX: use remote tensor from singleton * Update build.md to include OpenCL * NPU always requant to q4_0_128 * Optimize symmetric quant weight extraction: use single zp * Use Q8_0_C in token embd, lm_head, and for 5 and 6 bits quant * Update build.md * Support -ctk f32 * Initial stateful graph support * Update ggml/src/ggml-openvino/ggml-decoder.cpp Co-authored-by: Yamini Nimmagadda <yamini.nimmagadda@intel.com> * code cleanup * npu perf fix * requant to f16 for Q6 embed on NPU * Update ggml/src/ggml-openvino/ggml-decoder.cpp * Update ggml/src/ggml-openvino/ggml-openvino-extra.cpp * Create OPENVINO.md in llama.cpp backend docs * Update OPENVINO.md * Update OPENVINO.md * Update OPENVINO.md * Update build.md * Update OPENVINO.md * Update OPENVINO.md * Update OPENVINO.md * kq_mask naming fix * Syntax correction for workflows build file * Change ov backend buffer is_host to false * Fix llama-bench -p -n where p<=256 * Fix --direct-io 0 * Don't put kvcache on GPU in stateful mode * Remove hardcode names * Fix stateful shapes * Simplification for stateful and update output shape processing * Remove hardcode names * Avoid re-compilation in llama-bench * Extract zp directly instead of bias * Refactor weight tensor processing * create_weight_node accept non-ov backend buffer * remove changes in llama-graph.cpp * stateful masking fix (#38) Fix for stateful accuracy issues and cl_out_of_resources error in stateful GPU with larger context sizes. * Fix test-backend-ops crash glu, get_rows, scale, rms_norm, add * hardcoded name handling for rope_freqs.weight * Suppress logging and add error handling to allow test-backend-ops to complete * Fix MUL_MAT with broadcast; Add unsupported MUL_MAT FLASH_ATTN cases * Use bias instead of zp in test-backend-ops * Update OV in CI, Add OV CI Tests in GH Actions * Temp fix for multithreading bug * Update OV CI, fix review suggestions. * fix editorconfig-checker, update docs * Fix tabs to spaces for editorconfig-checker * fix editorconfig-checker * Update docs * updated model link to be GGUF model links * Remove GGML_CPU_REPACK=OFF * Skip permuted ADD and MUL * Removed static variables from utils.cpp * Removed initializing non-existing variable * Remove unused structs * Fix test-backend-ops for OV GPU * unify api calling * Update utils.cpp * When the dim is dynamic, throw an error, need to is stastic forst * Add interface compute_model_outputs(), which get the model output through computing the node use count & status in the cgraph to avoid the flag using * No need to return * Fix test-backend-ops for OV GPU LNL * Fix test-thread-safety * use the shape from infer request of output tensor create to avoid issue * fix dynamic output shape issue * fix issue for the unused node in tests * Remove unused lock * Add comment * Update openvino docs * update to OV release version 2026.0 * add ci ov-gpu self hosted runner * fix editorconfig * Fix perplexity * Rewrite the model inputs finding mechanism (#54) * Rewrite the model inputs finding logistic * Put stateful shape handle in get input shape * Put the iteration logistic in func * Added ggml-ci-intel-openvino-gpu and doc update * .hpp files converted to .h * fix ggml-ci-x64-intel-openvino-gpu * Fix for stateful execution bug in llama-bench * Minor updates after stateful llama-bench fix * Update ggml/src/ggml-openvino/utils.cpp Co-authored-by: Yamini Nimmagadda <yamini.nimmagadda@intel.com> * Remove multiple get_shape calls * Bring back mutex into compute * Fix VIEW op, which slice the input node * Added token_len_per_seq existence check before slicing masks and moved node retrieval inside guarded block to prevent missing-key access * Temp. fix for test requant errors * Update to OV ggml-ci to low-perf * ci : temporary disable "test-llama-archs" * ci : cache v4 -> v5, checkout v4 -> v6, fix runner tag * docs : update url * Fix OV link in docker and Update docs --------- Co-authored-by: Ravi Panchumarthy <ravi.panchumarthy@intel.com> Co-authored-by: Cavus Mustafa <mustafa.cavus@intel.com> Co-authored-by: Arshath <arshath.ramzan@intel.com> Co-authored-by: XuejunZhai <Xuejun.Zhai@intel.com> Co-authored-by: Yamini Nimmagadda <yamini.nimmagadda@intel.com> Co-authored-by: Xuejun Zhai <Xuejun.Zhai@intel> Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
This commit is contained in:
Zijun Yu
co-authored by
Yamini Nimmagadda
Ravi Panchumarthy
Cavus Mustafa
Arshath
XuejunZhai
Xuejun Zhai
Georgi Gerganov
Yamini Nimmagadda
Ravi Panchumarthy
Cavus Mustafa
Arshath
XuejunZhai
Xuejun Zhai
Georgi Gerganov
parent
77e20cc107
commit
9789c4ecdc
@@ -0,0 +1,74 @@
|
||||
#pragma once
|
||||
|
||||
#include <cstdint>
|
||||
#include <map>
|
||||
#include <openvino/core/node.hpp>
|
||||
#include <openvino/frontend/decoder.hpp>
|
||||
#include <string>
|
||||
|
||||
namespace ov {
|
||||
namespace frontend {
|
||||
namespace ggml {
|
||||
|
||||
class GgmlDecoder : public DecoderBase {
|
||||
public:
|
||||
virtual ov::Any get_attribute(const std::string& name) const = 0;
|
||||
|
||||
virtual PartialShape get_input_shape(int node_idx, const std::string& name) const = 0;
|
||||
|
||||
virtual std::vector<size_t> get_input_stride(int node_idx, const std::string& name) const = 0;
|
||||
|
||||
virtual element::Type get_input_type(int node_idx, const std::string& name) const = 0;
|
||||
|
||||
virtual size_t get_input_size() const = 0;
|
||||
|
||||
virtual size_t get_input_size(int node_idx) const = 0;
|
||||
|
||||
virtual void get_input_node(size_t input_port_idx,
|
||||
std::string& producer_name,
|
||||
std::string& producer_output_port_name,
|
||||
size_t& producer_output_port_index) const = 0;
|
||||
|
||||
virtual std::vector<std::string> get_input_names(int node_idx) const = 0;
|
||||
|
||||
virtual PartialShape get_output_shape(int node_idx) const = 0;
|
||||
|
||||
virtual element::Type get_output_type(const int node_idx) const = 0;
|
||||
|
||||
virtual int32_t* get_input_op_params(int node_idx, const std::string& name) const = 0;
|
||||
|
||||
virtual int32_t * get_output_op_params(int node_idx) const = 0;
|
||||
|
||||
virtual std::vector<std::string> get_output_names(int node_idx) const = 0;
|
||||
|
||||
virtual const std::string& get_op_type() const = 0;
|
||||
|
||||
virtual const std::string& get_op_type(int node_idx) const = 0;
|
||||
|
||||
virtual const std::string& get_op_name() const = 0;
|
||||
|
||||
virtual const std::string& get_op_name(int node_idx) const = 0;
|
||||
|
||||
virtual void visit_subgraph(std::function<void(std::shared_ptr<GgmlDecoder>, int node_idx)> node_visitor) const = 0;
|
||||
|
||||
virtual int get_op_case(int node_idx) const = 0;
|
||||
|
||||
virtual const std::map<std::string, std::shared_ptr<ov::Node>>& get_model_inputs() const = 0;
|
||||
virtual const std::map<std::string, std::shared_ptr<ov::Node>>& get_model_extra_inputs() const = 0;
|
||||
virtual const std::map<std::string, std::shared_ptr<ov::Node>>& get_model_weights() const = 0;
|
||||
virtual std::vector<std::string> get_model_output_names() const = 0;
|
||||
|
||||
virtual int32_t* get_rope_params() const = 0;
|
||||
|
||||
virtual std::map<std::string, std::string> get_kv_param_res_names() const = 0;
|
||||
|
||||
virtual bool is_static() const = 0;
|
||||
|
||||
virtual bool is_stateful() const = 0;
|
||||
|
||||
virtual int is_swa_layer(int layer) const = 0;
|
||||
};
|
||||
|
||||
} // namespace ggml
|
||||
} // namespace frontend
|
||||
} // namespace ov
|
||||
@@ -0,0 +1,27 @@
|
||||
#include "frontend.h"
|
||||
|
||||
#include "input_model.h"
|
||||
#include "op_table.h"
|
||||
#include "translate_session.h"
|
||||
|
||||
namespace ov {
|
||||
namespace frontend {
|
||||
namespace ggml {
|
||||
|
||||
FrontEnd::FrontEnd() {}
|
||||
|
||||
std::shared_ptr<Model> FrontEnd::convert(const InputModel::Ptr & model, bool naive) {
|
||||
auto ggml_model = std::dynamic_pointer_cast<ggml::InputModel>(model);
|
||||
FRONT_END_GENERAL_CHECK(ggml_model, "Invalid input model");
|
||||
std::shared_ptr<Model> converted_model;
|
||||
const auto & supported_ops = get_supported_ops();
|
||||
{
|
||||
TranslateSession translate_session(model, supported_ops, naive);
|
||||
converted_model = translate_session.get_converted_model();
|
||||
}
|
||||
return converted_model;
|
||||
}
|
||||
|
||||
} // namespace ggml
|
||||
} // namespace frontend
|
||||
} // namespace ov
|
||||
@@ -0,0 +1,23 @@
|
||||
// Copyright (C) 2018-2024 Intel Corporation
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
//
|
||||
|
||||
#pragma once
|
||||
|
||||
#include <openvino/frontend/frontend.hpp>
|
||||
|
||||
namespace ov {
|
||||
namespace frontend {
|
||||
namespace ggml {
|
||||
|
||||
class FrontEnd {
|
||||
public:
|
||||
using Ptr = std::shared_ptr<FrontEnd>;
|
||||
FrontEnd();
|
||||
|
||||
static std::shared_ptr<Model> convert(const InputModel::Ptr& model, bool naive = false);
|
||||
};
|
||||
|
||||
} // namespace ggml
|
||||
} // namespace frontend
|
||||
} // namespace ov
|
||||
@@ -0,0 +1,17 @@
|
||||
#include "input_model.h"
|
||||
|
||||
#include "decoder.h"
|
||||
|
||||
namespace ov {
|
||||
namespace frontend {
|
||||
namespace ggml {
|
||||
|
||||
InputModel::InputModel(const std::shared_ptr<GgmlDecoder> & gdecoder) : m_decoder(gdecoder) {}
|
||||
|
||||
const std::shared_ptr<GgmlDecoder> & InputModel::get_model_decoder() const {
|
||||
return m_decoder;
|
||||
}
|
||||
|
||||
} // namespace ggml
|
||||
} // namespace frontend
|
||||
} // namespace ov
|
||||
@@ -0,0 +1,29 @@
|
||||
#pragma once
|
||||
|
||||
#include <openvino/frontend/input_model.hpp>
|
||||
|
||||
#include "decoder.h"
|
||||
|
||||
namespace ov {
|
||||
namespace frontend {
|
||||
namespace ggml {
|
||||
|
||||
class FrontEnd;
|
||||
class GgmlDecoder;
|
||||
using ov::frontend::ggml::GgmlDecoder;
|
||||
|
||||
class InputModel : public ov::frontend::InputModel {
|
||||
friend class ::ov::frontend::ggml::FrontEnd;
|
||||
|
||||
public:
|
||||
explicit InputModel(const std::shared_ptr<GgmlDecoder>& gdecoder);
|
||||
|
||||
const std::shared_ptr<GgmlDecoder>& get_model_decoder() const;
|
||||
|
||||
private:
|
||||
std::shared_ptr<GgmlDecoder> m_decoder;
|
||||
};
|
||||
|
||||
} // namespace ggml
|
||||
} // namespace frontend
|
||||
} // namespace ov
|
||||
@@ -0,0 +1,112 @@
|
||||
#pragma once
|
||||
|
||||
#include <cstdint>
|
||||
#include <openvino/frontend/node_context.hpp>
|
||||
#include <string>
|
||||
|
||||
#include "decoder.h"
|
||||
|
||||
namespace ov {
|
||||
namespace frontend {
|
||||
namespace ggml {
|
||||
|
||||
class TranslateSession;
|
||||
|
||||
typedef std::map<std::string, Output<Node>> TensorMap;
|
||||
|
||||
class NodeContext : public frontend::NodeContext {
|
||||
public:
|
||||
NodeContext(const std::shared_ptr<GgmlDecoder>& decoder,
|
||||
std::shared_ptr<TensorMap>& tensor_map,
|
||||
int node_idx,
|
||||
TranslateSession* translate_session = nullptr)
|
||||
: ov::frontend::NodeContext(decoder->get_op_type(node_idx)),
|
||||
m_decoder(decoder),
|
||||
m_tensor_map(tensor_map),
|
||||
m_node_idx(node_idx),
|
||||
m_translate_session(translate_session) {
|
||||
m_input_names = decoder->get_input_names(m_node_idx);
|
||||
m_output_names = decoder->get_output_names(m_node_idx);
|
||||
}
|
||||
|
||||
TranslateSession* get_translate_session() const {
|
||||
return m_translate_session;
|
||||
}
|
||||
|
||||
const std::vector<std::string>& get_input_names() const { return m_input_names; }
|
||||
|
||||
size_t get_input_size() const override {
|
||||
return m_decoder->get_input_size(m_node_idx);
|
||||
}
|
||||
|
||||
ov::element::Type get_input_type(size_t index) const {
|
||||
return m_decoder->get_input_type(m_node_idx, m_input_names[index]);
|
||||
}
|
||||
|
||||
PartialShape get_input_shape(size_t input_index) const {
|
||||
return m_decoder->get_input_shape(m_node_idx, m_input_names[input_index]);
|
||||
}
|
||||
|
||||
std::vector<size_t> get_input_stride(size_t index) const {
|
||||
return m_decoder->get_input_stride(m_node_idx, m_input_names[index]);
|
||||
}
|
||||
|
||||
std::string get_output_name() const { return m_output_names[0]; }
|
||||
|
||||
PartialShape get_output_shape() const { return m_decoder->get_output_shape(m_node_idx); }
|
||||
|
||||
int32_t* get_input_op_params(size_t index) const {
|
||||
return m_decoder->get_input_op_params(m_node_idx, m_input_names[index]);
|
||||
}
|
||||
|
||||
int32_t * get_output_op_params() const { return m_decoder->get_output_op_params(m_node_idx); }
|
||||
|
||||
ov::element::Type get_output_type() const {
|
||||
return m_decoder->get_output_type(m_node_idx);
|
||||
}
|
||||
|
||||
Output<Node> get_input(int idx) const override {
|
||||
return m_tensor_map->at(m_input_names[idx]);
|
||||
}
|
||||
|
||||
Output<Node> get_input(const std::string& name) const override {
|
||||
if (m_tensor_map->find(name) == m_tensor_map->end()) {
|
||||
throw std::runtime_error("'" + name + "' not found in tensor map.");
|
||||
}
|
||||
return m_tensor_map->at(name);
|
||||
}
|
||||
|
||||
bool has_input(const std::string& name) const {
|
||||
return m_tensor_map->find(name) != m_tensor_map->end();
|
||||
}
|
||||
|
||||
const std::string& get_name() const override {
|
||||
return m_decoder->get_op_name(m_node_idx);
|
||||
}
|
||||
|
||||
ov::Any get_attribute_as_any(const std::string& name) const override {
|
||||
return m_decoder->get_attribute(name);
|
||||
}
|
||||
|
||||
int get_op_case() const {
|
||||
return m_decoder->get_op_case(m_node_idx);
|
||||
}
|
||||
|
||||
bool is_static() const { return m_decoder->is_static(); }
|
||||
|
||||
bool is_stateful() const { return m_decoder->is_stateful(); }
|
||||
|
||||
private:
|
||||
std::shared_ptr<GgmlDecoder> m_decoder;
|
||||
std::shared_ptr<TensorMap>& m_tensor_map;
|
||||
int m_node_idx;
|
||||
TranslateSession* m_translate_session;
|
||||
std::vector<std::string> m_input_names;
|
||||
std::vector<std::string> m_output_names;
|
||||
};
|
||||
|
||||
using CreatorFunction = std::function<ov::OutputVector(const ov::frontend::ggml::NodeContext&)>;
|
||||
|
||||
} // namespace ggml
|
||||
} // namespace frontend
|
||||
} // namespace ov
|
||||
@@ -0,0 +1,48 @@
|
||||
|
||||
#include "../node_context.h"
|
||||
#include "../op_table.h"
|
||||
#include "../utils.h"
|
||||
|
||||
#include <climits>
|
||||
#include <cstdint>
|
||||
#include <memory>
|
||||
#include <openvino/op/reshape.hpp>
|
||||
#include <openvino/op/slice.hpp>
|
||||
#include <vector>
|
||||
|
||||
namespace ov {
|
||||
namespace frontend {
|
||||
namespace ggml {
|
||||
namespace op {
|
||||
|
||||
OutputVector translate_cont(const NodeContext & context) {
|
||||
num_inputs_check(context, 1, 1);
|
||||
|
||||
int op_case = context.get_op_case();
|
||||
FRONT_END_CHECK_IMPLEMENTED(op_case == 1 || op_case == 2 || op_case == 3, "Unsupported CONT case");
|
||||
|
||||
auto src_shape = context.get_input_shape(0).to_shape();
|
||||
auto dst_shape = context.get_output_shape().to_shape();
|
||||
ov::Output<Node> res;
|
||||
|
||||
if (op_case == 1) {
|
||||
// The input comes from a PERMUTE
|
||||
throw std::runtime_error("Code of this case might be outdated");
|
||||
dst_shape[1] = -1;
|
||||
res = std::make_shared<ov::op::v1::Reshape>(
|
||||
context.get_input(0), ov::op::v0::Constant::create(ov::element::i64, {dst_shape.size()}, dst_shape), false);
|
||||
} else if (op_case == 2) {
|
||||
// The input comes from a TRANSPOSE
|
||||
return {context.get_input(0)};
|
||||
} else {
|
||||
// The input comes from a VIEW
|
||||
res = process_view_input(context, 0);
|
||||
}
|
||||
|
||||
return rename_outputs_with_suffix({res}, context.get_name());
|
||||
}
|
||||
|
||||
} // namespace op
|
||||
} // namespace ggml
|
||||
} // namespace frontend
|
||||
} // namespace ov
|
||||
@@ -0,0 +1,21 @@
|
||||
#include "../node_context.h"
|
||||
#include "../op_table.h"
|
||||
#include "../utils.h"
|
||||
|
||||
#include <memory>
|
||||
#include <openvino/op/convert.hpp>
|
||||
|
||||
namespace ov {
|
||||
namespace frontend {
|
||||
namespace ggml {
|
||||
namespace op {
|
||||
|
||||
OutputVector translate_cpy(const NodeContext & context) {
|
||||
auto res = std::make_shared<ov::op::v0::Convert>(context.get_input(0), context.get_output_type());
|
||||
return rename_outputs_with_suffix({res}, context.get_name());
|
||||
}
|
||||
|
||||
} // namespace op
|
||||
} // namespace ggml
|
||||
} // namespace frontend
|
||||
} // namespace ov
|
||||
@@ -0,0 +1,90 @@
|
||||
#include "../node_context.h"
|
||||
#include "../op_table.h"
|
||||
#include "../utils.h"
|
||||
|
||||
#include <cstdint>
|
||||
#include <memory>
|
||||
#include <openvino/op/broadcast.hpp>
|
||||
#include <openvino/op/concat.hpp>
|
||||
#include <openvino/op/constant.hpp>
|
||||
#include <openvino/op/convert.hpp>
|
||||
#include <openvino/op/reshape.hpp>
|
||||
#include <openvino/op/scaled_dot_product_attention.hpp>
|
||||
#include <openvino/op/transpose.hpp>
|
||||
#include <openvino/op/unsqueeze.hpp>
|
||||
#include <string>
|
||||
|
||||
namespace ov {
|
||||
namespace frontend {
|
||||
namespace ggml {
|
||||
namespace op {
|
||||
|
||||
OutputVector translate_flash_attn_ext(const NodeContext & context) {
|
||||
num_inputs_check(context, 4, 4);
|
||||
auto q_f32 = context.get_input(0);
|
||||
auto k = context.get_input(1);
|
||||
auto v = context.get_input(2);
|
||||
auto mask = context.get_input(3);
|
||||
|
||||
float * params = reinterpret_cast<float *>(context.get_output_op_params());
|
||||
float scale = params[0];
|
||||
// float max_bias = params[1];
|
||||
// float logit_softcap = params[2];
|
||||
|
||||
auto q = std::make_shared<ov::op::v0::Convert>(q_f32, ov::element::f16);
|
||||
auto scale_node = std::make_shared<ov::op::v0::Constant>(ov::element::f16, ov::Shape{}, std::vector<float>{scale});
|
||||
|
||||
ov::Output<ov::Node> mask_sliced, res;
|
||||
std::string mask_name = "KQ_mask_sliced";
|
||||
if (context.get_input_names()[3].find("swa") != std::string::npos) {
|
||||
mask_name = "KQ_mask_swa_sliced";
|
||||
}
|
||||
if (context.has_input(mask_name)) {
|
||||
mask_sliced = context.get_input(mask_name);
|
||||
} else {
|
||||
auto zero = ov::op::v0::Constant::create(ov::element::i64, {1}, {0});
|
||||
auto one = ov::op::v0::Constant::create(ov::element::i64, {1}, {1});
|
||||
auto two = ov::op::v0::Constant::create(ov::element::i64, {1}, {2});
|
||||
auto token_len = get_dimensions(q, {2});
|
||||
mask_sliced = std::make_shared<ov::op::v8::Slice>(mask, zero, token_len, one, two);
|
||||
}
|
||||
|
||||
if (mask_sliced.get_element_type() != ov::element::f16) {
|
||||
mask_sliced = std::make_shared<ov::op::v0::Convert>(mask_sliced, ov::element::f16);
|
||||
}
|
||||
|
||||
auto tile_kv = [&](int64_t num_heads, int64_t num_heads_kv, int64_t head_size, ov::Output<Node> kv) {
|
||||
int64_t factor = num_heads / num_heads_kv;
|
||||
if (factor > 1 && num_heads_kv > 1) {
|
||||
ov::Output<ov::Node> kv_broadcast_shape, kv_unsqueezed, new_kv_shape;
|
||||
auto unsqueeze_axes = ov::op::v0::Constant::create(ov::element::i64, Shape{}, {2});
|
||||
kv_unsqueezed = std::make_shared<ov::op::v0::Unsqueeze>(kv, unsqueeze_axes);
|
||||
|
||||
kv_broadcast_shape = ov::op::v0::Constant::create(
|
||||
ov::element::i64, {5}, {(int64_t) 1, (int64_t) 1, factor, (int64_t) 1, (int64_t) 1});
|
||||
new_kv_shape =
|
||||
ov::op::v0::Constant::create(ov::element::i64, {4}, {(int64_t) 0, num_heads, (int64_t) -1, head_size});
|
||||
|
||||
kv = std::make_shared<ov::op::v3::Broadcast>(kv_unsqueezed, kv_broadcast_shape,
|
||||
ov::op::BroadcastType::BIDIRECTIONAL);
|
||||
kv = std::make_shared<ov::op::v1::Reshape>(kv, new_kv_shape, true);
|
||||
}
|
||||
return kv;
|
||||
};
|
||||
|
||||
auto q_shape = context.get_input_shape(0).to_shape();
|
||||
auto k_shape = context.get_input_shape(1).to_shape();
|
||||
k = tile_kv(q_shape[1], k_shape[1], q_shape[3], k);
|
||||
v = tile_kv(q_shape[1], k_shape[1], q_shape[3], v);
|
||||
|
||||
auto sdpa = std::make_shared<ov::op::v13::ScaledDotProductAttention>(q, k, v, mask_sliced, scale_node, false);
|
||||
res = std::make_shared<ov::op::v1::Transpose>(sdpa,
|
||||
ov::op::v0::Constant::create(ov::element::i64, {4}, {0, 2, 1, 3}));
|
||||
res = std::make_shared<ov::op::v0::Convert>(res, ov::element::f32);
|
||||
return rename_outputs_with_suffix({res}, context.get_name());
|
||||
}
|
||||
|
||||
} // namespace op
|
||||
} // namespace ggml
|
||||
} // namespace frontend
|
||||
} // namespace ov
|
||||
@@ -0,0 +1,69 @@
|
||||
#include "../node_context.h"
|
||||
#include "../op_table.h"
|
||||
#include "../utils.h"
|
||||
|
||||
#include <openvino/core/node.hpp>
|
||||
#include <openvino/core/node_output.hpp>
|
||||
#include <openvino/op/constant.hpp>
|
||||
#include <openvino/op/convert.hpp>
|
||||
#include <openvino/op/gather.hpp>
|
||||
#include <openvino/op/squeeze.hpp>
|
||||
#include <openvino/op/unsqueeze.hpp>
|
||||
|
||||
namespace ov {
|
||||
namespace frontend {
|
||||
namespace ggml {
|
||||
namespace op {
|
||||
|
||||
OutputVector translate_get_rows(const NodeContext & context) {
|
||||
num_inputs_check(context, 2, 2);
|
||||
|
||||
int op_case = context.get_op_case();
|
||||
|
||||
Output<Node> res;
|
||||
auto data = context.get_input(0);
|
||||
auto indices = context.get_input(1);
|
||||
|
||||
if (op_case == 2) {
|
||||
// The input comes from a VIEW
|
||||
indices = process_view_input(context, 1);
|
||||
}
|
||||
|
||||
// data[1,b,x,y] ind[1,1,b,x'] test-backend-ops case
|
||||
// data[x,y] ind[1,1,1,x'] normal case
|
||||
indices =
|
||||
std::make_shared<ov::op::v0::Squeeze>(indices, ov::op::v0::Constant::create(ov::element::i64, {2}, {0, 1}));
|
||||
if (data.get_partial_shape().rank() == 4) {
|
||||
if (!(data.get_partial_shape()[1].is_dynamic()) && data.get_partial_shape()[1].get_length() == 1) {
|
||||
// Work-around for a bug in ov cpu plugin for test-backend-ops
|
||||
data = std::make_shared<ov::op::v0::Squeeze>(data,
|
||||
ov::op::v0::Constant::create(ov::element::i64, {2}, {0, 1}));
|
||||
auto axis = ov::op::v0::Constant::create(ov::element::i32, ov::Shape{}, {0});
|
||||
res = std::make_shared<ov::op::v8::Gather>(data, indices, axis);
|
||||
} else {
|
||||
auto axis = ov::op::v0::Constant::create(ov::element::i32, ov::Shape{}, {1});
|
||||
data =
|
||||
std::make_shared<ov::op::v0::Squeeze>(data, ov::op::v0::Constant::create(ov::element::i64, {1}, {0}));
|
||||
res = std::make_shared<ov::op::v8::Gather>(data, indices, axis, 1);
|
||||
}
|
||||
} else if (context.is_stateful() && data.get_partial_shape().rank() == 3) {
|
||||
auto axis = ov::op::v0::Constant::create(ov::element::i32, ov::Shape{}, {1});
|
||||
res = std::make_shared<ov::op::v8::Gather>(data, indices, axis, 1);
|
||||
} else {
|
||||
auto axis = ov::op::v0::Constant::create(ov::element::i32, ov::Shape{}, {0});
|
||||
res = std::make_shared<ov::op::v8::Gather>(data, indices, axis);
|
||||
}
|
||||
|
||||
if (res.get_element_type() != context.get_output_type()) {
|
||||
res = std::make_shared<ov::op::v0::Convert>(res, context.get_output_type());
|
||||
}
|
||||
if (!(context.is_stateful())) {
|
||||
res = std::make_shared<ov::op::v0::Unsqueeze>(res, ov::op::v0::Constant::create(ov::element::i64, {1}, {0}));
|
||||
}
|
||||
return rename_outputs_with_suffix({res}, context.get_name());
|
||||
}
|
||||
|
||||
} // namespace op
|
||||
} // namespace ggml
|
||||
} // namespace frontend
|
||||
} // namespace ov
|
||||
@@ -0,0 +1,61 @@
|
||||
#include "../node_context.h"
|
||||
#include "../op_table.h"
|
||||
#include "../utils.h"
|
||||
|
||||
#include <memory>
|
||||
#include <openvino/core/node_output.hpp>
|
||||
#include <openvino/op/constant.hpp>
|
||||
#include <openvino/op/gelu.hpp>
|
||||
#include <openvino/op/multiply.hpp>
|
||||
#include <openvino/op/sigmoid.hpp>
|
||||
#include <openvino/op/slice.hpp>
|
||||
|
||||
namespace ov {
|
||||
namespace frontend {
|
||||
namespace ggml {
|
||||
namespace op {
|
||||
|
||||
OutputVector translate_glu_geglu(const NodeContext & context) {
|
||||
num_inputs_check(context, 1, 2);
|
||||
|
||||
ov::Output<ov::Node> src0;
|
||||
ov::Output<ov::Node> src1;
|
||||
if (context.get_input_size() == 2) {
|
||||
src0 = context.get_input(0);
|
||||
src1 = context.get_input(1);
|
||||
} else {
|
||||
// GGML splits along ne[0] (OV last axis) using floor division: nc = ne[0] / 2.
|
||||
// Both halves are nc elements; if the dimension is odd, the last element is dropped.
|
||||
// Use Slice instead of Split to handle odd dimensions correctly.
|
||||
auto combined = context.get_input(0);
|
||||
auto combined_shape = combined.get_partial_shape();
|
||||
int64_t last_dim_val = combined_shape[combined_shape.rank().get_length() - 1].get_length();
|
||||
int64_t nc = last_dim_val / 2;
|
||||
|
||||
auto axis = ov::op::v0::Constant::create(ov::element::i64, {1}, {-1});
|
||||
auto step = ov::op::v0::Constant::create(ov::element::i64, {1}, {1});
|
||||
auto start0 = ov::op::v0::Constant::create(ov::element::i64, {1}, {0});
|
||||
auto stop0 = ov::op::v0::Constant::create(ov::element::i64, {1}, {nc});
|
||||
auto start1 = ov::op::v0::Constant::create(ov::element::i64, {1}, {nc});
|
||||
auto stop1 = ov::op::v0::Constant::create(ov::element::i64, {1}, {2 * nc});
|
||||
|
||||
src0 = std::make_shared<ov::op::v8::Slice>(combined, start0, stop0, step, axis);
|
||||
src1 = std::make_shared<ov::op::v8::Slice>(combined, start1, stop1, step, axis);
|
||||
}
|
||||
|
||||
int32_t * params = context.get_output_op_params();
|
||||
const int32_t swapped = params[1];
|
||||
if (swapped) {
|
||||
std::swap(src0, src1);
|
||||
}
|
||||
|
||||
auto gelu = std::make_shared<ov::op::v7::Gelu>(src0);
|
||||
auto res = std::make_shared<ov::op::v1::Multiply>(gelu, src1);
|
||||
|
||||
return rename_outputs_with_suffix({res}, context.get_name());
|
||||
}
|
||||
|
||||
} // namespace op
|
||||
} // namespace ggml
|
||||
} // namespace frontend
|
||||
} // namespace ov
|
||||
@@ -0,0 +1,62 @@
|
||||
#include "../node_context.h"
|
||||
#include "../op_table.h"
|
||||
#include "../utils.h"
|
||||
|
||||
#include <cstdint>
|
||||
#include <memory>
|
||||
#include <openvino/core/node_output.hpp>
|
||||
#include <openvino/op/constant.hpp>
|
||||
#include <openvino/op/multiply.hpp>
|
||||
#include <openvino/op/sigmoid.hpp>
|
||||
#include <openvino/op/slice.hpp>
|
||||
|
||||
namespace ov {
|
||||
namespace frontend {
|
||||
namespace ggml {
|
||||
namespace op {
|
||||
|
||||
OutputVector translate_glu_swiglu(const NodeContext & context) {
|
||||
num_inputs_check(context, 1, 2);
|
||||
|
||||
ov::Output<ov::Node> src0;
|
||||
ov::Output<ov::Node> src1;
|
||||
if (context.get_input_size() == 2) {
|
||||
src0 = context.get_input(0);
|
||||
src1 = context.get_input(1);
|
||||
} else {
|
||||
// GGML splits along ne[0] (OV last axis) using floor division: nc = ne[0] / 2.
|
||||
// Both halves are nc elements; if the dimension is odd, the last element is dropped.
|
||||
// Use Slice instead of Split to handle odd dimensions correctly.
|
||||
auto combined = context.get_input(0);
|
||||
auto combined_shape = combined.get_partial_shape();
|
||||
int64_t last_dim_val = combined_shape[combined_shape.rank().get_length() - 1].get_length();
|
||||
int64_t nc = last_dim_val / 2;
|
||||
|
||||
auto axis = ov::op::v0::Constant::create(ov::element::i64, {1}, {-1});
|
||||
auto step = ov::op::v0::Constant::create(ov::element::i64, {1}, {1});
|
||||
auto start0 = ov::op::v0::Constant::create(ov::element::i64, {1}, {0});
|
||||
auto stop0 = ov::op::v0::Constant::create(ov::element::i64, {1}, {nc});
|
||||
auto start1 = ov::op::v0::Constant::create(ov::element::i64, {1}, {nc});
|
||||
auto stop1 = ov::op::v0::Constant::create(ov::element::i64, {1}, {2 * nc});
|
||||
|
||||
src0 = std::make_shared<ov::op::v8::Slice>(combined, start0, stop0, step, axis);
|
||||
src1 = std::make_shared<ov::op::v8::Slice>(combined, start1, stop1, step, axis);
|
||||
}
|
||||
|
||||
int32_t * params = context.get_output_op_params();
|
||||
const int32_t swapped = params[1];
|
||||
if (swapped) {
|
||||
std::swap(src0, src1);
|
||||
}
|
||||
|
||||
auto sigmoid = std::make_shared<ov::op::v0::Sigmoid>(src0);
|
||||
auto silu = std::make_shared<ov::op::v1::Multiply>(src0, sigmoid);
|
||||
auto res = std::make_shared<ov::op::v1::Multiply>(silu, src1);
|
||||
|
||||
return rename_outputs_with_suffix({res}, context.get_name());
|
||||
}
|
||||
|
||||
} // namespace op
|
||||
} // namespace ggml
|
||||
} // namespace frontend
|
||||
} // namespace ov
|
||||
@@ -0,0 +1,90 @@
|
||||
#include "../node_context.h"
|
||||
#include "../op_table.h"
|
||||
#include "../utils.h"
|
||||
|
||||
#include <climits>
|
||||
#include <cstdint>
|
||||
#include <memory>
|
||||
#include <openvino/core/node.hpp>
|
||||
#include <openvino/core/node_output.hpp>
|
||||
#include <openvino/op/broadcast.hpp>
|
||||
#include <openvino/op/concat.hpp>
|
||||
#include <openvino/op/constant.hpp>
|
||||
#include <openvino/op/convert.hpp>
|
||||
#include <openvino/op/matmul.hpp>
|
||||
#include <openvino/op/reshape.hpp>
|
||||
#include <openvino/op/slice.hpp>
|
||||
#include <openvino/op/transpose.hpp>
|
||||
#include <openvino/op/unsqueeze.hpp>
|
||||
#include <openvino/op/util/op_types.hpp>
|
||||
#include <vector>
|
||||
|
||||
namespace ov {
|
||||
namespace frontend {
|
||||
namespace ggml {
|
||||
namespace op {
|
||||
|
||||
OutputVector translate_mulmat(const NodeContext & context) {
|
||||
num_inputs_check(context, 2, 2);
|
||||
|
||||
int op_case = context.get_op_case();
|
||||
|
||||
ov::Output<Node> res;
|
||||
ov::Output<ov::Node> B = context.get_input(0);
|
||||
ov::Output<ov::Node> A = context.get_input(1);
|
||||
|
||||
bool transpose_b = true;
|
||||
if (op_case == 2) {
|
||||
B = B.get_node_shared_ptr()->input_value(0);
|
||||
transpose_b = false;
|
||||
} else if (op_case == 3) {
|
||||
B = process_view_input(context, 0);
|
||||
A = process_view_input(context, 1);
|
||||
}
|
||||
if (A.get_element_type() != B.get_element_type()) {
|
||||
B = std::make_shared<ov::op::v0::Convert>(context.get_input(0), context.get_input_type(1));
|
||||
}
|
||||
|
||||
auto B_shape = context.get_input_shape(0).to_shape();
|
||||
auto A_shape = context.get_input_shape(1).to_shape();
|
||||
int64_t A_batch = A_shape[1];
|
||||
int64_t B_batch = B_shape[1];
|
||||
|
||||
auto A_batch_larger = A_batch > B_batch;
|
||||
auto batch_large = A_batch_larger ? A_batch : B_batch;
|
||||
auto batch_small = A_batch_larger ? B_batch : A_batch;
|
||||
|
||||
Output<Node> Z = A_batch_larger ? B : A;
|
||||
int64_t factor = batch_large / batch_small;
|
||||
if (factor > 1 && batch_small > 1) {
|
||||
auto batch_large_node = ov::op::v0::Constant::create(ov::element::i64, {1}, std::vector<int64_t>{batch_large});
|
||||
auto batch_small_node = ov::op::v0::Constant::create(ov::element::i64, {1}, std::vector<int64_t>{batch_small});
|
||||
auto factor_node = ov::op::v0::Constant::create(ov::element::i64, {1}, std::vector<int64_t>{factor});
|
||||
|
||||
auto unsqueeze_axes = ov::op::v0::Constant::create(ov::element::i64, Shape{}, {2});
|
||||
auto Z_unsqueezed = std::make_shared<ov::op::v0::Unsqueeze>(Z, unsqueeze_axes);
|
||||
|
||||
auto broadcast_shape = ov::op::v0::Constant::create(
|
||||
ov::element::i64, {5}, {(int64_t) 1, (int64_t) 1, factor, (int64_t) 1, (int64_t) 1});
|
||||
auto new_Z_shape = ov::op::v0::Constant::create(ov::element::i64, {4},
|
||||
{(int64_t) 0, batch_large, (int64_t) -1, (int64_t) A_shape[3]});
|
||||
|
||||
auto Z_broadcasted = std::make_shared<ov::op::v3::Broadcast>(Z_unsqueezed, broadcast_shape,
|
||||
ov::op::BroadcastType::BIDIRECTIONAL);
|
||||
Z = std::make_shared<ov::op::v1::Reshape>(Z_broadcasted, new_Z_shape, true);
|
||||
}
|
||||
if (A_batch_larger) {
|
||||
B = Z;
|
||||
} else {
|
||||
A = Z;
|
||||
}
|
||||
|
||||
res = std::make_shared<ov::op::v0::MatMul>(A, B, false, transpose_b);
|
||||
|
||||
return rename_outputs_with_suffix({res}, context.get_name());
|
||||
}
|
||||
|
||||
} // namespace op
|
||||
} // namespace ggml
|
||||
} // namespace frontend
|
||||
} // namespace ov
|
||||
@@ -0,0 +1,102 @@
|
||||
#include "../node_context.h"
|
||||
#include "../op_table.h"
|
||||
#include "../utils.h"
|
||||
|
||||
#include <climits>
|
||||
#include <cstdint>
|
||||
#include <memory>
|
||||
#include <openvino/core/node.hpp>
|
||||
#include <openvino/op/add.hpp>
|
||||
#include <openvino/op/concat.hpp>
|
||||
#include <openvino/op/constant.hpp>
|
||||
#include <openvino/op/reshape.hpp>
|
||||
#include <openvino/op/slice.hpp>
|
||||
#include <openvino/op/transpose.hpp>
|
||||
|
||||
namespace ov {
|
||||
namespace frontend {
|
||||
namespace ggml {
|
||||
namespace op {
|
||||
|
||||
OutputVector translate_permute(const NodeContext & context) {
|
||||
num_inputs_check(context, 1, 1);
|
||||
|
||||
int op_case = context.get_op_case();
|
||||
FRONT_END_CHECK_IMPLEMENTED(op_case == 1 || op_case == 2 || op_case == 3 || op_case == 4,
|
||||
"Unsupported PERMUTE case");
|
||||
|
||||
ov::Output<Node> res;
|
||||
auto src = context.get_input(0);
|
||||
auto perm = ov::op::v0::Constant::create(ov::element::i64, {4}, {0, 2, 1, 3});
|
||||
|
||||
if (op_case == 1 || context.is_stateful()) {
|
||||
res = std::make_shared<ov::op::v1::Transpose>(src, perm);
|
||||
} else if (op_case == 4) {
|
||||
auto output_shape = context.get_output_shape().to_shape();
|
||||
auto n_heads = ov::op::v0::Constant::create(ov::element::i64, {1}, {output_shape[1]});
|
||||
auto head_size = ov::op::v0::Constant::create(ov::element::i64, {1}, {output_shape[3]});
|
||||
auto n_seq_active = context.has_input("n_seq_active") ?
|
||||
context.get_input("n_seq_active") :
|
||||
ov::op::v0::Constant::create(ov::element::i64, {1}, {output_shape[0]});
|
||||
auto neg_one = ov::op::v0::Constant::create(ov::element::i64, {1}, {-1});
|
||||
|
||||
auto new_shape =
|
||||
std::make_shared<ov::op::v0::Concat>(ov::OutputVector{n_seq_active, neg_one, n_heads, head_size}, 0);
|
||||
|
||||
// // Alternative
|
||||
// auto zero = ov::op::v0::Constant::create(ov::element::i64, {1}, {0});
|
||||
// auto new_shape = std::make_shared<ov::op::v0::Concat>(ov::OutputVector{n_seq_active, neg_one, zero, zero}, 0);
|
||||
|
||||
auto reshaped = std::make_shared<ov::op::v1::Reshape>(src, new_shape, true);
|
||||
res = std::make_shared<ov::op::v1::Transpose>(reshaped, perm);
|
||||
} else {
|
||||
auto cache_shape = src.get_partial_shape();
|
||||
auto output_shape = context.get_output_shape().to_shape();
|
||||
int64_t head_size = output_shape[3];
|
||||
int64_t n_heads = output_shape[1];
|
||||
int64_t ctx_per_seq = cache_shape[2].is_static() ? cache_shape[2].get_length() : -1;
|
||||
int64_t n_seq = cache_shape[1].get_length();
|
||||
|
||||
Output<Node> attention_size;
|
||||
if (!context.has_input("attention_size")) {
|
||||
attention_size = ov::op::v0::Constant::create(ov::element::i64, {1}, {output_shape[2]});
|
||||
} else if (op_case == 2) {
|
||||
attention_size = context.get_input("attention_size");
|
||||
} else {
|
||||
attention_size = context.get_input("attention_size_swa");
|
||||
}
|
||||
|
||||
Output<Node> seq_active_start;
|
||||
Output<Node> seq_active_end;
|
||||
if (context.has_input("seq_active_start")) {
|
||||
seq_active_start = context.get_input("seq_active_start");
|
||||
seq_active_end = context.get_input("seq_active_end");
|
||||
} else {
|
||||
int64_t n_seq_active = output_shape[0];
|
||||
size_t offset = *((size_t *) context.get_input_op_params(0));
|
||||
int64_t seq_active_start_val = offset / context.get_input_stride(0)[0];
|
||||
int64_t seq_active_end_val = seq_active_start_val + n_seq_active;
|
||||
seq_active_start = ov::op::v0::Constant::create(ov::element::i64, {1}, {seq_active_start_val});
|
||||
seq_active_end = ov::op::v0::Constant::create(ov::element::i64, {1}, {seq_active_end_val});
|
||||
}
|
||||
|
||||
// 1. reshape to [n_seq, ctx_per_seq, n_heads, head_size]
|
||||
// 2. slice out the active sequences
|
||||
// 3. slice out the attention part in each sequence
|
||||
// 4. permute
|
||||
auto zero = ov::op::v0::Constant::create(ov::element::i64, {1}, {0});
|
||||
auto one = ov::op::v0::Constant::create(ov::element::i64, {1}, {1});
|
||||
|
||||
auto src_reshaped = std::make_shared<ov::op::v1::Reshape>(
|
||||
src, ov::op::v0::Constant::create(ov::element::i64, {4}, {n_seq, ctx_per_seq, n_heads, head_size}), false);
|
||||
auto slice1 = std::make_shared<ov::op::v8::Slice>(src_reshaped, seq_active_start, seq_active_end, one, zero);
|
||||
auto slice2 = std::make_shared<ov::op::v8::Slice>(slice1, zero, attention_size, one, one);
|
||||
res = std::make_shared<ov::op::v1::Transpose>(slice2, perm);
|
||||
}
|
||||
return rename_outputs_with_suffix({res}, context.get_name());
|
||||
}
|
||||
|
||||
} // namespace op
|
||||
} // namespace ggml
|
||||
} // namespace frontend
|
||||
} // namespace ov
|
||||
@@ -0,0 +1,83 @@
|
||||
#include "../node_context.h"
|
||||
#include "../op_table.h"
|
||||
#include "../utils.h"
|
||||
|
||||
#include <cstdint>
|
||||
#include <memory>
|
||||
#include <openvino/core/node.hpp>
|
||||
#include <openvino/core/node_output.hpp>
|
||||
#include <openvino/frontend/exception.hpp>
|
||||
#include <openvino/op/concat.hpp>
|
||||
#include <openvino/op/constant.hpp>
|
||||
#include <openvino/op/reshape.hpp>
|
||||
#include <stdexcept>
|
||||
#include <vector>
|
||||
|
||||
namespace ov {
|
||||
namespace frontend {
|
||||
namespace ggml {
|
||||
namespace op {
|
||||
|
||||
OutputVector translate_reshape(const NodeContext & context) {
|
||||
num_inputs_check(context, 1, 1);
|
||||
if (context.get_input_shape(0) == context.get_output_shape()) {
|
||||
return {context.get_input(0)};
|
||||
}
|
||||
|
||||
int op_case = context.get_op_case();
|
||||
FRONT_END_CHECK_IMPLEMENTED(
|
||||
op_case == 1 || op_case == 2 || op_case == 3 || op_case == 4 || op_case == 5 || op_case == 6,
|
||||
"Unsupported RESHAPE case");
|
||||
|
||||
auto output_shape = context.get_output_shape().to_shape();
|
||||
std::shared_ptr<ov::Node> new_shape_node;
|
||||
if (op_case == 1) {
|
||||
if (context.is_stateful()) {
|
||||
new_shape_node = ov::op::v0::Constant::create(
|
||||
ov::element::i64, {3},
|
||||
std::vector<int64_t>{-1, (int64_t) output_shape[2], (int64_t) output_shape[3]});
|
||||
} else {
|
||||
new_shape_node = ov::op::v0::Constant::create(
|
||||
ov::element::i64, {4},
|
||||
std::vector<int64_t>{(int64_t) output_shape[0], -1, (int64_t) output_shape[2], (int64_t) output_shape[3]});
|
||||
}
|
||||
} else if (op_case == 2) {
|
||||
new_shape_node = ov::op::v0::Constant::create(
|
||||
ov::element::i64, {4},
|
||||
std::vector<int64_t>{(int64_t) output_shape[0], (int64_t) output_shape[1], -1, (int64_t) output_shape[3]});
|
||||
|
||||
} else if (op_case == 3) {
|
||||
throw std::runtime_error("might be outdated RESHAPE case");
|
||||
new_shape_node = ov::op::v0::Constant::create(
|
||||
ov::element::i64, {4}, std::vector<int64_t>{(int64_t) output_shape[0], (int64_t) output_shape[1], -1, 1});
|
||||
|
||||
} else if (op_case == 4) {
|
||||
return {context.get_input(0).get_node_shared_ptr()->input_value(0)};
|
||||
|
||||
} else if (op_case == 5) {
|
||||
if (context.is_stateful()) {
|
||||
std::vector<int64_t> shape_vec = {1, -1, (int64_t) context.get_output_shape().to_shape()[3]};
|
||||
new_shape_node = ov::op::v0::Constant::create(ov::element::i64, {3}, shape_vec);
|
||||
} else {
|
||||
std::vector<int64_t> shape_vec = {1, 1, -1, (int64_t) context.get_output_shape().to_shape()[3]};
|
||||
new_shape_node = ov::op::v0::Constant::create(ov::element::i64, {4}, shape_vec);
|
||||
}
|
||||
|
||||
// // Alternative
|
||||
// auto token_len = context.get_input("token_len");
|
||||
// auto emb_size =
|
||||
// ov::op::v0::Constant::create(ov::element::i64, {1}, {(int64_t) context.get_output_shape().to_shape()[3]});
|
||||
// auto one = ov::op::v0::Constant::create(ov::element::i64, {1}, {1});
|
||||
// new_shape_node = std::make_shared<ov::op::v0::Concat>(ov::OutputVector{one, one, token_len, emb_size}, 0);
|
||||
|
||||
} else if (op_case == 6) {
|
||||
new_shape_node = ov::op::v0::Constant::create(ov::element::i64, {4}, context.get_output_shape().to_shape());
|
||||
}
|
||||
auto res = std::make_shared<ov::op::v1::Reshape>(context.get_input(0), new_shape_node, false);
|
||||
return rename_outputs_with_suffix({res}, context.get_name());
|
||||
}
|
||||
|
||||
} // namespace op
|
||||
} // namespace ggml
|
||||
} // namespace frontend
|
||||
} // namespace ov
|
||||
@@ -0,0 +1,46 @@
|
||||
#include "../node_context.h"
|
||||
#include "../op_table.h"
|
||||
#include "../utils.h"
|
||||
|
||||
#include <memory>
|
||||
#include <openvino/op/add.hpp>
|
||||
#include <openvino/op/constant.hpp>
|
||||
#include <openvino/op/divide.hpp>
|
||||
#include <openvino/op/multiply.hpp>
|
||||
#include <openvino/op/power.hpp>
|
||||
#include <openvino/op/reduce_mean.hpp>
|
||||
#include <openvino/op/sqrt.hpp>
|
||||
|
||||
namespace ov {
|
||||
namespace frontend {
|
||||
namespace ggml {
|
||||
namespace op {
|
||||
|
||||
OutputVector translate_rms_norm(const NodeContext & context) {
|
||||
num_inputs_check(context, 1, 1);
|
||||
|
||||
auto input_node = context.get_input(0);
|
||||
auto square = std::make_shared<ov::op::v1::Power>(
|
||||
input_node, ov::op::v0::Constant::create(ov::element::f32, ov::Shape{1}, {2.0f}));
|
||||
|
||||
auto mean = std::make_shared<ov::op::v1::ReduceMean>(
|
||||
square, ov::op::v0::Constant::create(ov::element::i64, ov::Shape{1}, {-1}), true);
|
||||
|
||||
float eps;
|
||||
memcpy(&eps, context.get_output_op_params(), sizeof(float));
|
||||
|
||||
auto rms = std::make_shared<ov::op::v0::Sqrt>(
|
||||
std::make_shared<ov::op::v1::Add>(mean, ov::op::v0::Constant::create(ov::element::f32, ov::Shape{1}, {eps})));
|
||||
|
||||
auto reciprocal =
|
||||
std::make_shared<ov::op::v1::Divide>(ov::op::v0::Constant::create(ov::element::f32, ov::Shape{1}, {1.0f}), rms);
|
||||
|
||||
auto res = std::make_shared<ov::op::v1::Multiply>(input_node, reciprocal);
|
||||
|
||||
return rename_outputs_with_suffix({res}, context.get_name());
|
||||
}
|
||||
|
||||
} // namespace op
|
||||
} // namespace ggml
|
||||
} // namespace frontend
|
||||
} // namespace ov
|
||||
@@ -0,0 +1,123 @@
|
||||
#include "../node_context.h"
|
||||
#include "../op_table.h"
|
||||
#include "../utils.h"
|
||||
|
||||
#include <cstdint>
|
||||
#include <memory>
|
||||
#include <openvino/core/node.hpp>
|
||||
#include <openvino/core/node_output.hpp>
|
||||
#include <openvino/op/add.hpp>
|
||||
#include <openvino/op/concat.hpp>
|
||||
#include <openvino/op/constant.hpp>
|
||||
#include <openvino/op/multiply.hpp>
|
||||
#include <openvino/op/reshape.hpp>
|
||||
#include <openvino/op/shape_of.hpp>
|
||||
#include <openvino/op/slice.hpp>
|
||||
#include <openvino/op/split.hpp>
|
||||
#include <openvino/op/subtract.hpp>
|
||||
#include <openvino/op/unsqueeze.hpp>
|
||||
#include <vector>
|
||||
|
||||
namespace ov {
|
||||
namespace frontend {
|
||||
namespace ggml {
|
||||
namespace op {
|
||||
|
||||
OutputVector translate_rope(const NodeContext & context) {
|
||||
num_inputs_check(context, 2, 3);
|
||||
|
||||
int op_case = context.get_op_case();
|
||||
|
||||
ov::Output<Node> res;
|
||||
|
||||
auto data_node = context.get_input(0).get_node_shared_ptr();
|
||||
auto output_shape = context.get_output_shape().to_shape();
|
||||
int32_t * op_params = context.get_output_op_params();
|
||||
|
||||
Output<Node> cos_theta_node;
|
||||
Output<Node> sin_theta_node;
|
||||
if (context.has_input("rope_cos")) {
|
||||
cos_theta_node = context.get_input("rope_cos");
|
||||
sin_theta_node = context.get_input("rope_sin");
|
||||
} else {
|
||||
auto inp_pos = context.get_input(1).get_node_shared_ptr();
|
||||
std::shared_ptr<ov::Node> rope_freqs_weight;
|
||||
if (context.get_input_size() == 3) {
|
||||
rope_freqs_weight = context.get_input(2).get_node_shared_ptr();
|
||||
}
|
||||
auto sin_cos = make_sin_cos(op_params, inp_pos, rope_freqs_weight);
|
||||
sin_theta_node = sin_cos.first;
|
||||
cos_theta_node = sin_cos.second;
|
||||
}
|
||||
|
||||
if (op_case == 2) {
|
||||
// The input comes from a VIEW
|
||||
int slice_len = output_shape[2] * output_shape[3];
|
||||
data_node = process_view_input(context, 0, slice_len).get_node_shared_ptr();
|
||||
if (context.is_stateful()) {
|
||||
auto data_shape = ov::op::v0::Constant::create(
|
||||
ov::element::i64, {3}, std::vector<int64_t>{-1, (int64_t) output_shape[2], (int64_t) output_shape[3]});
|
||||
data_node = std::make_shared<ov::op::v1::Reshape>(data_node, data_shape, false);
|
||||
} else {
|
||||
auto data_shape = ov::op::v0::Constant::create(
|
||||
ov::element::i64, {4}, std::vector<int64_t>{1, -1, (int64_t) output_shape[2], (int64_t) output_shape[3]});
|
||||
data_node = std::make_shared<ov::op::v1::Reshape>(data_node, data_shape, false);
|
||||
}
|
||||
}
|
||||
|
||||
const int mode = op_params[2];
|
||||
constexpr int ROPE_TYPE_NORMAL = 0;
|
||||
constexpr int ROPE_TYPE_NEOX = 2;
|
||||
|
||||
if (mode == ROPE_TYPE_NORMAL) {
|
||||
auto neg_one = ov::op::v0::Constant::create(ov::element::i64, {1}, {-1});
|
||||
auto zero = ov::op::v0::Constant::create(ov::element::i64, {1}, {0});
|
||||
auto one = ov::op::v0::Constant::create(ov::element::i64, {1}, {1});
|
||||
auto two = ov::op::v0::Constant::create(ov::element::i64, {1}, {2});
|
||||
auto end = ov::op::v0::Constant::create(ov::element::i64, {1}, {output_shape[3]});
|
||||
Output<Node> even_slice;
|
||||
Output<Node> odd_slice;
|
||||
int32_t unsqueeze_dim = context.is_stateful() ? 3 : 4;
|
||||
even_slice = std::make_shared<ov::op::v8::Slice>(data_node, zero, end, two, neg_one);
|
||||
odd_slice = std::make_shared<ov::op::v8::Slice>(data_node, one, end, two, neg_one);
|
||||
|
||||
Output<Node> first_half =
|
||||
std::make_shared<ov::op::v1::Subtract>(std::make_shared<ov::op::v1::Multiply>(even_slice, cos_theta_node),
|
||||
std::make_shared<ov::op::v1::Multiply>(odd_slice, sin_theta_node));
|
||||
Output<Node> second_half =
|
||||
std::make_shared<ov::op::v1::Add>(std::make_shared<ov::op::v1::Multiply>(even_slice, sin_theta_node),
|
||||
std::make_shared<ov::op::v1::Multiply>(odd_slice, cos_theta_node));
|
||||
|
||||
first_half = std::make_shared<ov::op::v0::Unsqueeze>(first_half,
|
||||
ov::op::v0::Constant::create(ov::element::i64, {1}, {unsqueeze_dim}));
|
||||
second_half = std::make_shared<ov::op::v0::Unsqueeze>(second_half,
|
||||
ov::op::v0::Constant::create(ov::element::i64, {1}, {unsqueeze_dim}));
|
||||
auto stack = std::make_shared<ov::op::v0::Concat>(OutputVector{first_half, second_half}, unsqueeze_dim);
|
||||
|
||||
auto data_shape = ov::op::v0::Constant::create(
|
||||
ov::element::i64, {4}, std::vector<int64_t>{1, -1, (int64_t) output_shape[2], (int64_t) output_shape[3]});
|
||||
res = std::make_shared<ov::op::v1::Reshape>(stack, data_shape, false);
|
||||
} else if (mode == ROPE_TYPE_NEOX) {
|
||||
auto data_split = std::make_shared<ov::op::v1::Split>(
|
||||
data_node, ov::op::v0::Constant::create(ov::element::i64, ov::Shape{}, {-1}), 2);
|
||||
Output<Node> slice_data_node_0 = data_split->outputs()[0];
|
||||
Output<Node> slice_data_node_1 = data_split->outputs()[1];
|
||||
|
||||
auto first_half_node = std::make_shared<ov::op::v1::Subtract>(
|
||||
std::make_shared<ov::op::v1::Multiply>(slice_data_node_0, cos_theta_node),
|
||||
std::make_shared<ov::op::v1::Multiply>(slice_data_node_1, sin_theta_node));
|
||||
|
||||
auto second_half_node = std::make_shared<ov::op::v1::Add>(
|
||||
std::make_shared<ov::op::v1::Multiply>(slice_data_node_0, sin_theta_node),
|
||||
std::make_shared<ov::op::v1::Multiply>(slice_data_node_1, cos_theta_node));
|
||||
|
||||
res = std::make_shared<ov::op::v0::Concat>(ov::OutputVector{first_half_node, second_half_node}, -1);
|
||||
}
|
||||
|
||||
return rename_outputs_with_suffix({res}, context.get_name());
|
||||
}
|
||||
|
||||
} // namespace op
|
||||
} // namespace ggml
|
||||
} // namespace frontend
|
||||
} // namespace ov
|
||||
@@ -0,0 +1,41 @@
|
||||
#include "../node_context.h"
|
||||
#include "../op_table.h"
|
||||
#include "../utils.h"
|
||||
|
||||
#include <openvino/op/add.hpp>
|
||||
#include <openvino/op/constant.hpp>
|
||||
#include <openvino/op/multiply.hpp>
|
||||
#include <vector>
|
||||
|
||||
namespace ov {
|
||||
namespace frontend {
|
||||
namespace ggml {
|
||||
namespace op {
|
||||
|
||||
OutputVector translate_scale(const NodeContext & context) {
|
||||
num_inputs_check(context, 1, 1);
|
||||
|
||||
float scale;
|
||||
float bias;
|
||||
memcpy(&scale, (float *) context.get_output_op_params() + 0, sizeof(float));
|
||||
memcpy(&bias, (float *) context.get_output_op_params() + 1, sizeof(float));
|
||||
|
||||
auto scale_node = std::make_shared<ov::op::v0::Constant>(ov::element::f32, ov::Shape{}, std::vector<float>{scale});
|
||||
auto scaled = std::make_shared<ov::op::v1::Multiply>(context.get_input(0), scale_node);
|
||||
|
||||
std::shared_ptr<ov::Node> res;
|
||||
if (bias != 0.0f) {
|
||||
auto bias_node =
|
||||
std::make_shared<ov::op::v0::Constant>(ov::element::f32, ov::Shape{}, std::vector<float>{bias});
|
||||
res = std::make_shared<ov::op::v1::Add>(scaled, bias_node);
|
||||
} else {
|
||||
res = scaled;
|
||||
}
|
||||
|
||||
return rename_outputs_with_suffix({res}, context.get_name());
|
||||
}
|
||||
|
||||
} // namespace op
|
||||
} // namespace ggml
|
||||
} // namespace frontend
|
||||
} // namespace ov
|
||||
@@ -0,0 +1,76 @@
|
||||
#include "../node_context.h"
|
||||
#include "../op_table.h"
|
||||
#include "../utils.h"
|
||||
|
||||
#include <cassert>
|
||||
#include <cstdint>
|
||||
#include <memory>
|
||||
#include <openvino/core/node.hpp>
|
||||
#include <openvino/core/node_output.hpp>
|
||||
#include <openvino/frontend/exception.hpp>
|
||||
#include <openvino/op/concat.hpp>
|
||||
#include <openvino/op/constant.hpp>
|
||||
#include <openvino/op/convert.hpp>
|
||||
#include <openvino/op/gather.hpp>
|
||||
#include <openvino/op/reshape.hpp>
|
||||
#include <openvino/op/scatter_update.hpp>
|
||||
#include <openvino/op/shape_of.hpp>
|
||||
#include <openvino/op/slice.hpp>
|
||||
#include <openvino/op/squeeze.hpp>
|
||||
#include <openvino/op/transpose.hpp>
|
||||
#include <vector>
|
||||
|
||||
namespace ov {
|
||||
namespace frontend {
|
||||
namespace ggml {
|
||||
namespace op {
|
||||
|
||||
OutputVector translate_set_rows(const NodeContext & context) {
|
||||
num_inputs_check(context, 3, 3);
|
||||
|
||||
auto data = context.get_input(0);
|
||||
auto indices = context.get_input(1);
|
||||
auto dst = context.get_input(2);
|
||||
|
||||
data = std::make_shared<ov::op::v0::Convert>(data, context.get_output_type());
|
||||
|
||||
auto dst_shape = context.get_output_shape().to_shape();
|
||||
|
||||
auto ind_squeezed =
|
||||
std::make_shared<ov::op::v0::Squeeze>(indices, ov::op::v0::Constant::create(ov::element::i64, {3}, {0, 1, 2}));
|
||||
auto data_reshaped = std::make_shared<ov::op::v1::Reshape>(
|
||||
data,
|
||||
ov::op::v0::Constant::create(ov::element::i64, {4},
|
||||
{(int64_t) 1, (int64_t) 1, (int64_t) -1, (int64_t) dst_shape[3]}),
|
||||
false);
|
||||
auto axes = ov::op::v0::Constant::create(ov::element::i64, ov::Shape{}, {2});
|
||||
|
||||
Output<Node> res;
|
||||
if (context.is_stateful()) {
|
||||
int concat_axis = 1;
|
||||
int64_t dim2 = dst.get_partial_shape()[2].get_length();
|
||||
int64_t dim3 = dst.get_partial_shape()[3].get_length();
|
||||
data = std::make_shared<ov::op::v1::Reshape>(
|
||||
data, ov::op::v0::Constant::create(ov::element::i64, {4}, {(int64_t) 1, (int64_t) -1, dim2, dim3}), false);
|
||||
res = std::make_shared<ov::op::v0::Concat>(OutputVector{dst, data}, concat_axis);
|
||||
} else {
|
||||
res = std::make_shared<ov::op::v3::ScatterUpdate>(dst, ind_squeezed, data_reshaped, axes);
|
||||
}
|
||||
|
||||
if (auto dst_reshape = std::dynamic_pointer_cast<ov::op::v1::Reshape>(dst.get_node_shared_ptr())) {
|
||||
// Fix the case of multiple sequences, reshape back to original shape [1, n_seq, ctx_per_seq, emb]
|
||||
// ctx_per_seq is not fixed due to llama-bench compatibility
|
||||
auto dst_shape_partial = dst_reshape->get_input_partial_shape(0);
|
||||
std::vector<int64_t> dst_shape = {dst_shape_partial[0].get_length(), dst_shape_partial[1].get_length(),
|
||||
dst_shape_partial[2].is_static() ? dst_shape_partial[2].get_length() : -1,
|
||||
dst_shape_partial[3].get_length()};
|
||||
res = std::make_shared<ov::op::v1::Reshape>(res, ov::op::v0::Constant::create(ov::element::i64, {4}, dst_shape),
|
||||
false);
|
||||
}
|
||||
return rename_outputs_with_suffix({res}, context.get_name());
|
||||
}
|
||||
|
||||
} // namespace op
|
||||
} // namespace ggml
|
||||
} // namespace frontend
|
||||
} // namespace ov
|
||||
@@ -0,0 +1,89 @@
|
||||
#include "../node_context.h"
|
||||
#include "../op_table.h"
|
||||
#include "../utils.h"
|
||||
|
||||
#include <climits>
|
||||
#include <cstdint>
|
||||
#include <memory>
|
||||
#include <openvino/core/node.hpp>
|
||||
#include <openvino/core/node_output.hpp>
|
||||
#include <openvino/op/add.hpp>
|
||||
#include <openvino/op/concat.hpp>
|
||||
#include <openvino/op/constant.hpp>
|
||||
#include <openvino/op/convert.hpp>
|
||||
#include <openvino/op/matmul.hpp>
|
||||
#include <openvino/op/multiply.hpp>
|
||||
#include <openvino/op/slice.hpp>
|
||||
#include <openvino/op/softmax.hpp>
|
||||
#include <vector>
|
||||
|
||||
namespace ov {
|
||||
namespace frontend {
|
||||
namespace ggml {
|
||||
namespace op {
|
||||
|
||||
OutputVector translate_soft_max(const NodeContext & context) {
|
||||
// TODO code is outdated
|
||||
num_inputs_check(context, 1, 2);
|
||||
|
||||
auto input_node = context.get_input(0).get_node_shared_ptr();
|
||||
ov::Output<Node> res;
|
||||
|
||||
float scale = 1.0f;
|
||||
float max_bias = 0.0f;
|
||||
auto * op_params = context.get_output_op_params();
|
||||
memcpy(&scale, (float *) op_params + 0, sizeof(float));
|
||||
memcpy(&max_bias, (float *) op_params + 1, sizeof(float));
|
||||
auto src0_shape = context.get_input_shape(0).get_shape();
|
||||
const uint32_t h = src0_shape[2];
|
||||
const uint32_t n_head = src0_shape[0];
|
||||
const uint32_t n_head_log2 = 1u << (uint32_t) floor(log2(n_head));
|
||||
|
||||
const float m0 = powf(2.0f, -(max_bias) / n_head_log2);
|
||||
const float m1 = powf(2.0f, -(max_bias / 2.0f) / n_head_log2);
|
||||
const float slope =
|
||||
(max_bias > 0.0f) ? h < n_head_log2 ? powf(m0, h + 1) : powf(m1, 2 * (h - n_head_log2) + 1) : 1.0f;
|
||||
|
||||
auto scale_node = std::make_shared<ov::op::v0::Constant>(ov::element::f32, ov::Shape{}, std::vector<float>{scale});
|
||||
auto scaled_input = std::make_shared<ov::op::v1::Multiply>(input_node, scale_node);
|
||||
|
||||
if (context.get_input_size() < 2) {
|
||||
res = std::make_shared<ov::op::v8::Softmax>(scaled_input, 2);
|
||||
return rename_outputs_with_suffix({res}, context.get_name());
|
||||
}
|
||||
|
||||
ov::Output<ov::Node> mask_node_sliced;
|
||||
if (context.has_input("KQ_mask_sliced")) {
|
||||
mask_node_sliced = context.get_input("KQ_mask_sliced");
|
||||
} else {
|
||||
auto token_len = get_dimensions(input_node, {1});
|
||||
auto mask_node = context.get_input(1);
|
||||
auto zero = ov::op::v0::Constant::create(ov::element::i64, {1}, {0});
|
||||
auto one = ov::op::v0::Constant::create(ov::element::i64, {1}, {1});
|
||||
mask_node_sliced = std::make_shared<ov::op::v8::Slice>(mask_node, zero, token_len, one, one);
|
||||
}
|
||||
|
||||
if (mask_node_sliced.get_element_type() != context.get_output_type()) {
|
||||
mask_node_sliced = std::make_shared<ov::op::v0::Convert>(mask_node_sliced, context.get_output_type());
|
||||
}
|
||||
|
||||
Output<Node> slope_mask;
|
||||
if (slope != 1.0f) {
|
||||
auto slope_node =
|
||||
std::make_shared<ov::op::v0::Constant>(ov::element::f32, ov::Shape{}, std::vector<float>{slope});
|
||||
slope_mask = std::make_shared<ov::op::v1::Multiply>(mask_node_sliced, slope_node);
|
||||
throw std::runtime_error("Slope != 1.0f in softmax has not been tested, verify it before use.");
|
||||
}
|
||||
slope_mask = mask_node_sliced;
|
||||
|
||||
auto input_slope_mask_node = std::make_shared<ov::op::v1::Add>(scaled_input, slope_mask);
|
||||
|
||||
res = std::make_shared<ov::op::v8::Softmax>(input_slope_mask_node, 2);
|
||||
|
||||
return rename_outputs_with_suffix({res}, context.get_name());
|
||||
}
|
||||
|
||||
} // namespace op
|
||||
} // namespace ggml
|
||||
} // namespace frontend
|
||||
} // namespace ov
|
||||
@@ -0,0 +1,23 @@
|
||||
#include "../node_context.h"
|
||||
#include "../op_table.h"
|
||||
#include "../utils.h"
|
||||
|
||||
#include <openvino/op/transpose.hpp>
|
||||
|
||||
namespace ov {
|
||||
namespace frontend {
|
||||
namespace ggml {
|
||||
namespace op {
|
||||
|
||||
OutputVector translate_transpose(const NodeContext & context) {
|
||||
num_inputs_check(context, 1, 1);
|
||||
|
||||
auto res = std::make_shared<ov::op::v1::Transpose>(
|
||||
context.get_input(0), ov::op::v0::Constant::create(ov::element::i64, {4}, {0, 1, 3, 2}));
|
||||
return rename_outputs_with_suffix({res}, context.get_name());
|
||||
}
|
||||
|
||||
} // namespace op
|
||||
} // namespace ggml
|
||||
} // namespace frontend
|
||||
} // namespace ov
|
||||
@@ -0,0 +1,27 @@
|
||||
#include "../node_context.h"
|
||||
#include "../op_table.h"
|
||||
#include "../utils.h"
|
||||
|
||||
#include <openvino/core/node_output.hpp>
|
||||
#include <openvino/op/multiply.hpp>
|
||||
#include <openvino/op/sigmoid.hpp>
|
||||
|
||||
namespace ov {
|
||||
namespace frontend {
|
||||
namespace ggml {
|
||||
namespace op {
|
||||
|
||||
OutputVector translate_unary_silu(const NodeContext & context) {
|
||||
num_inputs_check(context, 1, 1);
|
||||
|
||||
auto input = context.get_input(0);
|
||||
auto sigmoid = std::make_shared<ov::op::v0::Sigmoid>(input);
|
||||
auto res = std::make_shared<ov::op::v1::Multiply>(input, sigmoid);
|
||||
|
||||
return rename_outputs_with_suffix({res}, context.get_name());
|
||||
}
|
||||
|
||||
} // namespace op
|
||||
} // namespace ggml
|
||||
} // namespace frontend
|
||||
} // namespace ov
|
||||
@@ -0,0 +1,53 @@
|
||||
#include "../op_table.h"
|
||||
#include "../utils.h"
|
||||
#include <openvino/op/reshape.hpp>
|
||||
namespace ov {
|
||||
namespace frontend {
|
||||
namespace ggml {
|
||||
namespace op {
|
||||
|
||||
OutputVector translate_view(const NodeContext & context) {
|
||||
num_inputs_check(context, 1, 1);
|
||||
|
||||
if (context.get_op_case() == 2) {
|
||||
auto dst_shape = context.get_output_shape().to_shape();
|
||||
return rename_outputs_with_suffix({process_view_input(context, 0, dst_shape[2] * dst_shape[3])},
|
||||
context.get_name());
|
||||
}
|
||||
// op_case 3
|
||||
if (context.get_op_case() == 3) {
|
||||
auto input = context.get_input(0);
|
||||
auto input_ov_shape = input.get_partial_shape();
|
||||
|
||||
auto input_llama_shape = context.get_input_shape(0).to_shape();
|
||||
|
||||
// if the input ov shape size is different from the input llama shape size, it means the input is already reshaped and we need to reshape it back to the original shape before slicing
|
||||
if (input_ov_shape.size() != input_llama_shape.size()) {
|
||||
input = std::make_shared<ov::op::v1::Reshape>(input, ov::op::v0::Constant::create(ov::element::i64, {input_llama_shape.size()}, input_llama_shape), false);
|
||||
}
|
||||
|
||||
auto dst_shape = context.get_output_shape().to_shape();
|
||||
|
||||
// find the index of dst_shape that is different from input shape, and use that index to slice the input
|
||||
int slice_dim = -1;
|
||||
for (size_t i = 0; i < dst_shape.size(); ++i) {
|
||||
if (dst_shape[i] != input_llama_shape[i]) {
|
||||
slice_dim = i;
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
auto begin = ov::op::v0::Constant::create(ov::element::i64, {1}, {0});
|
||||
auto end = ov::op::v0::Constant::create(ov::element::i64, {1}, {dst_shape[slice_dim]});
|
||||
auto stride = ov::op::v0::Constant::create(ov::element::i64, {1}, {1});
|
||||
auto axes = ov::op::v0::Constant::create(ov::element::i64, {1}, {slice_dim});
|
||||
auto sliced = std::make_shared<ov::op::v8::Slice>(input, begin, end, stride, axes);
|
||||
return {sliced};
|
||||
}
|
||||
return {context.get_input(0)};
|
||||
}
|
||||
|
||||
} // namespace op
|
||||
} // namespace ggml
|
||||
} // namespace frontend
|
||||
} // namespace ov
|
||||
@@ -0,0 +1,46 @@
|
||||
#include "op_table.h"
|
||||
|
||||
#include "utils.h"
|
||||
|
||||
#include <openvino/op/add.hpp>
|
||||
#include <openvino/op/divide.hpp>
|
||||
#include <openvino/op/gather.hpp>
|
||||
#include <openvino/op/matmul.hpp>
|
||||
#include <openvino/op/multiply.hpp>
|
||||
#include <openvino/op/subtract.hpp>
|
||||
|
||||
namespace ov {
|
||||
namespace frontend {
|
||||
namespace ggml {
|
||||
|
||||
std::unordered_map<std::string, CreatorFunction> get_supported_ops() {
|
||||
using namespace ov::op;
|
||||
return {
|
||||
{"GGML_OP_ADD", op::translate_1to1_match_2_inputs<v1::Add> },
|
||||
{"GGML_OP_ADD1", op::translate_1to1_match_2_inputs<v1::Add> },
|
||||
{"GGML_OP_CONT", op::translate_cont },
|
||||
{"GGML_OP_DIV", op::translate_1to1_match_2_inputs<v1::Divide> },
|
||||
{"GGML_OP_GET_ROWS", op::translate_get_rows },
|
||||
{"GGML_OP_MUL", op::translate_1to1_match_2_inputs<v1::Multiply>},
|
||||
{"GGML_OP_MUL_MAT", op::translate_mulmat },
|
||||
{"GGML_OP_PERMUTE", op::translate_permute },
|
||||
{"GGML_OP_RESHAPE", op::translate_reshape },
|
||||
{"GGML_OP_RMS_NORM", op::translate_rms_norm },
|
||||
{"GGML_OP_ROPE", op::translate_rope },
|
||||
{"GGML_OP_SCALE", op::translate_scale },
|
||||
{"GGML_OP_SOFT_MAX", op::translate_soft_max },
|
||||
{"GGML_OP_SUB", op::translate_1to1_match_2_inputs<v1::Subtract>},
|
||||
{"GGML_OP_TRANSPOSE", op::translate_transpose },
|
||||
{"GGML_UNARY_OP_SILU", op::translate_unary_silu },
|
||||
{"GGML_OP_VIEW", op::translate_view },
|
||||
{"GGML_GLU_OP_SWIGLU", op::translate_glu_swiglu },
|
||||
{"GGML_GLU_OP_GEGLU", op::translate_glu_geglu },
|
||||
{"GGML_OP_SET_ROWS", op::translate_set_rows },
|
||||
{"GGML_OP_CPY", op::translate_cpy },
|
||||
{"GGML_OP_FLASH_ATTN_EXT", op::translate_flash_attn_ext },
|
||||
};
|
||||
}
|
||||
|
||||
} // namespace ggml
|
||||
} // namespace frontend
|
||||
} // namespace ov
|
||||
@@ -0,0 +1,39 @@
|
||||
#pragma once
|
||||
|
||||
#include "node_context.h"
|
||||
|
||||
namespace ov {
|
||||
namespace frontend {
|
||||
namespace ggml {
|
||||
|
||||
namespace op {
|
||||
|
||||
#define GGML_OP_CONVERTER(op) OutputVector op(const NodeContext& context)
|
||||
|
||||
GGML_OP_CONVERTER(translate_add);
|
||||
GGML_OP_CONVERTER(translate_cont);
|
||||
GGML_OP_CONVERTER(translate_get_rows);
|
||||
GGML_OP_CONVERTER(translate_mul);
|
||||
GGML_OP_CONVERTER(translate_mulmat);
|
||||
GGML_OP_CONVERTER(translate_permute);
|
||||
GGML_OP_CONVERTER(translate_reshape);
|
||||
GGML_OP_CONVERTER(translate_rms_norm);
|
||||
GGML_OP_CONVERTER(translate_rope);
|
||||
GGML_OP_CONVERTER(translate_scale);
|
||||
GGML_OP_CONVERTER(translate_unary_silu);
|
||||
GGML_OP_CONVERTER(translate_soft_max);
|
||||
GGML_OP_CONVERTER(translate_transpose);
|
||||
GGML_OP_CONVERTER(translate_view);
|
||||
GGML_OP_CONVERTER(translate_glu_swiglu);
|
||||
GGML_OP_CONVERTER(translate_glu_geglu);
|
||||
GGML_OP_CONVERTER(translate_set_rows);
|
||||
GGML_OP_CONVERTER(translate_cpy);
|
||||
GGML_OP_CONVERTER(translate_flash_attn_ext);
|
||||
|
||||
} // namespace op
|
||||
|
||||
std::unordered_map<std::string, CreatorFunction> get_supported_ops();
|
||||
|
||||
} // namespace ggml
|
||||
} // namespace frontend
|
||||
} // namespace ov
|
||||
@@ -0,0 +1,123 @@
|
||||
#include "eliminate_zp.h"
|
||||
|
||||
#include <openvino/core/graph_util.hpp>
|
||||
#include <openvino/core/parallel.hpp>
|
||||
#include <openvino/core/rt_info.hpp>
|
||||
#include <openvino/op/constant.hpp>
|
||||
#include <openvino/op/convert.hpp>
|
||||
#include <openvino/op/multiply.hpp>
|
||||
#include <openvino/op/subtract.hpp>
|
||||
#include <openvino/pass/pattern/op/label.hpp>
|
||||
#include <openvino/pass/pattern/op/pattern.hpp>
|
||||
#include <openvino/pass/pattern/op/wrap_type.hpp>
|
||||
|
||||
namespace ov {
|
||||
namespace frontend {
|
||||
namespace ggml {
|
||||
namespace pass {
|
||||
|
||||
EliminateZeroPoints::EliminateZeroPoints() {
|
||||
// Find pattern:
|
||||
// (Multiply Any(scale)
|
||||
// (Subtract (Convert Constant(data)))
|
||||
// (Convert Constant(zero_point)))
|
||||
// where zero_point is a scalar
|
||||
// If data is u4 and zp value is 8 (q4_0), Replace the Subtract with an i4 Constant whose value is data - zp_val
|
||||
// If data is u8 and zp value is 128 (q8_0) or 32 (q6_k), Replace the Subtract with an i8 Constant
|
||||
|
||||
auto m_data_constant = ov::pass::pattern::wrap_type<ov::op::v0::Constant>();
|
||||
auto m_data_convert = ov::pass::pattern::wrap_type<ov::op::v0::Convert>({m_data_constant});
|
||||
|
||||
auto m_zp_constant = ov::pass::pattern::wrap_type<ov::op::v0::Constant>();
|
||||
auto m_zp_convert = ov::pass::pattern::wrap_type<ov::op::v0::Convert>({m_zp_constant});
|
||||
|
||||
auto m_subtract = ov::pass::pattern::wrap_type<ov::op::v1::Subtract>({m_data_convert, m_zp_convert});
|
||||
auto m_scale = ov::pass::pattern::any_input();
|
||||
auto m_multiply = ov::pass::pattern::wrap_type<ov::op::v1::Multiply>({m_scale, m_subtract});
|
||||
|
||||
const auto callback = [=](ov::pass::pattern::Matcher & m) {
|
||||
const auto & pattern_map = m.get_pattern_value_map();
|
||||
|
||||
auto multiply_node =
|
||||
std::dynamic_pointer_cast<ov::op::v1::Multiply>(pattern_map.at(m_multiply).get_node_shared_ptr());
|
||||
auto subtract_node =
|
||||
std::dynamic_pointer_cast<ov::op::v1::Subtract>(pattern_map.at(m_subtract).get_node_shared_ptr());
|
||||
auto data_constant =
|
||||
std::dynamic_pointer_cast<ov::op::v0::Constant>(pattern_map.at(m_data_constant).get_node_shared_ptr());
|
||||
auto zp_constant =
|
||||
std::dynamic_pointer_cast<ov::op::v0::Constant>(pattern_map.at(m_zp_constant).get_node_shared_ptr());
|
||||
|
||||
if (!multiply_node || !subtract_node || !data_constant || !zp_constant) {
|
||||
return false;
|
||||
}
|
||||
|
||||
if (ov::shape_size(zp_constant->get_shape()) != 1) {
|
||||
return false;
|
||||
}
|
||||
|
||||
auto data_type = data_constant->get_element_type();
|
||||
auto zp_data = zp_constant->cast_vector<int>();
|
||||
|
||||
if (zp_data.empty()) {
|
||||
return false;
|
||||
}
|
||||
|
||||
int zp_value = zp_data[0];
|
||||
|
||||
bool should_eliminate = false;
|
||||
ov::element::Type target_type;
|
||||
|
||||
if (data_type == ov::element::u4 && zp_value == 8) {
|
||||
should_eliminate = true;
|
||||
target_type = ov::element::i4;
|
||||
} else if (data_type == ov::element::u8 && (zp_value == 128 || zp_value == 32)) {
|
||||
should_eliminate = true;
|
||||
target_type = ov::element::i8;
|
||||
}
|
||||
|
||||
if (!should_eliminate) {
|
||||
return false;
|
||||
}
|
||||
|
||||
auto data_shape = data_constant->get_shape();
|
||||
size_t total_elements = ov::shape_size(data_shape);
|
||||
|
||||
std::shared_ptr<ov::op::v0::Constant> new_constant;
|
||||
|
||||
// TODO improve performance
|
||||
if (data_type == ov::element::u4) {
|
||||
auto data_values = data_constant->cast_vector<uint8_t>();
|
||||
std::vector<int8_t> adjusted_values(total_elements);
|
||||
|
||||
ov::parallel_for(total_elements, [&](size_t i) {
|
||||
adjusted_values[i] = static_cast<int8_t>(static_cast<int>(data_values[i]) - 8);
|
||||
});
|
||||
|
||||
new_constant = std::make_shared<ov::op::v0::Constant>(target_type, data_shape, adjusted_values);
|
||||
} else if (data_type == ov::element::u8) {
|
||||
auto data_values = data_constant->cast_vector<uint8_t>();
|
||||
std::vector<int8_t> adjusted_values(total_elements);
|
||||
|
||||
ov::parallel_for(total_elements, [&, zp_value](size_t i) {
|
||||
adjusted_values[i] = static_cast<int8_t>(static_cast<int>(data_values[i]) - zp_value);
|
||||
});
|
||||
|
||||
new_constant = std::make_shared<ov::op::v0::Constant>(target_type, data_shape, adjusted_values);
|
||||
}
|
||||
|
||||
auto new_convert =
|
||||
std::make_shared<ov::op::v0::Convert>(new_constant, subtract_node->get_output_element_type(0));
|
||||
ov::replace_node(subtract_node, new_convert);
|
||||
|
||||
return true;
|
||||
};
|
||||
|
||||
register_matcher(
|
||||
std::make_shared<ov::pass::pattern::Matcher>(m_multiply, "ov::frontend::ggml::pass::EliminateZeroPoints"),
|
||||
callback);
|
||||
}
|
||||
|
||||
} // namespace pass
|
||||
} // namespace ggml
|
||||
} // namespace frontend
|
||||
} // namespace ov
|
||||
@@ -0,0 +1,17 @@
|
||||
#include "openvino/pass/matcher_pass.hpp"
|
||||
|
||||
namespace ov {
|
||||
namespace frontend {
|
||||
namespace ggml {
|
||||
namespace pass {
|
||||
|
||||
class EliminateZeroPoints : public ov::pass::MatcherPass {
|
||||
public:
|
||||
OPENVINO_MATCHER_PASS_RTTI("ov::frontend::ggml::pass::EliminateZeroPoints")
|
||||
EliminateZeroPoints();
|
||||
};
|
||||
|
||||
} // namespace pass
|
||||
} // namespace ggml
|
||||
} // namespace frontend
|
||||
} // namespace ov
|
||||
@@ -0,0 +1,60 @@
|
||||
#include "fuse_to_sdpa.h"
|
||||
|
||||
#include <openvino/core/graph_util.hpp>
|
||||
#include <openvino/core/rt_info.hpp>
|
||||
#include <openvino/op/add.hpp>
|
||||
#include <openvino/op/convert.hpp>
|
||||
#include <openvino/op/matmul.hpp>
|
||||
#include <openvino/op/multiply.hpp>
|
||||
#include <openvino/op/scaled_dot_product_attention.hpp>
|
||||
#include <openvino/op/softmax.hpp>
|
||||
#include <openvino/op/transpose.hpp>
|
||||
#include <openvino/pass/pattern/op/label.hpp>
|
||||
#include <openvino/pass/pattern/op/pattern.hpp>
|
||||
#include <openvino/pass/pattern/op/wrap_type.hpp>
|
||||
|
||||
namespace ov {
|
||||
namespace frontend {
|
||||
namespace ggml {
|
||||
namespace pass {
|
||||
|
||||
FuseToSDPA::FuseToSDPA() {
|
||||
// Not maintained since FLASH_ATTN_EXT has replaced this pattern
|
||||
const auto m_k = ov::pass::pattern::any_input();
|
||||
const auto m_q = ov::pass::pattern::any_input();
|
||||
const auto m_qk = ov::pass::pattern::wrap_type<ov::op::v0::MatMul>({m_q, m_k});
|
||||
const auto m_qk_f32 = ov::pass::pattern::wrap_type<ov::op::v0::Convert>({m_qk});
|
||||
const auto m_scale = ov::pass::pattern::any_input();
|
||||
const auto m_scaled_qk = ov::pass::pattern::wrap_type<ov::op::v1::Multiply>({m_qk_f32, m_scale});
|
||||
const auto m_mask = ov::pass::pattern::any_input();
|
||||
const auto m_masked_qk = ov::pass::pattern::wrap_type<ov::op::v1::Add>({m_scaled_qk, m_mask});
|
||||
const auto m_softmax_qk = ov::pass::pattern::wrap_type<ov::op::v8::Softmax>({m_masked_qk});
|
||||
const auto m_softmax_qk_f16 = ov::pass::pattern::wrap_type<ov::op::v0::Convert>({m_softmax_qk});
|
||||
const auto m_v = ov::pass::pattern::any_input();
|
||||
const auto m_qkv = ov::pass::pattern::wrap_type<ov::op::v0::MatMul>({m_softmax_qk_f16, m_v});
|
||||
|
||||
const auto callback = [=](ov::pass::pattern::Matcher & m) {
|
||||
auto & pattern_to_output = m.get_pattern_value_map();
|
||||
auto k = pattern_to_output[m_k];
|
||||
auto q = pattern_to_output[m_q];
|
||||
auto v = pattern_to_output[m_v];
|
||||
auto mask = pattern_to_output[m_mask];
|
||||
auto scale = pattern_to_output[m_scale];
|
||||
|
||||
auto mask_f16 = register_new_node<ov::op::v0::Convert>(mask, ov::element::f16);
|
||||
auto scale_f16 = register_new_node<ov::op::v0::Convert>(scale, ov::element::f16);
|
||||
auto sdpa = std::make_shared<ov::op::v13::ScaledDotProductAttention>(q, k, v, mask_f16, scale_f16, false);
|
||||
|
||||
ov::replace_node(m.get_match_root(), sdpa);
|
||||
ov::copy_runtime_info(m.get_matched_nodes(), sdpa);
|
||||
|
||||
return true;
|
||||
};
|
||||
register_matcher(std::make_shared<ov::pass::pattern::Matcher>(m_qkv, "ov::frontend::ggml::pass::FuseToSDPA"),
|
||||
callback);
|
||||
}
|
||||
|
||||
} // namespace pass
|
||||
} // namespace ggml
|
||||
} // namespace frontend
|
||||
} // namespace ov
|
||||
@@ -0,0 +1,17 @@
|
||||
#include "openvino/pass/matcher_pass.hpp"
|
||||
|
||||
namespace ov {
|
||||
namespace frontend {
|
||||
namespace ggml {
|
||||
namespace pass {
|
||||
|
||||
class FuseToSDPA : public ov::pass::MatcherPass {
|
||||
public:
|
||||
OPENVINO_MATCHER_PASS_RTTI("ov::frontend::ggml::pass::FuseToSDPA")
|
||||
FuseToSDPA();
|
||||
};
|
||||
|
||||
} // namespace pass
|
||||
} // namespace ggml
|
||||
} // namespace frontend
|
||||
} // namespace ov
|
||||
@@ -0,0 +1,29 @@
|
||||
#pragma once
|
||||
|
||||
#include "mark_decompression_convert_constant_folding.h"
|
||||
#include "openvino/pass/matcher_pass.hpp"
|
||||
#include "openvino/core/visibility.hpp"
|
||||
|
||||
#ifdef OPENVINO_STATIC_LIBRARY
|
||||
# define TRANSFORMATIONS_API
|
||||
#else
|
||||
# ifdef IMPLEMENT_OPENVINO_API
|
||||
# define TRANSFORMATIONS_API OPENVINO_CORE_EXPORTS
|
||||
# else
|
||||
# define TRANSFORMATIONS_API OPENVINO_CORE_IMPORTS
|
||||
# endif // IMPLEMENT_OPENVINO_API
|
||||
#endif // OPENVINO_STATIC_LIBRARY
|
||||
|
||||
namespace ov {
|
||||
namespace pass {
|
||||
|
||||
class TRANSFORMATIONS_API MarkCompressedFloatConstants;
|
||||
|
||||
} // namespace pass
|
||||
} // namespace ov
|
||||
|
||||
class ov::pass::MarkCompressedFloatConstants : public MatcherPass {
|
||||
public:
|
||||
OPENVINO_MATCHER_PASS_RTTI("MarkCompressedFloatConstants")
|
||||
MarkCompressedFloatConstants();
|
||||
};
|
||||
@@ -0,0 +1,58 @@
|
||||
#include "squeeze_matmul.h"
|
||||
|
||||
#include <openvino/core/graph_util.hpp>
|
||||
#include <openvino/core/rt_info.hpp>
|
||||
#include <openvino/op/constant.hpp>
|
||||
#include <openvino/op/matmul.hpp>
|
||||
#include <openvino/op/squeeze.hpp>
|
||||
#include <openvino/op/unsqueeze.hpp>
|
||||
#include <openvino/pass/pattern/op/label.hpp>
|
||||
#include <openvino/pass/pattern/op/pattern.hpp>
|
||||
#include <openvino/pass/pattern/op/wrap_type.hpp>
|
||||
|
||||
namespace opp = ov::pass::pattern;
|
||||
|
||||
namespace ov {
|
||||
namespace frontend {
|
||||
namespace ggml {
|
||||
namespace pass {
|
||||
|
||||
// For quantized models, NPUW expects the activation to be 3d in DQ(DynamicQuantization) opt, e.g. DQMatMulGQ2i
|
||||
SqueezeMatmul::SqueezeMatmul() {
|
||||
auto m_act = opp::any_input();
|
||||
auto m_wei = opp::any_input();
|
||||
auto m_matmul = opp::wrap_type<ov::op::v0::MatMul>({m_act, m_wei});
|
||||
|
||||
const auto callback = [=](ov::pass::pattern::Matcher & m) {
|
||||
const auto & pattern_map = m.get_pattern_value_map();
|
||||
auto matmul_node =
|
||||
std::dynamic_pointer_cast<ov::op::v0::MatMul>(pattern_map.at(m_matmul).get_node_shared_ptr());
|
||||
auto act = pattern_map.at(m_act);
|
||||
auto wei = pattern_map.at(m_wei);
|
||||
auto act_shape = act.get_partial_shape();
|
||||
auto wei_shape = wei.get_partial_shape();
|
||||
if (act_shape.rank().is_dynamic() || wei_shape.rank().is_dynamic()) {
|
||||
return false;
|
||||
}
|
||||
if (act_shape.rank().get_length() == 4 && wei_shape.rank().get_length() == 2) {
|
||||
auto axis = ov::op::v0::Constant::create(ov::element::i64, ov::Shape{1}, {0});
|
||||
auto squeezed_act = std::make_shared<ov::op::v0::Squeeze>(act, axis);
|
||||
auto new_matmul = std::make_shared<ov::op::v0::MatMul>(squeezed_act, wei, matmul_node->get_transpose_a(),
|
||||
matmul_node->get_transpose_b());
|
||||
auto unsqueezed_output = std::make_shared<ov::op::v0::Unsqueeze>(new_matmul, axis);
|
||||
unsqueezed_output->set_friendly_name(matmul_node->get_friendly_name());
|
||||
ov::copy_runtime_info(matmul_node, {squeezed_act, new_matmul, unsqueezed_output});
|
||||
ov::replace_node(matmul_node, unsqueezed_output);
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
};
|
||||
|
||||
register_matcher(std::make_shared<ov::pass::pattern::Matcher>(m_matmul, "ov::frontend::ggml::pass::SqueezeMatmul"),
|
||||
callback);
|
||||
}
|
||||
|
||||
} // namespace pass
|
||||
} // namespace ggml
|
||||
} // namespace frontend
|
||||
} // namespace ov
|
||||
@@ -0,0 +1,17 @@
|
||||
#include "openvino/pass/matcher_pass.hpp"
|
||||
|
||||
namespace ov {
|
||||
namespace frontend {
|
||||
namespace ggml {
|
||||
namespace pass {
|
||||
|
||||
class SqueezeMatmul : public ov::pass::MatcherPass {
|
||||
public:
|
||||
OPENVINO_MATCHER_PASS_RTTI("ov::frontend::ggml::pass::SqueezeMatmul")
|
||||
SqueezeMatmul();
|
||||
};
|
||||
|
||||
} // namespace pass
|
||||
} // namespace ggml
|
||||
} // namespace frontend
|
||||
} // namespace ov
|
||||
@@ -0,0 +1,293 @@
|
||||
#include "translate_session.h"
|
||||
|
||||
#include "ggml-openvino/openvino/node_context.h"
|
||||
#include "ggml-openvino/openvino/utils.h"
|
||||
#include "input_model.h"
|
||||
#include "pass/eliminate_zp.h"
|
||||
#include "pass/mark_decompression_convert_constant_folding.h"
|
||||
#include "pass/squeeze_matmul.h"
|
||||
|
||||
#include <cstdint>
|
||||
#include <cstdlib>
|
||||
#include <map>
|
||||
#include <memory>
|
||||
#include <openvino/core/node.hpp>
|
||||
#include <openvino/op/add.hpp>
|
||||
#include <openvino/op/broadcast.hpp>
|
||||
#include <openvino/op/concat.hpp>
|
||||
#include <openvino/op/convert.hpp>
|
||||
#include <openvino/op/convert_like.hpp>
|
||||
#include <openvino/op/cos.hpp>
|
||||
#include <openvino/op/divide.hpp>
|
||||
#include <openvino/op/gather.hpp>
|
||||
#include <openvino/op/multiply.hpp>
|
||||
#include <openvino/op/parameter.hpp>
|
||||
#include <openvino/op/range.hpp>
|
||||
#include <openvino/op/reshape.hpp>
|
||||
#include <openvino/op/result.hpp>
|
||||
#include <openvino/op/sin.hpp>
|
||||
#include <openvino/op/slice.hpp>
|
||||
#include <openvino/op/squeeze.hpp>
|
||||
#include <openvino/op/strided_slice.hpp>
|
||||
#include <openvino/op/transpose.hpp>
|
||||
#include <openvino/op/unsqueeze.hpp>
|
||||
#include <openvino/pass/constant_folding.hpp>
|
||||
#include <openvino/pass/make_stateful.hpp>
|
||||
#include <openvino/core/preprocess/pre_post_process.hpp>
|
||||
|
||||
namespace ov {
|
||||
namespace frontend {
|
||||
namespace ggml {
|
||||
|
||||
using namespace ov::op;
|
||||
|
||||
namespace {
|
||||
|
||||
ov::pass::MakeStateful::ParamResPairs get_kv_param_res_pairs(
|
||||
const std::shared_ptr<ov::Model> & model,
|
||||
const std::map<std::string, std::string> & kv_param_res_names) {
|
||||
ov::pass::MakeStateful::ParamResPairs pairs;
|
||||
const auto & params = model->get_parameters();
|
||||
const auto & results = model->get_results();
|
||||
|
||||
for (const auto & param_res : kv_param_res_names) {
|
||||
const auto & param_name = param_res.first;
|
||||
const auto & res_name = param_res.second;
|
||||
|
||||
auto param_it = std::find_if(params.begin(), params.end(), [&](const std::shared_ptr<v0::Parameter> & node) {
|
||||
return node->get_friendly_name() == param_name;
|
||||
});
|
||||
|
||||
OPENVINO_ASSERT(param_it != params.end(), "The tensor name ", param_name,
|
||||
" is not associated with any of "
|
||||
"Parameters in the network.");
|
||||
|
||||
auto res_it = std::find_if(results.begin(), results.end(), [&](const std::shared_ptr<v0::Result> & node) {
|
||||
return node->get_friendly_name() == res_name;
|
||||
});
|
||||
|
||||
OPENVINO_ASSERT(res_it != results.end(), "The tensor name ", res_name,
|
||||
" is not associated with any of "
|
||||
"Results in the network.");
|
||||
|
||||
std::shared_ptr<ov::op::v0::Parameter> param = *param_it;
|
||||
std::shared_ptr<ov::op::v0::Result> res = *res_it;
|
||||
pairs.emplace_back(param, res);
|
||||
}
|
||||
return pairs;
|
||||
}
|
||||
|
||||
void add_sliced_mask(TensorMap & tensor_map, GgmlDecoder & ggml_model_decoder) {
|
||||
|
||||
auto create_sliced_mask = [&](const std::string & mask_name, const std::string & sliced_name, bool is_static) {
|
||||
if ((tensor_map.find(mask_name) != tensor_map.end()) &&
|
||||
(tensor_map.find("token_len_per_seq") != tensor_map.end())) {
|
||||
auto token_len_per_seq = tensor_map.at("token_len_per_seq").get_node_shared_ptr();
|
||||
auto mask = tensor_map.at(mask_name).get_node_shared_ptr();
|
||||
std::shared_ptr<ov::Node> mask_sliced;
|
||||
if (is_static) {
|
||||
mask_sliced = mask;
|
||||
} else if (ggml_model_decoder.is_stateful()) {
|
||||
auto zero_2d = ov::op::v0::Constant::create(ov::element::i64, {2}, {0,0});
|
||||
auto one_2d = ov::op::v0::Constant::create(ov::element::i64, {2}, {1,1});
|
||||
auto zero_1d = ov::op::v0::Constant::create(ov::element::i64, {1}, {0});
|
||||
auto three_1d = ov::op::v0::Constant::create(ov::element::i64, {1}, {3});
|
||||
auto neg_one_1d = ov::op::v0::Constant::create(ov::element::i64, {1}, {-1});
|
||||
auto axes = ov::op::v0::Constant::create(ov::element::i64, {2}, {-2,-1});
|
||||
auto inp_pos = tensor_map.at("inp_pos").get_node_shared_ptr();
|
||||
auto gather_inp_pos = std::make_shared<ov::op::v8::Gather>(inp_pos, neg_one_1d, three_1d);
|
||||
auto reshaped_inp_pos = std::make_shared<ov::op::v1::Reshape>(gather_inp_pos, ov::op::v0::Constant::create(ov::element::i64, {1}, {1}), false);
|
||||
auto inp_pos_incremented = std::make_shared<ov::op::v1::Add>(reshaped_inp_pos, ov::op::v0::Constant::create(ov::element::i32, ov::Shape{1}, {1}));
|
||||
auto stop = std::make_shared<ov::op::v0::Concat>(ov::OutputVector{token_len_per_seq, std::make_shared<v1::ConvertLike>(inp_pos_incremented, token_len_per_seq)}, 0);
|
||||
mask_sliced =
|
||||
std::make_shared<ov::op::v8::Slice>(mask, zero_2d, stop, one_2d, axes);
|
||||
mask_sliced = std::make_shared<ov::op::v0::Convert>(mask_sliced, ov::element::f16);
|
||||
mask_sliced->set_friendly_name(sliced_name);
|
||||
} else {
|
||||
auto zero = ov::op::v0::Constant::create(ov::element::i64, {1}, {0});
|
||||
auto one = ov::op::v0::Constant::create(ov::element::i64, {1}, {1});
|
||||
auto two = ov::op::v0::Constant::create(ov::element::i64, {1}, {2});
|
||||
mask_sliced = std::make_shared<ov::op::v8::Slice>(mask, zero, token_len_per_seq, one, two);
|
||||
mask_sliced = std::make_shared<ov::op::v0::Convert>(mask_sliced, ov::element::f16);
|
||||
mask_sliced->set_friendly_name(sliced_name);
|
||||
}
|
||||
tensor_map.insert({sliced_name, mask_sliced->output(0)});
|
||||
}
|
||||
};
|
||||
|
||||
create_sliced_mask("self_kq_mask", "KQ_mask_sliced", ggml_model_decoder.is_static());
|
||||
create_sliced_mask("self_kq_mask_swa", "KQ_mask_swa_sliced", ggml_model_decoder.is_static());
|
||||
}
|
||||
|
||||
void add_rope_sin_cos(TensorMap & tensor_map, GgmlDecoder & ggml_model_decoder) {
|
||||
int32_t * rope_params = ggml_model_decoder.get_rope_params();
|
||||
if (tensor_map.find("inp_pos") == tensor_map.end() || rope_params == nullptr) {
|
||||
return;
|
||||
}
|
||||
auto inp_pos = tensor_map.at("inp_pos").get_node_shared_ptr();
|
||||
std::shared_ptr<ov::Node> rope_freqs_weight;
|
||||
if (tensor_map.find("rope_freqs.weight") != tensor_map.end()) {
|
||||
rope_freqs_weight = tensor_map.at("rope_freqs.weight").get_node_shared_ptr();
|
||||
}
|
||||
|
||||
auto sin_cos = make_sin_cos(rope_params, inp_pos, rope_freqs_weight);
|
||||
auto sin_theta = sin_cos.first;
|
||||
auto cos_theta = sin_cos.second;
|
||||
|
||||
cos_theta.get_node_shared_ptr()->set_friendly_name("rope_cos");
|
||||
sin_theta.get_node_shared_ptr()->set_friendly_name("rope_sin");
|
||||
tensor_map.insert({"rope_cos", cos_theta});
|
||||
tensor_map.insert({"rope_sin", sin_theta});
|
||||
}
|
||||
|
||||
// Create common patterns
|
||||
void preprocess(TensorMap & tensor_map, GgmlDecoder & ggml_model_decoder) {
|
||||
add_sliced_mask(tensor_map, ggml_model_decoder);
|
||||
add_rope_sin_cos(tensor_map, ggml_model_decoder);
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
TranslateSession::TranslateSession(const frontend::InputModel::Ptr & input_model,
|
||||
const std::unordered_map<std::string, CreatorFunction> & translator_map,
|
||||
bool naive) :
|
||||
m_input_model(input_model),
|
||||
m_translator_map(translator_map),
|
||||
m_ov_model(nullptr),
|
||||
m_naive(naive) {}
|
||||
|
||||
std::shared_ptr<Model> TranslateSession::get_converted_model() {
|
||||
if (m_ov_model) {
|
||||
return m_ov_model;
|
||||
}
|
||||
m_ov_model = translate_graph(m_input_model);
|
||||
return m_ov_model;
|
||||
}
|
||||
|
||||
std::shared_ptr<Model> TranslateSession::translate_graph(const frontend::InputModel::Ptr & input_model) {
|
||||
ov::ParameterVector params;
|
||||
ov::ResultVector results;
|
||||
auto tensor_map = std::make_shared<TensorMap>();
|
||||
std::shared_ptr<Model> resulting_model;
|
||||
|
||||
const auto & ggml_model = std::dynamic_pointer_cast<InputModel>(input_model);
|
||||
std::shared_ptr<GgmlDecoder> ggml_model_decoder = ggml_model->get_model_decoder();
|
||||
|
||||
for (const auto & it : ggml_model_decoder->get_model_inputs()) {
|
||||
params.push_back(std::dynamic_pointer_cast<ov::op::v0::Parameter>(it.second));
|
||||
(*tensor_map)[it.first] = it.second;
|
||||
}
|
||||
|
||||
for (const auto & it : ggml_model_decoder->get_model_extra_inputs()) {
|
||||
if (std::dynamic_pointer_cast<ov::op::v0::Parameter>(it.second)) {
|
||||
params.push_back(std::dynamic_pointer_cast<ov::op::v0::Parameter>(it.second));
|
||||
}
|
||||
(*tensor_map)[it.first] = it.second;
|
||||
}
|
||||
|
||||
for (const auto & it : ggml_model_decoder->get_model_weights()) {
|
||||
(*tensor_map)[it.first] = it.second;
|
||||
}
|
||||
|
||||
auto node_visitor = [&](std::shared_ptr<GgmlDecoder> decoder, int node_idx) {
|
||||
auto operation_type = decoder->get_op_type(node_idx);
|
||||
if (operation_type == "GGML_OP_NONE") {
|
||||
return;
|
||||
}
|
||||
|
||||
ov::OutputVector converted_outputs;
|
||||
auto it = m_translator_map.find(operation_type);
|
||||
FRONT_END_OP_CONVERSION_CHECK(it != m_translator_map.end(), "Translation for operation type ", operation_type,
|
||||
" is not implemented.");
|
||||
NodeContext node_context(decoder, tensor_map, node_idx, this);
|
||||
converted_outputs = it->second(node_context);
|
||||
|
||||
const auto & node_output_names = decoder->get_output_names(node_idx);
|
||||
FRONT_END_OP_CONVERSION_CHECK(node_output_names.size() == converted_outputs.size(), "Number of ",
|
||||
operation_type, " outputs greater than number of converted outputs, which are ",
|
||||
node_output_names.size(), " and ", converted_outputs.size(), " respectively.");
|
||||
|
||||
for (size_t i = 0; i < node_output_names.size(); ++i) {
|
||||
auto output_name = node_output_names[i];
|
||||
if (i < converted_outputs.size() && converted_outputs[i].get_node_shared_ptr() != nullptr) {
|
||||
(*tensor_map)[output_name] = converted_outputs[i];
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
if (!m_naive) {
|
||||
preprocess(*tensor_map, *ggml_model_decoder);
|
||||
}
|
||||
ggml_model_decoder->visit_subgraph(node_visitor);
|
||||
|
||||
for (const auto & name : ggml_model_decoder->get_model_output_names()) {
|
||||
FRONT_END_GENERAL_CHECK(tensor_map->find(name) != tensor_map->end(),
|
||||
"Output name not found in tensor map: ", name);
|
||||
auto result = std::make_shared<v0::Result>(tensor_map->at(name));
|
||||
result->set_friendly_name(name);
|
||||
results.push_back(result);
|
||||
}
|
||||
|
||||
ov::ParameterVector used_params;
|
||||
for (const auto & param : params) {
|
||||
if (!param->output(0).get_target_inputs().empty()) {
|
||||
used_params.push_back(param);
|
||||
}
|
||||
}
|
||||
// if (auto diff = params.size() - used_params.size()) {
|
||||
// GGML_LOG_INFO("%zu parameters are not used in the model.", diff);
|
||||
// }
|
||||
resulting_model = std::make_shared<Model>(results, used_params);
|
||||
|
||||
apply_transformations(resulting_model);
|
||||
return resulting_model;
|
||||
}
|
||||
|
||||
std::shared_ptr<Model> TranslateSession::apply_transformations(std::shared_ptr<Model> model) {
|
||||
auto ggml_model_decoder = std::dynamic_pointer_cast<InputModel>(m_input_model)->get_model_decoder();
|
||||
{
|
||||
ov::pass::Manager manager;
|
||||
manager.set_per_pass_validation(true);
|
||||
manager.register_pass<ov::pass::MarkCompressedFloatConstants>();
|
||||
|
||||
if (ggml_model_decoder->is_stateful()) {
|
||||
const auto kv_param_res_names = ggml_model_decoder->get_kv_param_res_names();
|
||||
const auto kv_param_res_pairs = get_kv_param_res_pairs(model, kv_param_res_names);
|
||||
manager.register_pass<ov::pass::MakeStateful>(kv_param_res_pairs);
|
||||
}
|
||||
|
||||
if (ggml_model_decoder->is_static()) {
|
||||
manager.register_pass<pass::EliminateZeroPoints>();
|
||||
manager.register_pass<pass::SqueezeMatmul>();
|
||||
}
|
||||
manager.run_passes(model);
|
||||
if (ggml_model_decoder->is_stateful()) {
|
||||
auto output_names = ggml_model_decoder->get_model_output_names();
|
||||
std::map<std::string, int> model_output_indexes;
|
||||
for (size_t i=0; i<output_names.size(); i++) {
|
||||
model_output_indexes.insert(std::make_pair(output_names[i], i));
|
||||
}
|
||||
ov::preprocess::PrePostProcessor ppp(model);
|
||||
for (size_t i=0; i<model->get_output_size(); i++) {
|
||||
auto output_friendly_name = model->output(i).get_node_shared_ptr()->get_friendly_name();
|
||||
auto output_id = model_output_indexes[output_friendly_name];
|
||||
auto model_output_shape = model->output(i).get_partial_shape();
|
||||
auto decoder_output_shape = ggml_model_decoder->get_output_shape(output_id);
|
||||
if (model_output_shape.rank().is_static() && decoder_output_shape.rank().is_static()
|
||||
&& model_output_shape.rank().get_length() + 1 == decoder_output_shape.rank().get_length()
|
||||
&& decoder_output_shape[0].is_static() && decoder_output_shape[0].get_length() == 1) {
|
||||
ppp.output(i).postprocess().custom([](const ov::Output<ov::Node>& node) {
|
||||
auto axes = ov::op::v0::Constant::create(ov::element::i32, ov::Shape{1}, {0});
|
||||
return std::make_shared<ov::op::v0::Unsqueeze>(node, axes);
|
||||
});
|
||||
}
|
||||
}
|
||||
model = ppp.build();
|
||||
}
|
||||
}
|
||||
return model;
|
||||
}
|
||||
|
||||
} // namespace ggml
|
||||
} // namespace frontend
|
||||
} // namespace ov
|
||||
@@ -0,0 +1,28 @@
|
||||
#pragma once
|
||||
|
||||
#include "input_model.h"
|
||||
#include "node_context.h"
|
||||
|
||||
namespace ov {
|
||||
namespace frontend {
|
||||
namespace ggml {
|
||||
|
||||
class TranslateSession {
|
||||
public:
|
||||
TranslateSession(const frontend::InputModel::Ptr& input_model,
|
||||
const std::unordered_map<std::string, CreatorFunction>& translator_map, bool naive = false);
|
||||
|
||||
std::shared_ptr<Model> get_converted_model();
|
||||
std::shared_ptr<Model> translate_graph(const frontend::InputModel::Ptr& input_model);
|
||||
|
||||
private:
|
||||
std::shared_ptr<Model> apply_transformations(std::shared_ptr<Model> model);
|
||||
const frontend::InputModel::Ptr m_input_model;
|
||||
const std::unordered_map<std::string, CreatorFunction>& m_translator_map;
|
||||
std::shared_ptr<Model> m_ov_model;
|
||||
bool m_naive;
|
||||
};
|
||||
|
||||
} // namespace ggml
|
||||
} // namespace frontend
|
||||
} // namespace ov
|
||||
@@ -0,0 +1,226 @@
|
||||
#include "utils.h"
|
||||
|
||||
#include "ggml-impl.h"
|
||||
|
||||
#include <cstddef>
|
||||
#include <ctime>
|
||||
#include <memory>
|
||||
#include <openvino/op/add.hpp>
|
||||
#include <openvino/op/clamp.hpp>
|
||||
#include <openvino/op/convert.hpp>
|
||||
#include <openvino/op/cos.hpp>
|
||||
#include <openvino/op/divide.hpp>
|
||||
#include <openvino/op/gather.hpp>
|
||||
#include <openvino/op/maximum.hpp>
|
||||
#include <openvino/op/multiply.hpp>
|
||||
#include <openvino/op/shape_of.hpp>
|
||||
#include <openvino/op/sin.hpp>
|
||||
#include <openvino/op/squeeze.hpp>
|
||||
#include <openvino/op/subtract.hpp>
|
||||
#include <openvino/op/transpose.hpp>
|
||||
#include <string>
|
||||
|
||||
namespace ov {
|
||||
namespace frontend {
|
||||
namespace ggml {
|
||||
|
||||
std::string getCurrentTime() {
|
||||
std::time_t now = std::time(nullptr);
|
||||
char buf[100];
|
||||
std::strftime(buf, sizeof(buf), "%Y-%m-%d %H:%M:%S", std::localtime(&now));
|
||||
return buf;
|
||||
}
|
||||
|
||||
void num_inputs_check(const NodeContext & context, size_t min_inputs, size_t max_inputs) {
|
||||
auto input_size = context.get_input_size();
|
||||
FRONT_END_OP_CONVERSION_CHECK(input_size >= min_inputs, "Got less inputs than expected");
|
||||
FRONT_END_OP_CONVERSION_CHECK(input_size <= max_inputs, "Got more inputs than expected");
|
||||
}
|
||||
|
||||
int non_cont_dim(std::vector<size_t> ne, std::vector<size_t> nb) {
|
||||
int dim = nb.size() - 1;
|
||||
size_t bytes = nb[dim];
|
||||
for (int i = dim; i > 0; i--) {
|
||||
bytes *= ne[i];
|
||||
if (bytes != nb[i - 1]) {
|
||||
return i;
|
||||
}
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
std::shared_ptr<ov::Node> get_dimensions(const std::shared_ptr<ov::op::v3::ShapeOf> & shape,
|
||||
const std::vector<int> & dims) {
|
||||
using namespace ov::op;
|
||||
const auto zero = v0::Constant::create(ov::element::i32, ov::Shape{}, {0});
|
||||
const auto dims_const = v0::Constant::create(ov::element::i32, ov::Shape{dims.size()}, dims);
|
||||
return std::make_shared<v8::Gather>(shape, dims_const, zero);
|
||||
}
|
||||
|
||||
std::shared_ptr<ov::Node> get_dimensions(const std::shared_ptr<ov::Node> & node, const std::vector<int> & dims) {
|
||||
return get_dimensions(std::make_shared<ov::op::v3::ShapeOf>(node), dims);
|
||||
}
|
||||
|
||||
OutputVector rename_outputs_with_suffix(const OutputVector & outputs, const std::string & suffix) {
|
||||
for (const auto & output : outputs) {
|
||||
auto node = output.get_node_shared_ptr();
|
||||
std::string name = node->get_friendly_name();
|
||||
name += "_";
|
||||
name += suffix;
|
||||
node->set_friendly_name(name);
|
||||
// std::cout << name << " " << output.get_partial_shape() << std::endl;
|
||||
}
|
||||
return outputs;
|
||||
}
|
||||
|
||||
namespace {
|
||||
ov::Output<ov::Node> rope_yarn_ramp_mix(int n_dims, const float corr_dims[2], float ext_factor) {
|
||||
int half_n_dims = n_dims / 2;
|
||||
std::vector<float> dim_ids_vec(half_n_dims);
|
||||
std::iota(dim_ids_vec.begin(), dim_ids_vec.end(), 0);
|
||||
auto dim_ids = ov::op::v0::Constant::create(ov::element::f32, Shape{1, 1, 1, (size_t) half_n_dims}, dim_ids_vec);
|
||||
auto corr_low = ov::op::v0::Constant::create(ov::element::f32, Shape{1, 1, 1, 1}, {corr_dims[0]});
|
||||
auto corr_high = ov::op::v0::Constant::create(ov::element::f32, Shape{1, 1, 1, 1}, {corr_dims[1]});
|
||||
auto denom = std::make_shared<ov::op::v1::Maximum>(
|
||||
std::make_shared<ov::op::v1::Subtract>(corr_high, corr_low),
|
||||
ov::op::v0::Constant::create(ov::element::f32, Shape{1, 1, 1, 1}, {0.001f}));
|
||||
auto ramp_y =
|
||||
std::make_shared<ov::op::v1::Divide>(std::make_shared<ov::op::v1::Subtract>(dim_ids, corr_low), denom);
|
||||
auto ramp_clamped = std::make_shared<ov::op::v0::Clamp>(ramp_y, 0.0f, 1.0f);
|
||||
auto ext_factor_node = ov::op::v0::Constant::create(ov::element::f32, Shape{}, {ext_factor});
|
||||
auto ramp_mix = std::make_shared<ov::op::v1::Multiply>(ramp_clamped, ext_factor_node);
|
||||
return ramp_mix;
|
||||
}
|
||||
|
||||
float ggml_rope_yarn_corr_dim(int n_dims, int n_ctx_orig, float n_rot, float base) {
|
||||
#ifndef M_PI
|
||||
# define M_PI 3.14159265358979323846
|
||||
#endif
|
||||
return n_dims * logf(n_ctx_orig / (n_rot * 2 * (float) M_PI)) / (2 * logf(base));
|
||||
}
|
||||
|
||||
void ggml_rope_yarn_corr_dims(int n_dims,
|
||||
int n_ctx_orig,
|
||||
float freq_base,
|
||||
float beta_fast,
|
||||
float beta_slow,
|
||||
float dims[2]) {
|
||||
float start = floorf(ggml_rope_yarn_corr_dim(n_dims, n_ctx_orig, beta_fast, freq_base));
|
||||
float end = ceilf(ggml_rope_yarn_corr_dim(n_dims, n_ctx_orig, beta_slow, freq_base));
|
||||
dims[0] = std::max(0.0f, start);
|
||||
dims[1] = std::min(static_cast<float>(n_dims - 1), end);
|
||||
}
|
||||
} // namespace
|
||||
|
||||
std::pair<ov::Output<Node>, ov::Output<Node>> make_sin_cos(int32_t * rope_params,
|
||||
std::shared_ptr<ov::Node> inp_pos,
|
||||
std::shared_ptr<ov::Node> rope_freqs_weight,
|
||||
bool stateful) {
|
||||
if (stateful) {
|
||||
inp_pos = std::make_shared<ov::op::v0::Squeeze>(inp_pos, ov::op::v0::Constant::create(ov::element::i64, {1}, {0}));
|
||||
inp_pos = std::make_shared<ov::op::v0::Convert>(inp_pos, ov::element::f32);
|
||||
auto pos_perm =
|
||||
std::make_shared<ov::op::v0::Constant>(ov::element::i64, ov::Shape{3}, std::vector<int64_t>{2, 1, 0});
|
||||
inp_pos = std::make_shared<ov::op::v1::Transpose>(inp_pos, pos_perm);
|
||||
} else {
|
||||
inp_pos = std::make_shared<ov::op::v0::Convert>(inp_pos, ov::element::f32);
|
||||
auto pos_perm =
|
||||
std::make_shared<ov::op::v0::Constant>(ov::element::i64, ov::Shape{4}, std::vector<int64_t>{0, 3, 1, 2});
|
||||
inp_pos = std::make_shared<ov::op::v1::Transpose>(inp_pos, pos_perm);
|
||||
}
|
||||
|
||||
float freq_base;
|
||||
float freq_scale;
|
||||
float ext_factor;
|
||||
float attn_factor;
|
||||
float beta_fast;
|
||||
float beta_slow;
|
||||
const int n_dims = rope_params[1];
|
||||
const int n_ctx_orig = rope_params[4];
|
||||
memcpy(&freq_base, rope_params + 5, sizeof(float));
|
||||
memcpy(&freq_scale, rope_params + 6, sizeof(float));
|
||||
memcpy(&ext_factor, rope_params + 7, sizeof(float));
|
||||
memcpy(&attn_factor, rope_params + 8, sizeof(float));
|
||||
memcpy(&beta_fast, rope_params + 9, sizeof(float));
|
||||
memcpy(&beta_slow, rope_params + 10, sizeof(float));
|
||||
|
||||
const float theta_scale = powf(freq_base, -2.0f / n_dims);
|
||||
|
||||
float corr_dims[2];
|
||||
ggml_rope_yarn_corr_dims(n_dims, n_ctx_orig, freq_base, beta_fast, beta_slow, corr_dims);
|
||||
|
||||
std::vector<float> factor(n_dims / 2);
|
||||
factor[0] = 1.0f;
|
||||
for (size_t i = 1; i < factor.size(); i++) {
|
||||
factor[i] = theta_scale * factor[i - 1];
|
||||
}
|
||||
|
||||
Output<Node> freq_factors;
|
||||
if (stateful) {
|
||||
freq_factors =
|
||||
std::make_shared<ov::op::v0::Constant>(ov::element::f32, ov::Shape{1, 1, factor.size()}, factor);
|
||||
} else {
|
||||
freq_factors =
|
||||
std::make_shared<ov::op::v0::Constant>(ov::element::f32, ov::Shape{1, 1, 1, factor.size()}, factor);
|
||||
}
|
||||
if (rope_freqs_weight) {
|
||||
freq_factors = std::make_shared<ov::op::v1::Divide>(freq_factors, rope_freqs_weight);
|
||||
}
|
||||
|
||||
auto theta_extrap = std::make_shared<ov::op::v1::Multiply>(freq_factors, inp_pos);
|
||||
auto theta_interp = std::make_shared<ov::op::v1::Multiply>(
|
||||
theta_extrap, ov::op::v0::Constant::create(ov::element::f32, {1}, {freq_scale}));
|
||||
|
||||
Output<Node> theta;
|
||||
float mscale = attn_factor;
|
||||
if (ext_factor == 0.0f) {
|
||||
theta = theta_interp;
|
||||
} else {
|
||||
auto ramp_mix = rope_yarn_ramp_mix(n_dims, corr_dims, ext_factor);
|
||||
Output<Node> one;
|
||||
if (stateful) {
|
||||
one = ov::op::v0::Constant::create(ov::element::f32, Shape{1, 1, 1}, {1.0f});
|
||||
} else {
|
||||
one = ov::op::v0::Constant::create(ov::element::f32, Shape{1, 1, 1, 1}, {1.0f});
|
||||
}
|
||||
auto one_minus_ramp = std::make_shared<ov::op::v1::Subtract>(one, ramp_mix);
|
||||
|
||||
theta = std::make_shared<ov::op::v1::Add>(std::make_shared<ov::op::v1::Multiply>(theta_interp, one_minus_ramp),
|
||||
std::make_shared<ov::op::v1::Multiply>(theta_extrap, ramp_mix));
|
||||
mscale *= (1.0f + 0.1f * std::log(1.0f / freq_scale));
|
||||
}
|
||||
|
||||
Output<Node> cos_theta = std::make_shared<ov::op::v0::Cos>(theta);
|
||||
Output<Node> sin_theta = std::make_shared<ov::op::v0::Sin>(theta);
|
||||
|
||||
auto mscale_node = ov::op::v0::Constant::create(ov::element::f32, Shape{}, {mscale});
|
||||
|
||||
cos_theta = std::make_shared<ov::op::v1::Multiply>(cos_theta, mscale_node);
|
||||
sin_theta = std::make_shared<ov::op::v1::Multiply>(sin_theta, mscale_node);
|
||||
return std::make_pair(sin_theta, cos_theta);
|
||||
}
|
||||
|
||||
ov::Output<ov::Node> process_view_input(const NodeContext & context, int input_index, int slice_len) {
|
||||
// Only works for VIEW operations that slice at the lowest dimension
|
||||
// If the VIEW also reshape the result, `slice_len` should be provided
|
||||
auto input = context.get_input(input_index);
|
||||
auto * op_params = (size_t *) context.get_input_op_params(input_index);
|
||||
auto src1_stride = context.get_input_stride(input_index);
|
||||
|
||||
int64_t split_addr = op_params[0] / src1_stride[3];
|
||||
if (slice_len == 0) {
|
||||
slice_len = context.get_input_shape(input_index)[3].get_length();
|
||||
}
|
||||
int64_t slice_end = split_addr + slice_len;
|
||||
|
||||
auto begin = ov::op::v0::Constant::create(ov::element::i64, {1}, {split_addr});
|
||||
auto end = ov::op::v0::Constant::create(ov::element::i64, {1}, {slice_end});
|
||||
auto stride = ov::op::v0::Constant::create(ov::element::i64, {1}, {1});
|
||||
auto axes = ov::op::v0::Constant::create(ov::element::i64, {1}, {context.is_stateful() ? 2 : 3});
|
||||
auto sliced = std::make_shared<ov::op::v8::Slice>(input, begin, end, stride, axes);
|
||||
return sliced;
|
||||
}
|
||||
|
||||
} // namespace ggml
|
||||
} // namespace frontend
|
||||
} // namespace ov
|
||||
@@ -0,0 +1,85 @@
|
||||
#pragma once
|
||||
|
||||
#include <memory>
|
||||
#include <openvino/core/node.hpp>
|
||||
#include <openvino/op/shape_of.hpp>
|
||||
#include <openvino/op/slice.hpp>
|
||||
#include <utility>
|
||||
|
||||
#include "node_context.h"
|
||||
|
||||
namespace ov {
|
||||
namespace frontend {
|
||||
namespace ggml {
|
||||
|
||||
std::string getCurrentTime();
|
||||
|
||||
void dump_ov_model(std::shared_ptr<ov::Model> model);
|
||||
|
||||
void num_inputs_check(const NodeContext& context, size_t min_inputs, size_t max_inputs);
|
||||
|
||||
int non_cont_dim(std::vector<size_t> ne, std::vector<size_t> nb);
|
||||
|
||||
template <typename T>
|
||||
std::vector<int> argsort_descend(const std::vector<T>& v) {
|
||||
std::vector<int> idx(v.size());
|
||||
std::iota(idx.begin(), idx.end(), 0);
|
||||
std::sort(idx.begin(), idx.end(), [&v](int i1, int i2) {
|
||||
return v[i1] > v[i2];
|
||||
});
|
||||
return idx;
|
||||
}
|
||||
|
||||
template <typename T>
|
||||
std::vector<T> sorted_descend(std::vector<T> v) {
|
||||
std::sort(v.begin(), v.end(), [](T a, T b) {
|
||||
return a > b;
|
||||
});
|
||||
return v;
|
||||
}
|
||||
|
||||
template <typename T>
|
||||
bool is_permuted(const std::vector<T>& strides) {
|
||||
for (size_t i = 0; i < strides.size() - 1; ++i) {
|
||||
if (strides[i] < strides[i + 1]) {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
template <typename T>
|
||||
std::vector<T> permute(const std::vector<T>& x, const std::vector<int>& perm) {
|
||||
std::vector<T> result;
|
||||
result.reserve(perm.size());
|
||||
for (int i : perm) {
|
||||
result.push_back(x[i]);
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
std::shared_ptr<ov::Node> get_dimensions(const std::shared_ptr<ov::op::v3::ShapeOf>& shape,
|
||||
const std::vector<int>& dims);
|
||||
std::shared_ptr<ov::Node> get_dimensions(const std::shared_ptr<ov::Node>& node, const std::vector<int>& dims);
|
||||
|
||||
OutputVector rename_outputs_with_suffix(const OutputVector& outputs, const std::string& suffix);
|
||||
|
||||
std::pair<ov::Output<Node>, ov::Output<Node>> make_sin_cos(int32_t* rope_params,
|
||||
std::shared_ptr<ov::Node> inp_pos,
|
||||
std::shared_ptr<ov::Node> rope_freqs_weight = nullptr,
|
||||
bool stateful = false);
|
||||
|
||||
ov::Output<ov::Node> process_view_input(const NodeContext& context, int input_index, int slice_len = 0);
|
||||
|
||||
namespace op {
|
||||
template <typename T>
|
||||
OutputVector translate_1to1_match_2_inputs(const NodeContext& context) {
|
||||
num_inputs_check(context, 2, 2);
|
||||
auto res = std::make_shared<T>(context.get_input(0), context.get_input(1));
|
||||
return rename_outputs_with_suffix({res}, context.get_name());
|
||||
}
|
||||
} // namespace op
|
||||
|
||||
} // namespace ggml
|
||||
} // namespace frontend
|
||||
} // namespace ov
|
||||
Reference in New Issue
Block a user