* ggml-openvino : Qwen3.5 MoE perf (#312) Squash of ravi9/llama.cpp#312: - ggml-openvino: add detailed inference profiling (Yu, Zijun) - ggml-openvino: use remote output tensors by default (Yu, Zijun) - ggml-openvino: optimize single-sequence recurrent state (Yu, Zijun) - opt1: remove recurrent reset for single sequence, opt2: direct gdn outputs (break parallel sequence) (Yu, Zijun) - fix parallel sequences (Yu, Zijun) - ggml-openvino: simplify graph cache key (ynimmaga) - enable stateful for qwen35 single sequence (Yu, Zijun) - Fix after rebasing (Yu, Zijun) - Add k-requant option q4_asym64 (Yu, Zijun) - Fix qwen35 llama-bench -p 0 (Yu, Zijun) - Simplify RESHAPE translation (Yu, Zijun) - openvino: fuse MoE routing (Yu, Zijun) - openvino: fuse GDN qk normalization (Yu, Zijun) - openvino: enable GPU MoE fusion by default (Yu, Zijun) - ggml-openvino: add cache_only mode to import cached compiled model on disk directly (Yu, Zijun) - openvino : report the device allocation limit to ggml (Łukasz Ślusarczyk) - Fix windows build (Yu, Zijun) Co-authored-by: ynimmaga <ynimmaga@users.noreply.github.com> Co-authored-by: Łukasz Ślusarczyk <lukasz.slusarczyk@intel.com> * ggml-openvino: Update doc of compiled model cache * openvino: implement PRD-compliant device enumeration and memory reporting * openvino: fix multi-device listing issues from review - Only the device selected by GGML_OPENVINO_DEVICE reports as GPU; the other OpenVINO devices report as IGPU so llama.cpp does not offload to them. Initializing a non-selected device logs a warning. - Name devices OPENVINO<i> again and show the OpenVINO id in the description. Raw "CPU" names shadowed the ggml CPU backend. - Support GPU.N: create the OpenCL queue on OpenVINO's own context for the selected device, and replace "GPU"/"NPU" string comparisons with ggml_openvino_is_gpu()/ggml_openvino_is_npu(). - An unavailable GGML_OPENVINO_DEVICE is now an error that lists the available devices, instead of silently falling back to CPU. - Memory: cap iGPU/NPU free memory at system available memory, fall back to system memory instead of 0/0 when the plugin lacks memory properties, and ignore host USM allocations in GPU usage. - Initialize the device config once under a lock, even if OpenCL setup fails. - Fix supports_op return type for non-selected devices (build error). * openvino : take USM entry points from the selected device platform clGetExtensionFunctionAddressForPlatform was called on the first platform returned by clGetPlatformIDs. The address it returns is only valid for the platform it was queried on, and the first platform is not always the one that holds the device OpenVINO selected. On a host whose first platform comes from another vendor the lookup returns null, and then every read, write and memset on a GPU buffer fails with "clEnqueueMemcpyINTEL not available". Look both entry points up in init(), on the platform of the device OpenVINO picked, and keep them in the device config next to the command queue. Assisted-by: Claude Opus 5 * openvino: fuse MoE experts for models with a fused gate_up weight FuseMoeCompressed only matches models whose gate and up projections are separate GatherMatmul ops. gemma-4 packs both into one expert weight and splits the result after the GEMM, so its MoE block stayed unfused and ran the expert GEMMs as per-token GEMVs. Add FuseMoeCompressedFusedGateUp, which matches that shape (one GatherMatmul -> Slice/Slice -> Gelu(ERF) -> Multiply) and folds it into the same MOECompressed op, using GEMM3_SWIGLU with GEGLU_ERF. The fused weight, scale and zero point are split into gate/up halves by copying raw bytes, since a graph Slice would be rewritten to StridedSlice and constant folded, whose reference evaluator crashes on sub-byte types. gemma-4 also applies a per-expert output scale to the down projection before the router weights. MOECompressed takes only one per-expert weight, so that scale is folded into the routing weights, which is exact. The op reads the zero point straight off a weight port and needs an integer Constant there, so the matcher requires one and leaves natively quantized experts (exact f16 zp) to the unfused path. gemma-4-26B-A4B on Arc B390, GGML_OPENVINO_REQUANT_KQUANT=q4_asym64_all, llama-bench -p 512 -n 128 -r 2, against a GGML_OPENVINO_MOE_OP=0 baseline: pp512 66.16 -> 1608.73 t/s, tg128 25.94 -> 26.46 t/s. Perplexity over 12 chunks is unchanged (1451.3 +/- 177.9 unfused vs 1427.6 +/- 175.1 fused). No effect without that requant option, on models with separate gate/up weights, or on CPU. test-backend-ops -b OPENVINO0 is unchanged by this commit: two MUL_MAT_ID m_v cases fail, the same two on the unmodified base. * openvino: fix rank-3 axis handling so MoE works under stateful execution Stateful execution drops the leading size-1 batch dim, so OV tensors are rank 3 while GgmlOvDecoder::get_shape/get_stride still report GGML_MAX_DIMS=4 reversed entries. Several MoE ops derive OV axis indices straight from that metadata, so they picked the wrong axis. A MoE model with GGML_OPENVINO_STATEFUL_EXECUTION=1 aborts while building the graph: Check 'is_axis_valid(axis, r)' failed at src/core/src/validation_util.cpp:336 While validating node 'opset11::TopK ... _ffn_moe_probs ...' Axis 3 out of the tensor rank range [-3, 2]. Fix idiom throughout: take the axis from the real OV rank, or shift a metadata-derived axis down by metadata_rank - actual_rank. argsort.cpp the router top-k axis is 2 on rank 3, not 3. This is the abort quoted above. add.cpp the MoE expert-sum bypass collapses the 8-ADD chain into one ReduceSum on hardcoded axis 2, which on rank 3 reduces n_embd instead of the expert axis. Now rank-2, with the following Unsqueeze at rank-3. get_rows.cpp squeezing a hardcoded {0,1} also strips the batch dim whenever it is 1, which is every decode step. Squeeze down to the trailing two dims instead. mul_mat_id.cpp pick the reshape dims by actual rank, and skip the trailing Unsqueeze that re-adds the batch dim. view.cpp the expert-plane slice had the Slice axis, dst_ov_axis, the ShapeOf+Gather index and the Reshape target all rank-4. utils.cpp process_view_input_new's "translate_view already resolved this VIEW, skip re-slicing" shortcut required equal ranks. 4 vs 3 never matched, so every resolved expert plane got re-sliced. Now compares the common trailing dims. Same axis shift for the Slice in the view-chain walker. Stateless is unchanged by construction: every edit is gated on the actual rank, so axis_shift == 0 reproduces the previous code exactly. Checked on OV-CPU by diffing greedy output against the unmodified base for dense gemma-4-E2B, granite-1b-a400m and gemma-4-26B-A4B; all identical. granite-1b-a400m on OV-CPU aborts with the error above before this change; after it, it generates and is byte-identical to stateless. Dense gemma-4-E2B is identical stateless vs stateful both before and after. test-backend-ops -b OPENVINO0 is unchanged: two pre-existing MUL_MAT_ID m_v cases fail, the same two on the unmodified base. gemma-4-26B-A4B is a poor correctness vehicle here. On OV it already drifts into degenerate repetition a few tokens in, in stateless as much as stateful, and the two modes diverge somewhere inside that degenerate region instead of matching token for token. Each mode is self-reproducible across runs. Known limitation: FuseMoeCompressedFusedGateUp does not match the rank-3 graph, so a MoE model run with GGML_OPENVINO_STATEFUL_EXECUTION=1 loses the prefill fusion while gaining decode. gemma-4-26B-A4B on Arc B390, GGML_OPENVINO_REQUANT_KQUANT=q4_asym64_all, llama-bench -p 512 -n 128 -r 2: unfused (GGML_OPENVINO_MOE_OP=0) pp512 66.16 tg128 25.94 fused, stateless (default) pp512 1608.73 tg128 26.46 fused, stateful pp512 66.18 tg128 29.91 Stateful is opt-in and off by default, and MoE did not run there at all before this, so nothing that previously worked regresses. Making the pass match rank 3 is the follow-up. * OpenVINO Backend: Upgrade graph cache to use node_idx, src_idx, node type * ggml-openvino : enable more comprehensive conv fusion * enable conv ops * Reject kernel size 0 and support IM2COL_3D * openvino : abort when the GPU remote context cannot be created init() logged the error and returned, which left the device name a GPU but remote_context empty. The remote buffer and tensor paths assert only on the device being a GPU and then dereference that empty optional. Those paths have no host fallback, and a device that OpenVINO listed should have a working OpenCL context, so stop instead of continuing. An OpenCL stack that is broken as a whole is still caught earlier by the device availability check, which falls back to CPU. Assisted-by: Claude Opus 5 * openvino : fix build warnings The single-argument form of the OpenVINO RTTI macros is the intended one, but their selector macro leaves __VA_ARGS__ empty, which -Wpedantic reports on every pass and op header. Turn that warning off for this backend only, the way ggml-cuda and ggml-sycl already do for their own third-party warnings. Also drop a break and a dead assignment around a GGML_ABORT, which is noreturn. Assisted-by: Claude Opus 5 * OpenVINO Backend: Support common MTMD ops * ggml-openvino: give a reshaping view its own ov::Tensor * ggml-openvino : compute HARDSIGMOID and EXPM1 in f32 HARDSIGMOID used a 1/6 constant in the input type, which is not exact in bf16, and EXPM1 lost precision for small inputs in f16. Both now compute in f32 and convert back, except on NPU where the f32 path gives wrong results. Fixes the HARDSIGMOID/EXPM1 test-backend-ops failures on GPU. * ggml-openvino : update device selection and --list-devices Show the selecting GGML_OPENVINO_DEVICE value and active device in --list-devices, startup logs, and backend tests. Clarify OpenVINO selection uses GGML_OPENVINO_DEVICE, not -dev. * openvino : remove unreachable OpenCL queue checks A remote buffer exists only on a GPU device, and init() aborts there if the queue cannot be created, so the queue is never null at these call sites. Assisted-by: Claude Opus 5 * openvino : update OpenVINO to 2026.4.1 and GPU drivers to 26.35.39758.10 * docs : update OpenVINO validated models and GPU driver version * ggml-openvino : skip empty views when giving a reshaping view its own tensor A zero-size view can sit at the end of a GPU USM buffer (Qwen3.5 recurrent cache). Wrapping it as a remote tensor throws "shared USM buffer has smaller size (0)". Assisted-by: Claude * ggml-openvino : rebind the cached decoder when llama passes a different graph llama keeps separate graphs for batches with and without outputs. llama-server splits the prompt into chunks for context checkpoints, so a cached decoder could be reused with a graph built in other memory and bind the previous chunk's input tensors. SWA and recurrent models then lost most of the prompt in llama-cli and llama-server. Assisted-by: Claude * docs : update OpenVINO validated models Smoke test on Lunar Lake (32 GB) with the two fixes above. Re-add the Qwen3.5 and gemma models. Assisted-by: Claude --------- Co-authored-by: Yu, Zijun <zijun.yu@intel.com> Co-authored-by: ynimmaga <ynimmaga@users.noreply.github.com> Co-authored-by: Łukasz Ślusarczyk <lukasz.slusarczyk@intel.com> Co-authored-by: haarika-madaka <haarika.madaka@intel.com> Co-authored-by: Mustafa Cavus <mustafa.cavus@intel.com> Co-authored-by: Mostafa Faheem <mostafaaafaheem@gmail.com>
814 lines
44 KiB
Markdown
814 lines
44 KiB
Markdown
# OpenVINO Backend for llama.cpp
|
|
|
|
> [!NOTE]
|
|
> Performance and memory optimizations, accuracy validation, broader quantization coverage, broader operator and model support are work in progress.
|
|
|
|
[OpenVINO](https://docs.openvino.ai/) is an open-source toolkit for optimizing and deploying high-performance AI inference, specifically designed for Intel hardware, including CPUs, GPUs, and NPUs, in the cloud, on-premises, and on the edge. [OpenVINO backend for llama.cpp](../../ggml/src/ggml-openvino) enables hardware-accelerated inference on **Intel® CPUs, GPUs, and NPUs** while remaining compatible with the existing **GGUF model ecosystem**. The backend translates GGML compute graphs into OpenVINO graphs and leverages graph compilation, kernel fusion, and device-specific optimizations to improve inference performance on supported Intel hardware.
|
|
|
|
The OpenVINO backend is implemented in `ggml/src/ggml-openvino` and provides a translation layer for core GGML operations. The OpenVINO backend replaces the standard GGML graph execution path with Intel's OpenVINO inference engine. This approach allows the same GGUF model file to run on Intel CPUs, Intel GPUs (integrated and discrete), and Intel NPUs without changes to the model or the rest of the llama.cpp stack. When a `ggml_cgraph` is dispatched to OpenVINO backend, it:
|
|
|
|
- Walks the GGML graph and identifies inputs, outputs, weights, and KV cache tensors.
|
|
- Translates the GGML operations into an `ov::Model` using OpenVINO's frontend API.
|
|
- Compiles and caches the model for the target device.
|
|
- Binds GGML tensor memory to OpenVINO inference tensors and runs inference.
|
|
|
|
For guidance on contributing to the OpenVINO backend, see the [OpenVINO Backend Contributing Guide](https://github.com/ravi9/llamacpp-ov-dev-guide/blob/main/contributing-llamacpp-ov.md).
|
|
|
|
## Contents
|
|
|
|
- [Supported Devices](#supported-devices)
|
|
- [Supported Model Precisions](#supported-model-precisions)
|
|
- [Supported Llama.cpp Tools](#supported-llamacpp-tools)
|
|
- [Validated Models](#validated-models)
|
|
- [Build Instructions](#build-instructions)
|
|
- [0. Prerequisites](#0-prerequisites)
|
|
- [1. Install OpenVINO Runtime](#1-install-openvino-runtime)
|
|
- [2. Build llama.cpp with OpenVINO Backend](#2-build-llamacpp-with-openvino-backend)
|
|
- [Ubuntu Build Script](#ubuntu-build-script)
|
|
- [Windows Build Script](#windows-build-script)
|
|
- [3. Download Sample Model](#3-download-sample-model)
|
|
- [4. Run Inference with OpenVINO Backend](#4-run-inference-with-openvino-backend)
|
|
- [5. Docker Build](#5-docker-build)
|
|
- [GGML OpenVINO Backend Runtime Configurations](#ggml-openvino-backend-runtime-configurations)
|
|
- [Known Limitations](#known-limitations)
|
|
- [Work in Progress](#work-in-progress)
|
|
|
|
## Supported Devices
|
|
|
|
OpenVINO backend supports the following hardware:
|
|
|
|
- Intel CPUs
|
|
- Intel GPUs (integrated and discrete)
|
|
- Intel NPUs
|
|
|
|
Although OpenVINO supports a wide range of [Intel hardware](https://docs.openvino.ai/2026/about-openvino/release-notes-openvino/system-requirements.html), the llama.cpp OpenVINO backend has been validated specifically on AI PCs such as the Intel® Core™ Ultra Series 1 and Series 2.
|
|
|
|
## Supported Model Precisions
|
|
|
|
- `FP16`
|
|
- `BF16` (on Intel Xeon)
|
|
- `Q8_0`
|
|
- `Q4_0`
|
|
- `Q4_1`
|
|
- `Q4_K`
|
|
- `Q4_K_M`
|
|
- `Q5_K` (converted to `Q8_0_C` at runtime by default)
|
|
- `Q6_K` (converted to `Q8_0_C` at runtime by default)
|
|
|
|
> [!NOTE]
|
|
> Accuracy validation and performance optimizations for quantized models are a work in progress.
|
|
|
|
**CPU and GPU Quantization Details:**
|
|
- `Q5_K` and `Q6_K` tensors are converted to `Q8_0_C`
|
|
|
|
**NPU Quantization Details:**
|
|
- Primary supported quantization scheme is `Q4_0`
|
|
- `Q6_K` tensors are requantized to `Q4_0_128` in general. For embedding weights, `Q6_K` tensors are requantized to `Q8_0_C` except for the token embedding matrix which is dequantized to fp16
|
|
|
|
**Additional Notes:**
|
|
- Both `Q4_0` and `Q4_1` models use `Q6_K` for the token embedding tensor and the final matmul weight tensor (often the same tensor)
|
|
- `Q4_0` models may produce some `Q4_1` tensors if an imatrix is provided during quantization using `llama-quantize`
|
|
- `Q4_K_M` models may include both `Q6_K` and `Q5_K` tensors (observed in Phi-3)
|
|
- `Q5_1` tensors are dequantized natively (weights, scales, and zero-points extracted directly)
|
|
|
|
## Supported Llama.cpp Tools
|
|
|
|
The OpenVINO backend integrates with the standard llama.cpp tools listed below.
|
|
However, all the tools coverage across all devices is not uniform and exhaustive validation is work in progress.
|
|
|
|
- llama-bench
|
|
- llama-cli
|
|
- llama-completion
|
|
- llama-embedding
|
|
- llama-perplexity
|
|
- llama-run
|
|
- llama-server
|
|
- llama-simple
|
|
|
|
## Validated Models
|
|
|
|
Although, the validated models below were tested with `llama-cli` using the `Q4_K_M` quantization format on Intel® Core™ Ultra Series 2 (Lunar Lake), the OpenVINO backend is expected to work across a broader range of [Intel hardware](https://docs.openvino.ai/2026/about-openvino/release-notes-openvino/system-requirements.html), [supported model precisions](#supported-model-precisions), [supported llama.cpp tools](#supported-llamacpp-tools) and additional model architectures.
|
|
|
|
> [!NOTE]
|
|
> Extensive accuracy validation, performance optimizations, and broader architecture coverage are work in progress.
|
|
|
|
**Legend & Test Configuration:**
|
|
- **Status:** ✓ = Passed | ~ = Accuracy issues | ✗ = Failed or Unsupported
|
|
- **Execution Modes:**
|
|
- **SL** = Stateless (`GGML_OPENVINO_STATEFUL_EXECUTION=0`)
|
|
- **SF** = Stateful (`GGML_OPENVINO_STATEFUL_EXECUTION=1`)
|
|
- Note: The NPU operates in stateless mode only.
|
|
- **Validation system:** Intel® Core™ Ultra 5 238V (Lunar Lake) | 32 GB RAM | Ubuntu 24.04 | Intel Graphics Compiler 2.41.5 | Intel OpenCL GPU Driver 26.35.39758.10-0 | Intel NPU Driver 1.38.0.
|
|
- See [Known Limitations](#known-limitations) for context on observed failures.
|
|
|
|
| Model | CPU (SL / SF) | GPU (SL / SF) | NPU (SL) |
|
|
| :--- | :---: | :---: | :---: |
|
|
| [bartowski/Llama-3.2-1B-Instruct-Q4_K_M](https://huggingface.co/bartowski/Llama-3.2-1B-Instruct-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |
|
|
| [bartowski/Llama-3.2-3B-Instruct-Q4_K_M](https://huggingface.co/bartowski/Llama-3.2-3B-Instruct-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |
|
|
| [bartowski/Meta-Llama-3.1-8B-Instruct-Q4_K_M](https://huggingface.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |
|
|
| | | | |
|
|
| [Qwen/qwen2.5-1.5b-instruct-q4_k_m](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |
|
|
| [Qwen/qwen2.5-coder-7b-instruct-q4_k_m](https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |
|
|
| [bartowski/Qwen_Qwen3-0.6B-Q4_K_M](https://huggingface.co/bartowski/Qwen_Qwen3-0.6B-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |
|
|
| [bartowski/Qwen_Qwen3-1.7B-Q4_K_M](https://huggingface.co/bartowski/Qwen_Qwen3-1.7B-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |
|
|
| [Qwen/Qwen3-4B-Q4_K_M](https://huggingface.co/Qwen/Qwen3-4B-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |
|
|
| [lm-kit/Qwen3-8B-Q4_K_M](https://huggingface.co/lm-kit/qwen-3-8b-instruct-gguf) | ✓ / ✓ | ✓ / ✓ | ✓ |
|
|
| [bartowski/Qwen_Qwen3.5-0.8B-Q4_K_M](https://huggingface.co/bartowski/Qwen_Qwen3.5-0.8B-GGUF) | ✓ / ✓ | ✓ / ~ | ✗ |
|
|
| [bartowski/Qwen_Qwen3.5-2B-Q4_K_M](https://huggingface.co/bartowski/Qwen_Qwen3.5-2B-GGUF) | ✓ / ✓ | ✓ / ~ | ✗ |
|
|
| [bartowski/Qwen_Qwen3.5-4B-Q4_K_M](https://huggingface.co/bartowski/Qwen_Qwen3.5-4B-GGUF) | ✓ / ✓ | ✓ / ~ | ✗ |
|
|
| [lmstudio-community/Qwen3.5-9B-Q4_K_M](https://huggingface.co/lmstudio-community/Qwen3.5-9B-GGUF) | ✓ / ✓ | ✓ / ~ | ✗ |
|
|
| | | | |
|
|
| [unsloth/gemma-3-4b-it-Q4_K_M](https://huggingface.co/unsloth/gemma-3-4b-it-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |
|
|
| [bartowski/google_gemma-4-E2B-it-Q4_K_M](https://huggingface.co/bartowski/google_gemma-4-E2B-it-GGUF) | ✓ / ✓ | ✓ / ~ | ~ |
|
|
| [bartowski/google_gemma-4-E4B-it-Q4_K_M](https://huggingface.co/bartowski/google_gemma-4-E4B-it-GGUF) | ✓ / ✓ | ✗ / ✗ | ✓ |
|
|
| [bartowski/gemma-4-12B-it-Q4_K_M](https://huggingface.co/bartowski/gemma-4-12B-it-GGUF) | ✓ / ✓ | ✓ / ✓ | ✗ |
|
|
| | | | |
|
|
| [bartowski/Phi-3-mini-4k-instruct-Q4_K_M](https://huggingface.co/bartowski/Phi-3-mini-4k-instruct-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |
|
|
| [bartowski/Phi-3.5-mini-instruct-Q4_K_M](https://huggingface.co/bartowski/Phi-3.5-mini-instruct-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |
|
|
| [bartowski/microsoft_Phi-4-mini-instruct-Q4_K_M](https://huggingface.co/bartowski/microsoft_Phi-4-mini-instruct-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |
|
|
| | | | |
|
|
| [bartowski/Mistral-7B-Instruct-v0.3-Q4_K_M](https://huggingface.co/bartowski/Mistral-7B-Instruct-v0.3-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |
|
|
| [QuantFactory/Ministral-3b-instruct.Q4_K_M](https://huggingface.co/QuantFactory/Ministral-3b-instruct-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |
|
|
| [bartowski/Ministral-8B-Instruct-2410-Q4_K_M](https://huggingface.co/bartowski/Ministral-8B-Instruct-2410-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |
|
|
| | | | |
|
|
| [bartowski/DeepSeek-R1-Distill-Llama-8B-Q4_K_M](https://huggingface.co/bartowski/DeepSeek-R1-Distill-Llama-8B-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |
|
|
| [bartowski/DeepSeek-R1-Distill-Qwen-7B-Q4_K_M](https://huggingface.co/bartowski/DeepSeek-R1-Distill-Qwen-7B-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |
|
|
| | | | |
|
|
| [ibm-granite/granite-4.0-350m-Q4_K_M](https://huggingface.co/ibm-granite/granite-4.0-350m-GGUF) | ✓ / ✓ | ~ / ~ | ✓ |
|
|
| [ibm-granite/granite-4.0-micro-Q4_K_M](https://huggingface.co/ibm-granite/granite-4.0-micro-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |
|
|
| [ibm-granite/granite-4.0-1b-Q4_K_M](https://huggingface.co/ibm-granite/granite-4.0-1b-GGUF) | ✓ / ✓ | ~ / ~ | ~ |
|
|
| [ibm-research/granite-3.2-8b-instruct-Q4_K_M](https://huggingface.co/ibm-research/granite-3.2-8b-instruct-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |
|
|
| | | | |
|
|
| [HuggingFaceTB/smollm2-1.7b-instruct-q4_k_m](https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B-Instruct-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |
|
|
| [openbmb/MiniCPM-V-2_6-Q4_K_M](https://huggingface.co/openbmb/MiniCPM-V-2_6-gguf) | ✓ / ✓ | ✓ / ✓ | ✓ |
|
|
| [bartowski/tencent_Hunyuan-7B-Instruct-Q4_K_M](https://huggingface.co/bartowski/tencent_Hunyuan-7B-Instruct-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |
|
|
| [LGAI-EXAONE/EXAONE-3.5-7.8B-Instruct-Q4_K_M](https://huggingface.co/LGAI-EXAONE/EXAONE-3.5-7.8B-Instruct-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |
|
|
| [bartowski/prism-ml_Bonsai-8B-unpacked-Q4_K_M](https://huggingface.co/bartowski/prism-ml_Bonsai-8B-unpacked-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |
|
|
| | | | |
|
|
| [gpustack/bge-m3-Q4_K_M.gguf](https://huggingface.co/gpustack/bge-m3-GGUF) | ✓ | ✗ | ✗ |
|
|
|
|
## Build Instructions
|
|
|
|
### 0. Prerequisites
|
|
|
|
- Linux or Windows system with Intel hardware (CPU, GPU, or NPU)
|
|
- **For Intel GPU or NPU Usage**: Install the appropriate hardware drivers for your Intel GPU or NPU. For detailed instructions, see: [Additional Configurations for Hardware Acceleration](https://docs.openvino.ai/2026/get-started/install-openvino/configurations.html).
|
|
|
|
- **Linux:**
|
|
- Git, CMake, and Ninja software tools are needed for building.
|
|
```bash
|
|
sudo apt-get update
|
|
sudo apt-get install -y build-essential libcurl4-openssl-dev libtbb12 cmake ninja-build python3-pip curl wget tar
|
|
```
|
|
- OpenCL
|
|
```bash
|
|
sudo apt install ocl-icd-opencl-dev opencl-headers opencl-clhpp-headers intel-opencl-icd
|
|
```
|
|
|
|
- **Windows:**
|
|
- Download and install [Microsoft Visual Studio 2022 Build Tools](https://aka.ms/vs/17/release/vs_BuildTools.exe). During installation, select the **"Desktop development with C++"** workload.
|
|
|
|
- Install required tools:
|
|
```powershell
|
|
# Windows PowerShell
|
|
winget install Git.Git
|
|
winget install GNU.Wget
|
|
winget install Ninja-build.Ninja
|
|
```
|
|
|
|
- Install **OpenCL** using **vcpkg**:
|
|
```powershell
|
|
# Windows PowerShell
|
|
cd C:\
|
|
git clone https://github.com/microsoft/vcpkg
|
|
cd vcpkg
|
|
.\bootstrap-vcpkg.bat
|
|
.\vcpkg install opencl
|
|
# Optional but recommended: Integrate vcpkg with Visual Studio / CMake:
|
|
.\vcpkg integrate install
|
|
```
|
|
|
|
### 1. Install OpenVINO Runtime
|
|
|
|
- Follow the guide to install OpenVINO Runtime from an archive file: [Linux](https://docs.openvino.ai/2026/get-started/install-openvino/install-openvino-archive-linux.html) | [Windows](https://docs.openvino.ai/2026/get-started/install-openvino/install-openvino-archive-windows.html)
|
|
|
|
- Verify OpenVINO is initialized properly:
|
|
```bash
|
|
echo $OpenVINO_DIR
|
|
```
|
|
|
|
### 2. Build llama.cpp with OpenVINO Backend
|
|
|
|
Clone llama.cpp repo and build :
|
|
|
|
```bash
|
|
git clone https://github.com/ggml-org/llama.cpp
|
|
cd llama.cpp
|
|
```
|
|
|
|
- **Linux:**
|
|
```bash
|
|
source /opt/intel/openvino/setupvars.sh
|
|
cmake -B build/ReleaseOV -G Ninja -DCMAKE_BUILD_TYPE=Release -DGGML_OPENVINO=ON
|
|
cmake --build build/ReleaseOV --parallel
|
|
```
|
|
|
|
- **Windows:** Open **x64 Native Tools Command Prompt for VS** (so the MSVC toolchain is on `PATH`), then run:
|
|
|
|
```cmd
|
|
C:\Intel\openvino\setupvars.bat
|
|
cmake -B build\ReleaseOV -G Ninja -DCMAKE_BUILD_TYPE=Release -DGGML_OPENVINO=ON -DCMAKE_TOOLCHAIN_FILE=C:\vcpkg\scripts\buildsystems\vcpkg.cmake
|
|
cmake --build build\ReleaseOV --parallel
|
|
```
|
|
|
|
> [!NOTE]
|
|
> The Windows install path is `C:\Intel\openvino` (no spaces) to avoid quoting problems some CMake/Ninja toolchains have with `C:\Program Files (x86)\...`. Adjust to wherever you installed OpenVINO Runtime. From `cmd`, run `C:\Intel\openvino\setupvars.bat`; from PowerShell, run `& "C:\Intel\openvino\setupvars.ps1"` instead. Once the build is finished you can launch the binaries from any `cmd` or `PowerShell` window after sourcing the matching `setupvars` script for that shell.
|
|
|
|
#### Ubuntu Build Script
|
|
|
|
For Ubuntu24 users, the following shell script automates the prerequisite installs (build tools, OpenCL ICD), the OpenVINO Runtime download/extract/setup, and the Ninja-based llama.cpp build.
|
|
Save the following as `build-llamacpp-ov.sh` next to where you want the `llama.cpp` folder to land, then run it:
|
|
|
|
```bash
|
|
chmod +x build-llamacpp-ov.sh
|
|
./build-llamacpp-ov.sh
|
|
```
|
|
|
|
<details>
|
|
<summary>Click to expand <code>build-llamacpp-ov.sh</code></summary>
|
|
|
|
```bash
|
|
#!/usr/bin/env bash
|
|
# ============================================
|
|
# llama.cpp OpenVINO Build Script (Ninja)
|
|
# ============================================
|
|
set -euo pipefail
|
|
|
|
OPENVINO_VERSION_MAJOR="2026.4.1"
|
|
OPENVINO_VERSION_FULL="2026.4.1.22982.07f9c262b05"
|
|
|
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
OPENVINO_INSTALL_DIR="/opt/intel/openvino_${OPENVINO_VERSION_MAJOR}"
|
|
OPENVINO_LINK_DIR="/opt/intel/openvino"
|
|
OPENVINO_TGZ="${SCRIPT_DIR}/openvino.tgz"
|
|
OPENVINO_URL="https://storage.openvinotoolkit.org/repositories/openvino/packages/${OPENVINO_VERSION_MAJOR}/linux/openvino_toolkit_ubuntu24_${OPENVINO_VERSION_FULL}_x86_64.tgz"
|
|
|
|
echo "============================================"
|
|
echo "Installing prerequisites (apt)..."
|
|
echo "============================================"
|
|
sudo apt-get update
|
|
sudo apt-get install -y \
|
|
build-essential libcurl4-openssl-dev libtbb12 \
|
|
cmake ninja-build python3-pip \
|
|
curl wget tar git
|
|
|
|
echo "============================================"
|
|
echo "Installing OpenCL runtime + headers..."
|
|
echo "============================================"
|
|
sudo apt-get install -y \
|
|
ocl-icd-opencl-dev opencl-headers opencl-clhpp-headers intel-opencl-icd
|
|
|
|
cd "${SCRIPT_DIR}"
|
|
|
|
# ============================================
|
|
# Clone llama.cpp if missing
|
|
# ============================================
|
|
if [[ ! -f "llama.cpp/CMakeLists.txt" ]]; then
|
|
echo "Cloning llama.cpp..."
|
|
git clone https://github.com/ggml-org/llama.cpp
|
|
fi
|
|
|
|
# ============================================
|
|
# Setup OpenVINO: download & extract to /opt/intel/openvino_${OPENVINO_VERSION_MAJOR},
|
|
# then point /opt/intel/openvino at it via symlink so the active version is swappable.
|
|
# ============================================
|
|
if [[ -f "${OPENVINO_INSTALL_DIR}/setupvars.sh" ]]; then
|
|
echo "OpenVINO ${OPENVINO_VERSION_MAJOR} already installed at ${OPENVINO_INSTALL_DIR}. Skipping download."
|
|
else
|
|
echo "OpenVINO not found at ${OPENVINO_INSTALL_DIR}. Starting download..."
|
|
curl -L -o "${OPENVINO_TGZ}" "${OPENVINO_URL}"
|
|
|
|
echo "Extracting OpenVINO to ${OPENVINO_INSTALL_DIR}..."
|
|
sudo mkdir -p "${OPENVINO_INSTALL_DIR}"
|
|
sudo tar -xzf "${OPENVINO_TGZ}" -C "${OPENVINO_INSTALL_DIR}" --strip-components=1
|
|
rm -f "${OPENVINO_TGZ}"
|
|
fi
|
|
|
|
# Refresh symlink: /opt/intel/openvino -> /opt/intel/openvino_${OPENVINO_VERSION_MAJOR}
|
|
sudo ln -sfn "${OPENVINO_INSTALL_DIR}" "${OPENVINO_LINK_DIR}"
|
|
|
|
OPENVINO_ROOT="${OPENVINO_LINK_DIR}"
|
|
echo "OpenVINO Ready: ${OPENVINO_ROOT} -> ${OPENVINO_INSTALL_DIR}"
|
|
|
|
# Install OpenVINO's own runtime dependencies (one-time per system).
|
|
if [[ -x "${OPENVINO_ROOT}/install_dependencies/install_openvino_dependencies.sh" ]]; then
|
|
echo "============================================"
|
|
echo "Installing OpenVINO runtime dependencies..."
|
|
echo "============================================"
|
|
echo "Y" | sudo -E "${OPENVINO_ROOT}/install_dependencies/install_openvino_dependencies.sh"
|
|
fi
|
|
|
|
# ============================================
|
|
# Clean old build cache
|
|
# ============================================
|
|
cd "${SCRIPT_DIR}/llama.cpp"
|
|
if [[ -d "build/ReleaseOV" ]]; then
|
|
echo "Removing old build directory..."
|
|
rm -rf "build/ReleaseOV"
|
|
fi
|
|
|
|
echo "============================================"
|
|
echo "Configuring with CMake..."
|
|
echo "============================================"
|
|
set +u
|
|
source "${OPENVINO_ROOT}/setupvars.sh"
|
|
set -u
|
|
|
|
cmake -B build/ReleaseOV -G Ninja \
|
|
-DCMAKE_BUILD_TYPE=Release \
|
|
-DGGML_OPENVINO=ON
|
|
|
|
cmake --build build/ReleaseOV --parallel
|
|
|
|
echo "============================================"
|
|
echo "Build completed successfully!"
|
|
echo "============================================"
|
|
echo "Binaries: $(pwd)/build/ReleaseOV/bin"
|
|
echo
|
|
echo "NOTE: To run, source setupvars.sh and pick a device:"
|
|
echo " source /opt/intel/openvino/setupvars.sh"
|
|
echo " export GGML_OPENVINO_DEVICE=CPU # or GPU / NPU"
|
|
echo " ./build/ReleaseOV/bin/llama-cli -m model.gguf"
|
|
```
|
|
|
|
> [!NOTE]
|
|
> The script pins OpenVINO `2026.4.1` via the `OPENVINO_VERSION_MAJOR` / `OPENVINO_VERSION_FULL` variables at the top — edit them to track a different release.
|
|
|
|
</details>
|
|
|
|
#### Windows Build Script
|
|
|
|
For Windows users, the following `.bat` script automates the prerequisite installs (Git, Ninja, CMake, Visual Studio 2022 Build Tools, vcpkg + OpenCL), the OpenVINO Runtime download/extract, and the Ninja-based llama.cpp build.
|
|
Save the following as `build-llamacpp-ov.bat` next to where you want the `llama.cpp` to land, then run it from either **Command Prompt** or **PowerShell**:
|
|
|
|
```cmd
|
|
:: Command Prompt
|
|
build-llamacpp-ov.bat
|
|
```
|
|
|
|
```powershell
|
|
# PowerShell
|
|
.\build-llamacpp-ov.bat
|
|
```
|
|
|
|
<details>
|
|
<summary>Click to expand <code>build-llamacpp-ov.bat</code></summary>
|
|
|
|
```bat
|
|
@echo off
|
|
setlocal enabledelayedexpansion
|
|
|
|
REM ============================================
|
|
REM llama.cpp OpenVINO Build Script (Ninja)
|
|
REM ============================================
|
|
|
|
set "OPENVINO_VERSION_MAJOR=2026.4.1"
|
|
set "OPENVINO_VERSION_FULL=2026.4.1.22982.07f9c262b05"
|
|
|
|
set "SCRIPT_DIR=%~dp0"
|
|
set "VCPKG_DIR=C:\vcpkg"
|
|
set "OPENVINO_INSTALL_DIR=C:\Intel\openvino_%OPENVINO_VERSION_MAJOR%"
|
|
set "OPENVINO_LINK_DIR=C:\Intel\openvino"
|
|
set "OPENVINO_ZIP=%SCRIPT_DIR%openvino.zip"
|
|
set "OPENVINO_EXTRACT_TMP=%SCRIPT_DIR%openvino_extract_tmp"
|
|
set "OPENVINO_URL=https://storage.openvinotoolkit.org/repositories/openvino/packages/%OPENVINO_VERSION_MAJOR%/windows/openvino_toolkit_windows_%OPENVINO_VERSION_FULL%_x86_64.zip"
|
|
|
|
echo ============================================
|
|
echo Installing prerequisites...
|
|
echo ============================================
|
|
winget install --id Git.Git -e --accept-source-agreements --accept-package-agreements 2>nul
|
|
winget install --id Ninja-build.Ninja -e --accept-source-agreements --accept-package-agreements 2>nul
|
|
winget install --id Kitware.CMake -e --accept-source-agreements --accept-package-agreements 2>nul
|
|
|
|
REM Ensure Visual Studio Build Tools are installed.
|
|
echo Checking for Visual Studio Build Tools...
|
|
set "VSWHERE=%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe"
|
|
set "VS_INSTALLED="
|
|
if exist "%VSWHERE%" (
|
|
for /f "usebackq tokens=*" %%i in (`"%VSWHERE%" -latest -products * -requires Microsoft.VisualStudio.Component.VC.Tools.x86.x64 -property installationPath 2^>nul`) do (
|
|
set "VS_INSTALLED=%%i"
|
|
)
|
|
)
|
|
if defined VS_INSTALLED (
|
|
echo Visual Studio with VC++ x86/x64 tools already present at "!VS_INSTALLED!". Skipping winget install.
|
|
) else (
|
|
winget install --id Microsoft.VisualStudio.2022.BuildTools -e --override "--wait --passive --add Microsoft.VisualStudio.Workload.VCTools --includeRecommended" --accept-source-agreements --accept-package-agreements
|
|
if errorlevel 1 (
|
|
echo WARNING: winget could not install Visual Studio Build Tools automatically.
|
|
echo Install manually from https://aka.ms/vs/17/release/vs_BuildTools.exe ^(select the "Desktop development with C++" workload^)
|
|
echo and re-run this script from a "Developer Command Prompt for VS 2022".
|
|
)
|
|
)
|
|
|
|
echo ============================================
|
|
echo Installing OpenCL via vcpkg...
|
|
echo ============================================
|
|
if not exist "%VCPKG_DIR%" (
|
|
git clone https://github.com/microsoft/vcpkg "%VCPKG_DIR%"
|
|
cd /d "%VCPKG_DIR%"
|
|
call bootstrap-vcpkg.bat
|
|
call vcpkg integrate install
|
|
)
|
|
cd /d "%VCPKG_DIR%"
|
|
call vcpkg install opencl
|
|
|
|
cd /d "%SCRIPT_DIR%"
|
|
|
|
REM ============================================
|
|
REM Clone llama.cpp if missing
|
|
REM ============================================
|
|
if not exist "llama.cpp\CMakeLists.txt" (
|
|
echo Cloning llama.cpp...
|
|
git clone https://github.com/ggml-org/llama.cpp
|
|
)
|
|
|
|
cd /d "llama.cpp"
|
|
set "SCRIPT_DIR=%CD%"
|
|
|
|
REM ============================================
|
|
REM Setup OpenVINO: download & extract to C:\Intel\openvino_%OPENVINO_VERSION_MAJOR%,
|
|
REM then point C:\Intel\openvino at it via a directory junction (mklink /J).
|
|
REM ============================================
|
|
|
|
if exist "%OPENVINO_INSTALL_DIR%\setupvars.bat" (
|
|
echo OpenVINO %OPENVINO_VERSION_MAJOR% already installed at "%OPENVINO_INSTALL_DIR%". Skipping download.
|
|
) else (
|
|
echo OpenVINO not found at "%OPENVINO_INSTALL_DIR%". Starting download...
|
|
|
|
curl -L -o "%OPENVINO_ZIP%" "%OPENVINO_URL%"
|
|
if errorlevel 1 (
|
|
echo ERROR: Download failed.
|
|
exit /b 1
|
|
)
|
|
|
|
echo Extracting OpenVINO...
|
|
if exist "%OPENVINO_EXTRACT_TMP%" rmdir /s /q "%OPENVINO_EXTRACT_TMP%"
|
|
mkdir "%OPENVINO_EXTRACT_TMP%"
|
|
tar -xf "%OPENVINO_ZIP%" -C "%OPENVINO_EXTRACT_TMP%"
|
|
if errorlevel 1 (
|
|
echo ERROR: Extraction failed.
|
|
exit /b 1
|
|
)
|
|
|
|
REM Move the single top-level folder contents into the versioned install dir.
|
|
set "OPENVINO_EXTRACTED="
|
|
for /d %%i in ("%OPENVINO_EXTRACT_TMP%\*") do set "OPENVINO_EXTRACTED=%%i"
|
|
if not defined OPENVINO_EXTRACTED (
|
|
echo ERROR: Could not locate extracted OpenVINO folder under "%OPENVINO_EXTRACT_TMP%".
|
|
exit /b 1
|
|
)
|
|
if not exist "%OPENVINO_INSTALL_DIR%" mkdir "%OPENVINO_INSTALL_DIR%"
|
|
xcopy /e /i /y /q "!OPENVINO_EXTRACTED!\*" "%OPENVINO_INSTALL_DIR%\" >nul
|
|
if errorlevel 1 (
|
|
echo ERROR: Failed to copy OpenVINO from "!OPENVINO_EXTRACTED!" to "%OPENVINO_INSTALL_DIR%".
|
|
echo Re-run this script from an elevated Command Prompt ^(Run as administrator^) if access is denied.
|
|
exit /b 1
|
|
)
|
|
|
|
rmdir /s /q "%OPENVINO_EXTRACT_TMP%"
|
|
del "%OPENVINO_ZIP%"
|
|
)
|
|
|
|
REM Refresh junction: C:\Intel\openvino -> C:\Intel\openvino_<version>.
|
|
REM `mklink /J` creates a directory junction (no admin / Developer Mode required).
|
|
if exist "%OPENVINO_LINK_DIR%" rmdir "%OPENVINO_LINK_DIR%"
|
|
mklink /J "%OPENVINO_LINK_DIR%" "%OPENVINO_INSTALL_DIR%" >nul
|
|
if errorlevel 1 (
|
|
echo ERROR: Failed to create junction "%OPENVINO_LINK_DIR%" -^> "%OPENVINO_INSTALL_DIR%".
|
|
echo If "%OPENVINO_LINK_DIR%" already exists as a regular non-empty folder, remove it manually and re-run.
|
|
exit /b 1
|
|
)
|
|
|
|
set "OPENVINO_ROOT=%OPENVINO_LINK_DIR%"
|
|
echo OpenVINO Ready: %OPENVINO_ROOT% -^> %OPENVINO_INSTALL_DIR%
|
|
|
|
|
|
echo ============================================
|
|
echo Setting up compiler environment...
|
|
echo ============================================
|
|
REM Locate Visual Studio Build Tools vcvars64.bat
|
|
set "VSWHERE=%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe"
|
|
if exist "%VSWHERE%" (
|
|
for /f "usebackq tokens=*" %%i in (`"%VSWHERE%" -latest -products Microsoft.VisualStudio.Product.BuildTools -property installationPath`) do (
|
|
set "VS_PATH=%%i"
|
|
)
|
|
)
|
|
if defined VS_PATH (
|
|
call "%VS_PATH%\VC\Auxiliary\Build\vcvars64.bat" >nul
|
|
) else (
|
|
echo WARNING: Visual Studio Build Tools not found. Compiler may be missing.
|
|
)
|
|
|
|
REM ============================================
|
|
REM Clean old build cache
|
|
REM ============================================
|
|
if exist "build\ReleaseOV" (
|
|
echo Removing old build directory ...
|
|
rmdir /s /q "build\ReleaseOV"
|
|
)
|
|
|
|
echo ============================================
|
|
echo Configuring with CMake...
|
|
echo ============================================
|
|
call "%OPENVINO_ROOT%\setupvars.bat" >nul 2>nul
|
|
|
|
cmake -B build\ReleaseOV -G Ninja ^
|
|
-DCMAKE_BUILD_TYPE=Release ^
|
|
-DGGML_OPENVINO=ON ^
|
|
-DCMAKE_TOOLCHAIN_FILE="%VCPKG_DIR%\scripts\buildsystems\vcpkg.cmake"
|
|
|
|
if errorlevel 1 (
|
|
echo If you continue to face CMAKE errors, make sure to install:
|
|
echo winget install Microsoft.VisualStudio.2022.BuildTools
|
|
echo Then run the "Developer Command Prompt for VS 2022" and launch this script from there.
|
|
exit /b 1
|
|
)
|
|
|
|
cmake --build build\ReleaseOV --config Release
|
|
if errorlevel 1 exit /b 1
|
|
|
|
echo ============================================
|
|
echo Build completed successfully!
|
|
echo ============================================
|
|
echo Binaries: %CD%\build\ReleaseOV\bin
|
|
echo.
|
|
echo NOTE: To run, source setupvars.bat and pick a device:
|
|
echo call "C:\Intel\openvino\setupvars.bat"
|
|
echo set GGML_OPENVINO_DEVICE=CPU ^&^& REM or GPU / NPU
|
|
echo build\ReleaseOV\bin\llama-cli.exe -m model.gguf
|
|
echo.
|
|
|
|
endlocal
|
|
```
|
|
|
|
> [!NOTE]
|
|
> The script pins OpenVINO `2026.4.1` via the `OPENVINO_VERSION_MAJOR` / `OPENVINO_VERSION_FULL` variables at the top — edit them to track a different release. From any new shell, source the matching `setupvars` script via the junction — `call "C:\Intel\openvino\setupvars.bat"` from `cmd`, or `& "C:\Intel\openvino\setupvars.ps1"` from PowerShell. If `winget` cannot register Visual Studio Build Tools on first run, install them once manually and re-run the script from an elevated **Developer Command Prompt for VS 2022**.
|
|
|
|
</details>
|
|
|
|
|
|
### 3. Download Sample Model
|
|
|
|
Download sample model for testing.
|
|
|
|
```bash
|
|
# Linux
|
|
mkdir -p ~/models/
|
|
wget https://huggingface.co/bartowski/Llama-3.2-1B-Instruct-GGUF/resolve/main/Llama-3.2-1B-Instruct-Q4_K_M.gguf \
|
|
-O ~/models/Llama-3.2-1B-Instruct-Q4_K_M.gguf
|
|
|
|
# Windows PowerShell
|
|
mkdir C:\models
|
|
Invoke-WebRequest -Uri https://huggingface.co/bartowski/Llama-3.2-1B-Instruct-GGUF/resolve/main/Llama-3.2-1B-Instruct-Q4_K_M.gguf -OutFile C:\models\Llama-3.2-1B-Instruct-Q4_K_M.gguf
|
|
|
|
# Windows Command Line
|
|
mkdir C:\models
|
|
curl -L https://huggingface.co/bartowski/Llama-3.2-1B-Instruct-GGUF/resolve/main/Llama-3.2-1B-Instruct-Q4_K_M.gguf -o C:\models\Llama-3.2-1B-Instruct-Q4_K_M.gguf
|
|
```
|
|
|
|
### 4. Run Inference with OpenVINO Backend
|
|
|
|
When using the OpenVINO backend, the first inference token may have slightly higher latency due to on-the-fly conversion to the OpenVINO graph. Subsequent tokens and runs will be faster.
|
|
|
|
> [!NOTE]
|
|
> Default context size is set to the model training context, which may be very large. For example, 131072 for Llama 3.2 1B, which may result in lower performance, especially on edge/laptop devices. Use `-c` to limit context size in supported llama.cpp tools for better performance. For example, `-c 512`.
|
|
|
|
```bash
|
|
# If device is unset or unavailable, defaults to CPU.
|
|
# If the system has multiple GPUs, use GPU.0 or GPU.1 to explicitly target a specific GPU.
|
|
|
|
# Linux
|
|
export GGML_OPENVINO_DEVICE=GPU
|
|
# Optional: enable stateful execution for improved GPU performance (recommended).
|
|
export GGML_OPENVINO_STATEFUL_EXECUTION=1
|
|
# To run llama-simple:
|
|
./build/ReleaseOV/bin/llama-simple -m ~/models/Llama-3.2-1B-Instruct-Q4_K_M.gguf -n 50 "The story of AI is "
|
|
# To run in chat mode:
|
|
./build/ReleaseOV/bin/llama-cli -m ~/models/Llama-3.2-1B-Instruct-Q4_K_M.gguf -c 1024
|
|
# To run llama-bench, -fa 1 is needed
|
|
GGML_OPENVINO_STATEFUL_EXECUTION=1 GGML_OPENVINO_DEVICE=GPU ./build/ReleaseOV/bin/llama-bench -m ~/models/Llama-3.2-1B-Instruct-Q4_K_M.gguf -fa 1
|
|
|
|
# NPU: keep context small to avoid failures from very large model context windows.
|
|
export GGML_OPENVINO_DEVICE=NPU
|
|
./build/ReleaseOV/bin/llama-cli -m ~/models/Llama-3.2-1B-Instruct-Q4_K_M.gguf -c 512
|
|
|
|
# Windows Command Line
|
|
set GGML_OPENVINO_DEVICE=GPU
|
|
# Optional: enable stateful execution for improved GPU performance (recommended).
|
|
set GGML_OPENVINO_STATEFUL_EXECUTION=1
|
|
# Windows PowerShell
|
|
$env:GGML_OPENVINO_DEVICE = "GPU"
|
|
$env:GGML_OPENVINO_STATEFUL_EXECUTION = "1"
|
|
|
|
# To run llama-simple
|
|
build\ReleaseOV\bin\llama-simple.exe -m "C:\models\Llama-3.2-1B-Instruct-Q4_K_M.gguf" -n 50 "The story of AI is "
|
|
# To run in chat mode:
|
|
build\ReleaseOV\bin\llama-cli.exe -m "C:\models\Llama-3.2-1B-Instruct-Q4_K_M.gguf" -c 1024
|
|
# To run llama-bench, -fa 1 is needed
|
|
build\ReleaseOV\bin\llama-bench.exe -m "C:\models\Llama-3.2-1B-Instruct-Q4_K_M.gguf" -fa 1
|
|
|
|
# NPU: keep context small to avoid failures from very large model context windows.
|
|
# Windows Command Line
|
|
set GGML_OPENVINO_DEVICE=NPU
|
|
# Windows PowerShell
|
|
$env:GGML_OPENVINO_DEVICE = "NPU"
|
|
build\ReleaseOV\bin\llama-cli.exe -m "C:\models\Llama-3.2-1B-Instruct-Q4_K_M.gguf" -c 512
|
|
```
|
|
> [!NOTE]
|
|
> On systems with multiple GPUs, use `GPU.0` or `GPU.1` to explicitly target specific GPU. A device that is not available is an error (no fallback to CPU), and the error message lists the available OpenVINO devices with their names. Run `llama-cli --list-devices` to see the valid values: each OpenVINO device shows the `GGML_OPENVINO_DEVICE=<value>` to set, and `(selected)` marks the active one. Select the OpenVINO device with this variable, not with `-dev`. See [OpenVINO GPU Device](https://docs.openvino.ai/2026/openvino-workflow/running-inference/inference-devices-and-modes/gpu-device.html) for more details.
|
|
|
|
### 5. Docker Build
|
|
|
|
You can build and run llama.cpp with OpenVINO backend using Docker.
|
|
|
|
```bash
|
|
# Build the base runtime image with compiled shared libraries and minimal dependencies.
|
|
docker build -t llama-openvino:base -f .devops/openvino.Dockerfile .
|
|
|
|
# Build the complete image with all binaries, Python tools, gguf-py library, and model conversion utilities.
|
|
docker build --target=full -t llama-openvino:full -f .devops/openvino.Dockerfile .
|
|
|
|
# Build a minimal CLI-only image containing just the llama-cli executable.
|
|
docker build --target=light -t llama-openvino:light -f .devops/openvino.Dockerfile .
|
|
|
|
# Builds a server-only image with llama-server executable, health check endpoint, and REST API support.
|
|
docker build --target=server -t llama-openvino:server -f .devops/openvino.Dockerfile .
|
|
|
|
# If you are behind a proxy:
|
|
docker build --build-arg http_proxy=$http_proxy --build-arg https_proxy=$https_proxy --target=server -t llama-openvino:server -f .devops/openvino.Dockerfile .
|
|
```
|
|
|
|
Run llama.cpp with OpenVINO backend Docker container.
|
|
Save sample models in `~/models` as [shown above](#3-download-sample-model). It will be mounted to the container in the examples below.
|
|
|
|
|
|
```bash
|
|
# Run Docker container
|
|
docker run --rm -it -v ~/models:/models llama-openvino:light --no-warmup -c 1024 -m /models/Llama-3.2-1B-Instruct-Q4_K_M.gguf
|
|
|
|
# With Intel GPU access (iGPU or dGPU)
|
|
docker run --rm -it -v ~/models:/models \
|
|
--device=/dev/dri --group-add=$(stat -c "%g" /dev/dri/render* | head -n 1) -u $(id -u):$(id -g) \
|
|
--env=GGML_OPENVINO_DEVICE=GPU --env=GGML_OPENVINO_STATEFUL_EXECUTION=1 \
|
|
llama-openvino:light --no-warmup -c 1024 -m /models/Llama-3.2-1B-Instruct-Q4_K_M.gguf
|
|
|
|
# With Intel NPU access
|
|
docker run --rm -it -v ~/models:/models \
|
|
--device=/dev/accel --group-add=$(stat -c "%g" /dev/dri/render* | head -n 1) -u $(id -u):$(id -g) \
|
|
--env=GGML_OPENVINO_DEVICE=NPU \
|
|
llama-openvino:light --no-warmup -c 1024 -m /models/Llama-3.2-1B-Instruct-Q4_K_M.gguf
|
|
```
|
|
|
|
Run Llama.cpp Server with OpenVINO Backend.
|
|
> [!NOTE]
|
|
> `llama-server` with OpenVINO backend supports only one chat session/thread, when `GGML_OPENVINO_STATEFUL_EXECUTION=1` is enabled.
|
|
|
|
```bash
|
|
# Run the llama-openvino:server Docker container (CPU)
|
|
docker run --rm -it -p 8080:8080 -v ~/models:/models llama-openvino:server --no-warmup -m /models/Llama-3.2-1B-Instruct-Q4_K_M.gguf -c 1024 --host 0.0.0.0
|
|
|
|
# Run the llama-openvino:server Docker container with Intel GPU access (iGPU or dGPU)
|
|
docker run --rm -it -v ~/models:/models \
|
|
--device=/dev/dri --group-add=$(stat -c "%g" /dev/dri/render* | head -n 1) -u $(id -u):$(id -g) \
|
|
-p 8080:8080 --env=GGML_OPENVINO_DEVICE=GPU \
|
|
llama-openvino:server --no-warmup -c 1024 -m /models/Llama-3.2-1B-Instruct-Q4_K_M.gguf --host 0.0.0.0
|
|
|
|
# Run the llama-openvino:server Docker container with Intel NPU access
|
|
docker run --rm -it -v ~/models:/models \
|
|
--device=/dev/accel --group-add=$(stat -c "%g" /dev/dri/render* | head -n 1) -u $(id -u):$(id -g) \
|
|
-p 8080:8080 --env=GGML_OPENVINO_DEVICE=NPU \
|
|
llama-openvino:server --no-warmup -c 1024 -m /models/Llama-3.2-1B-Instruct-Q4_K_M.gguf --host 0.0.0.0
|
|
|
|
# Or Using llama-server executable
|
|
./build/ReleaseOV/bin/llama-server -m ~/models/Llama-3.2-1B-Instruct-Q4_K_M.gguf --port 8080 -c 1024
|
|
|
|
# Option 1: Open your browser to http://localhost:8080 to access the web UI for the llama.cpp server.
|
|
# Option 2: In a NEW terminal, test the server with curl
|
|
|
|
# If you are behind a proxy, make sure to set NO_PROXY to avoid proxy for localhost
|
|
export NO_PROXY=localhost,127.0.0.1
|
|
|
|
# Test health endpoint
|
|
curl -f http://localhost:8080/health
|
|
|
|
# Test with a simple prompt
|
|
curl -X POST "http://localhost:8080/v1/chat/completions" -H "Content-Type: application/json" \
|
|
-d '{"messages":[{"role":"user","content":"Write a poem about OpenVINO"}],"max_tokens":100}' | jq .
|
|
```
|
|
|
|
## GGML OpenVINO Backend Runtime Configurations
|
|
|
|
The OpenVINO backend can be configured using the following environment variables at runtime to control device selection, caching, debugging, and profiling behavior.
|
|
Boolean flags follow a uniform convention: set to a **positive integer** (e.g. `1`) to enable; unset, empty, `0`, negative, or non-numeric values are treated as disabled.
|
|
|
|
| Variable | Type | Default | Description |
|
|
|-----------------------------------|-----------|------------|-------------------------------------------------------------------------------------------------------------|
|
|
| `GGML_OPENVINO_DEVICE` | String | `CPU` | Specify the target device (CPU, GPU, NPU). On systems with multiple GPUs, use `GPU.0` or `GPU.1` to explicitly target specific GPU. A device that is not available is an error (no fallback to CPU), and the error message lists the available OpenVINO devices with their names. See [OpenVINO GPU Device](https://docs.openvino.ai/2026/openvino-workflow/running-inference/inference-devices-and-modes/gpu-device.html). When set to **NPU**, static compilation mode is enabled for optimal performance. |
|
|
| `GGML_OPENVINO_CACHE_DIR` | String | `not set` | Directory for OpenVINO's separate plugin cache. On NPU, this sets `NPUW_CACHE_DIR`. |
|
|
| `GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR` | String | `not set` | Directory for standalone compiled blobs with weights. Dynamic CPU/GPU graphs can import matching blobs on later runs. |
|
|
| `GGML_OPENVINO_COMPILED_MODEL_CACHE_ONLY` | Boolean | `0` | Require an existing compiled blob and skip weight uploads and compilation. Requires Linux or Windows mmap loading and a full dynamic CPU/GPU graph on OpenVINO. |
|
|
| `GGML_OPENVINO_PREFILL_CHUNK_SIZE`| Integer | `256` | Token chunk size for **NPU** prefill (NPU-only; ignored on CPU/GPU). Must be a positive integer; otherwise the default is used. |
|
|
| `GGML_OPENVINO_NPU_COMPILE_CONFIG` | String | `not set` | NPU-only compiler mode parameters forwarded to OpenVINO as `NPU_COMPILATION_MODE_PARAMS`, for example `optimization-level=3`. |
|
|
| `GGML_OPENVINO_STATEFUL_EXECUTION`| Boolean | `0` | Keep KV and supported recurrent caches inside the model. Single-slot CPU/GPU execution only. |
|
|
| `GGML_OPENVINO_DISABLE_CACHE` | Boolean | `0` | Disable the in-process compiled-model / decoder cache (cache is on by default). Set to `1` to disable. |
|
|
| `GGML_OPENVINO_DISABLE_KV_SLICE` | Boolean | `0` | Disable the KV-cache input-tensor slicing optimization (slicing is on by default on CPU/GPU). Set to `1` to disable. |
|
|
| `GGML_OPENVINO_DISABLE_KV_STATE_RELAYOUT` | Boolean | `0` | Disable the stateful KV-state sequence-axis relayout (relayout is on by default). It moves the KV state sequence axis from dim 1 to dim 2, so the GPU plugin can append new tokens in place instead of copying the whole state every token, and the reader side no longer transposes the whole accumulated state. Set to `1` to disable. |
|
|
| `GGML_OPENVINO_MANUAL_GQA_ATTN` | Boolean | device-based | Tri-state. When **unset**, manual GQA attention is enabled by default on `GPU` and disabled on other devices. Set to a positive integer to force-enable, or `0` to force-disable. |
|
|
| `GGML_OPENVINO_MEMORY_OPTIMIZE` | Boolean | `0` | Umbrella switch for compile-time memory reductions. Enables `GGML_OPENVINO_REDUCE_COMPILE_MEM` and, on GPU, `GGML_OPENVINO_RELEASE_WEIGHTS` unless those fine-grained variables are explicitly set. |
|
|
| `GGML_OPENVINO_REDUCE_COMPILE_MEM`| Boolean | inherits from `GGML_OPENVINO_MEMORY_OPTIMIZE` | Reduce compile-time host memory use by streaming weight requantization and avoiding extra weight-node materialization where possible. Set explicitly to override the umbrella switch. |
|
|
| `GGML_OPENVINO_RELEASE_WEIGHTS` | Boolean | inherits from `GGML_OPENVINO_MEMORY_OPTIMIZE` on GPU | GPU-only. Release host weight buffers after the compiled model cache can reuse the device/plugin copy. Requires stable graph shapes; dynamic workloads that need recompilation should leave this disabled. |
|
|
| `GGML_OPENVINO_SPILL_DIR` | String | `not set` | Directory for a disk-backed weight buffer. When set, the repacked weight buffer is mapped from an unlinked file on this path instead of anonymous memory, so its pages are reclaimable under memory pressure instead of staying pinned, cutting the load-time host memory peak. Must point at real storage; a tmpfs mount (e.g. `/tmp` on many systems) backs it with RAM and makes the peak worse. |
|
|
| `GGML_OPENVINO_REQUANT_KQUANT` | String | `not set` | Requantize Q6_K/Q5_K weights (and matching MoE expert weights) to a 4-bit target instead of the default Q8_0_C, trading accuracy for less memory traffic. One of `q4_asym64` (Q6_K/Q5_K only, keeps a real zero point at group 64), `q4_asym64_all` (also requantizes Q4_K), `q4_sym128` (Q6_K/Q5_K only), `q4_sym128_all` (Q4_K too, drops its per-group zero point), or `native` (no requantization). |
|
|
| `GGML_OPENVINO_PROFILING` | Integer | `0` | `1` logs execution timing; `2` or higher also enables OpenVINO and OpenCL profiling. |
|
|
| `GGML_OPENVINO_DEBUG_NODE` | String | `not set` | Add the named graph nodes as compiled outputs for debugging. Separate multiple names with commas. |
|
|
| `GGML_OPENVINO_MOE_OP` | Boolean | `1` | On GPU, set to `0` to keep the unfused GatherMatmul path. |
|
|
| `GGML_OPENVINO_DUMP_CGRAPH` | Boolean | `0` | Dump the GGML compute graph to `cgraph_ov.txt`. |
|
|
| `GGML_OPENVINO_DUMP_IR` | Boolean | `0` | Serialize OpenVINO IR files with timestamps. |
|
|
| `GGML_OPENVINO_DEBUG_INPUT` | Boolean | `0` | Enable input debugging and print input tensor info. |
|
|
| `GGML_OPENVINO_DEBUG_OUTPUT` | Boolean | `0` | Enable output debugging and print output tensor info. |
|
|
| `GGML_OPENVINO_PRINT_CGRAPH_TENSOR_ADDRESS` | Boolean | `0` | Print tensor address map once. |
|
|
| `GGML_OPENVINO_LOG_UNSUPPORTED_OPS`| Boolean | `0` | Log warning messages with tensor details and rejection reasons for any ops not supported by the OpenVINO backend. Emits at `WARN` level (requires `--log-verbosity >= 2`, enabled by default). |
|
|
|
|
> [!NOTE]
|
|
> - `GGML_OPENVINO_STATEFUL_EXECUTION` is an **Experimental** feature for managing caches internally inside the OpenVINO model on CPUs and GPUs. Use a single slot (`-np 1`). KV caches retain the append-based state layout and sequence-axis optimization. Qwen3.5 adds recurrent cache states in their GGML layouts. Qwen3.5 requires an unsplit graph with model caching enabled and no recurrent rollback. A prompt starting at position 0 resets all states. State save/restore, sequence rewind, context shift, and mid-sequence graph replacement are unsupported. Stateful execution is not effective on NPUs.
|
|
> - `GGML_OPENVINO_LOG_UNSUPPORTED_OPS` emits logs at `WARN` level (`GGML_LOG_WARN`), which requires application log verbosity `--log-verbosity >= 2` (or `-lv 2`).
|
|
> - With `GGML_OPENVINO_COMPILED_MODEL_CACHE_ONLY=1`, use the same compilation settings as the export run. One directory can hold blobs for different models and settings; `GGML_OPENVINO_SPILL_DIR` does not affect the cache key and is ignored in cache-only mode. See [Compiled model cache](../../ggml/src/ggml-openvino/README.md) for the workflow and restrictions.
|
|
|
|
### Example Usage
|
|
|
|
#### GPU Inference with Profiling
|
|
|
|
```bash
|
|
# If the system has multiple GPUs, use GPU.0 or GPU.1 to explicitly target a specific GPU.
|
|
|
|
# Linux
|
|
export GGML_OPENVINO_CACHE_DIR=/tmp/ov_cache
|
|
export GGML_OPENVINO_PROFILING=1
|
|
export GGML_OPENVINO_DEVICE=GPU
|
|
export GGML_OPENVINO_STATEFUL_EXECUTION=1
|
|
|
|
./build/ReleaseOV/bin/llama-simple -m ~/models/Llama-3.2-1B-Instruct-Q4_K_M.gguf -n 50 "The story of AI is "
|
|
|
|
# Windows Command Line
|
|
set GGML_OPENVINO_CACHE_DIR=C:\tmp\ov_cache
|
|
set GGML_OPENVINO_PROFILING=1
|
|
set GGML_OPENVINO_DEVICE=GPU
|
|
set GGML_OPENVINO_STATEFUL_EXECUTION=1
|
|
|
|
# Windows PowerShell
|
|
$env:GGML_OPENVINO_CACHE_DIR = "C:\tmp\ov_cache"
|
|
$env:GGML_OPENVINO_PROFILING = "1"
|
|
$env:GGML_OPENVINO_DEVICE = "GPU"
|
|
$env:GGML_OPENVINO_STATEFUL_EXECUTION = "1"
|
|
|
|
build\ReleaseOV\bin\llama-simple.exe -m "C:\models\Llama-3.2-1B-Instruct-Q4_K_M.gguf" -n 50 "The story of AI is "
|
|
|
|
```
|
|
|
|
## Known Limitations
|
|
|
|
**General (all devices)**
|
|
|
|
- Llama.cpp OpenVINO backend currently supports a subset of GGML ops and text-only models. Unsupported ops or unsupported op shapes/cases fail during OpenVINO translation.
|
|
- Multimodal features (audio/image/video) are a work in progress.
|
|
- Limited Embedding and Reranking model support.
|
|
- Llama.cpp tool coverage across CPU/GPU/NPU is not uniform.
|
|
|
|
**Tool-specific**
|
|
|
|
- `llama-bench`: requires `-fa 1` (flash-attention).
|
|
- `llama-cli --context-shift`: stateless only (`GGML_OPENVINO_STATEFUL_EXECUTION=0`). In stateful mode the KV cache is owned by the OpenVINO model and cannot be shifted externally.
|
|
- `llama-server`: only one chat session/thread when `GGML_OPENVINO_STATEFUL_EXECUTION=1`.
|
|
|
|
**GPU-specific**
|
|
|
|
- `llama-server -np > 1`: concurrent requests are batched together, which may slightly reduce per-request throughput.
|
|
|
|
**NPU-specific**
|
|
|
|
- Default context resolves to the model's training context (e.g. 131072 for Llama 3.2 1B), which can OOM or fail or degrade performance on NPU. Inspect the resolved value with `-lv 3`.
|
|
- **Workaround:** Pass an explicit `-c <N>`, e.g. `-c 1024`.
|
|
- NPU device uses a static graph with a fixed prefill chunk size (defaults to 256), configurable with `GGML_OPENVINO_PREFILL_CHUNK_SIZE`. Large prefill/batch settings may need tuning.
|
|
- `llama-server -np > 1` (multiple parallel sequences) is not supported.
|
|
- `llama-perplexity`: requires `-b 512` or smaller.
|
|
|
|
> [!NOTE]
|
|
> The OpenVINO backend is actively under development. Fixes and improvements are underway, and this document will continue to be updated.
|
|
|
|
## Work in Progress
|
|
|
|
- Performance and memory optimizations
|
|
- Accuracy validation
|
|
- Broader quantization coverage
|
|
- Support for additional model architectures
|