Files
llama.cpp/docs/backend/OPENVINO.md
T
9a080f44b5 ggml-openvino: fix OpenVINO CI, minor fixes and optimizations (#32590)
* ggml-openvino: round the GPU KV length up to a power of two

Review notes (follow-ups, not blocking):
- convert_ggml_input_to_ov pads the KQ mask into a std::vector and then
  copies it into the ov::Tensor. Writing the padded rows straight into
  padded.data() avoids the second allocation and copy (up to 16 MB per
  call at a 16K context).
- The padding is skipped when the mask is not contiguous, and the KV
  inputs are then sliced to more rows than the mask has columns. Masks
  are always contiguous today; a GGML_ASSERT would make that explicit.
- compute_llm_params reads GGML_OPENVINO_STATEFUL_EXECUTION directly;
  is_stateful_enabled() is local to ggml-openvino.cpp.

* ggml-openvino: Optimize model recompiles of prefill stage

Optimize the model recompiles of prefill stage when attended size
changed between prompts. This can save model recompile time for
llama-bench and llama-server.

* ggml-openvino: Optimize model recompiles of prefill stage

Optimize the model recompiles of prefill stage when attended size
changed between prompts. This can save model recompile time for
llama-bench and llama-server.

* ggml-openvino: Remove solution C and keep A as default

* ggml-openvino: match the output count before reusing a compiled NPU prefill

The static prefill model has a fixed inp_out_ids shape [1,1,1,output_len].
The new reuse check compared only the attention sizes, so llama-perplexity,
which alternates 1 and 512 outputs per ubatch, reused the model compiled
for 512 outputs on a 1-output ubatch and failed with "got [1,1,1,1]
expecting [1,1,1,512]". Store the output count of the cached prefill
model and require it to match.

* ggml-openvino: Fix LFM2 context create on GPU and DSpark speculative decoding

* ggml-openvino: remove leftovers from merging dev into the LFM2/DSpark fix

- compute_model_outputs exported OUTPUT-flagged nodes twice: dev's block
  ran before the new one and still exported result_norm, so the
  result_norm exclusion had no effect. Keep only the new block.
- compute_node_dynamic_dims checked is_activation_input twice in a row.
  Remove the second copy.
- process_view_input_new created the dynamic Split on split_dim. With
  stateful execution the input drops the leading size-1 axis, so use
  split_axis, which already accounts for the rank difference.

* ggml-openvino: hash the output flags into the model-cache fingerprint

The fingerprint hashed the ops, names and weights but not the outputs, so
a blob compiled with another output set was imported. With the result_norm
change, an NPU blob written before it gave wrong output, and enabling
llama_set_embeddings_layer_inp() reused a blob without the layer-input
outputs. Hash each node's output flag and bump the cache version to 8.

* ggml-openvino: support f16 SCALE and f16/bf16 TRI

test-backend-ops now runs SCALE on f16 and TRI on f16 and bf16. Both
translators built their constants as f32, and Multiply, Add and Select
need all inputs in one element type, so building the graph threw for
these cases.

SCALE now converts a non-f32 input to f32, applies the scale and bias,
and converts back once, as the CPU backend computes it. TRI creates its
zero constant in the input's element type. f32 graphs are unchanged.

* ggml-openvino: [Qwen3.6 A3B]fix static-shape ids view of the last MoE
 layer

The same-stride single-dim Slice in process_view_input_new was followed by a Reshape
whose target baked the static chunk size into the token axis. After the output-ids
GET_ROWS the last layer runs on n_outputs tokens (1 in the prefill graph), so the
Reshape failed shape validation for qwen35moe with FORCE_STATIC or on the NPU.
Skip the Reshape in static graphs when the slice already keeps the rank;
the dynamic graphs are unchanged.

* enable mamba based models on OV CPU-GPU

* revert changes in tests

* docs : add Falcon-H1 to the OpenVINO validated models

Falcon-H1 runs on CPU and GPU since Mamba SSM support was added. Add its
row and the known limitations: multi-sequence batches fail, GPU accuracy
is slightly lower than CPU, and the NPU fails with an element type
mismatch.

* ggml-openvino: simplify the SSM_SCAN build check and clean up

- Drop the SelectiveSSM configure check and its guards: OpenVINO 2026.4.1
  always provides the op. Drop the no-op LANGUAGE CXX source property.
- Drop the semicolon after OPENVINO_OP("SelectiveSSM"). GCC 15 reports it
  with -Wextra-semi, which fails builds with fatal warnings.
- Add ggml_openvino_is_static() for the repeated NPU or
  GGML_OPENVINO_FORCE_STATIC check.
- Share the f16 and f32 KQ mask padding code.
- Drop the redundant inline on in-class static functions.

---------

Co-authored-by: Suhas Somashekar <suhas.somashekar@intel.com>
Co-authored-by: yzhou51 <yaowei.zhou@intel.com>
Co-authored-by: Ananth, Swathi <swathi.ananth@intel.com>
Co-authored-by: Beheshti <nazanin.beheshti@intel.com>
2026-11-19 07:22:26 -08:00

50 KiB

OpenVINO Backend for llama.cpp

Note

Performance and memory optimizations, accuracy validation, broader quantization coverage, broader operator and model support are work in progress.

OpenVINO is an open-source toolkit for optimizing and deploying high-performance AI inference, specifically designed for Intel hardware, including CPUs, GPUs (integrated and discrete), and NPUs, in the cloud, on-premises, and on the edge. The OpenVINO backend runs the same GGUF model files on Intel® CPUs, GPUs and NPUs, without changes to the model or the rest of llama.cpp.

When a ggml_cgraph is dispatched to the OpenVINO backend, it:

  • Walks the GGML graph and identifies inputs, outputs, weights, and KV cache tensors.
  • Translates the GGML operations into an ov::Model using OpenVINO's frontend API.
  • Compiles and caches the model for the target device.
  • Binds GGML tensor memory to OpenVINO inference tensors and runs inference.

For guidance on contributing to the OpenVINO backend, see the OpenVINO Backend Contributing Guide.

Contents

Quick Start

On Ubuntu, with the prerequisites and the OpenVINO Runtime installed:

# 1. Build
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
source /opt/intel/openvino/setupvars.sh
cmake -B build/ReleaseOV -G Ninja -DCMAKE_BUILD_TYPE=Release -DGGML_OPENVINO=ON
cmake --build build/ReleaseOV --parallel

# 2. Download a model
mkdir -p ~/models/
wget https://huggingface.co/bartowski/Llama-3.2-1B-Instruct-GGUF/resolve/main/Llama-3.2-1B-Instruct-Q4_K_M.gguf \
     -O ~/models/Llama-3.2-1B-Instruct-Q4_K_M.gguf

# 3. Run on CPU, GPU or NPU
export GGML_OPENVINO_DEVICE=GPU
./build/ReleaseOV/bin/llama-cli -m ~/models/Llama-3.2-1B-Instruct-Q4_K_M.gguf -c 1024

The Ubuntu Build Script and the Windows Build Script do the setup in one step: they install the prerequisites and the OpenVINO Runtime, and build llama.cpp with the OpenVINO backend.

See Build Instructions for Windows and Docker, and Runtime Configuration for the environment variables.

Supported Devices

OpenVINO backend supports the following hardware:

  • Intel CPUs
  • Intel GPUs (integrated and discrete)
  • Intel NPUs

Although OpenVINO supports a wide range of Intel hardware, the llama.cpp OpenVINO backend has been validated specifically on AI PCs such as the Intel® Core™ Ultra Series 2 (Lunar Lake) and Series 3 (Panther Lake).

Supported Model Precisions

  • FP16
  • BF16 (on Intel Xeon)
  • Q8_0
  • Q4_0
  • Q4_1
  • Q4_K
  • Q5_0 (converted at runtime, see the details below)
  • Q5_1
  • Q5_K (converted at runtime, see the details below)
  • Q6_K (converted at runtime, see the details below)

Q4_K_M files mix Q4_K tensors with Q6_K tensors, and sometimes Q5_K tensors.

Quantization details for each device

All devices:

  • FP16 and BF16 tensors are used as they are.
  • A quantized token embedding (token_embd.weight) and output weight (output.weight) are converted to Q8_0_C.

CPU and GPU:

  • Q4_0, Q4_1, Q4_K, Q5_1 and Q8_0 tensors keep their layout: weights, scales and zero points are extracted directly.
  • Q5_K and Q6_K tensors are converted to Q8_0_C. GGML_OPENVINO_REQUANT_KQUANT selects a 4-bit layout instead.
  • Q5_0 tensors are converted to Q4_0_64.

GPU:

  • Q4_K, Q5_K and Q6_K MoE expert tensors are converted to Q4_1_64. This lets the GPU plugin use its fused MoE op, which is much faster for prompt processing. GGML_OPENVINO_REQUANT_KQUANT does not change this. GGML_OPENVINO_MOE_OP=0 turns the fused MoE op off.

NPU:

  • All other quantized tensors are requantized to Q4_0_128.
  • For a Q6_K token embedding, GGML_OPENVINO_NPU_TOKEN_EMBD_QUANT selects another layout.
  • The validated models use Q4_K_M files.

Layout names:

  • Q8_0_C: 8-bit, one scale per row.
  • Q4_0_128 and Q4_0_64: 4-bit, symmetric, one scale per 128 or 64 weights.

Additional Notes:

  • Both Q4_0 and Q4_1 models use Q6_K for the token embedding tensor and the final matmul weight tensor (often the same tensor)
  • Q4_0 models may produce some Q4_1 tensors if an imatrix is provided during quantization using llama-quantize
  • Q4_K_M models may include both Q6_K and Q5_K tensors (observed in Phi-3)

Supported llama.cpp Tools

The OpenVINO backend works with the standard llama.cpp tools listed below. Tool coverage is not uniform across devices.

  • llama-bench
  • llama-cli
  • llama-completion
  • llama-embedding
  • llama-perplexity
  • llama-server
  • llama-simple

Validated Models

The models below were tested with llama-cli and Q4_K_M files on an Intel® Core™ Ultra Series 2 (Lunar Lake). The OpenVINO backend is expected to work on a broader range of Intel hardware, model precisions, llama.cpp tools and model architectures.

Legend & Test Configuration:

  • Status: ✓ = Passed | ~ = Accuracy issues | ✗ = Failed or Unsupported
  • Execution mode: stateless (GGML_OPENVINO_STATEFUL_EXECUTION=0) on every device.
  • Validation system: Intel® Core™ Ultra 7 258V (Lunar Lake) | 32 GB RAM | Ubuntu 26.04.1 | Intel Graphics Compiler 2.42.6 | Intel OpenCL GPU Driver 26.40.40196.8-0 | Intel NPU Driver 1.38.0.
  • See Known Limitations for context on observed failures.
Validated models
Model CPU GPU NPU
bartowski/Llama-3.2-1B-Instruct-Q4_K_M ✓ ✓ ✓
bartowski/Llama-3.2-3B-Instruct-Q4_K_M ✓ ✓ ✓
bartowski/Meta-Llama-3.1-8B-Instruct-Q4_K_M ✓ ✓ ✓
Qwen/qwen2.5-1.5b-instruct-q4_k_m ✓ ✓ ✓
Qwen/qwen2.5-coder-7b-instruct-q4_k_m ✓ ✓ ✓
bartowski/Qwen_Qwen3-0.6B-Q4_K_M ✓ ✓ ✓
bartowski/Qwen_Qwen3-1.7B-Q4_K_M ✓ ✓ ✓
Qwen/Qwen3-4B-Q4_K_M ✓ ✓ ✓
lm-kit/Qwen3-8B-Q4_K_M ✓ ✓ ✓
bartowski/Qwen_Qwen3.5-0.8B-Q4_K_M ✓ ✓ ✓
bartowski/Qwen_Qwen3.5-2B-Q4_K_M ✓ ✓ ✓
bartowski/Qwen_Qwen3.5-4B-Q4_K_M ✓ ✓ ✓
lmstudio-community/Qwen3.5-9B-Q4_K_M ✓ ✓ ✓
unsloth/gemma-3-4b-it-Q4_K_M ✓ ✓ ✓
bartowski/google_gemma-4-E2B-it-Q4_K_M ✓ ✓ ✓
bartowski/google_gemma-4-E4B-it-Q4_K_M ✓ ✓ ✓
bartowski/gemma-4-12B-it-Q4_K_M ✓ ✓ ✗
bartowski/Phi-3-mini-4k-instruct-Q4_K_M ✓ ✓ ✓
bartowski/Phi-3.5-mini-instruct-Q4_K_M ✓ ✓ ✓
bartowski/microsoft_Phi-4-mini-instruct-Q4_K_M ✓ ✓ ✓
bartowski/Mistral-7B-Instruct-v0.3-Q4_K_M ✓ ✓ ✓
QuantFactory/Ministral-3b-instruct.Q4_K_M ✓ ✓ ✓
bartowski/Ministral-8B-Instruct-2410-Q4_K_M ✓ ✓ ✓
bartowski/DeepSeek-R1-Distill-Llama-8B-Q4_K_M ✓ ✓ ✓
bartowski/DeepSeek-R1-Distill-Qwen-7B-Q4_K_M ✓ ✓ ✓
ibm-granite/granite-4.0-350m-Q4_K_M ✓ ✓ ✓
ibm-granite/granite-4.0-micro-Q4_K_M ✓ ✓ ✓
ibm-granite/granite-4.0-1b-Q4_K_M ✓ ✓ ~
ibm-research/granite-3.2-8b-instruct-Q4_K_M ✓ ✓ ✓
HuggingFaceTB/smollm2-1.7b-instruct-q4_k_m ✓ ✓ ✓
openbmb/MiniCPM-V-2_6-Q4_K_M ✓ ✓ ✓
openbmb/MiniCPM-V-4_5-Q4_K_M ✓ ✓ ✓
bartowski/tencent_Hunyuan-7B-Instruct-Q4_K_M ✓ ✓ ✓
LGAI-EXAONE/EXAONE-3.5-7.8B-Instruct-Q4_K_M ✓ ✓ ✓
bartowski/prism-ml_Bonsai-8B-unpacked-Q4_K_M ✓ ✓ ✓
LiquidAI/LFM2.5-2.6B-Q4_K_M ✓ ✓ ✗
bartowski/inclusionAI_Ling-mini-2.0-Q4_K_M ✓ ✓ ✗
tiiuae/Falcon-H1-0.5B-Instruct-Q4_K_M ✓ ✓ ✗
Qwen/Qwen3-Embedding-0.6B-Q8_0 ✓ ✓ ✗
ggml-org/bge-m3-Q8_0 ✓ ✓ ✗
nomic-ai/nomic-embed-text-v1.5-Q8_0 ✓ ✓ ✗

The embedding models were tested with llama-embedding and Q8_0 files.

Build Instructions

0. Prerequisites

Linux

Build tools (Git, CMake, Ninja):

sudo apt-get update
sudo apt-get install -y build-essential libcurl4-openssl-dev libtbb12 cmake ninja-build python3-pip curl wget tar

OpenCL:

sudo apt install ocl-icd-opencl-dev opencl-headers opencl-clhpp-headers intel-opencl-icd
Windows
  1. Download and install Microsoft Visual Studio 2022 Build Tools. During installation, select the "Desktop development with C++" workload.

  2. Install the required tools:

    # Windows PowerShell
    winget install Git.Git
    winget install GNU.Wget
    winget install Ninja-build.Ninja
    
  3. Install OpenCL using vcpkg:

    # Windows PowerShell
    cd C:\
    git clone https://github.com/microsoft/vcpkg
    cd vcpkg
    .\bootstrap-vcpkg.bat
    .\vcpkg install opencl
    # Optional but recommended: Integrate vcpkg with Visual Studio / CMake:
    .\vcpkg integrate install
    

1. Install OpenVINO Runtime

  • Follow the guide to install OpenVINO Runtime from an archive file: Linux | Windows

  • Verify OpenVINO is initialized properly:

    echo $OpenVINO_DIR
    

2. Build llama.cpp with OpenVINO Backend

Clone the llama.cpp repository:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp

Build on Linux:

source /opt/intel/openvino/setupvars.sh
cmake -B build/ReleaseOV -G Ninja -DCMAKE_BUILD_TYPE=Release -DGGML_OPENVINO=ON
cmake --build build/ReleaseOV --parallel
Build on Windows

Open x64 Native Tools Command Prompt for VS (so the MSVC toolchain is on PATH), then run:

C:\Intel\openvino\setupvars.bat
cmake -B build\ReleaseOV -G Ninja -DCMAKE_BUILD_TYPE=Release -DGGML_OPENVINO=ON -DCMAKE_TOOLCHAIN_FILE=C:\vcpkg\scripts\buildsystems\vcpkg.cmake
cmake --build build\ReleaseOV --parallel

Note

The Windows install path is C:\Intel\openvino (no spaces) to avoid quoting problems some CMake/Ninja toolchains have with C:\Program Files (x86)\.... Adjust to wherever you installed OpenVINO Runtime. From cmd, run C:\Intel\openvino\setupvars.bat; from PowerShell, run & "C:\Intel\openvino\setupvars.ps1" instead. Once the build is finished you can launch the binaries from any cmd or PowerShell window after sourcing the matching setupvars script for that shell.

Ubuntu Build Script

For Ubuntu users, the following shell script automates the prerequisite installs (build tools, OpenCL ICD), the OpenVINO Runtime download/extract/setup, and the Ninja-based llama.cpp build. Save the following as build-llamacpp-ov.sh next to where you want the llama.cpp folder to land, then run it:

chmod +x build-llamacpp-ov.sh
./build-llamacpp-ov.sh
Click to expand build-llamacpp-ov.sh
#!/usr/bin/env bash
# ============================================
# llama.cpp OpenVINO Build Script (Ninja)
# ============================================
set -euo pipefail

OPENVINO_VERSION_MAJOR="2026.4.1"
OPENVINO_VERSION_FULL="2026.4.1.22982.07f9c262b05"

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
OPENVINO_INSTALL_DIR="/opt/intel/openvino_${OPENVINO_VERSION_MAJOR}"
OPENVINO_LINK_DIR="/opt/intel/openvino"
OPENVINO_TGZ="${SCRIPT_DIR}/openvino.tgz"
OPENVINO_URL="https://storage.openvinotoolkit.org/repositories/openvino/packages/${OPENVINO_VERSION_MAJOR}/linux/openvino_toolkit_ubuntu24_${OPENVINO_VERSION_FULL}_x86_64.tgz"

echo "============================================"
echo "Installing prerequisites (apt)..."
echo "============================================"
sudo apt-get update
sudo apt-get install -y \
    build-essential libcurl4-openssl-dev libtbb12 \
    cmake ninja-build python3-pip \
    curl wget tar git

echo "============================================"
echo "Installing OpenCL runtime + headers..."
echo "============================================"
sudo apt-get install -y \
    ocl-icd-opencl-dev opencl-headers opencl-clhpp-headers intel-opencl-icd

cd "${SCRIPT_DIR}"

# ============================================
# Clone llama.cpp if missing
# ============================================
if [[ ! -f "llama.cpp/CMakeLists.txt" ]]; then
    echo "Cloning llama.cpp..."
    git clone https://github.com/ggml-org/llama.cpp
fi

# ============================================
# Setup OpenVINO: download & extract to /opt/intel/openvino_${OPENVINO_VERSION_MAJOR},
# then point /opt/intel/openvino at it via symlink so the active version is swappable.
# ============================================
if [[ -f "${OPENVINO_INSTALL_DIR}/setupvars.sh" ]]; then
    echo "OpenVINO ${OPENVINO_VERSION_MAJOR} already installed at ${OPENVINO_INSTALL_DIR}. Skipping download."
else
    echo "OpenVINO not found at ${OPENVINO_INSTALL_DIR}. Starting download..."
    curl -L -o "${OPENVINO_TGZ}" "${OPENVINO_URL}"

    echo "Extracting OpenVINO to ${OPENVINO_INSTALL_DIR}..."
    sudo mkdir -p "${OPENVINO_INSTALL_DIR}"
    sudo tar -xzf "${OPENVINO_TGZ}" -C "${OPENVINO_INSTALL_DIR}" --strip-components=1
    rm -f "${OPENVINO_TGZ}"
fi

# Refresh symlink: /opt/intel/openvino -> /opt/intel/openvino_${OPENVINO_VERSION_MAJOR}
sudo ln -sfn "${OPENVINO_INSTALL_DIR}" "${OPENVINO_LINK_DIR}"

OPENVINO_ROOT="${OPENVINO_LINK_DIR}"
echo "OpenVINO Ready: ${OPENVINO_ROOT} -> ${OPENVINO_INSTALL_DIR}"

# Install OpenVINO's own runtime dependencies (one-time per system).
if [[ -x "${OPENVINO_ROOT}/install_dependencies/install_openvino_dependencies.sh" ]]; then
    echo "============================================"
    echo "Installing OpenVINO runtime dependencies..."
    echo "============================================"
    echo "Y" | sudo -E "${OPENVINO_ROOT}/install_dependencies/install_openvino_dependencies.sh"
fi

# ============================================
# Clean old build cache
# ============================================
cd "${SCRIPT_DIR}/llama.cpp"
if [[ -d "build/ReleaseOV" ]]; then
    echo "Removing old build directory..."
    rm -rf "build/ReleaseOV"
fi

echo "============================================"
echo "Configuring with CMake..."
echo "============================================"
set +u
source "${OPENVINO_ROOT}/setupvars.sh"
set -u

cmake -B build/ReleaseOV -G Ninja \
    -DCMAKE_BUILD_TYPE=Release \
    -DGGML_OPENVINO=ON

cmake --build build/ReleaseOV --parallel

echo "============================================"
echo "Build completed successfully!"
echo "============================================"
echo "Binaries: $(pwd)/build/ReleaseOV/bin"
echo
echo "NOTE: To run, source setupvars.sh and pick a device:"
echo "  source /opt/intel/openvino/setupvars.sh"
echo "  export GGML_OPENVINO_DEVICE=CPU   # or GPU / NPU"
echo "  ./build/ReleaseOV/bin/llama-cli -m model.gguf"

Note

The script pins OpenVINO 2026.4.1 via the OPENVINO_VERSION_MAJOR / OPENVINO_VERSION_FULL variables at the top — edit them to track a different release.

Windows Build Script

For Windows users, the following .bat script automates the prerequisite installs (Git, Ninja, CMake, Visual Studio 2022 Build Tools, vcpkg + OpenCL), the OpenVINO Runtime download/extract, and the Ninja-based llama.cpp build. Save the following as build-llamacpp-ov.bat next to where you want the llama.cpp to land, then run it from either Command Prompt or PowerShell:

:: Command Prompt
build-llamacpp-ov.bat
# PowerShell
.\build-llamacpp-ov.bat
Click to expand build-llamacpp-ov.bat
@echo off
setlocal enabledelayedexpansion

REM ============================================
REM llama.cpp OpenVINO Build Script (Ninja)
REM ============================================

set "OPENVINO_VERSION_MAJOR=2026.4.1"
set "OPENVINO_VERSION_FULL=2026.4.1.22982.07f9c262b05"

set "SCRIPT_DIR=%~dp0"
set "VCPKG_DIR=C:\vcpkg"
set "OPENVINO_INSTALL_DIR=C:\Intel\openvino_%OPENVINO_VERSION_MAJOR%"
set "OPENVINO_LINK_DIR=C:\Intel\openvino"
set "OPENVINO_ZIP=%SCRIPT_DIR%openvino.zip"
set "OPENVINO_EXTRACT_TMP=%SCRIPT_DIR%openvino_extract_tmp"
set "OPENVINO_URL=https://storage.openvinotoolkit.org/repositories/openvino/packages/%OPENVINO_VERSION_MAJOR%/windows/openvino_toolkit_windows_%OPENVINO_VERSION_FULL%_x86_64.zip"

echo ============================================
echo Installing prerequisites...
echo ============================================
winget install --id Git.Git -e --accept-source-agreements --accept-package-agreements 2>nul
winget install --id Ninja-build.Ninja -e --accept-source-agreements --accept-package-agreements 2>nul
winget install --id Kitware.CMake -e --accept-source-agreements --accept-package-agreements 2>nul

REM Ensure Visual Studio Build Tools are installed.
echo Checking for Visual Studio Build Tools...
set "VSWHERE=%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe"
set "VS_INSTALLED="
if exist "%VSWHERE%" (
    for /f "usebackq tokens=*" %%i in (`"%VSWHERE%" -latest -products * -requires Microsoft.VisualStudio.Component.VC.Tools.x86.x64 -property installationPath 2^>nul`) do (
        set "VS_INSTALLED=%%i"
    )
)
if defined VS_INSTALLED (
    echo Visual Studio with VC++ x86/x64 tools already present at "!VS_INSTALLED!". Skipping winget install.
) else (
    winget install --id Microsoft.VisualStudio.2022.BuildTools -e --override "--wait --passive --add Microsoft.VisualStudio.Workload.VCTools --includeRecommended" --accept-source-agreements --accept-package-agreements
    if errorlevel 1 (
        echo WARNING: winget could not install Visual Studio Build Tools automatically.
        echo Install manually from https://aka.ms/vs/17/release/vs_BuildTools.exe ^(select the "Desktop development with C++" workload^)
        echo and re-run this script from a "Developer Command Prompt for VS 2022".
    )
)

echo ============================================
echo Installing OpenCL via vcpkg...
echo ============================================
if not exist "%VCPKG_DIR%" (
    git clone https://github.com/microsoft/vcpkg "%VCPKG_DIR%"
    cd /d "%VCPKG_DIR%"
    call bootstrap-vcpkg.bat
    call vcpkg integrate install
)
cd /d "%VCPKG_DIR%"
call vcpkg install opencl

cd /d "%SCRIPT_DIR%"

REM ============================================
REM Clone llama.cpp if missing
REM ============================================
if not exist "llama.cpp\CMakeLists.txt" (
    echo Cloning llama.cpp...
    git clone https://github.com/ggml-org/llama.cpp
)

cd /d "llama.cpp"
set "SCRIPT_DIR=%CD%"

REM ============================================
REM Setup OpenVINO: download & extract to C:\Intel\openvino_%OPENVINO_VERSION_MAJOR%,
REM then point C:\Intel\openvino at it via a directory junction (mklink /J).
REM ============================================

if exist "%OPENVINO_INSTALL_DIR%\setupvars.bat" (
    echo OpenVINO %OPENVINO_VERSION_MAJOR% already installed at "%OPENVINO_INSTALL_DIR%". Skipping download.
) else (
    echo OpenVINO not found at "%OPENVINO_INSTALL_DIR%". Starting download...

    curl -L -o "%OPENVINO_ZIP%" "%OPENVINO_URL%"
    if errorlevel 1 (
        echo ERROR: Download failed.
        exit /b 1
    )

    echo Extracting OpenVINO...
    if exist "%OPENVINO_EXTRACT_TMP%" rmdir /s /q "%OPENVINO_EXTRACT_TMP%"
    mkdir "%OPENVINO_EXTRACT_TMP%"
    tar -xf "%OPENVINO_ZIP%" -C "%OPENVINO_EXTRACT_TMP%"
    if errorlevel 1 (
        echo ERROR: Extraction failed.
        exit /b 1
    )

    REM Move the single top-level folder contents into the versioned install dir.
    set "OPENVINO_EXTRACTED="
    for /d %%i in ("%OPENVINO_EXTRACT_TMP%\*") do set "OPENVINO_EXTRACTED=%%i"
    if not defined OPENVINO_EXTRACTED (
        echo ERROR: Could not locate extracted OpenVINO folder under "%OPENVINO_EXTRACT_TMP%".
        exit /b 1
    )
    if not exist "%OPENVINO_INSTALL_DIR%" mkdir "%OPENVINO_INSTALL_DIR%"
    xcopy /e /i /y /q "!OPENVINO_EXTRACTED!\*" "%OPENVINO_INSTALL_DIR%\" >nul
    if errorlevel 1 (
        echo ERROR: Failed to copy OpenVINO from "!OPENVINO_EXTRACTED!" to "%OPENVINO_INSTALL_DIR%".
        echo Re-run this script from an elevated Command Prompt ^(Run as administrator^) if access is denied.
        exit /b 1
    )

    rmdir /s /q "%OPENVINO_EXTRACT_TMP%"
    del "%OPENVINO_ZIP%"
)

REM Refresh junction: C:\Intel\openvino -> C:\Intel\openvino_<version>.
REM `mklink /J` creates a directory junction (no admin / Developer Mode required).
if exist "%OPENVINO_LINK_DIR%" rmdir "%OPENVINO_LINK_DIR%"
mklink /J "%OPENVINO_LINK_DIR%" "%OPENVINO_INSTALL_DIR%" >nul
if errorlevel 1 (
    echo ERROR: Failed to create junction "%OPENVINO_LINK_DIR%" -^> "%OPENVINO_INSTALL_DIR%".
    echo If "%OPENVINO_LINK_DIR%" already exists as a regular non-empty folder, remove it manually and re-run.
    exit /b 1
)

set "OPENVINO_ROOT=%OPENVINO_LINK_DIR%"
echo OpenVINO Ready: %OPENVINO_ROOT% -^> %OPENVINO_INSTALL_DIR%


echo ============================================
echo Setting up compiler environment...
echo ============================================
REM Locate Visual Studio Build Tools vcvars64.bat
set "VSWHERE=%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe"
if exist "%VSWHERE%" (
    for /f "usebackq tokens=*" %%i in (`"%VSWHERE%" -latest -products Microsoft.VisualStudio.Product.BuildTools -property installationPath`) do (
        set "VS_PATH=%%i"
    )
)
if defined VS_PATH (
    call "%VS_PATH%\VC\Auxiliary\Build\vcvars64.bat" >nul
) else (
    echo WARNING: Visual Studio Build Tools not found. Compiler may be missing.
)

REM ============================================
REM Clean old build cache
REM ============================================
if exist "build\ReleaseOV" (
    echo Removing old build directory ...
    rmdir /s /q "build\ReleaseOV"
)

echo ============================================
echo Configuring with CMake...
echo ============================================
call "%OPENVINO_ROOT%\setupvars.bat" >nul 2>nul

cmake -B build\ReleaseOV -G Ninja ^
    -DCMAKE_BUILD_TYPE=Release ^
    -DGGML_OPENVINO=ON ^
    -DCMAKE_TOOLCHAIN_FILE="%VCPKG_DIR%\scripts\buildsystems\vcpkg.cmake"

if errorlevel 1 (
    echo If you continue to face CMAKE errors, make sure to install:
    echo   winget install Microsoft.VisualStudio.2022.BuildTools
    echo   Then run the "Developer Command Prompt for VS 2022" and launch this script from there.
    exit /b 1
)

cmake --build build\ReleaseOV --config Release
if errorlevel 1 exit /b 1

echo ============================================
echo Build completed successfully!
echo ============================================
echo Binaries: %CD%\build\ReleaseOV\bin
echo.
echo NOTE: To run, source setupvars.bat and pick a device:
echo   call "C:\Intel\openvino\setupvars.bat"
echo   set GGML_OPENVINO_DEVICE=CPU   ^&^& REM or GPU / NPU
echo   build\ReleaseOV\bin\llama-cli.exe -m model.gguf
echo.

endlocal

Note

The script pins OpenVINO 2026.4.1 via the OPENVINO_VERSION_MAJOR / OPENVINO_VERSION_FULL variables at the top — edit them to track a different release. From any new shell, source the matching setupvars script via the junction — call "C:\Intel\openvino\setupvars.bat" from cmd, or & "C:\Intel\openvino\setupvars.ps1" from PowerShell. If winget cannot register Visual Studio Build Tools on first run, install them once manually and re-run the script from an elevated Developer Command Prompt for VS 2022.

3. Download Sample Model

Download a sample model for testing.

mkdir -p ~/models/
wget https://huggingface.co/bartowski/Llama-3.2-1B-Instruct-GGUF/resolve/main/Llama-3.2-1B-Instruct-Q4_K_M.gguf \
     -O ~/models/Llama-3.2-1B-Instruct-Q4_K_M.gguf
Windows
# Windows PowerShell
mkdir C:\models
Invoke-WebRequest -Uri https://huggingface.co/bartowski/Llama-3.2-1B-Instruct-GGUF/resolve/main/Llama-3.2-1B-Instruct-Q4_K_M.gguf -OutFile C:\models\Llama-3.2-1B-Instruct-Q4_K_M.gguf
REM Windows Command Prompt
mkdir C:\models
curl -L https://huggingface.co/bartowski/Llama-3.2-1B-Instruct-GGUF/resolve/main/Llama-3.2-1B-Instruct-Q4_K_M.gguf -o C:\models\Llama-3.2-1B-Instruct-Q4_K_M.gguf

4. Run Inference with OpenVINO Backend

The first run of a model converts and compiles it for the device. This takes seconds on CPU and GPU, and can take minutes on the NPU for large models. Later tokens are faster, and the compiled model cache makes later runs start faster.

Note

Default context size is set to the model training context, which may be very large. For example, 131072 for Llama 3.2 1B, which may result in lower performance, especially on edge/laptop devices. Use -c to limit context size in supported llama.cpp tools for better performance:

  • CPU and GPU: for example -c 1024.
  • NPU: keep the context small, for example -c 512, or -c 256 for larger models. Decode may slow down as the context grows.
# If the device is unset, the CPU is used.
export GGML_OPENVINO_DEVICE=GPU
# To run llama-simple:
./build/ReleaseOV/bin/llama-simple -m ~/models/Llama-3.2-1B-Instruct-Q4_K_M.gguf -n 50 "The story of AI is "
# To run in chat mode:
./build/ReleaseOV/bin/llama-cli -m ~/models/Llama-3.2-1B-Instruct-Q4_K_M.gguf -c 1024
# To run llama-bench:
GGML_OPENVINO_DEVICE=GPU ./build/ReleaseOV/bin/llama-bench -m ~/models/Llama-3.2-1B-Instruct-Q4_K_M.gguf

# NPU: keep context small to avoid failures from very large model context windows.
export GGML_OPENVINO_DEVICE=NPU
./build/ReleaseOV/bin/llama-cli -m ~/models/Llama-3.2-1B-Instruct-Q4_K_M.gguf -c 512
Windows
REM Windows Command Prompt
set GGML_OPENVINO_DEVICE=GPU
REM To run llama-simple:
build\ReleaseOV\bin\llama-simple.exe -m "C:\models\Llama-3.2-1B-Instruct-Q4_K_M.gguf" -n 50 "The story of AI is "
REM To run in chat mode:
build\ReleaseOV\bin\llama-cli.exe -m "C:\models\Llama-3.2-1B-Instruct-Q4_K_M.gguf" -c 1024
REM To run llama-bench:
build\ReleaseOV\bin\llama-bench.exe -m "C:\models\Llama-3.2-1B-Instruct-Q4_K_M.gguf"

REM NPU: keep the context small.
set GGML_OPENVINO_DEVICE=NPU
build\ReleaseOV\bin\llama-cli.exe -m "C:\models\Llama-3.2-1B-Instruct-Q4_K_M.gguf" -c 512
# Windows PowerShell
$env:GGML_OPENVINO_DEVICE = "GPU"
build\ReleaseOV\bin\llama-cli.exe -m "C:\models\Llama-3.2-1B-Instruct-Q4_K_M.gguf" -c 1024

# NPU: keep the context small.
$env:GGML_OPENVINO_DEVICE = "NPU"
build\ReleaseOV\bin\llama-cli.exe -m "C:\models\Llama-3.2-1B-Instruct-Q4_K_M.gguf" -c 512

Note

  • Select the OpenVINO device with GGML_OPENVINO_DEVICE, not with -dev.
  • On systems with multiple GPUs, use GPU.0 or GPU.1 to explicitly target specific GPU.
  • To split a model across several GPUs, set a comma-separated list, e.g. GPU.0,GPU.1, and use -sm layer.
  • A device that is not available is an error (no fallback to CPU). The error message lists the available OpenVINO devices with their names.
  • Run llama-cli --list-devices to see the valid values: each OpenVINO device shows the GGML_OPENVINO_DEVICE=<value> to set, and (selected) marks the active one.
  • See OpenVINO GPU Device for more details.

5. Docker Build

You can build and run llama.cpp with OpenVINO backend using Docker.

Build the images
# Build the base runtime image with compiled shared libraries and minimal dependencies.
docker build -t llama-openvino:base -f .devops/openvino.Dockerfile .

# Build the complete image with all binaries, Python tools, gguf-py library, and model conversion utilities.
docker build --target=full -t llama-openvino:full -f .devops/openvino.Dockerfile .

# Build a minimal CLI-only image containing just the llama-cli executable.
docker build --target=light -t llama-openvino:light -f .devops/openvino.Dockerfile .

# Builds a server-only image with llama-server executable, health check endpoint, and REST API support.
docker build --target=server -t llama-openvino:server -f .devops/openvino.Dockerfile .

# If you are behind a proxy:
docker build --build-arg http_proxy=$http_proxy --build-arg https_proxy=$https_proxy --target=server -t llama-openvino:server -f .devops/openvino.Dockerfile .
Run a container (CPU, GPU, NPU)

Run llama.cpp with OpenVINO backend Docker container. Save sample models in ~/models as shown above. It will be mounted to the container in the examples below.

#  Run Docker container
docker run --rm -it -v ~/models:/models llama-openvino:light --no-warmup -c 1024 -m /models/Llama-3.2-1B-Instruct-Q4_K_M.gguf

# With Intel GPU access (iGPU or dGPU)
docker run --rm -it -v ~/models:/models \
--device=/dev/dri --group-add=$(stat -c "%g" /dev/dri/render* | head -n 1) -u $(id -u):$(id -g) \
--env=GGML_OPENVINO_DEVICE=GPU \
llama-openvino:light --no-warmup -c 1024 -m /models/Llama-3.2-1B-Instruct-Q4_K_M.gguf

# With Intel NPU access
docker run --rm -it -v ~/models:/models \
--device=/dev/accel --group-add=$(stat -c "%g" /dev/accel/accel* | head -n 1) -u $(id -u):$(id -g) \
--env=GGML_OPENVINO_DEVICE=NPU \
llama-openvino:light --no-warmup -c 512 -m /models/Llama-3.2-1B-Instruct-Q4_K_M.gguf
Run llama-server (container or local build) and test it
# Run the llama-openvino:server Docker container (CPU)
docker run --rm -it -p 8080:8080 -v ~/models:/models llama-openvino:server --no-warmup -m /models/Llama-3.2-1B-Instruct-Q4_K_M.gguf -c 1024 --host 0.0.0.0

# Run the llama-openvino:server Docker container with Intel GPU access (iGPU or dGPU)
docker run --rm -it -v ~/models:/models \
--device=/dev/dri --group-add=$(stat -c "%g" /dev/dri/render* | head -n 1) -u $(id -u):$(id -g) \
-p 8080:8080 --env=GGML_OPENVINO_DEVICE=GPU  \
llama-openvino:server --no-warmup -c 1024 -m /models/Llama-3.2-1B-Instruct-Q4_K_M.gguf --host 0.0.0.0

# Run the llama-openvino:server Docker container with Intel NPU access
docker run --rm -it -v ~/models:/models \
--device=/dev/accel --group-add=$(stat -c "%g" /dev/accel/accel* | head -n 1) -u $(id -u):$(id -g) \
-p 8080:8080 --env=GGML_OPENVINO_DEVICE=NPU \
llama-openvino:server --no-warmup -c 512 -m /models/Llama-3.2-1B-Instruct-Q4_K_M.gguf --host 0.0.0.0

# Or Using llama-server executable
./build/ReleaseOV/bin/llama-server -m ~/models/Llama-3.2-1B-Instruct-Q4_K_M.gguf --port 8080 -c 1024

# Option 1: Open your browser to http://localhost:8080 to open Llama Chat, the built-in chat app of the llama.cpp server.
# Option 2: In a NEW terminal, test the server with curl

# If you are behind a proxy, make sure to set NO_PROXY to avoid proxy for localhost
export NO_PROXY=localhost,127.0.0.1

# Test health endpoint
curl -f http://localhost:8080/health

# Test with a simple prompt
curl -X POST "http://localhost:8080/v1/chat/completions" -H "Content-Type: application/json" \
 -d '{"messages":[{"role":"user","content":"Write a poem about OpenVINO"}],"max_tokens":100}' | jq .

Runtime Configuration

The backend needs no environment variable: it runs on the CPU with its defaults. Set GGML_OPENVINO_DEVICE to use the GPU or the NPU. All other variables are optional.

export GGML_OPENVINO_DEVICE=GPU            # String
export GGML_OPENVINO_PROFILING=1           # Integer
export GGML_OPENVINO_EXACT_ZERO_POINT=1    # Boolean: 1 (on) or 0 (off)

# Show the variables in effect (log verbosity 4)
./build/ReleaseOV/bin/llama-cli -m ~/models/Llama-3.2-1B-Instruct-Q4_K_M.gguf -c 1024 -lv 4 2>&1 | grep "settings from the environment"
  • The backend reads the variables once, at start. Unset or empty means the default.
  • To lower the memory a model needs, see Tips for Memory Optimization.
  • A value that is not valid stops the backend. A GGML_OPENVINO_* name that the backend does not know gives a warning:
ggml-openvino: GGML_OPENVINO_EXACT_ZERO_POINT=yes is not valid, use 1 (on) or 0 (off)
ggml-openvino: unknown environment variable GGML_OPENVINO_DEVCE is ignored

Basic

Variable Device Type Default Description
GGML_OPENVINO_DEVICE all String CPU Target device: CPU, GPU or NPU. For several GPUs, see the device notes. Examples: GGML_OPENVINO_DEVICE=NPU, GGML_OPENVINO_DEVICE=GPU.0,GPU.1.
GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR all String not set Directory for the backend's cache of compiled models. A later start with the same model and settings loads faster. See Compiled model cache. Example: GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR=/tmp/ov_cache.
GGML_OPENVINO_NPU_CONFIG NPU String not set OpenVINO properties for the NPU plugin, as comma-separated KEY=VALUE pairs. See Plugin properties. Example: GGML_OPENVINO_NPU_CONFIG="NPU_COMPILER_TYPE=PLUGIN".
GGML_OPENVINO_GPU_CONFIG GPU String not set OpenVINO properties for the GPU plugin, as comma-separated KEY=VALUE pairs. See Plugin properties. Example: GGML_OPENVINO_GPU_CONFIG="INFERENCE_PRECISION_HINT=f16".
GGML_OPENVINO_CPU_CONFIG CPU String not set OpenVINO properties for the CPU plugin, as comma-separated KEY=VALUE pairs. See Plugin properties. Example: GGML_OPENVINO_CPU_CONFIG="INFERENCE_PRECISION_HINT=f16".
GGML_OPENVINO_PROFILING all Integer 0 1 logs execution timing. 2 or higher also turns on OpenVINO and OpenCL profiling. Example: GGML_OPENVINO_PROFILING=1.

Plugin properties

GGML_OPENVINO_NPU_CONFIG, GGML_OPENVINO_GPU_CONFIG and GGML_OPENVINO_CPU_CONFIG pass OpenVINO plugin properties to the device in their name.

# Format: comma-separated KEY=VALUE pairs. The backend uses the variable of the selected device only.
export GGML_OPENVINO_DEVICE=NPU
export GGML_OPENVINO_NPU_CONFIG="NPU_COMPILER_TYPE=PLUGIN,COMPILATION_NUM_THREADS=4"

# CPU: compute in FP16, as the GPU and the NPU do. This helps to debug the accuracy of a quantized model.
export GGML_OPENVINO_DEVICE=CPU
export GGML_OPENVINO_CPU_CONFIG="INFERENCE_PRECISION_HINT=f16"
  • Property names and values are those of OpenVINO, for example YES and NO. They are not the 1 and 0 of the GGML_OPENVINO_* variables.
  • The backend applies the variable last, so it overrides the properties that the backend sets.
  • OpenVINO checks the properties when the model loads. If it rejects one, the load fails and the error names the variable.
  • All properties and their values are in the OpenVINO documentation: NPU device, GPU device, CPU device.
  • NPU: the backend sets several properties by default. They are listed in the developer documentation.
  • GPU and CPU: the backend sets no property, except CACHE_DIR and CACHE_MODE=OPTIMIZE_SIZE when GGML_OPENVINO_CACHE_DIR is set.

NPU_COMPILER_TYPE selects the NPU compiler. Both forms below work; use the first one. If both are set, GGML_OPENVINO_NPU_CONFIG takes precedence.

# Values: DRIVER (the compiler in the NPU driver; backend default), PLUGIN (the compiler in OpenVINO),
# PREFER_PLUGIN (OpenVINO default: PLUGIN when it is available, DRIVER if not).
export GGML_OPENVINO_NPU_CONFIG="NPU_COMPILER_TYPE=PLUGIN"   # recommended form
export GGML_OPENVINO_NPU_COMPILER_TYPE=PLUGIN                # older name, still works

Advanced tuning

The defaults fit most models. The variables below trade one property for another, for example speed for accuracy.

Advanced tuning variables
Variable Device Type Default Description
GGML_OPENVINO_PREFILL_CHUNK_SIZE NPU Integer 256 Number of tokens that the NPU processes in one prefill step. A larger value can make long prompts faster, and makes the first load longer. Zero or a negative value means the default. Example: GGML_OPENVINO_PREFILL_CHUNK_SIZE=512.
GGML_OPENVINO_NPU_TOKEN_EMBD_QUANT NPU String q8_0_c How a Q6_K token_embd.weight is stored for the NPU: q8_0_c (8-bit, one scale per row), f16, q4_0_128 (4-bit, one scale per 128 weights) or q4_0_c (4-bit, one scale per row). The 4-bit layouts use less memory at some cost in accuracy; f16 uses more memory. Keep the default unless memory is tight.
GGML_OPENVINO_GDN_DECOMPOSE all Boolean auto Translate GATED_DELTA_NET (Qwen3.5) into MatMul operations instead of the OpenVINO GatedDeltaNet op. Unset: on for the Lunar Lake NPU, off elsewhere. 0 turns it off, 1 turns it on for any device. Try 1 for Qwen3.5 on the Panther Lake NPU.
GGML_OPENVINO_MANUAL_GQA_ATTN CPU, GPU Boolean on for GPU Build grouped-query attention from separate MatMul operations. Unset: on for GPU, off for CPU and NPU. 1 turns it on, 0 turns it off. Try 1 on the CPU for faster decode.
GGML_OPENVINO_REQUANT_KQUANT CPU, GPU String not set Requantize Q6_K and Q5_K weights to a 4-bit layout instead of Q8_0_C: faster decode for lower accuracy. One of q4_asym64, q4_asym64_all (also Q4_K), q4_sym128, q4_sym128_all (also Q4_K) or native (no requantization). On GPU with Q4_K_M files use q4_asym64_all, because q4_asym64 does not compile there.
GGML_OPENVINO_EXACT_ZERO_POINT CPU, GPU Boolean 0 Keep the exact zero point of asymmetric weights (Q4_1, Q4_K, Q5_1, Q5_K) instead of rounding it to an integer. This improves accuracy and lowers speed. Use it when accuracy matters more than speed.

Compiled model cache

The cache keeps compiled models on disk, so later starts of the same model load faster. Use the same model and the same settings in every run.

# CPU or GPU: the first run compiles the model and writes the cache. Later runs read it.
export GGML_OPENVINO_DEVICE=GPU
export GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR=/tmp/ov_cache
./build/ReleaseOV/bin/llama-cli -m ~/models/Llama-3.2-1B-Instruct-Q4_K_M.gguf -c 1024

# CPU or GPU, later runs (optional): require the cached model and skip the weight upload and the compilation.
export GGML_OPENVINO_COMPILED_MODEL_CACHE_ONLY=1

# NPU: add the weightless bank.
export GGML_OPENVINO_DEVICE=NPU
export GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR=/tmp/ov_cache
export GGML_OPENVINO_NPU_WEIGHTLESS_BANK=1
./build/ReleaseOV/bin/llama-cli -m ~/models/Llama-3.2-1B-Instruct-Q4_K_M.gguf -c 512

# NPU, later runs (optional): use less memory. The run fails if a cached model is missing.
export GGML_OPENVINO_SELF_CONTAINED_MMAP=1
Cache variables
Variable Device Type Default Description
GGML_OPENVINO_COMPILED_MODEL_CACHE_ONLY CPU, GPU Boolean 0 Require an existing cached model and skip the weight upload and the compilation. Needs mmap model loading (the default on Linux and Windows) and a model that runs fully on OpenVINO.
GGML_OPENVINO_NPU_WEIGHTLESS_BANK NPU Boolean 0 Requires GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR. Write the prefill and decode models without weights, plus one shared weights file. Later runs read the weights from that file and keep one copy in memory.
GGML_OPENVINO_SELF_CONTAINED_MMAP NPU Boolean 0 Later runs only. Requires GGML_OPENVINO_NPU_WEIGHTLESS_BANK=1 and an existing cache. Use the memory-mapped GGUF file as the model buffer, which lowers the host memory peak. Fails if a cached model is missing; run once without it to build the cache.
GGML_OPENVINO_CACHE_DIR all String not set Directory for the cache of the OpenVINO plugin itself. GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR is the recommended cache.
  • One directory can hold the cached models of different models and settings.
  • NPU: each prefill length enters the cache in a run without GGML_OPENVINO_SELF_CONTAINED_MMAP.
  • See Compiled model cache for the workflow and the restrictions.

Deprecated

Variable Device Type Default Description
GGML_OPENVINO_STATEFUL_EXECUTION CPU, GPU Boolean 0 Deprecated, to be removed. Keep the KV cache inside the OpenVINO model. Use the default, stateless execution (0).
Limits of stateful execution
  • CPU and GPU only; it has no effect on the NPU.
  • One slot only (-np 1).
  • State save and restore, sequence rewind, context shift and recurrent rollback are not supported.
  • Qwen3.5 needs a graph that runs fully on OpenVINO, with model caching on.
  • A prompt that starts at position 0 resets all states.

Debugging and development

Debug output, switches that turn a default optimization off, experimental options and older variable names are in the developer documentation.

Example Usage

GPU Inference with Profiling

export GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR=/tmp/ov_cache
export GGML_OPENVINO_PROFILING=1
export GGML_OPENVINO_DEVICE=GPU

./build/ReleaseOV/bin/llama-simple -m ~/models/Llama-3.2-1B-Instruct-Q4_K_M.gguf -n 50 "The story of AI is "
Windows
REM Windows Command Prompt
set GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR=C:\tmp\ov_cache
set GGML_OPENVINO_PROFILING=1
set GGML_OPENVINO_DEVICE=GPU
build\ReleaseOV\bin\llama-simple.exe -m "C:\models\Llama-3.2-1B-Instruct-Q4_K_M.gguf" -n 50 "The story of AI is "
# Windows PowerShell
$env:GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR = "C:\tmp\ov_cache"
$env:GGML_OPENVINO_PROFILING = "1"
$env:GGML_OPENVINO_DEVICE = "GPU"
build\ReleaseOV\bin\llama-simple.exe -m "C:\models\Llama-3.2-1B-Instruct-Q4_K_M.gguf" -n 50 "The story of AI is "

Known Limitations

General (all devices)

  • Llama.cpp OpenVINO backend currently supports a subset of GGML ops and text-only models. Unsupported ops or unsupported op shapes/cases fail during OpenVINO translation.
  • Multimodal features (audio/image/video) are a work in progress.
  • Embedding models: reranking models are not validated.
  • Llama.cpp tool coverage across CPU/GPU/NPU is not uniform.
  • Mamba-based models (e.g. Falcon-H1): llama-perplexity with several sequences per batch fails; set -b to the same value as -c. GPU accuracy is slightly lower than CPU.

GPU-specific

  • Multiple GPUs (GGML_OPENVINO_DEVICE=GPU.0,GPU.1 with -sm layer):
    • -sm tensor is not supported.
    • Copies between GPUs go through host memory. There is no asynchronous copy and no peer-to-peer copy.
    • Only GPUs can be combined. A list with several NPUs, or with a GPU and the CPU or the NPU, is rejected.
  • MoE models use the GPU plugin's fused MoE op, which runs in f16 only:
    • granitemoe models (e.g. granite-3.1-1b-a400m) give wrong output.
      • Workaround: GGML_OPENVINO_MOE_OP=0. Output is correct, but prompt processing is much slower.
    • GGML_OPENVINO_GPU_CONFIG="INFERENCE_PRECISION_HINT=f32" fails to compile.
  • llama-perplexity with MoE models: set -b to the same value as -c, so that each batch holds one sequence.

NPU-specific

  • Default context resolves to the model's training context (e.g. 131072 for Llama 3.2 1B), which can OOM or fail or degrade performance on NPU. Inspect the resolved value with -lv 3.
    • Workaround: Pass an explicit -c <N>, e.g. -c 512.
  • NPU device uses a static graph with a fixed prefill chunk size (defaults to 256), configurable with GGML_OPENVINO_PREFILL_CHUNK_SIZE. Large prefill/batch settings may need tuning.
  • llama-server -np > 1 (multiple parallel sequences) is not supported.
  • llama-perplexity: requires -b 512 or smaller.
  • Flash attention must stay on, which is the default (-fa auto). With -fa off, the run fails on the NPU.
  • llama-embedding fails on the NPU.
  • Mamba-based models (e.g. Falcon-H1) fail on the NPU with Failed to set tensor (element type mismatch).
  • Lunar Lake NPU: decode slows down sharply when the context is large. The context size at which this starts depends on the model.
    • Workaround: Pass the smallest context that fits the use, e.g. -c 512, or -c 256 for larger models.
  • Panther Lake NPU: the load of a Qwen3.5 model can end with a segmentation fault in the NPU compiler of OpenVINO 2026.4.1.
    • Workaround: Set GGML_OPENVINO_NPU_CONFIG=COMPILATION_NUM_THREADS=1.
  • Panther Lake NPU: GGML_OPENVINO_NPU_TOKEN_EMBD_QUANT=f16 and q4_0_128 can give wrong output. Keep the default.
  • Lunar Lake NPU: granite-4.0-1b gives wrong output. The activations of the model exceed the FP16 range, and the NPU computes in FP16. The model runs correctly on CPU and GPU.
  • Lunar Lake NPU: gemma-4-12B does not run. The first inference fails with ZE_RESULT_ERROR_UNINITIALIZED from the NPU driver. The model runs on CPU and GPU.
  • The first load of a model on the NPU compiles it for the device, which can take minutes for large models.

Note

The OpenVINO backend is actively under development. Fixes and improvements are underway, and this document will continue to be updated.

Work in Progress

  • Performance and memory optimizations
  • Accuracy validation
  • Broader quantization coverage
  • Support for additional model architectures