* ggml-et: Add performance logging * ggml-et: Quants helpers * ggml-et: Add MUL_MAT kernel * ggml-et: Add ROPE kernel * ggml-et: Add RMS_NORM kernel * ggml-et: Add GLU kernel * ggml-et: Add SOFT_MAX kernel * ggml-et: Add GET_ROWS kernel * ggml-et: Add CONT kernel * ggml-et: Add SET_ROWS kernel * ggml-et: Add MUL_MAT_ID kernel * ggml-et: Build et kernels as part of ggml * ggml-et: Embed kernels with fs fallback * ggml-et: Build fixes * ggml-et: Add MUL_MAT F32xF32 op * ggml_et: Add MUL_MAT_ID op * ggml-et: Disable offloading for debug * ggml-et: Refactor out block ops * ggml-et: ggml backend API changes * ggml-et: Add RESHAPE/TRANSPOSE to supported * ggml-et: Add CONT_F16 * ggml-et: Add supported ops doc * gglm-et: Initial doc * ggml-et: Remove runtime import hacks We can now import the runtime by a simple find_package(), so we can cleanup the CMakeLists.txt. * ggml-et: Fix GET_ROWS kernel Fix lost batch dimension. Also clean vibe-comments. * ggml-et: Fix SET_ROWS kernel Remove incorrect broadcasting guard. * ggml-et: Use custom instruction for fp32->fp16 * ggml-et: Vectorize set_rows fp32->fp16 * ggml-et: Fix ROPE kernel (yarn) ggml-et: fix et_logf WIP: Fix ramp WIP: fix ROPE! * ggml-et: Better sinf * ggml-et: Fix SOFT_MAX Add `max_bias` and `sink` support. * ggml-et: Fix CONT Reorder from contiguous write to read with atomic stores. * ggml-et: Fix elmap kernel Remainder handlin * ggml-et: Fix MUL_MAT MUL_MAT_ID remainders * ggml-et: Fix ET-SOC reference * ggml-et: Fix embed kernels scripts for old python This allows GGML-ET to build on pre-3.8 python. * Add sysemu support with compile time flag `-DGGML_ET_SYSEMU=ON` (#6) * Example using ET-Soc-1 emulator configuration Example usage: ```bash cmake -B build -DGGML_CUDA=OFF -DGGML_ET=ON -DLLAMA_CURL=OFF -DGGML_CCACHE=ON cmake --build build --config Release -j $(nproc) time ./build/bin/test-backend-ops ./build/bin/llama-server \ --model Qwen3-0.6B-Q8_0.gguf \ --alias Qwen3-0.6B-Q8_0 \ -fa 0 \ --ctx-size 1024 \ --no-warmup \ --host 127.0.0.1 \ --port 8080 ``` * build: proper dep tracking for kernels * support host using MOLD linker * initial multi core GET_ROW F32 implementation * vectorized q8 dequant * wip: cland warning clenaups and initial logging refactor * wip: message default message cleanup * chore: message cleanups * cmake cleanup * migrate to use platform provided functions * cmake back into subdir * support et_print() in kernels * fix: repair kernel building * perf: operations run async by default * debug: proper kernel dep tracking and error detection on kenrel launch * fix: kernel binary dep tracking and fixing get_rows_f32 erroring * perf: back to doing async kernel runs by default * perf: vectorize and parallel device memset * merge matmul work * misc: align allocation and enable all offload * misc: delete deadcode and respect memory limits * fix: repair tensor debug print * fix: loosen RMS_NORM op percision * feat: Q4_0 GET_ROWS * perf: FP32 MUL_MAT using TensorFMA * update limitations * perf: redue L1 load in compute_block_dot_product_q8_0 * feat: save kernel mapping (name to id) when profiling is enabled * chore: memops cleanup * perf: parallelize softmax by rows * perf: vectorize 2nd phase of softmax * perf: ban GET_ROWS from offloaded * perf: vectorize and non-atomic for eltwise ops and sub support * perf: vectorize normal rope * perf: glu runs in parallel * merge: manually merge saqib's work on kernel fixes * perf: more vectorized RoPE * perf: parallelize mul_mat_id * perf: parallelize set_rows_f32 * perf: vectorize softmax * feat: support kernel fusion and fuse RMS_NORM + MUL * fix: mostly resolve test-backend-ops failure in SOFT_MAX and ROPE * fix: bump max rope dims for gemma * feat: GeGLU and SCALE support to fully offload Gemma * perf: faster device memset * feat: get_rows supporting Q4_K and avoid cont cache coherent issues * better F32 MM * feat: NORM for ET backend * feat: SQR for ET backend * feat: UNARY on ET * feat: el_map support broadcasting for ET * feat: SUM_ROWS in ET backend * feat: more ops in ET backend * feat: WKV* operators in ET backend * perf: parallelize operators across cacheline instead of row * perf: parallelize get_rows on cacheline * wip: baseline FlashAttention for ET backend * wip: enough FA and CPY f32->f16 to run llama 3.1 fully offloaded with FA on * feat: f16 x f16 -> f32 MM using matrix engine * wip: f16 FlashAttention using matrix engine * wip: clean up * feat: barriers * perf: optimize FA_F16 in ET * perf: vectorize pack_k_for_transpose16 * perf: prefetch next loop matrix tile * perf: FlashAttention 2nd MM uses TensorFMA and optimizations * cleanup: flashattention reorg * perf: optimizations and fixes * feat: L2SCP API and make FlashAttention support DV = 256 for gemma * perf: parallelize norms beyond single row * feat: GATED_DELTA_NET support and relaxed L2_NORM requirment * feat: loosen RMS_NORM, NORM, ROPE contingous req too * feat: repeat supports brocasting on dim 0 and loosen cont check * feat: FILL and DIAG operator * feat: loosen UNARY support chcek * feat: TRI support * feat: SOLVE_TRI support * feat: basic SET support * feat: loosen CONT req * perf: fp16_to_fp32 use ASM * feat: IMROPE support * feat: PAD support * feat: global barrier * fix: view must live on the same backend as backing tensor * feat: relax CONCAT in ET backend * feat: dead simple CUMSUM implementation * feat: basic SSM_CONV support * feat: loosen CONCAT req * feat: relax GATED_DELTA_NET and add SET support proper * cleanup: cleanup LCM math * feat: SWIGLU single input * feat: SSM_SCAN support * feat: el_map supports non aligned tensors in best effort * feat: basic GROUP_NORM support * feat: loosen MUL_MAT capablities slightly * feat: loosen MUL_MAT and GET_ROWS and add IM2COL * feat: special case for softmax 1x1x1x1 * feat: loosen SOFT_MAX req in ET backend * fix: el_map unaligned acse fixes * perf: optimize zero_acc_vec in flash_attn_ext_f16_me * perf: use hart 1 for packing in MM and FA for FP16 * feat: kernel semaphore * perf: better instruction sequence in FlashAttention * fix: gated_delta_net with proper masking * perf: better parallelization for GATED_DELTA_NET * perf: parallelize SSM_CONV over nr * perf: vectorize SSM_CONV * perf: optimize MUL_MAT for q8 * feat: support Gemma 4 * fix: support multi-device * feat: broader GLU support * feat: unary ops supports view * fix: repair fp16 MM using matrix engine * perf: handle large N GEMV better * perf: better q8_0 MM * perf: better set_rows * add back deleted files * fix: repair after merge * feat: POC version of uberkernel * feat: RMS_NORM in uberkernel * feat: add more kernels into usage * chore: clean up uberkernel compilation * perf: faster flash attention * perf: opt flash attention for large seq length * feat: loosen op bounds. clamp and mean support * perf: vectorize ssm_scan * perf: slightly faster FA * perf: FlashAttention parallel MM and load * perf: fuse Q8 MM and ADD * feat: basic conv kernel for ET * softMAx_test * set_rows_f32 * get_rows and cont * testing * set_rows_exp * Junk addition * Narrowing the issue * Update flash_attn_ext_f16_me.c Focusing FA_ext_f16_me * test * Eviction updated * Detailed cache eviction debug * mulmat * removeal of `BUILD_FOR_UBERKERNEL` flag * cleaning... * fix: balance FCC0 count * feat: implement mul_mat and mul_mat_id for Q4_0 type * optimize uberkernel plan upload * add mul_mat q4 into uberkernel * enable gating flush to just uberkernel * update docs for ET * update op support for ET * et-backend: optimize Q4_0 and Q8_0 mul_mat_id row accumulations * et-backend: specialize mul_mat_id kernels for Q4_0 and Q8_0 * et-backend: fix RoPE YaRN corr_dim formula and handle degenerate inputs * test-backend-ops: add DeepSeek-V2-Lite RoPE test coverage * et-backend: add Q4_0 mul_mat matrix-engine kernel using TensorFMA32 * et-backend: vectorize Q4_0 matrix-engine dequantization * et-backend: support hybrid matrix/vector engine execution for Q4_0 mul_mat tail * et-backend: run partial-N tiles on matrix engine for Q4_0 mul_mat * et-backend: route Q4_0 mul_mat N < 53 to vecdot for better prefill latency * Update uberkernel.c * Update unary_f32.c * gemma 4 * bisect gemma4: enable scale_f32 only * bisect gemma4: +rms_norm_f32 * bisect gemma4: +rms_norm_mul_f32 * bisect gemma4: disable rms_norm_mul_f32 -- BREAKS OUTPUT * bisect gemma4: +rope_f32 (skip rms_norm_mul) * bisect gemma4: +el_map_f32 * bisect gemma4: +softmax_f32 * bisect gemma4: +get_rows_f32 * bisect gemma4: +glu_f32 * bisect gemma4: +mul_mat_f32 +mul_mat_f32_matrix_engine * bisect gemma4: +mul_mat_f16 +mul_mat_f16_matrix_engine * bisect gemma4: +mul_mat_Q8_0 +mul_mat_Q4_0 * bisect gemma4: +flash_attn_ext_f32 +flash_attn_ext_f16_me * bisect gemma4: +mul_mat_id_f32 * bisect gemma4: +sum_rows_f32 * bisect gemma4: +cont_f16 * bisect gemma4: +fill_f32 * bisect gemma4: +unary_f32 (all ops re-enabled except rms_norm_mul) * Update rms_norm_mul_f32.c * bisect2 gemma4 n64: +scale_f32 only * bisect2 gemma4 n64: +rms_norm_f32 +rope_f32 * bisect2 gemma4 n64: +rms_norm_mul_f32 (with ET_UBERKERNEL eviction fix) * bisect2 gemma4 n64: +el_map +get_rows +glu +softmax (skip rms_norm_mul) * bisect2 gemma4 n64: all ops enabled except rms_norm_mul * bisect2 n64: test unary+cont+fill+sum_rows (no mul_mat/flash_attn) * bisect2 n64: +mul_mat_f32 +mul_mat_f32_matrix_engine * bisect2 n64: +mul_mat_f16 +mul_mat_f16_matrix_engine * bisect2 n64: +mul_mat_Q8_0 +mul_mat_Q4_0 * bisect2 n64: +mul_mat_Q8_0 only (disable Q4_0) * bisect2 n64: +mul_mat_Q4_0 only (Q8_0 breaks) * bisect2 n64: +mul_mat_id +flash_attn_ext (skip Q8_0) * run-3: matmul + rms_norm_mul * run-4 * Revert "run-4" * run5 * changes after cleanup * cleanup before upstream * restrict changes into ET backend * move kernel embedding from Python to CMake * move uberkernel gen into CMake * apply clang format * update CMake style * update to match C and C++ style * use source ggml and quant headers instead of ET's * MROPE support * absorb view ops into same branch as none * fix bad rebase * add marty1885 to codeowners * oops * remove redundant newline * fix CI editor warnings --------- Co-authored-by: Vidas <vidas@nuolat.lt> Co-authored-by: Gianluca Guida <glguida@tlbflush.org> Co-authored-by: Gianluca Guida <gianluca@nekko.ai> Co-authored-by: ubergarm <leimgrub@gmail.com> Co-authored-by: SaqibAkram-10xE <saqib.akram@10xengineers.ai> Co-authored-by: Rehan Qasim <rehan.qasim@10xengineers.ai>
519 lines
20 KiB
CMake
519 lines
20 KiB
CMake
include(CheckCXXCompilerFlag)
|
|
include("../cmake/common.cmake")
|
|
|
|
add_compile_definitions(GGML_SCHED_MAX_COPIES=${GGML_SCHED_MAX_COPIES})
|
|
|
|
# enable libstdc++ assertions for debug builds
|
|
if (CMAKE_SYSTEM_NAME MATCHES "Linux")
|
|
add_compile_definitions($<$<CONFIG:Debug>:_GLIBCXX_ASSERTIONS>)
|
|
endif()
|
|
|
|
if (NOT MSVC)
|
|
if (GGML_SANITIZE_THREAD)
|
|
add_compile_options(-fsanitize=thread)
|
|
link_libraries (-fsanitize=thread)
|
|
endif()
|
|
|
|
if (GGML_SANITIZE_ADDRESS)
|
|
add_compile_options(-fsanitize=address -fno-omit-frame-pointer)
|
|
link_libraries (-fsanitize=address)
|
|
endif()
|
|
|
|
if (GGML_SANITIZE_UNDEFINED)
|
|
add_compile_options(-fsanitize=undefined)
|
|
link_libraries (-fsanitize=undefined)
|
|
endif()
|
|
endif()
|
|
|
|
if (GGML_FATAL_WARNINGS)
|
|
if (CMAKE_CXX_COMPILER_ID MATCHES "GNU" OR CMAKE_CXX_COMPILER_ID MATCHES "Clang")
|
|
list(APPEND C_FLAGS -Werror)
|
|
list(APPEND CXX_FLAGS -Werror)
|
|
elseif (CMAKE_CXX_COMPILER_ID STREQUAL "MSVC")
|
|
add_compile_options(/WX)
|
|
endif()
|
|
endif()
|
|
|
|
if (GGML_ALL_WARNINGS)
|
|
if (NOT MSVC)
|
|
list(APPEND WARNING_FLAGS -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function)
|
|
list(APPEND C_FLAGS -Wshadow -Wstrict-prototypes -Wpointer-arith -Wmissing-prototypes
|
|
-Werror=implicit-int -Werror=implicit-function-declaration)
|
|
list(APPEND CXX_FLAGS -Wmissing-declarations -Wmissing-noreturn)
|
|
|
|
list(APPEND C_FLAGS ${WARNING_FLAGS})
|
|
list(APPEND CXX_FLAGS ${WARNING_FLAGS})
|
|
|
|
ggml_get_flags(${CMAKE_CXX_COMPILER_ID} ${CMAKE_CXX_COMPILER_VERSION})
|
|
|
|
add_compile_options("$<$<COMPILE_LANGUAGE:C>:${C_FLAGS};${GF_C_FLAGS}>"
|
|
"$<$<COMPILE_LANGUAGE:CXX>:${CXX_FLAGS};${GF_CXX_FLAGS}>")
|
|
else()
|
|
# todo : msvc
|
|
set(C_FLAGS "")
|
|
set(CXX_FLAGS "")
|
|
endif()
|
|
endif()
|
|
|
|
if (GGML_LTO)
|
|
include(CheckIPOSupported)
|
|
check_ipo_supported(RESULT result OUTPUT output)
|
|
if (result)
|
|
set(CMAKE_INTERPROCEDURAL_OPTIMIZATION TRUE)
|
|
else()
|
|
message(WARNING "IPO is not supported: ${output}")
|
|
endif()
|
|
endif()
|
|
|
|
if (GGML_CCACHE AND NOT CMAKE_C_COMPILER_LAUNCHER AND NOT CMAKE_CXX_COMPILER_LAUNCHER)
|
|
find_program(GGML_CCACHE_FOUND ccache)
|
|
find_program(GGML_SCCACHE_FOUND sccache)
|
|
|
|
if (GGML_CCACHE_FOUND OR GGML_SCCACHE_FOUND)
|
|
if(GGML_CCACHE_FOUND)
|
|
set(GGML_CCACHE_VARIANT ccache)
|
|
else()
|
|
set(GGML_CCACHE_VARIANT sccache)
|
|
endif()
|
|
# TODO: should not be set globally
|
|
if (GGML_SYCL AND GGML_CCACHE_FOUND AND WIN32)
|
|
set_property(GLOBAL PROPERTY RULE_LAUNCH_COMPILE "ccache compiler_type=icl")
|
|
else ()
|
|
set_property(GLOBAL PROPERTY RULE_LAUNCH_COMPILE "${GGML_CCACHE_VARIANT}")
|
|
endif ()
|
|
set(ENV{CCACHE_SLOPPINESS} time_macros)
|
|
message(STATUS "${GGML_CCACHE_VARIANT} found, compilation results will be cached. Disable with GGML_CCACHE=OFF.")
|
|
else()
|
|
message(STATUS "Warning: ccache not found - consider installing it for faster compilation or disable this warning with GGML_CCACHE=OFF")
|
|
endif ()
|
|
endif()
|
|
|
|
# this version of Apple ld64 is buggy
|
|
execute_process(
|
|
COMMAND ${CMAKE_C_COMPILER} ${CMAKE_EXE_LINKER_FLAGS} -Wl,-v
|
|
ERROR_VARIABLE output
|
|
OUTPUT_QUIET
|
|
)
|
|
|
|
if (output MATCHES "dyld-1015\.7")
|
|
add_compile_definitions(HAVE_BUGGY_APPLE_LINKER)
|
|
endif()
|
|
|
|
# architecture specific
|
|
# TODO: probably these flags need to be tweaked on some architectures
|
|
# feel free to update the Makefile for your architecture and send a pull request or issue
|
|
message(STATUS "CMAKE_SYSTEM_PROCESSOR: ${CMAKE_SYSTEM_PROCESSOR}")
|
|
if (MSVC)
|
|
string(TOLOWER "${CMAKE_GENERATOR_PLATFORM}" CMAKE_GENERATOR_PLATFORM_LWR)
|
|
message(STATUS "CMAKE_GENERATOR_PLATFORM: ${CMAKE_GENERATOR_PLATFORM}")
|
|
else ()
|
|
set(CMAKE_GENERATOR_PLATFORM_LWR "")
|
|
endif ()
|
|
ggml_get_system_arch()
|
|
message(STATUS "GGML_SYSTEM_ARCH: ${GGML_SYSTEM_ARCH}")
|
|
|
|
if (NOT MSVC)
|
|
if (GGML_STATIC)
|
|
if (UNIX AND NOT APPLE)
|
|
set(CMAKE_FIND_LIBRARY_SUFFIXES ".a;.so")
|
|
endif()
|
|
add_link_options(-static)
|
|
if (MINGW)
|
|
add_link_options(-static-libgcc -static-libstdc++)
|
|
endif()
|
|
endif()
|
|
if (GGML_GPROF)
|
|
add_compile_options(-pg)
|
|
endif()
|
|
endif()
|
|
|
|
#
|
|
# POSIX conformance
|
|
#
|
|
|
|
# clock_gettime came in POSIX.1b (1993)
|
|
# CLOCK_MONOTONIC came in POSIX.1-2001 / SUSv3 as optional
|
|
# posix_memalign came in POSIX.1-2001 / SUSv3
|
|
# M_PI is an XSI extension since POSIX.1-2001 / SUSv3, came in XPG1 (1985)
|
|
|
|
# Somehow in OpenBSD whenever POSIX conformance is specified
|
|
# some string functions rely on locale_t availability,
|
|
# which was introduced in POSIX.1-2008, forcing us to go higher
|
|
if (CMAKE_SYSTEM_NAME MATCHES "OpenBSD")
|
|
add_compile_definitions(_XOPEN_SOURCE=700)
|
|
elseif (CMAKE_SYSTEM_NAME MATCHES "AIX")
|
|
# Don't define _XOPEN_SOURCE. We need _ALL_SOURCE, which is the default,
|
|
# in order to define _SC_PHYS_PAGES.
|
|
else()
|
|
add_compile_definitions(_XOPEN_SOURCE=600)
|
|
endif()
|
|
|
|
# Data types, macros and functions related to controlling CPU affinity and
|
|
# some memory allocation are available on Linux through GNU extensions in libc
|
|
if (CMAKE_SYSTEM_NAME MATCHES "Linux" OR CMAKE_SYSTEM_NAME MATCHES "Android")
|
|
add_compile_definitions(_GNU_SOURCE)
|
|
endif()
|
|
|
|
# RLIMIT_MEMLOCK came in BSD, is not specified in POSIX.1,
|
|
# and on macOS its availability depends on enabling Darwin extensions
|
|
# similarly on DragonFly, enabling BSD extensions is necessary
|
|
if (
|
|
CMAKE_SYSTEM_NAME MATCHES "Darwin" OR
|
|
CMAKE_SYSTEM_NAME MATCHES "iOS" OR
|
|
CMAKE_SYSTEM_NAME MATCHES "tvOS" OR
|
|
CMAKE_SYSTEM_NAME MATCHES "DragonFly"
|
|
)
|
|
add_compile_definitions(_DARWIN_C_SOURCE)
|
|
endif()
|
|
|
|
# alloca is a non-standard interface that is not visible on BSDs when
|
|
# POSIX conformance is specified, but not all of them provide a clean way
|
|
# to enable it in such cases
|
|
if (CMAKE_SYSTEM_NAME MATCHES "FreeBSD")
|
|
add_compile_definitions(__BSD_VISIBLE)
|
|
endif()
|
|
if (CMAKE_SYSTEM_NAME MATCHES "NetBSD")
|
|
add_compile_definitions(_NETBSD_SOURCE)
|
|
endif()
|
|
if (CMAKE_SYSTEM_NAME MATCHES "OpenBSD")
|
|
add_compile_definitions(_BSD_SOURCE)
|
|
endif()
|
|
|
|
if (WIN32)
|
|
add_compile_definitions(_CRT_SECURE_NO_WARNINGS)
|
|
endif()
|
|
|
|
# ggml
|
|
|
|
if (GGML_BACKEND_DL AND NOT BUILD_SHARED_LIBS)
|
|
message(FATAL_ERROR "GGML_BACKEND_DL requires BUILD_SHARED_LIBS")
|
|
endif()
|
|
|
|
add_library(ggml-base
|
|
../include/ggml.h
|
|
../include/ggml-alloc.h
|
|
../include/ggml-backend.h
|
|
../include/ggml-cpp.h
|
|
../include/ggml-opt.h
|
|
../include/gguf.h
|
|
ggml.c
|
|
ggml.cpp
|
|
ggml-alloc.c
|
|
ggml-backend.cpp
|
|
ggml-backend-meta.cpp
|
|
ggml-opt.cpp
|
|
ggml-threading.cpp
|
|
ggml-threading.h
|
|
ggml-quants.c
|
|
ggml-quants.h
|
|
gguf.cpp)
|
|
|
|
set_target_properties(ggml-base PROPERTIES
|
|
VERSION ${GGML_VERSION}
|
|
SOVERSION ${GGML_VERSION_MAJOR}
|
|
)
|
|
|
|
target_include_directories(ggml-base PRIVATE .)
|
|
if (GGML_BACKEND_DL)
|
|
target_compile_definitions(ggml-base PUBLIC GGML_BACKEND_DL)
|
|
endif()
|
|
|
|
if (GGML_SCHED_NO_REALLOC)
|
|
target_compile_definitions(ggml-base PUBLIC GGML_SCHED_NO_REALLOC)
|
|
endif()
|
|
|
|
if (GGML_OPENMP)
|
|
find_package(OpenMP)
|
|
if (OpenMP_FOUND)
|
|
set(GGML_OPENMP_ENABLED "ON" CACHE INTERNAL "")
|
|
else()
|
|
set(GGML_OPENMP_ENABLED "OFF" CACHE INTERNAL "")
|
|
message(WARNING "OpenMP not found")
|
|
endif()
|
|
else()
|
|
set(GGML_OPENMP_ENABLED "OFF" CACHE INTERNAL "")
|
|
endif()
|
|
|
|
if (GGML_OPENMP_ENABLED)
|
|
target_compile_definitions(ggml-base PRIVATE GGML_USE_OPENMP)
|
|
target_link_libraries(ggml-base PRIVATE OpenMP::OpenMP_C OpenMP::OpenMP_CXX)
|
|
endif()
|
|
|
|
add_library(ggml
|
|
ggml-backend-dl.cpp
|
|
ggml-backend-reg.cpp)
|
|
add_library(ggml::ggml ALIAS ggml)
|
|
|
|
set_target_properties(ggml PROPERTIES
|
|
VERSION ${GGML_VERSION}
|
|
SOVERSION ${GGML_VERSION_MAJOR}
|
|
)
|
|
|
|
if (GGML_BACKEND_DIR)
|
|
if (NOT GGML_BACKEND_DL)
|
|
message(FATAL_ERROR "GGML_BACKEND_DIR requires GGML_BACKEND_DL")
|
|
endif()
|
|
target_compile_definitions(ggml PUBLIC GGML_BACKEND_DIR="${GGML_BACKEND_DIR}")
|
|
endif()
|
|
|
|
target_link_libraries(ggml PUBLIC ggml-base)
|
|
|
|
if (CMAKE_SYSTEM_NAME MATCHES "Linux")
|
|
target_link_libraries(ggml PRIVATE dl)
|
|
endif()
|
|
|
|
function(ggml_add_backend_library backend)
|
|
if (GGML_BACKEND_DL)
|
|
add_library(${backend} MODULE ${ARGN})
|
|
# write the shared library to the output directory
|
|
set_target_properties(${backend} PROPERTIES LIBRARY_OUTPUT_DIRECTORY ${CMAKE_RUNTIME_OUTPUT_DIRECTORY})
|
|
target_compile_definitions(${backend} PRIVATE GGML_BACKEND_DL)
|
|
add_dependencies(ggml ${backend})
|
|
if (GGML_BACKEND_DIR)
|
|
install(TARGETS ${backend} LIBRARY DESTINATION ${GGML_BACKEND_DIR})
|
|
else()
|
|
install(TARGETS ${backend} LIBRARY DESTINATION ${CMAKE_INSTALL_BINDIR})
|
|
endif()
|
|
else()
|
|
add_library(${backend} ${ARGN})
|
|
target_link_libraries(ggml PUBLIC ${backend})
|
|
install(TARGETS ${backend} LIBRARY)
|
|
endif()
|
|
|
|
target_link_libraries(${backend} PRIVATE ggml-base)
|
|
target_include_directories(${backend} PRIVATE ..)
|
|
|
|
if (${BUILD_SHARED_LIBS})
|
|
target_compile_definitions(${backend} PRIVATE GGML_BACKEND_BUILD)
|
|
target_compile_definitions(${backend} PUBLIC GGML_BACKEND_SHARED)
|
|
endif()
|
|
|
|
# Set versioning properties for all backend libraries
|
|
# Building a MODULE library with a version is not supported on macOS (https://gitlab.kitware.com/cmake/cmake/-/issues/20782)
|
|
if (NOT (APPLE AND GGML_BACKEND_DL))
|
|
set_target_properties(${backend} PROPERTIES
|
|
VERSION ${GGML_VERSION}
|
|
SOVERSION ${GGML_VERSION_MAJOR}
|
|
)
|
|
endif()
|
|
|
|
if(NOT GGML_AVAILABLE_BACKENDS)
|
|
set(GGML_AVAILABLE_BACKENDS "${backend}"
|
|
CACHE INTERNAL "List of backends for cmake package")
|
|
else()
|
|
list(FIND GGML_AVAILABLE_BACKENDS "${backend}" has_backend)
|
|
if(has_backend EQUAL -1)
|
|
set(GGML_AVAILABLE_BACKENDS "${GGML_AVAILABLE_BACKENDS};${backend}"
|
|
CACHE INTERNAL "List of backends for cmake package")
|
|
endif()
|
|
endif()
|
|
endfunction()
|
|
|
|
function(ggml_add_backend backend)
|
|
string(TOUPPER "GGML_${backend}" backend_id)
|
|
if (${backend_id})
|
|
string(TOLOWER "ggml-${backend}" backend_target)
|
|
add_subdirectory(${backend_target})
|
|
message(STATUS "Including ${backend} backend")
|
|
if (NOT GGML_BACKEND_DL)
|
|
string(TOUPPER "GGML_USE_${backend}" backend_use)
|
|
target_compile_definitions(ggml PUBLIC ${backend_use})
|
|
endif()
|
|
endif()
|
|
endfunction()
|
|
|
|
function(ggml_add_cpu_backend_variant tag_name)
|
|
set(GGML_CPU_TAG_NAME ${tag_name})
|
|
# other: OPENMP LLAMAFILE CPU_HBM
|
|
if (GGML_SYSTEM_ARCH STREQUAL "x86")
|
|
foreach (feat NATIVE
|
|
SSE42
|
|
AVX AVX2 BMI2 AVX_VNNI FMA F16C
|
|
AVX512 AVX512_VBMI AVX512_VNNI AVX512_BF16
|
|
AMX_TILE AMX_INT8 AMX_BF16)
|
|
set(GGML_${feat} OFF)
|
|
endforeach()
|
|
|
|
foreach (feat ${ARGN})
|
|
set(GGML_${feat} ON)
|
|
endforeach()
|
|
elseif (GGML_SYSTEM_ARCH STREQUAL "ARM")
|
|
foreach (feat ${ARGN})
|
|
set(GGML_INTERNAL_${feat} ON)
|
|
endforeach()
|
|
elseif (GGML_SYSTEM_ARCH STREQUAL "PowerPC")
|
|
foreach (feat ${ARGN})
|
|
set(GGML_INTERNAL_${feat} ON)
|
|
endforeach()
|
|
elseif (GGML_SYSTEM_ARCH STREQUAL "s390x")
|
|
foreach (feat VXE2 NNPA)
|
|
set(GGML_INTERNAL_${feat} OFF)
|
|
endforeach()
|
|
|
|
foreach (feat ${ARGN})
|
|
set(GGML_INTERNAL_${feat} ON)
|
|
endforeach()
|
|
elseif (GGML_SYSTEM_ARCH STREQUAL "riscv64")
|
|
foreach (feat RVV)
|
|
set(GGML_INTERNAL_${feat} OFF)
|
|
endforeach()
|
|
|
|
foreach (feat ${ARGN})
|
|
set(GGML_INTERNAL_${feat} ON)
|
|
endforeach()
|
|
endif()
|
|
|
|
ggml_add_cpu_backend_variant_impl(${tag_name})
|
|
endfunction()
|
|
|
|
ggml_add_backend(CPU)
|
|
|
|
if (GGML_CPU_ALL_VARIANTS)
|
|
if (NOT GGML_BACKEND_DL)
|
|
message(FATAL_ERROR "GGML_CPU_ALL_VARIANTS requires GGML_BACKEND_DL")
|
|
elseif (GGML_CPU_ARM_ARCH)
|
|
message(FATAL_ERROR "Cannot use both GGML_CPU_ARM_ARCH and GGML_CPU_ALL_VARIANTS")
|
|
endif()
|
|
if (GGML_SYSTEM_ARCH STREQUAL "x86")
|
|
ggml_add_cpu_backend_variant(x64)
|
|
ggml_add_cpu_backend_variant(sse42 SSE42)
|
|
ggml_add_cpu_backend_variant(sandybridge SSE42 AVX)
|
|
if (NOT MSVC)
|
|
# __FMA__ and __F16C__ are not defined in MSVC, however they are implied with AVX2/AVX512
|
|
ggml_add_cpu_backend_variant(ivybridge SSE42 AVX F16C)
|
|
ggml_add_cpu_backend_variant(piledriver SSE42 AVX F16C FMA)
|
|
endif()
|
|
ggml_add_cpu_backend_variant(haswell SSE42 AVX F16C FMA AVX2 BMI2)
|
|
ggml_add_cpu_backend_variant(skylakex SSE42 AVX F16C FMA AVX2 BMI2 AVX512)
|
|
ggml_add_cpu_backend_variant(cannonlake SSE42 AVX F16C FMA AVX2 BMI2 AVX512 AVX512_VBMI)
|
|
ggml_add_cpu_backend_variant(cascadelake SSE42 AVX F16C FMA AVX2 BMI2 AVX512 AVX512_VNNI)
|
|
ggml_add_cpu_backend_variant(icelake SSE42 AVX F16C FMA AVX2 BMI2 AVX512 AVX512_VBMI AVX512_VNNI)
|
|
if (NOT MSVC)
|
|
# MSVC 2022 doesn't support BF16 intrinsics without `/arch:AVX10.1` ?!
|
|
# https://learn.microsoft.com/en-us/cpp/intrinsics/x64-amd64-intrinsics-list?view=msvc-170
|
|
# https://learn.microsoft.com/en-us/cpp/build/reference/arch-x64?view=msvc-170
|
|
ggml_add_cpu_backend_variant(cooperlake SSE42 AVX F16C FMA AVX2 BMI2 AVX512 AVX512_VNNI AVX512_BF16)
|
|
ggml_add_cpu_backend_variant(zen4 SSE42 AVX F16C FMA AVX2 BMI2 AVX512 AVX512_VBMI AVX512_VNNI AVX512_BF16)
|
|
endif()
|
|
ggml_add_cpu_backend_variant(alderlake SSE42 AVX F16C FMA AVX2 BMI2 AVX_VNNI)
|
|
if (NOT MSVC)
|
|
# MSVC doesn't support AMX
|
|
ggml_add_cpu_backend_variant(sapphirerapids SSE42 AVX F16C FMA AVX2 BMI2 AVX512 AVX512_VBMI AVX512_VNNI AVX512_BF16 AMX_TILE AMX_INT8)
|
|
endif()
|
|
elseif(GGML_SYSTEM_ARCH STREQUAL "ARM")
|
|
if (CMAKE_SYSTEM_NAME MATCHES "Linux")
|
|
# Many of these features are optional so we build versions with popular
|
|
# combinations and name the backends based on the version they were
|
|
# first released with
|
|
ggml_add_cpu_backend_variant(armv8.0_1)
|
|
ggml_add_cpu_backend_variant(armv8.2_1 DOTPROD)
|
|
ggml_add_cpu_backend_variant(armv8.2_2 DOTPROD FP16_VECTOR_ARITHMETIC)
|
|
ggml_add_cpu_backend_variant(armv8.2_3 DOTPROD FP16_VECTOR_ARITHMETIC SVE)
|
|
ggml_add_cpu_backend_variant(armv8.6_1 DOTPROD FP16_VECTOR_ARITHMETIC SVE MATMUL_INT8)
|
|
ggml_add_cpu_backend_variant(armv8.6_2 DOTPROD FP16_VECTOR_ARITHMETIC SVE MATMUL_INT8 SVE2)
|
|
ggml_add_cpu_backend_variant(armv9.2_1 DOTPROD FP16_VECTOR_ARITHMETIC SVE MATMUL_INT8 SME)
|
|
ggml_add_cpu_backend_variant(armv9.2_2 DOTPROD FP16_VECTOR_ARITHMETIC SVE MATMUL_INT8 SVE2 SME)
|
|
elseif (CMAKE_SYSTEM_NAME MATCHES "Android")
|
|
# Android-specific backends with SoC-compatible feature sets
|
|
ggml_add_cpu_backend_variant(android_armv8.0_1)
|
|
ggml_add_cpu_backend_variant(android_armv8.2_1 DOTPROD)
|
|
ggml_add_cpu_backend_variant(android_armv8.2_2 DOTPROD FP16_VECTOR_ARITHMETIC)
|
|
ggml_add_cpu_backend_variant(android_armv8.6_1 DOTPROD FP16_VECTOR_ARITHMETIC MATMUL_INT8)
|
|
ggml_add_cpu_backend_variant(android_armv9.0_1 DOTPROD MATMUL_INT8 FP16_VECTOR_ARITHMETIC SVE2)
|
|
ggml_add_cpu_backend_variant(android_armv9.2_1 DOTPROD MATMUL_INT8 FP16_VECTOR_ARITHMETIC SVE SME)
|
|
ggml_add_cpu_backend_variant(android_armv9.2_2 DOTPROD MATMUL_INT8 FP16_VECTOR_ARITHMETIC SVE SVE2 SME)
|
|
elseif (APPLE)
|
|
ggml_add_cpu_backend_variant(apple_m1 DOTPROD)
|
|
ggml_add_cpu_backend_variant(apple_m2_m3 DOTPROD MATMUL_INT8)
|
|
ggml_add_cpu_backend_variant(apple_m4 DOTPROD MATMUL_INT8 NOSVE SME)
|
|
else()
|
|
message(FATAL_ERROR "Unsupported ARM target OS: ${CMAKE_SYSTEM_NAME}")
|
|
endif()
|
|
elseif (GGML_SYSTEM_ARCH STREQUAL "PowerPC")
|
|
if (CMAKE_SYSTEM_NAME MATCHES "Linux")
|
|
ggml_add_cpu_backend_variant(power0)
|
|
ggml_add_cpu_backend_variant(power7_1 POWER7)
|
|
ggml_add_cpu_backend_variant(power7_2 POWER7 VSX)
|
|
ggml_add_cpu_backend_variant(power8_1 POWER8)
|
|
ggml_add_cpu_backend_variant(power8_2 POWER8 VSX)
|
|
ggml_add_cpu_backend_variant(power9 POWER9 VSX)
|
|
ggml_add_cpu_backend_variant(power10 POWER10 VSX)
|
|
# POWER11 backend: only if compiler supports -mcpu=power11
|
|
check_cxx_compiler_flag("-mcpu=power11" GGML_CXX_SUPPORTS_POWER11)
|
|
if (GGML_CXX_SUPPORTS_POWER11)
|
|
message(STATUS "Compiler supports -mcpu=power11, enabling POWER11 backend")
|
|
ggml_add_cpu_backend_variant(power11 POWER11 VSX)
|
|
else()
|
|
message(STATUS "Skipping POWER11 backend: compiler does not support -mcpu=power11")
|
|
endif()
|
|
else()
|
|
message(FATAL_ERROR "Unsupported PowerPC target OS: ${CMAKE_SYSTEM_NAME}")
|
|
endif()
|
|
elseif (GGML_SYSTEM_ARCH STREQUAL "s390x")
|
|
if (CMAKE_SYSTEM_NAME MATCHES "Linux")
|
|
ggml_add_cpu_backend_variant(z15 Z15 VXE2)
|
|
ggml_add_cpu_backend_variant(z16 Z16 VXE2 NNPA)
|
|
else()
|
|
message(FATAL_ERROR "Unsupported s390x target OS: ${CMAKE_SYSTEM_NAME}")
|
|
endif()
|
|
elseif (GGML_SYSTEM_ARCH STREQUAL "riscv64")
|
|
if (CMAKE_SYSTEM_NAME MATCHES "Linux")
|
|
ggml_add_cpu_backend_variant(riscv64_0)
|
|
ggml_add_cpu_backend_variant(riscv64_v RVV)
|
|
else()
|
|
message(FATAL_ERROR "Unsupported RISC-V target OS: ${CMAKE_SYSTEM_NAME}")
|
|
endif()
|
|
else()
|
|
message(FATAL_ERROR "GGML_CPU_ALL_VARIANTS not yet supported with ${GGML_SYSTEM_ARCH} on ${CMAKE_SYSTEM_NAME}")
|
|
endif()
|
|
elseif (GGML_CPU)
|
|
ggml_add_cpu_backend_variant_impl("")
|
|
endif()
|
|
|
|
ggml_add_backend(BLAS)
|
|
ggml_add_backend(CANN)
|
|
ggml_add_backend(CUDA)
|
|
ggml_add_backend(ET)
|
|
ggml_add_backend(HIP)
|
|
ggml_add_backend(METAL)
|
|
ggml_add_backend(MUSA)
|
|
ggml_add_backend(RPC)
|
|
ggml_add_backend(VirtGPU)
|
|
ggml_add_backend(SYCL)
|
|
ggml_add_backend(Vulkan)
|
|
ggml_add_backend(WebGPU)
|
|
ggml_add_backend(zDNN)
|
|
ggml_add_backend(OpenCL)
|
|
ggml_add_backend(Hexagon)
|
|
ggml_add_backend(ZenDNN)
|
|
ggml_add_backend(OPENVINO)
|
|
|
|
foreach (target ggml-base ggml)
|
|
target_include_directories(${target} PUBLIC $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/../include> $<INSTALL_INTERFACE:include>)
|
|
target_compile_features (${target} PRIVATE c_std_11 cxx_std_17) # don't bump
|
|
endforeach()
|
|
|
|
target_link_libraries(ggml-base PRIVATE Threads::Threads)
|
|
|
|
if (DEFINED MATH_LIBRARY)
|
|
target_link_libraries(ggml-base PRIVATE ${MATH_LIBRARY})
|
|
elseif (NOT WIN32 AND NOT DEFINED ENV{ONEAPI_ROOT})
|
|
target_link_libraries(ggml-base PRIVATE m)
|
|
endif()
|
|
|
|
if (CMAKE_SYSTEM_NAME MATCHES "Android")
|
|
target_link_libraries(ggml-base PRIVATE dl)
|
|
endif()
|
|
|
|
if(CMAKE_SYSTEM_NAME MATCHES "visionOS")
|
|
target_compile_definitions(ggml-base PUBLIC _DARWIN_C_SOURCE)
|
|
endif()
|
|
|
|
if (BUILD_SHARED_LIBS)
|
|
foreach (target ggml-base ggml)
|
|
set_target_properties(${target} PROPERTIES POSITION_INDEPENDENT_CODE ON)
|
|
target_compile_definitions(${target} PRIVATE GGML_BUILD)
|
|
target_compile_definitions(${target} PUBLIC GGML_SHARED)
|
|
endforeach()
|
|
endif()
|