Author SHA1 Message Date
Lumpiasty 8a545c7820 ggml-vulkan: implement host buffer pinning for faster H2D uploads
Assisted-by: opencode
2026-07-12 15:51:29 +02:00
Anirban Kar 5e7f6271c0 docs(readme): add usage + benchmark instructions for the MoE-offload optimizations 2026-07-08 23:56:46 +05:30
thecodacus 5f83fbbe7c ggml : size prefetch slots per layer and fix fallback use-after-free
Profiling showed the 2-slot rotation stalls ~5.5ms per layer waiting for
the down_exps upload: 3 tensors per MoE layer need 3 slots for a full
layer of lookahead. Default is now 3 (one layer), configurable via the
env var value, degrading to however many slots fit on OOM.

Also fixes a use-after-free: on a failed slot regrow the repointed
staging tensors kept dangling pointers into freed device memory, which
graph reuse carried into later evals. Staging repoints are now restored
right after kernel launch, slots are sized once from the graph max, and
allocation happens before freeing.

Qwen3.6-35B-A3B pp2048 on RTX 3060: 1643 -> 1880 t/s (mainline: 1143).
2026-07-02 12:42:55 +00:00
thecodacus 1163cb3493 ggml : overlap offloaded expert weight uploads with compute
At large batch sizes virtually every expert is used, so the per-layer
routing-ids readback that mainline waits on buys nothing while forcing a
full device sync per expert tensor (3x per MoE layer). Above a batch
threshold, upload the full expert tensors through a second backend
instance on the same device (own stream) with two event-ordered staging
slots, so uploads for tensor N+1 overlap compute of tensor N.

Qwen3.6-35B-A3B pp2048 on RTX 3060 (-ncmoe 26, ub 2048): 1383 -> 1663 t/s.
Generation output verified token-identical; decode path unaffected.
Opt-in via GGML_SCHED_PREFETCH_EXPERTS=1.
2026-07-02 12:16:39 +00:00
thecodacus 20f5994bfe llama : pin mmap-backed CPU weights for faster H2D uploads
Wire the existing GGML_CUDA_REGISTER_HOST path back up: after model load,
cudaHostRegister the mmap pages backing weights kept in system memory.
Recovers pageable-copy losses when MoE experts are streamed to the GPU
during prefill (n-cpu-moe): Qwen3.6-35B-A3B pp2048 1144 -> 1385 t/s on
RTX 3060. Opt-in via GGML_CUDA_REGISTER_HOST=1, unchanged otherwise.
2026-07-02 09:44:53 +00:00
4055 changed files with 223616 additions and 819257 deletions

No files matched your search

+6 -11
View File
@@ -19,16 +19,12 @@ FROM docker.io/node:$NODE_VERSION AS web
ARG APP_VERSION ARG APP_VERSION
WORKDIR /app/ui/apps/chat WORKDIR /app/tools/ui
COPY ui/apps/chat/package.json ui/apps/chat/package-lock.json ./ COPY tools/ui/package.json tools/ui/package-lock.json ./
RUN npm ci RUN npm ci
COPY ui/apps/chat/ ./ COPY tools/ui/ ./
# the app compiles Llama UI and Llama Voice from their sources next to it
COPY ui/lib/ui/package.json /app/ui/lib/ui/package.json
COPY ui/lib/ui/src/lib/ /app/ui/lib/ui/src/lib/
COPY ui/lib/voice/src/ /app/ui/lib/voice/src/
RUN LLAMA_BUILD_NUMBER="$APP_VERSION" npm run build RUN LLAMA_BUILD_NUMBER="$APP_VERSION" npm run build
FROM ${CANN_BASE_IMAGE} AS build FROM ${CANN_BASE_IMAGE} AS build
@@ -44,7 +40,7 @@ WORKDIR /app
# -- Copy project files -- # -- Copy project files --
COPY . . COPY . .
COPY --from=web /app/ui/apps/chat/dist ui/apps/chat/dist COPY --from=web /app/tools/ui/dist tools/ui/dist
# -- Set CANN environment variables (required for compilation) -- # -- Set CANN environment variables (required for compilation) --
# Using ENV instead of `source` allows environment variables to persist across the entire image layer # Using ENV instead of `source` allows environment variables to persist across the entire image layer
@@ -149,7 +145,7 @@ ENTRYPOINT ["/app/tools.sh"]
# ============================================================================== # ==============================================================================
FROM base AS light FROM base AS light
COPY --from=build /app/full/llama /app/full/llama-cli /app/full/llama-completion /app/ COPY --from=build /app/full/llama /app/full/llama-cli /app/full/llama-completion /app
ENTRYPOINT [ "/app/llama-cli" ] ENTRYPOINT [ "/app/llama-cli" ]
@@ -159,9 +155,8 @@ ENTRYPOINT [ "/app/llama-cli" ]
FROM base AS server FROM base AS server
ENV LLAMA_ARG_HOST=0.0.0.0 ENV LLAMA_ARG_HOST=0.0.0.0
ENV LLAMA_ARG_PORT=8080
COPY --from=build /app/full/llama /app/full/llama-server /app/ COPY --from=build /app/full/llama /app/full/llama-server /app
HEALTHCHECK --interval=5m CMD [ "curl", "-f", "http://localhost:8080/health" ] HEALTHCHECK --interval=5m CMD [ "curl", "-f", "http://localhost:8080/health" ]
+10 -13
View File
@@ -1,4 +1,4 @@
ARG UBUNTU_VERSION=26.04 ARG UBUNTU_VERSION=24.04
ARG BUILD_DATE=N/A ARG BUILD_DATE=N/A
ARG APP_VERSION=N/A ARG APP_VERSION=N/A
ARG APP_REVISION=N/A ARG APP_REVISION=N/A
@@ -9,16 +9,12 @@ FROM docker.io/node:$NODE_VERSION AS web
ARG APP_VERSION ARG APP_VERSION
WORKDIR /app/ui/apps/chat WORKDIR /app/tools/ui
COPY ui/apps/chat/package.json ui/apps/chat/package-lock.json ./ COPY tools/ui/package.json tools/ui/package-lock.json ./
RUN npm ci RUN npm ci
COPY ui/apps/chat/ ./ COPY tools/ui/ ./
# the app compiles Llama UI and Llama Voice from their sources next to it
COPY ui/lib/ui/package.json /app/ui/lib/ui/package.json
COPY ui/lib/ui/src/lib/ /app/ui/lib/ui/src/lib/
COPY ui/lib/voice/src/ /app/ui/lib/voice/src/
RUN LLAMA_BUILD_NUMBER="$APP_VERSION" npm run build RUN LLAMA_BUILD_NUMBER="$APP_VERSION" npm run build
FROM docker.io/ubuntu:$UBUNTU_VERSION AS build FROM docker.io/ubuntu:$UBUNTU_VERSION AS build
@@ -26,13 +22,15 @@ FROM docker.io/ubuntu:$UBUNTU_VERSION AS build
ARG TARGETARCH ARG TARGETARCH
RUN apt-get update && \ RUN apt-get update && \
apt-get install -y build-essential git cmake libssl-dev apt-get install -y gcc-14 g++-14 build-essential git cmake libssl-dev
ENV CC=gcc-14 CXX=g++-14
WORKDIR /app WORKDIR /app
COPY . . COPY . .
COPY --from=web /app/ui/apps/chat/dist ui/apps/chat/dist COPY --from=web /app/tools/ui/dist tools/ui/dist
RUN if [ "$TARGETARCH" = "amd64" ] || [ "$TARGETARCH" = "arm64" ]; then \ RUN if [ "$TARGETARCH" = "amd64" ] || [ "$TARGETARCH" = "arm64" ]; then \
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=OFF -DLLAMA_BUILD_TESTS=OFF -DGGML_BACKEND_DL=ON -DGGML_CPU_ALL_VARIANTS=ON; \ cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=OFF -DLLAMA_BUILD_TESTS=OFF -DGGML_BACKEND_DL=ON -DGGML_CPU_ALL_VARIANTS=ON; \
@@ -106,7 +104,7 @@ ENTRYPOINT ["/app/tools.sh"]
### Light, CLI only ### Light, CLI only
FROM base AS light FROM base AS light
COPY --from=build /app/full/llama /app/full/llama-cli /app/full/llama-completion /app/ COPY --from=build /app/full/llama /app/full/llama-cli /app/full/llama-completion /app
WORKDIR /app WORKDIR /app
@@ -116,9 +114,8 @@ ENTRYPOINT [ "/app/llama-cli" ]
FROM base AS server FROM base AS server
ENV LLAMA_ARG_HOST=0.0.0.0 ENV LLAMA_ARG_HOST=0.0.0.0
ENV LLAMA_ARG_PORT=8080
COPY --from=build /app/full/llama /app/full/llama-server /app/ COPY --from=build /app/full/llama /app/full/llama-server /app
WORKDIR /app WORKDIR /app
+6 -14
View File
@@ -17,16 +17,12 @@ FROM docker.io/node:$NODE_VERSION AS web
ARG APP_VERSION ARG APP_VERSION
WORKDIR /app/ui/apps/chat WORKDIR /app/tools/ui
COPY ui/apps/chat/package.json ui/apps/chat/package-lock.json ./ COPY tools/ui/package.json tools/ui/package-lock.json ./
RUN npm ci RUN npm ci
COPY ui/apps/chat/ ./ COPY tools/ui/ ./
# the app compiles Llama UI and Llama Voice from their sources next to it
COPY ui/lib/ui/package.json /app/ui/lib/ui/package.json
COPY ui/lib/ui/src/lib/ /app/ui/lib/ui/src/lib/
COPY ui/lib/voice/src/ /app/ui/lib/voice/src/
RUN LLAMA_BUILD_NUMBER="$APP_VERSION" npm run build RUN LLAMA_BUILD_NUMBER="$APP_VERSION" npm run build
FROM ${BASE_CUDA_DEV_CONTAINER} AS build FROM ${BASE_CUDA_DEV_CONTAINER} AS build
@@ -44,7 +40,7 @@ WORKDIR /app
COPY . . COPY . .
COPY --from=web /app/ui/apps/chat/dist ui/apps/chat/dist COPY --from=web /app/tools/ui/dist tools/ui/dist
RUN if [ "${CUDA_DOCKER_ARCH}" != "default" ]; then \ RUN if [ "${CUDA_DOCKER_ARCH}" != "default" ]; then \
export CMAKE_ARGS="-DCMAKE_CUDA_ARCHITECTURES=${CUDA_DOCKER_ARCH}"; \ export CMAKE_ARGS="-DCMAKE_CUDA_ARCHITECTURES=${CUDA_DOCKER_ARCH}"; \
@@ -88,9 +84,6 @@ RUN apt-get update \
&& find /var/cache/apt/archives /var/lib/apt/lists -not -name lock -type f -delete \ && find /var/cache/apt/archives /var/lib/apt/lists -not -name lock -type f -delete \
&& find /var/cache -type f -delete && find /var/cache -type f -delete
# Some hosts (e.g. GKE) only mount the driver here, and CUDA 12.8.1-12.9.1 base images don't set these paths
ENV LD_LIBRARY_PATH=/usr/local/nvidia/lib64:/usr/local/nvidia/lib:${LD_LIBRARY_PATH}
COPY --from=build /app/lib/ /app COPY --from=build /app/lib/ /app
### Full ### Full
@@ -120,7 +113,7 @@ ENTRYPOINT ["/app/tools.sh"]
### Light, CLI only ### Light, CLI only
FROM base AS light FROM base AS light
COPY --from=build /app/full/llama /app/full/llama-cli /app/full/llama-completion /app/ COPY --from=build /app/full/llama /app/full/llama-cli /app/full/llama-completion /app
WORKDIR /app WORKDIR /app
@@ -130,9 +123,8 @@ ENTRYPOINT [ "/app/llama-cli" ]
FROM base AS server FROM base AS server
ENV LLAMA_ARG_HOST=0.0.0.0 ENV LLAMA_ARG_HOST=0.0.0.0
ENV LLAMA_ARG_PORT=8080
COPY --from=build /app/full/llama /app/full/llama-server /app/ COPY --from=build /app/full/llama /app/full/llama-server /app
WORKDIR /app WORKDIR /app
+19 -31
View File
@@ -1,4 +1,4 @@
ARG ONEAPI_VERSION=2026.2.0-devel-ubuntu24.04 ARG ONEAPI_VERSION=2025.3.3-0-devel-ubuntu24.04
ARG BUILD_DATE=N/A ARG BUILD_DATE=N/A
ARG APP_VERSION=N/A ARG APP_VERSION=N/A
ARG APP_REVISION=N/A ARG APP_REVISION=N/A
@@ -11,41 +11,32 @@ FROM docker.io/node:$NODE_VERSION AS web
ARG APP_VERSION ARG APP_VERSION
WORKDIR /app/ui/apps/chat WORKDIR /app/tools/ui
COPY ui/apps/chat/package.json ui/apps/chat/package-lock.json ./ COPY tools/ui/package.json tools/ui/package-lock.json ./
RUN npm ci RUN npm ci
COPY ui/apps/chat/ ./ COPY tools/ui/ ./
# the app compiles Llama UI and Llama Voice from their sources next to it
COPY ui/lib/ui/package.json /app/ui/lib/ui/package.json
COPY ui/lib/ui/src/lib/ /app/ui/lib/ui/src/lib/
COPY ui/lib/voice/src/ /app/ui/lib/voice/src/
RUN LLAMA_BUILD_NUMBER="$APP_VERSION" npm run build RUN LLAMA_BUILD_NUMBER="$APP_VERSION" npm run build
FROM docker.io/intel/deep-learning-essentials:$ONEAPI_VERSION AS build FROM docker.io/intel/deep-learning-essentials:$ONEAPI_VERSION AS build
RUN echo "Download oneDNN installation package" && \
wget https://registrationcenter-download.intel.com/akdlm/IRC_NAS/46452a4a-7b7f-49f2-be4c-602a900180bb/intel-onednn-2026.1.0.625_offline.sh -O intel-onednn_offline.sh && \
echo "Install oneDNN components" && \
bash intel-onednn_offline.sh -s -a --silent --eula accept
ARG GGML_SYCL_F16=ON ARG GGML_SYCL_F16=ON
ARG LEVEL_ZERO_VERSION=1.34.0 ARG LEVEL_ZERO_VERSION=1.28.2
ARG LEVEL_ZERO_UBUNTU_VERSION=u24.04 ARG LEVEL_ZERO_UBUNTU_VERSION=u24.04
RUN apt-get update && \ RUN apt-get update && \
apt-get install -y git libssl-dev wget ca-certificates && \ apt-get install -y git libssl-dev wget ca-certificates && \
cd /tmp && \ cd /tmp && \
wget -q "https://github.com/oneapi-src/level-zero/releases/download/v${LEVEL_ZERO_VERSION}/libze1_${LEVEL_ZERO_VERSION}%2B${LEVEL_ZERO_UBUNTU_VERSION}_amd64.deb" -O libze1.deb && \ wget -q "https://github.com/oneapi-src/level-zero/releases/download/v${LEVEL_ZERO_VERSION}/level-zero_${LEVEL_ZERO_VERSION}%2B${LEVEL_ZERO_UBUNTU_VERSION}_amd64.deb" -O level-zero.deb && \
wget -q "https://github.com/oneapi-src/level-zero/releases/download/v${LEVEL_ZERO_VERSION}/libze-dev_${LEVEL_ZERO_VERSION}%2B${LEVEL_ZERO_UBUNTU_VERSION}_amd64.deb" -O libze-dev.deb && \ wget -q "https://github.com/oneapi-src/level-zero/releases/download/v${LEVEL_ZERO_VERSION}/level-zero-devel_${LEVEL_ZERO_VERSION}%2B${LEVEL_ZERO_UBUNTU_VERSION}_amd64.deb" -O level-zero-devel.deb && \
apt-get -o Dpkg::Options::="--force-overwrite" install -y ./libze1.deb ./libze-dev.deb && \ apt-get -o Dpkg::Options::="--force-overwrite" install -y ./level-zero.deb ./level-zero-devel.deb && \
rm -f /tmp/libze1.deb /tmp/libze-dev.deb rm -f /tmp/level-zero.deb /tmp/level-zero-devel.deb
WORKDIR /app WORKDIR /app
COPY . . COPY . .
COPY --from=web /app/ui/apps/chat/dist ui/apps/chat/dist COPY --from=web /app/tools/ui/dist tools/ui/dist
RUN if [ "${GGML_SYCL_F16}" = "ON" ]; then \ RUN if [ "${GGML_SYCL_F16}" = "ON" ]; then \
echo "GGML_SYCL_F16 is set" \ echo "GGML_SYCL_F16 is set" \
@@ -54,7 +45,7 @@ RUN if [ "${GGML_SYCL_F16}" = "ON" ]; then \
fi && \ fi && \
echo "Building with dynamic libs" && \ echo "Building with dynamic libs" && \
cmake -B build -DGGML_NATIVE=OFF -DGGML_SYCL=ON -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx -DGGML_BACKEND_DL=ON -DGGML_CPU_ALL_VARIANTS=ON -DLLAMA_BUILD_TESTS=OFF ${OPT_SYCL_F16} && \ cmake -B build -DGGML_NATIVE=OFF -DGGML_SYCL=ON -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx -DGGML_BACKEND_DL=ON -DGGML_CPU_ALL_VARIANTS=ON -DLLAMA_BUILD_TESTS=OFF ${OPT_SYCL_F16} && \
cmake --build build --config Release -j$((($(nproc)+1)/2)) cmake --build build --config Release -j$(nproc)
RUN mkdir -p /app/lib && \ RUN mkdir -p /app/lib && \
find build -name "*.so*" -exec cp -P {} /app/lib \; find build -name "*.so*" -exec cp -P {} /app/lib \;
@@ -83,8 +74,6 @@ LABEL org.opencontainers.image.created=$BUILD_DATE \
org.opencontainers.image.url=$IMAGE_URL \ org.opencontainers.image.url=$IMAGE_URL \
org.opencontainers.image.source=$IMAGE_SOURCE org.opencontainers.image.source=$IMAGE_SOURCE
COPY --from=build /opt/intel/oneapi/dnnl /opt/intel/oneapi/dnnl
#Following versions are for multiple GPUs, since 26.x has known issue: #Following versions are for multiple GPUs, since 26.x has known issue:
# https://github.com/ggml-org/llama.cpp/issues/21747, # https://github.com/ggml-org/llama.cpp/issues/21747,
# https://github.com/intel/compute-runtime/issues/921. # https://github.com/intel/compute-runtime/issues/921.
@@ -95,11 +84,11 @@ COPY --from=build /opt/intel/oneapi/dnnl /opt/intel/oneapi/dnnl
#ARG IGDGMM_VERSION=22.8.2 #ARG IGDGMM_VERSION=22.8.2
ARG IGC_VERSION=v2.42.6 ARG IGC_VERSION=v2.34.4
ARG IGC_VERSION_FULL=2_2.42.6+23070 ARG IGC_VERSION_FULL=2_2.34.4+21428
ARG COMPUTE_RUNTIME_VERSION=26.40.40196.8 ARG COMPUTE_RUNTIME_VERSION=26.18.38308.1
ARG COMPUTE_RUNTIME_VERSION_FULL=26.40.40196.8-0 ARG COMPUTE_RUNTIME_VERSION_FULL=26.18.38308.1-0
ARG IGDGMM_VERSION=22.10.2 ARG IGDGMM_VERSION=22.10.0
RUN mkdir /tmp/neo/ && cd /tmp/neo/ \ RUN mkdir /tmp/neo/ && cd /tmp/neo/ \
&& wget https://github.com/intel/intel-graphics-compiler/releases/download/$IGC_VERSION/intel-igc-core-${IGC_VERSION_FULL}_amd64.deb \ && wget https://github.com/intel/intel-graphics-compiler/releases/download/$IGC_VERSION/intel-igc-core-${IGC_VERSION_FULL}_amd64.deb \
&& wget https://github.com/intel/intel-graphics-compiler/releases/download/$IGC_VERSION/intel-igc-opencl-${IGC_VERSION_FULL}_amd64.deb \ && wget https://github.com/intel/intel-graphics-compiler/releases/download/$IGC_VERSION/intel-igc-opencl-${IGC_VERSION_FULL}_amd64.deb \
@@ -113,7 +102,7 @@ RUN mkdir /tmp/neo/ && cd /tmp/neo/ \
&& dpkg --install *.deb && dpkg --install *.deb
RUN apt-get update \ RUN apt-get update \
&& apt-get install -y libgomp1 curl ffmpeg openssl libssl-dev \ && apt-get install -y libgomp1 curl ffmpeg \
&& apt autoremove -y \ && apt autoremove -y \
&& apt clean -y \ && apt clean -y \
&& rm -rf /tmp/* /var/tmp/* \ && rm -rf /tmp/* /var/tmp/* \
@@ -152,7 +141,7 @@ ENTRYPOINT ["/app/tools.sh"]
FROM base AS light FROM base AS light
COPY --from=build /app/lib/ /app COPY --from=build /app/lib/ /app
COPY --from=build /app/full/llama /app/full/llama-cli /app/full/llama-completion /app/ COPY --from=build /app/full/llama /app/full/llama-cli /app/full/llama-completion /app
WORKDIR /app WORKDIR /app
@@ -162,10 +151,9 @@ ENTRYPOINT [ "/app/llama-cli" ]
FROM base AS server FROM base AS server
ENV LLAMA_ARG_HOST=0.0.0.0 ENV LLAMA_ARG_HOST=0.0.0.0
ENV LLAMA_ARG_PORT=8080
COPY --from=build /app/lib/ /app COPY --from=build /app/lib/ /app
COPY --from=build /app/full/llama /app/full/llama-server /app/ COPY --from=build /app/full/llama /app/full/llama-server /app
WORKDIR /app WORKDIR /app
+11 -21
View File
@@ -1,9 +1,10 @@
ARG UBUNTU_VERSION=22.04 ARG UBUNTU_VERSION=22.04
# This needs to generally match the container host's environment. # This needs to generally match the container host's environment.
ARG MUSA_VERSION=rc4.3.0
# Target the MUSA build image # Target the MUSA build image
ARG BASE_MUSA_DEV_CONTAINER=registry.mthreads.com/mcconline/musa_sdk:5.2.0-devel-ubuntu${UBUNTU_VERSION}-s5000 ARG BASE_MUSA_DEV_CONTAINER=docker.io/mthreads/musa:${MUSA_VERSION}-devel-ubuntu${UBUNTU_VERSION}-amd64
ARG BASE_MUSA_RUN_CONTAINER=registry.mthreads.com/mcconline/musa_sdk:5.2.0-runtime-ubuntu${UBUNTU_VERSION}-s5000 ARG BASE_MUSA_RUN_CONTAINER=docker.io/mthreads/musa:${MUSA_VERSION}-runtime-ubuntu${UBUNTU_VERSION}-amd64
ARG BUILD_DATE=N/A ARG BUILD_DATE=N/A
ARG APP_VERSION=N/A ARG APP_VERSION=N/A
@@ -15,16 +16,12 @@ FROM docker.io/node:$NODE_VERSION AS web
ARG APP_VERSION ARG APP_VERSION
WORKDIR /app/ui/apps/chat WORKDIR /app/tools/ui
COPY ui/apps/chat/package.json ui/apps/chat/package-lock.json ./ COPY tools/ui/package.json tools/ui/package-lock.json ./
RUN npm ci RUN npm ci
COPY ui/apps/chat/ ./ COPY tools/ui/ ./
# the app compiles Llama UI and Llama Voice from their sources next to it
COPY ui/lib/ui/package.json /app/ui/lib/ui/package.json
COPY ui/lib/ui/src/lib/ /app/ui/lib/ui/src/lib/
COPY ui/lib/voice/src/ /app/ui/lib/voice/src/
RUN LLAMA_BUILD_NUMBER="$APP_VERSION" npm run build RUN LLAMA_BUILD_NUMBER="$APP_VERSION" npm run build
FROM ${BASE_MUSA_DEV_CONTAINER} AS build FROM ${BASE_MUSA_DEV_CONTAINER} AS build
@@ -40,16 +37,13 @@ RUN apt-get update && \
python3-pip \ python3-pip \
git \ git \
libssl-dev \ libssl-dev \
libgomp1 \ libgomp1
musa-mualg-5-2 \
musa-muthrust-5-2 \
libmthreads-compute
WORKDIR /app WORKDIR /app
COPY . . COPY . .
COPY --from=web /app/ui/apps/chat/dist ui/apps/chat/dist COPY --from=web /app/tools/ui/dist tools/ui/dist
RUN if [ "${MUSA_DOCKER_ARCH}" != "default" ]; then \ RUN if [ "${MUSA_DOCKER_ARCH}" != "default" ]; then \
export CMAKE_ARGS="-DMUSA_ARCHITECTURES=${MUSA_DOCKER_ARCH}"; \ export CMAKE_ARGS="-DMUSA_ARCHITECTURES=${MUSA_DOCKER_ARCH}"; \
@@ -86,16 +80,13 @@ LABEL org.opencontainers.image.created=$BUILD_DATE \
org.opencontainers.image.source=$IMAGE_SOURCE org.opencontainers.image.source=$IMAGE_SOURCE
RUN apt-get update \ RUN apt-get update \
&& apt-get install -y libgomp1 curl ffmpeg libmthreads-compute \ && apt-get install -y libgomp1 curl ffmpeg \
&& apt autoremove -y \ && apt autoremove -y \
&& apt clean -y \ && apt clean -y \
&& rm -rf /tmp/* /var/tmp/* \ && rm -rf /tmp/* /var/tmp/* \
&& find /var/cache/apt/archives /var/lib/apt/lists -not -name lock -type f -delete \ && find /var/cache/apt/archives /var/lib/apt/lists -not -name lock -type f -delete \
&& find /var/cache -type f -delete && find /var/cache -type f -delete
# The MUSA runtime image does not register its library directory
RUN echo "/usr/local/musa/lib" > /etc/ld.so.conf.d/musa-runtime.conf && ldconfig
COPY --from=build /app/lib/ /app COPY --from=build /app/lib/ /app
### Full ### Full
@@ -124,7 +115,7 @@ ENTRYPOINT ["/app/tools.sh"]
### Light, CLI only ### Light, CLI only
FROM base AS light FROM base AS light
COPY --from=build /app/full/llama /app/full/llama-cli /app/full/llama-completion /app/ COPY --from=build /app/full/llama /app/full/llama-cli /app/full/llama-completion /app
WORKDIR /app WORKDIR /app
@@ -134,9 +125,8 @@ ENTRYPOINT [ "/app/llama-cli" ]
FROM base AS server FROM base AS server
ENV LLAMA_ARG_HOST=0.0.0.0 ENV LLAMA_ARG_HOST=0.0.0.0
ENV LLAMA_ARG_PORT=8080
COPY --from=build /app/full/llama /app/full/llama-server /app/ COPY --from=build /app/full/llama /app/full/llama-server /app
WORKDIR /app WORKDIR /app
+18 -115
View File
@@ -22,51 +22,27 @@
spirv-headers, spirv-headers,
nodejs, nodejs,
importNpmLock, importNpmLock,
syclToolkit ? null,
level-zero,
ocl-icd,
intel-graphics-compiler,
intel-compute-runtime,
makeBinaryWrapper,
useBlas ? useBlas ?
builtins.all (x: !x) [ builtins.all (x: !x) [
useCuda useCuda
useMetalKit useMetalKit
useRocm useRocm
useVulkan useVulkan
useSycl
] ]
&& blas.meta.available, && blas.meta.available,
useCuda ? config.cudaSupport, useCuda ? config.cudaSupport,
useMetalKit ? stdenv.hostPlatform.isAarch64 && stdenv.hostPlatform.isDarwin, useMetalKit ? stdenv.isAarch64 && stdenv.isDarwin,
# Increases the runtime closure size by ~700M # Increases the runtime closure size by ~700M
useMpi ? false, useMpi ? false,
useRocm ? config.rocmSupport, useRocm ? config.rocmSupport,
rocmGpuTargets ? builtins.concatStringsSep ";" rocmPackages.clr.gpuTargets, rocmGpuTargets ? builtins.concatStringsSep ";" rocmPackages.clr.gpuTargets,
useVulkan ? false, useVulkan ? false,
useRpc ? false, useRpc ? false,
useSycl ? false,
# Rebuild, not a runtime switch. cmake default is off.
syclF16 ? false,
# null keeps JIT. AOT makes icpx run ocloc, which DPC++ does not ship.
syclDeviceArch ? null,
syclMaxParallelLinkJobs ? null,
llamaVersion ? "0.0.0", # Arbitrary version, substituted by the flake llamaVersion ? "0.0.0", # Arbitrary version, substituted by the flake
# CUDA needs its backend stdenv or libstdc++ mismatches downstream. # It's necessary to consistently use backendStdenv when building with CUDA support,
# SYCL needs icpx which is provided by the oneAPI stdenv # otherwise we get libstdc++ errors downstream.
effectiveStdenv ? effectiveStdenv ? if useCuda then cudaPackages.backendStdenv else stdenv,
if useCuda && useSycl then
throw "llama-cpp: useCuda and useSycl are mutually exclusive"
else if useCuda then
cudaPackages.backendStdenv
else if useSycl then
if syclToolkit == null then
throw "llama-cpp: useSycl requires syclToolkit, consider setting it to intel-oneapi-toolkit"
else
syclToolkit.stdenv
else
stdenv,
enableStatic ? effectiveStdenv.hostPlatform.isStatic, enableStatic ? effectiveStdenv.hostPlatform.isStatic,
precompileMetalShaders ? false, precompileMetalShaders ? false,
useWebUi ? true, useWebUi ? true,
@@ -89,8 +65,7 @@ let
++ lib.optionals useMetalKit [ "MetalKit" ] ++ lib.optionals useMetalKit [ "MetalKit" ]
++ lib.optionals useMpi [ "MPI" ] ++ lib.optionals useMpi [ "MPI" ]
++ lib.optionals useRocm [ "ROCm" ] ++ lib.optionals useRocm [ "ROCm" ]
++ lib.optionals useVulkan [ "Vulkan" ] ++ lib.optionals useVulkan [ "Vulkan" ];
++ lib.optionals useSycl [ "SYCL" ];
pnameSuffix = pnameSuffix =
strings.optionalString (suffices != [ ]) strings.optionalString (suffices != [ ])
@@ -117,7 +92,7 @@ let
cudaBuildInputs = with cudaPackages; [ cudaBuildInputs = with cudaPackages; [
cuda_cudart cuda_cudart
cccl # <nv/target> cuda_cccl # <nv/target>
libcublas libcublas
]; ];
@@ -133,26 +108,6 @@ let
shaderc shaderc
spirv-headers spirv-headers
]; ];
syclAot = useSycl && syclDeviceArch != null && syclDeviceArch != "";
# source ocloc binary from intel-compute-runtime
oclocAot = runCommand "ocloc-aot" { nativeBuildInputs = [ makeBinaryWrapper ]; } ''
mkdir -p "$out/bin"
shopt -s nullglob
exact=(${lib.getBin intel-compute-runtime}/bin/ocloc)
vers=(${lib.getBin intel-compute-runtime}/bin/ocloc-*)
if [ ''${#exact[@]} -eq 1 ]; then
src=''${exact[0]}
elif [ ''${#vers[@]} -eq 1 ]; then
src=''${vers[0]}
else
echo "expected one ocloc binary from intel-compute-runtime" >&2
exit 1
fi
makeWrapper "$src" "$out/bin/ocloc" \
--prefix LD_LIBRARY_PATH : ${lib.makeLibraryPath [ intel-graphics-compiler ]}
'';
in in
effectiveStdenv.mkDerivation (finalAttrs: { effectiveStdenv.mkDerivation (finalAttrs: {
@@ -178,13 +133,11 @@ effectiveStdenv.mkDerivation (finalAttrs: {
src = lib.cleanSource ../../.; src = lib.cleanSource ../../.;
}; };
# Builds Llama Chat locally, taking care not to require updating any sha256 hash. # Builds the webui locally, taking care not to require updating any sha256 hash.
webui = stdenvNoCC.mkDerivation { webui = stdenvNoCC.mkDerivation {
pname = "webui"; pname = "webui";
version = llamaVersion; version = llamaVersion;
# the app compiles Llama UI from its source in ../../lib/ui, so the source is ui/ as a whole src = lib.cleanSource ../../tools/ui;
src = lib.cleanSource ../../ui;
sourceRoot = "source/apps/chat";
nativeBuildInputs = [ nativeBuildInputs = [
nodejs nodejs
@@ -193,7 +146,7 @@ effectiveStdenv.mkDerivation (finalAttrs: {
# no sha256 required when using buildNodeModules # no sha256 required when using buildNodeModules
npmDeps = importNpmLock.buildNodeModules { npmDeps = importNpmLock.buildNodeModules {
npmRoot = ../../ui/apps/chat; npmRoot = ../../tools/ui;
inherit nodejs; inherit nodejs;
}; };
@@ -203,8 +156,8 @@ effectiveStdenv.mkDerivation (finalAttrs: {
}; };
postPatch = lib.optionalString useWebUi '' postPatch = lib.optionalString useWebUi ''
cp -r ${finalAttrs.webui} ui/apps/chat/dist cp -r ${finalAttrs.webui} tools/ui/dist
chmod -R u+w ui/apps/chat/dist chmod -R u+w tools/ui/dist
''; '';
# With PR#6015 https://github.com/ggml-org/llama.cpp/pull/6015, # With PR#6015 https://github.com/ggml-org/llama.cpp/pull/6015,
@@ -213,14 +166,7 @@ effectiveStdenv.mkDerivation (finalAttrs: {
# `xcrun` is used find the path of the Metal compiler, which is varible # `xcrun` is used find the path of the Metal compiler, which is varible
# and not on $PATH # and not on $PATH
# see https://github.com/ggml-org/llama.cpp/pull/6118 for discussion # see https://github.com/ggml-org/llama.cpp/pull/6118 for discussion
__noChroot = effectiveStdenv.hostPlatform.isDarwin && useMetalKit && precompileMetalShaders; __noChroot = effectiveStdenv.isDarwin && useMetalKit && precompileMetalShaders;
# Do not fortify under SYCL as it breaks compiled memcpy invocations
# _FORTIFY_SOURCE rewrites memcpy -> __memcpy_chk
hardeningDisable = optionals useSycl [
"fortify"
"fortify3"
];
nativeBuildInputs = nativeBuildInputs =
[ [
@@ -235,28 +181,19 @@ effectiveStdenv.mkDerivation (finalAttrs: {
autoAddDriverRunpath autoAddDriverRunpath
] ]
++ optionals (effectiveStdenv.hostPlatform.isGnu && enableStatic) [ glibc.static ] ++ optionals (effectiveStdenv.hostPlatform.isGnu && enableStatic) [ glibc.static ]
++ optionals (effectiveStdenv.hostPlatform.isDarwin && useMetalKit && precompileMetalShaders) [ xcrunHost ] ++ optionals (effectiveStdenv.isDarwin && useMetalKit && precompileMetalShaders) [ xcrunHost ];
++ optionals useSycl [
autoAddDriverRunpath
makeBinaryWrapper
]
++ optionals syclAot [ oclocAot ];
buildInputs = buildInputs =
optionals effectiveStdenv.hostPlatform.isDarwin darwinBuildInputs optionals effectiveStdenv.isDarwin darwinBuildInputs
++ optionals useCuda cudaBuildInputs ++ optionals useCuda cudaBuildInputs
++ optionals useMpi [ mpi ] ++ optionals useMpi [ mpi ]
++ optionals useRocm rocmBuildInputs ++ optionals useRocm rocmBuildInputs
++ optionals useBlas [ blas ] ++ optionals useBlas [ blas ]
++ optionals useVulkan vulkanBuildInputs ++ optionals useVulkan vulkanBuildInputs
++ optionals useSycl [
level-zero
ocl-icd
intel-graphics-compiler
]
++ [ openssl ]; ++ [ openssl ];
cmakeFlags = [ cmakeFlags =
[
(cmakeBool "LLAMA_BUILD_SERVER" true) (cmakeBool "LLAMA_BUILD_SERVER" true)
(cmakeBool "LLAMA_BUILD_WEBUI" useWebUi) (cmakeBool "LLAMA_BUILD_WEBUI" useWebUi)
(cmakeBool "BUILD_SHARED_LIBS" (!enableStatic)) (cmakeBool "BUILD_SHARED_LIBS" (!enableStatic))
@@ -285,16 +222,6 @@ effectiveStdenv.mkDerivation (finalAttrs: {
++ optionals useMetalKit [ ++ optionals useMetalKit [
(lib.cmakeFeature "CMAKE_C_FLAGS" "-D__ARM_FEATURE_DOTPROD=1") (lib.cmakeFeature "CMAKE_C_FLAGS" "-D__ARM_FEATURE_DOTPROD=1")
(cmakeBool "GGML_METAL_EMBED_LIBRARY" (!precompileMetalShaders)) (cmakeBool "GGML_METAL_EMBED_LIBRARY" (!precompileMetalShaders))
]
++ optionals useSycl [
(cmakeBool "GGML_SYCL" true)
(cmakeBool "GGML_SYCL_F16" syclF16)
]
++ optionals syclAot [
(cmakeFeature "GGML_SYCL_DEVICE_ARCH" syclDeviceArch)
]
++ optionals (syclAot && syclMaxParallelLinkJobs != null) [
(cmakeFeature "GGML_SYCL_MAX_PARALLEL_LINK_JOBS" (toString syclMaxParallelLinkJobs))
]; ];
# Environment variables needed for ROCm # Environment variables needed for ROCm
@@ -310,39 +237,15 @@ effectiveStdenv.mkDerivation (finalAttrs: {
cp $src/include/llama.h $out/include/ cp $src/include/llama.h $out/include/
''; '';
# Fix SYCL binary calling missing important libraries
# Also use Level Zero + sysman INSTEAD of OpenCL
# due to OpenCL reporting free memory as total memory,
# breaking `-fit` functionality
postFixup = lib.optionalString useSycl ''
driverLibs=${
lib.makeLibraryPath [
intel-graphics-compiler
ocl-icd
]
}
for b in "$out"/bin/*; do
[ -f "$b" ] && [ -x "$b" ] || continue
patchelf --print-interpreter "$b" >/dev/null 2>&1 || continue
patchelf --add-needed libOpenCL.so.1 --add-needed libigc.so.2 \
--add-rpath "$driverLibs" "$b"
wrapProgram "$b" \
--set-default ONEAPI_DEVICE_SELECTOR level_zero:gpu \
--set-default ZES_ENABLE_SYSMAN 1
done
'';
meta = { meta = {
# Configurations we don't want even the CI to evaluate. Results in the # Configurations we don't want even the CI to evaluate. Results in the
# "unsupported platform" messages. This is mostly a no-op, because # "unsupported platform" messages. This is mostly a no-op, because
# cudaPackages would've refused to evaluate anyway. # cudaPackages would've refused to evaluate anyway.
badPlatforms = badPlatforms = optionals useCuda lib.platforms.darwin;
optionals useCuda lib.platforms.darwin
++ optionals useSycl (lib.subtractLists [ "x86_64-linux" ] lib.platforms.all);
# Configurations that are known to result in build failures. Can be # Configurations that are known to result in build failures. Can be
# overridden by importing Nixpkgs with `allowBroken = true`. # overridden by importing Nixpkgs with `allowBroken = true`.
broken = (useMetalKit && !effectiveStdenv.hostPlatform.isDarwin); broken = (useMetalKit && !effectiveStdenv.isDarwin);
description = "Inference of LLaMA model in pure C/C++${descriptionSuffix}"; description = "Inference of LLaMA model in pure C/C++${descriptionSuffix}";
homepage = "https://github.com/ggml-org/llama.cpp/"; homepage = "https://github.com/ggml-org/llama.cpp/";
+4
View File
@@ -22,6 +22,10 @@ let
gitpython gitpython
tabulate tabulate
# for examples/pydantic-models-to-grammar-examples.py
docstring-parser
pydantic
]; ];
llama-python-test-deps = with python3Packages; [ llama-python-test-deps = with python3Packages; [
+15 -23
View File
@@ -1,18 +1,18 @@
ARG OPENVINO_VERSION_MAJOR=2026.4.1 ARG OPENVINO_VERSION_MAJOR=2026.2.1
ARG OPENVINO_VERSION_FULL=2026.4.1.22982.07f9c262b05 ARG OPENVINO_VERSION_FULL=2026.2.1.21919.ede283a88e3
ARG UBUNTU_VERSION=24.04 ARG UBUNTU_VERSION=24.04
# Intel GPU driver versions. https://github.com/intel/compute-runtime/releases # Intel GPU driver versions. https://github.com/intel/compute-runtime/releases
ARG IGC_VERSION=v2.42.6 ARG IGC_VERSION=v2.36.3
ARG IGC_VERSION_FULL=2_2.42.6+23070 ARG IGC_VERSION_FULL=2_2.36.3+21719
ARG COMPUTE_RUNTIME_VERSION=26.40.40196.8 ARG COMPUTE_RUNTIME_VERSION=26.22.38646.4
ARG COMPUTE_RUNTIME_VERSION_FULL=26.40.40196.8-0 ARG COMPUTE_RUNTIME_VERSION_FULL=26.22.38646.4-0
ARG IGDGMM_VERSION=22.10.2 ARG IGDGMM_VERSION=22.10.0
# Intel NPU driver versions. https://github.com/intel/linux-npu-driver/releases # Intel NPU driver versions. https://github.com/intel/linux-npu-driver/releases
ARG NPU_DRIVER_VERSION=v1.38.0 ARG NPU_DRIVER_VERSION=v1.33.0
ARG NPU_DRIVER_FULL=v1.38.0.20260910-34487311128 ARG NPU_DRIVER_FULL=v1.33.0.20260529-26625960453
ARG LIBZE1_VERSION=1.32.0-1~24.04~ppa1 ARG LIBZE1_VERSION=1.27.0-1~24.04~ppa2
# Optional proxy build arguments # Optional proxy build arguments
ARG http_proxy= ARG http_proxy=
@@ -28,16 +28,12 @@ FROM docker.io/node:$NODE_VERSION AS web
ARG APP_VERSION ARG APP_VERSION
WORKDIR /app/ui/apps/chat WORKDIR /app/tools/ui
COPY ui/apps/chat/package.json ui/apps/chat/package-lock.json ./ COPY tools/ui/package.json tools/ui/package-lock.json ./
RUN npm ci RUN npm ci
COPY ui/apps/chat/ ./ COPY tools/ui/ ./
# the app compiles Llama UI and Llama Voice from their sources next to it
COPY ui/lib/ui/package.json /app/ui/lib/ui/package.json
COPY ui/lib/ui/src/lib/ /app/ui/lib/ui/src/lib/
COPY ui/lib/voice/src/ /app/ui/lib/voice/src/
RUN LLAMA_BUILD_NUMBER="$APP_VERSION" npm run build RUN LLAMA_BUILD_NUMBER="$APP_VERSION" npm run build
## Build Image ## Build Image
@@ -87,16 +83,13 @@ WORKDIR /app
COPY . . COPY . .
COPY --from=web /app/ui/apps/chat/dist ui/apps/chat/dist COPY --from=web /app/tools/ui/dist tools/ui/dist
# Build Stage # Build Stage
RUN bash -c "source ${OpenVINO_DIR}/setupvars.sh && \ RUN bash -c "source ${OpenVINO_DIR}/setupvars.sh && \
cmake -B build/ReleaseOV -G Ninja \ cmake -B build/ReleaseOV -G Ninja \
-DCMAKE_BUILD_TYPE=Release \ -DCMAKE_BUILD_TYPE=Release \
-DLLAMA_BUILD_TESTS=OFF \ -DLLAMA_BUILD_TESTS=OFF \
-DGGML_NATIVE=OFF \
-DGGML_BACKEND_DL=ON \
-DGGML_CPU_ALL_VARIANTS=ON \
-DGGML_OPENVINO=ON && \ -DGGML_OPENVINO=ON && \
cmake --build build/ReleaseOV --parallel " cmake --build build/ReleaseOV --parallel "
@@ -177,7 +170,7 @@ RUN --mount=type=cache,target=/var/cache/intel-npu,sharing=locked \
fi; \ fi; \
DEB=/var/cache/intel-npu/libze1_${LIBZE1_VERSION}_amd64.deb; \ DEB=/var/cache/intel-npu/libze1_${LIBZE1_VERSION}_amd64.deb; \
if [ ! -f "$DEB" ]; then \ if [ ! -f "$DEB" ]; then \
wget -q -O "$DEB" https://snapshot.ppa.launchpadcontent.net/kobuk-team/intel-graphics/ubuntu/20260830T100000Z/pool/main/l/level-zero-loader/libze1_${LIBZE1_VERSION}_amd64.deb; \ wget -q -O "$DEB" https://snapshot.ppa.launchpadcontent.net/kobuk-team/intel-graphics/ubuntu/20260324T100000Z/pool/main/l/level-zero-loader/libze1_${LIBZE1_VERSION}_amd64.deb; \
fi; \ fi; \
mkdir /tmp/npu/ && cd /tmp/npu/ && tar -xf "$TGZ" && cp "$DEB" .; \ mkdir /tmp/npu/ && cd /tmp/npu/ && tar -xf "$TGZ" && cp "$DEB" .; \
apt-get update; \ apt-get update; \
@@ -231,7 +224,6 @@ ENTRYPOINT [ "/app/llama-cli" ]
FROM base AS server FROM base AS server
ENV LLAMA_ARG_HOST=0.0.0.0 ENV LLAMA_ARG_HOST=0.0.0.0
ENV LLAMA_ARG_PORT=8080
COPY --from=build /app/full/llama /app/full/llama-server /app/ COPY --from=build /app/full/llama /app/full/llama-server /app/
+7 -11
View File
@@ -17,16 +17,12 @@ FROM docker.io/node:$NODE_VERSION AS web
ARG APP_VERSION ARG APP_VERSION
WORKDIR /app/ui/apps/chat WORKDIR /app/tools/ui
COPY ui/apps/chat/package.json ui/apps/chat/package-lock.json ./ COPY tools/ui/package.json tools/ui/package-lock.json ./
RUN npm ci RUN npm ci
COPY ui/apps/chat/ ./ COPY tools/ui/ ./
# the app compiles Llama UI and Llama Voice from their sources next to it
COPY ui/lib/ui/package.json /app/ui/lib/ui/package.json
COPY ui/lib/ui/src/lib/ /app/ui/lib/ui/src/lib/
COPY ui/lib/voice/src/ /app/ui/lib/voice/src/
RUN LLAMA_BUILD_NUMBER="$APP_VERSION" npm run build RUN LLAMA_BUILD_NUMBER="$APP_VERSION" npm run build
### Build image ### Build image
@@ -56,11 +52,12 @@ WORKDIR /app
COPY . . COPY . .
COPY --from=web /app/ui/apps/chat/dist ui/apps/chat/dist COPY --from=web /app/tools/ui/dist tools/ui/dist
RUN HIPCXX="$(hipconfig -l)/clang" HIP_PATH="$(hipconfig -R)" \ RUN HIPCXX="$(hipconfig -l)/clang" HIP_PATH="$(hipconfig -R)" \
cmake -S . -B build \ cmake -S . -B build \
-DGGML_HIP=ON \ -DGGML_HIP=ON \
-DGGML_HIP_ROCWMMA_FATTN=ON \
-DAMDGPU_TARGETS="$ROCM_DOCKER_ARCH" \ -DAMDGPU_TARGETS="$ROCM_DOCKER_ARCH" \
-DGGML_BACKEND_DL=ON -DGGML_CPU_ALL_VARIANTS=ON \ -DGGML_BACKEND_DL=ON -DGGML_CPU_ALL_VARIANTS=ON \
-DCMAKE_BUILD_TYPE=Release -DLLAMA_BUILD_TESTS=OFF \ -DCMAKE_BUILD_TYPE=Release -DLLAMA_BUILD_TESTS=OFF \
@@ -130,7 +127,7 @@ ENTRYPOINT ["/app/tools.sh"]
### Light, CLI only ### Light, CLI only
FROM base AS light FROM base AS light
COPY --from=build /app/full/llama /app/full/llama-cli /app/full/llama-completion /app/ COPY --from=build /app/full/llama /app/full/llama-cli /app/full/llama-completion /app
WORKDIR /app WORKDIR /app
@@ -140,9 +137,8 @@ ENTRYPOINT [ "/app/llama-cli" ]
FROM base AS server FROM base AS server
ENV LLAMA_ARG_HOST=0.0.0.0 ENV LLAMA_ARG_HOST=0.0.0.0
ENV LLAMA_ARG_PORT=8080
COPY --from=build /app/full/llama /app/full/llama-server /app/ COPY --from=build /app/full/llama /app/full/llama-server /app
WORKDIR /app WORKDIR /app
-1
View File
@@ -133,7 +133,6 @@ ENTRYPOINT [ "/llama.cpp/bin/llama-cli" ]
FROM base AS server FROM base AS server
ENV LLAMA_ARG_HOST=0.0.0.0 ENV LLAMA_ARG_HOST=0.0.0.0
ENV LLAMA_ARG_PORT=8080
WORKDIR /llama.cpp/bin WORKDIR /llama.cpp/bin
+17 -16
View File
@@ -9,16 +9,12 @@ FROM docker.io/node:$NODE_VERSION AS web
ARG APP_VERSION ARG APP_VERSION
WORKDIR /app/ui/apps/chat WORKDIR /app/tools/ui
COPY ui/apps/chat/package.json ui/apps/chat/package-lock.json ./ COPY tools/ui/package.json tools/ui/package-lock.json ./
RUN npm ci RUN npm ci
COPY ui/apps/chat/ ./ COPY tools/ui/ ./
# the app compiles Llama UI and Llama Voice from their sources next to it
COPY ui/lib/ui/package.json /app/ui/lib/ui/package.json
COPY ui/lib/ui/src/lib/ /app/ui/lib/ui/src/lib/
COPY ui/lib/voice/src/ /app/ui/lib/voice/src/
RUN LLAMA_BUILD_NUMBER="$APP_VERSION" npm run build RUN LLAMA_BUILD_NUMBER="$APP_VERSION" npm run build
FROM docker.io/ubuntu:$UBUNTU_VERSION AS build FROM docker.io/ubuntu:$UBUNTU_VERSION AS build
@@ -35,7 +31,7 @@ WORKDIR /app
COPY . . COPY . .
COPY --from=web /app/ui/apps/chat/dist ui/apps/chat/dist COPY --from=web /app/tools/ui/dist tools/ui/dist
RUN cmake -B build -DGGML_NATIVE=OFF -DGGML_VULKAN=ON -DLLAMA_BUILD_TESTS=OFF -DGGML_BACKEND_DL=ON -DGGML_CPU_ALL_VARIANTS=ON && \ RUN cmake -B build -DGGML_NATIVE=OFF -DGGML_VULKAN=ON -DLLAMA_BUILD_TESTS=OFF -DGGML_BACKEND_DL=ON -DGGML_CPU_ALL_VARIANTS=ON && \
cmake --build build --config Release -j$(nproc) cmake --build build --config Release -j$(nproc)
@@ -86,14 +82,20 @@ COPY --from=build /app/full /app
WORKDIR /app WORKDIR /app
ENV PATH="/root/.venv/bin:/root/.local/bin:${PATH}"
# Flag for compatibility with pip
ARG UV_INDEX_STRATEGY="unsafe-best-match"
RUN apt-get update \ RUN apt-get update \
&& apt-get install -y \ && apt-get install -y \
build-essential \
curl \
git \ git \
python3 \ ca-certificates \
python3-pip \ && curl -LsSf https://astral.sh/uv/install.sh | sh \
python3-wheel \ && uv python install 3.13 \
&& pip install --break-system-packages --upgrade setuptools \ && uv venv --python 3.13 /root/.venv \
&& pip install --break-system-packages -r requirements.txt \ && uv pip install --python /root/.venv/bin/python -r requirements.txt \
&& apt autoremove -y \ && apt autoremove -y \
&& apt clean -y \ && apt clean -y \
&& rm -rf /tmp/* /var/tmp/* \ && rm -rf /tmp/* /var/tmp/* \
@@ -105,7 +107,7 @@ ENTRYPOINT ["/app/tools.sh"]
### Light, CLI only ### Light, CLI only
FROM base AS light FROM base AS light
COPY --from=build /app/full/llama /app/full/llama-cli /app/full/llama-completion /app/ COPY --from=build /app/full/llama /app/full/llama-cli /app/full/llama-completion /app
WORKDIR /app WORKDIR /app
@@ -115,9 +117,8 @@ ENTRYPOINT [ "/app/llama-cli" ]
FROM base AS server FROM base AS server
ENV LLAMA_ARG_HOST=0.0.0.0 ENV LLAMA_ARG_HOST=0.0.0.0
ENV LLAMA_ARG_PORT=8080
COPY --from=build /app/full/llama /app/full/llama-server /app/ COPY --from=build /app/full/llama /app/full/llama-server /app
WORKDIR /app WORKDIR /app
+6 -11
View File
@@ -9,16 +9,12 @@ FROM docker.io/node:$NODE_VERSION AS web
ARG APP_VERSION ARG APP_VERSION
WORKDIR /app/ui/apps/chat WORKDIR /app/tools/ui
COPY ui/apps/chat/package.json ui/apps/chat/package-lock.json ./ COPY tools/ui/package.json tools/ui/package-lock.json ./
RUN npm ci RUN npm ci
COPY ui/apps/chat/ ./ COPY tools/ui/ ./
# the app compiles Llama UI and Llama Voice from their sources next to it
COPY ui/lib/ui/package.json /app/ui/lib/ui/package.json
COPY ui/lib/ui/src/lib/ /app/ui/lib/ui/src/lib/
COPY ui/lib/voice/src/ /app/ui/lib/voice/src/
RUN LLAMA_BUILD_NUMBER="$APP_VERSION" npm run build RUN LLAMA_BUILD_NUMBER="$APP_VERSION" npm run build
FROM docker.io/ubuntu:$UBUNTU_VERSION AS build FROM docker.io/ubuntu:$UBUNTU_VERSION AS build
@@ -32,7 +28,7 @@ WORKDIR /app
COPY . . COPY . .
COPY --from=web /app/ui/apps/chat/dist ui/apps/chat/dist COPY --from=web /app/tools/ui/dist tools/ui/dist
RUN cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=OFF -DLLAMA_BUILD_TESTS=OFF -DGGML_BACKEND_DL=ON -DGGML_CPU_ALL_VARIANTS=ON -DGGML_ZENDNN=ON && \ RUN cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=OFF -DLLAMA_BUILD_TESTS=OFF -DGGML_BACKEND_DL=ON -DGGML_CPU_ALL_VARIANTS=ON -DGGML_ZENDNN=ON && \
cmake --build build -j $(nproc) cmake --build build -j $(nproc)
@@ -101,7 +97,7 @@ ENTRYPOINT ["/app/tools.sh"]
### Light, CLI only ### Light, CLI only
FROM base AS light FROM base AS light
COPY --from=build /app/full/llama /app/full/llama-cli /app/full/llama-completion /app/ COPY --from=build /app/full/llama /app/full/llama-cli /app/full/llama-completion /app
WORKDIR /app WORKDIR /app
@@ -111,9 +107,8 @@ ENTRYPOINT [ "/app/llama-cli" ]
FROM base AS server FROM base AS server
ENV LLAMA_ARG_HOST=0.0.0.0 ENV LLAMA_ARG_HOST=0.0.0.0
ENV LLAMA_ARG_PORT=8080
COPY --from=build /app/full/llama /app/full/llama-server /app/ COPY --from=build /app/full/llama /app/full/llama-server /app
WORKDIR /app WORKDIR /app
+1 -1
View File
@@ -10,7 +10,7 @@
build*/ build*/
ui/apps/chat/node_modules/ tools/ui/node_modules/
models/* models/*
+1 -1
View File
@@ -1,5 +1,5 @@
{ {
"Exclude": ["^\\.gitmodules$", "stb_image\\.h", "examples/test-cmake/build/", "examples/test-cmake/build-subdir/"], "Exclude": ["^\\.gitmodules$", "stb_image\\.h"],
"Disable": { "Disable": {
"IndentSize": true "IndentSize": true
} }
+1 -4
View File
@@ -45,10 +45,7 @@ insert_final_newline = unset
trim_trailing_whitespace = unset trim_trailing_whitespace = unset
insert_final_newline = unset insert_final_newline = unset
[vendor/**.patch] [tools/ui/**]
trim_trailing_whitespace = unset
[ui/**]
indent_style = unset indent_style = unset
indent_size = unset indent_size = unset
end_of_line = unset end_of_line = unset
-2
View File
@@ -5,8 +5,6 @@ exclude =
# Do not traverse examples and tools # Do not traverse examples and tools
examples, examples,
tools, tools,
# Do not traverse .venv directories
.venv*,
# Do not include package initializers # Do not include package initializers
__init__.py, __init__.py,
# No need to traverse our git directory # No need to traverse our git directory
+1 -1
View File
@@ -1,4 +1,4 @@
blank_issues_enabled: false blank_issues_enabled: true
contact_links: contact_links:
- name: Got an idea? - name: Got an idea?
url: https://github.com/ggml-org/llama.cpp/discussions/categories/ideas url: https://github.com/ggml-org/llama.cpp/discussions/categories/ideas
-95
View File
@@ -1,95 +0,0 @@
name: "ccache-buckets"
description: "Save/restore latest GitHub Actions ccache matching a key prefix to/from HF buckets"
inputs:
key:
description: "Cache key prefix to match and load"
required: true
folder:
description: "Bucket folder containing ccache files"
required: true
evict-old-files:
description: "Corresponds to the ccache --evict-older-than AGE option, where AGE is the number of seconds or days followed by the 's' or 'd' suffix respectively."
default: ''
save:
description: "Save ccache"
required: false
default: false
type: boolean
hf_bucket:
description: 'Hugging Face buckets path'
required: true
runs:
using: "composite"
steps:
- name: Install Hugging Face Hub CLI
shell: bash
run: |
python3 -m venv .venv-hf
.venv-hf/bin/pip install -U huggingface_hub==1.28.0
- name: Restore ccache from buckets
if: ${{ inputs.save != 'true' }}
shell: bash
run: |
set +e -uo pipefail
source .venv-hf/bin/activate
CCACHE_DIR=$(ccache -k cache_dir)
if [[ -d "$CCACHE_DIR" ]]; then
CACHE_PATH=$(hf buckets list "hf://buckets/${{ inputs.hf_bucket }}/${{ inputs.folder }}" --json | jq -r '[.[] | select(.type == "file") | select(.path | startswith("${{ inputs.folder }}/${{ inputs.key }}") and endswith(".tar.gz"))] | sort_by(.path) | last | .path // ""')
if [[ -n "$CACHE_PATH" ]]; then
echo "Restoring ccache from '$CACHE_PATH'."
hf buckets cp "hf://buckets/${{ inputs.hf_bucket }}/$CACHE_PATH" ccache_bucket.tar.gz
mkdir -p ccache_bucket
if tar -xzf ccache_bucket.tar.gz -C ccache_bucket; then
rm -rf "$CCACHE_DIR"
mv ccache_bucket "$CCACHE_DIR"
ccache -z
fi
rm ccache_bucket.tar.gz
else
echo "No ccache found."
fi
else
echo "'$CCACHE_DIR' not found."
fi
- name: Save ccache to buckets
if: ${{ inputs.save == 'true' }}
shell: bash
run: |
if [[ -n "$HF_TOKEN" ]]; then
set +e -uo pipefail
source .venv-hf/bin/activate
CCACHE_DIR=$(ccache -k cache_dir)
if [[ -d "$CCACHE_DIR" ]]; then
ccache -s
if [[ -n "${{ inputs.evict-old-files }}" ]]; then
ccache --evict-older-than "${{ inputs.evict-old-files }}"
fi
DATESTAMP=$(date -u +'%Y-%m-%dT%H:%M:%SZ')
CACHEFILE="${{ inputs.key }}-$DATESTAMP.tar.gz"
if tar -czf ccache_bucket.tar.gz -C "$CCACHE_DIR" .; then
hf buckets cp ccache_bucket.tar.gz "hf://buckets/${{ inputs.hf_bucket }}/${{ inputs.folder }}/$CACHEFILE"
fi
rm ccache_bucket.tar.gz
else
echo "'$CCACHE_DIR' not found."
fi
fi
- name: Remove old ccache files from buckets
if: ${{ inputs.save == 'true' }}
shell: bash
run: |
if [[ -n "$HF_TOKEN" ]]; then
set +e -uo pipefail
source .venv-hf/bin/activate
CACHE_FILES=$(hf buckets list "hf://buckets/${{ inputs.hf_bucket }}/${{ inputs.folder }}" --json | jq -r '[.[] | select(.type == "file") | select((.uploaded_at | .[:19]+"Z" | fromdateiso8601) < (now - 5 * 60)) | select(.path | startswith("${{ inputs.folder }}/${{ inputs.key }}") and endswith(".tar.gz"))] | sort_by(.path)[:-1] | .[] | [.path // ""] | @tsv')
if [[ -n "$CACHE_FILES" ]]; then
echo "Removing old ccache files..."
while IFS=$'\t' read -r CACHE_PATH; do
hf buckets rm "hf://buckets/${{ inputs.hf_bucket }}/$CACHE_PATH" -y
done <<< "$CACHE_FILES"
fi
fi
+10 -38
View File
@@ -1,50 +1,22 @@
# note: place this as the last step of the job, so the new cache is saved by "Post ccache" right after the old one is cleared
name: "ccache-clear" name: "ccache-clear"
description: "Delete GitHub Actions caches matching a key prefix, oldest first" description: "Delete all GitHub Actions caches matching a key prefix"
inputs: inputs:
key: key:
description: "Cache key prefix to match and delete" description: "Cache key prefix to match and delete"
required: true required: true
older:
description: "Only delete caches created more than this long ago (e.g. 90m, 1h, 1d). By default all matching caches are deleted"
required: false
default: ""
min:
description: "Stop deleting if fewer than this many caches would remain (e.g. 1). By default there is no minimum"
required: false
default: "0"
dry-run:
description: "Only print the caches that would be deleted, without deleting them"
required: false
default: "false"
runs: runs:
using: "composite" using: "composite"
steps: steps:
- name: Install GitHub CLI if missing
shell: bash
run: |
# e.g. in container jobs, where it is not preinstalled
if ! command -v gh >/dev/null 2>&1; then
echo "GitHub CLI not found, installing..."
if ! command -v curl >/dev/null 2>&1; then
apt-get update >/dev/null 2>&1 || true
apt-get install -y curl >/dev/null 2>&1 || true
fi
mkdir -p -m 755 /etc/apt/keyrings
curl -fsSL https://cli.github.com/packages/githubcli-archive-keyring.gpg | tee /etc/apt/keyrings/githubcli-archive-keyring.gpg >/dev/null
chmod go+r /etc/apt/keyrings/githubcli-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" > /etc/apt/sources.list.d/github-cli.list
apt-get update >/dev/null 2>&1 || true
apt-get install -y gh || { echo "Failed to install GitHub CLI (gh)" >&2; exit 1; }
fi
command -v gh >/dev/null 2>&1 || { echo "GitHub CLI (gh) is required but could not be installed" >&2; exit 1; }
- name: Clear caches - name: Clear caches
shell: bash shell: bash
run: | run: |
bash scripts/ccache-clear.sh \ CACHES=$(gh cache list --key "ccache-${{ inputs.key }}" --json id,key --jq '.[] | "\(.id) \(.key)"' 2>/dev/null)
--key "${{ inputs.key }}" \ if [ -z "$CACHES" ]; then
--older "${{ inputs.older }}" \ echo "No caches found with key prefix: ${{ inputs.key }}"
--min "${{ inputs.min }}" \ exit 0
${{ inputs.dry-run == 'true' && '--dry-run' || '' }} fi
while read -r id key; do
echo "Deleting cache: $id ($key)"
gh cache delete "$id"
done <<< "$CACHES"
+1 -3
View File
@@ -14,9 +14,7 @@ runs:
run: | run: |
BUILD_NUMBER="$(git rev-list --count HEAD)" BUILD_NUMBER="$(git rev-list --count HEAD)"
SHORT_HASH="$(git rev-parse --short=7 HEAD)" SHORT_HASH="$(git rev-parse --short=7 HEAD)"
if [[ "$GITHUB_REF_TYPE" == "tag" && "${{ env.BRANCH_NAME }}" =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]]; then if [[ "${{ env.BRANCH_NAME }}" == "master" ]]; then
echo "name=${{ env.BRANCH_NAME }}" >> $GITHUB_OUTPUT
elif [[ "${{ env.BRANCH_NAME }}" == "master" || "${{ env.BRANCH_NAME }}" == "b${BUILD_NUMBER}" ]]; then
echo "name=b${BUILD_NUMBER}" >> $GITHUB_OUTPUT echo "name=b${BUILD_NUMBER}" >> $GITHUB_OUTPUT
else else
SAFE_NAME=$(echo "${{ env.BRANCH_NAME }}" | tr '/' '-') SAFE_NAME=$(echo "${{ env.BRANCH_NAME }}" | tr '/' '-')
@@ -0,0 +1,20 @@
name: "Linux - Setup Vulkan SDK"
description: "Setup Vulkan SDK for Linux"
inputs:
path:
description: "Installation path"
required: true
version:
description: "Vulkan SDK version"
required: true
runs:
using: "composite"
steps:
- name: Setup Vulkan SDK
id: setup
uses: ./.github/actions/unarchive-tar
with:
url: https://sdk.lunarg.com/sdk/download/${{ inputs.version }}/linux/vulkan_sdk.tar.xz
path: ${{ inputs.path }}
strip: 1
+27 -63
View File
@@ -4,9 +4,6 @@ inputs:
cuda_version: cuda_version:
description: "CUDA toolkit version" description: "CUDA toolkit version"
required: true required: true
cuda_arch:
description: "CUDA target architecture"
required: true
runs: runs:
using: "composite" using: "composite"
@@ -25,9 +22,7 @@ runs:
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/visual_studio_integration/windows-x86_64/visual_studio_integration-windows-x86_64-11.7.91-archive.zip" curl -O "https://developer.download.nvidia.com/compute/cuda/redist/visual_studio_integration/windows-x86_64/visual_studio_integration-windows-x86_64-11.7.91-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_nvprof/windows-x86_64/cuda_nvprof-windows-x86_64-11.7.101-archive.zip" curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_nvprof/windows-x86_64/cuda_nvprof-windows-x86_64-11.7.101-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_cccl/windows-x86_64/cuda_cccl-windows-x86_64-11.7.91-archive.zip" curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_cccl/windows-x86_64/cuda_cccl-windows-x86_64-11.7.91-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_nvml_dev/windows-x86_64/cuda_nvml_dev-windows-x86_64-11.7.91-archive.zip"
unzip '*.zip' -d "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7" unzip '*.zip' -d "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7"
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7\cuda_nvml_dev-windows-x86_64-11.7.91-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7\cuda_cudart-windows-x86_64-11.7.99-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7" /E /I /H /Y xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7\cuda_cudart-windows-x86_64-11.7.99-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7\cuda_nvcc-windows-x86_64-11.7.99-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7" /E /I /H /Y xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7\cuda_nvcc-windows-x86_64-11.7.99-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7\cuda_nvrtc-windows-x86_64-11.7.99-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7" /E /I /H /Y xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7\cuda_nvrtc-windows-x86_64-11.7.99-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7" /E /I /H /Y
@@ -56,9 +51,7 @@ runs:
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/visual_studio_integration/windows-x86_64/visual_studio_integration-windows-x86_64-12.4.127-archive.zip" curl -O "https://developer.download.nvidia.com/compute/cuda/redist/visual_studio_integration/windows-x86_64/visual_studio_integration-windows-x86_64-12.4.127-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_nvprof/windows-x86_64/cuda_nvprof-windows-x86_64-12.4.127-archive.zip" curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_nvprof/windows-x86_64/cuda_nvprof-windows-x86_64-12.4.127-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_cccl/windows-x86_64/cuda_cccl-windows-x86_64-12.4.127-archive.zip" curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_cccl/windows-x86_64/cuda_cccl-windows-x86_64-12.4.127-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_nvml_dev/windows-x86_64/cuda_nvml_dev-windows-x86_64-12.4.127-archive.zip"
unzip '*.zip' -d "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4" unzip '*.zip' -d "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4"
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4\cuda_nvml_dev-windows-x86_64-12.4.127-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4\cuda_cudart-windows-x86_64-12.4.127-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4" /E /I /H /Y xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4\cuda_cudart-windows-x86_64-12.4.127-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4\cuda_nvcc-windows-x86_64-12.4.131-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4" /E /I /H /Y xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4\cuda_nvcc-windows-x86_64-12.4.131-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4\cuda_nvrtc-windows-x86_64-12.4.127-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4" /E /I /H /Y xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4\cuda_nvrtc-windows-x86_64-12.4.127-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4" /E /I /H /Y
@@ -89,9 +82,7 @@ runs:
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_profiler_api/windows-x86_64/cuda_profiler_api-windows-x86_64-13.1.80-archive.zip" curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_profiler_api/windows-x86_64/cuda_profiler_api-windows-x86_64-13.1.80-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/visual_studio_integration/windows-x86_64/visual_studio_integration-windows-x86_64-13.1.68-archive.zip" curl -O "https://developer.download.nvidia.com/compute/cuda/redist/visual_studio_integration/windows-x86_64/visual_studio_integration-windows-x86_64-13.1.68-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_cccl/windows-x86_64/cuda_cccl-windows-x86_64-13.1.78-archive.zip" curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_cccl/windows-x86_64/cuda_cccl-windows-x86_64-13.1.78-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_nvml_dev/windows-x86_64/cuda_nvml_dev-windows-x86_64-13.1.68-archive.zip"
unzip '*.zip' -d "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.1" unzip '*.zip' -d "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.1"
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.1\cuda_nvml_dev-windows-x86_64-13.1.68-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.1" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.1\cuda_crt-windows-x86_64-13.1.80-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.1" /E /I /H /Y xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.1\cuda_crt-windows-x86_64-13.1.80-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.1" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.1\cuda_cudart-windows-x86_64-13.1.80-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.1" /E /I /H /Y xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.1\cuda_cudart-windows-x86_64-13.1.80-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.1" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.1\cuda_nvcc-windows-x86_64-13.1.80-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.1" /E /I /H /Y xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.1\cuda_nvcc-windows-x86_64-13.1.80-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.1" /E /I /H /Y
@@ -106,60 +97,33 @@ runs:
echo "CUDA_PATH=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.1" | Out-File -FilePath $env:GITHUB_ENV -Append -Encoding utf8 echo "CUDA_PATH=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.1" | Out-File -FilePath $env:GITHUB_ENV -Append -Encoding utf8
echo "CUDA_PATH_V13_1=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.1" | Out-File -FilePath $env:GITHUB_ENV -Append -Encoding utf8 echo "CUDA_PATH_V13_1=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.1" | Out-File -FilePath $env:GITHUB_ENV -Append -Encoding utf8
- name: Install Cuda Toolkit 13.5 for x64 - name: Install Cuda Toolkit 13.3
if: ${{ inputs.cuda_version == '13.5' && inputs.cuda_arch == 'x64' }} if: ${{ inputs.cuda_version == '13.3' }}
shell: pwsh shell: pwsh
run: | run: |
mkdir -p "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5" mkdir -p "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.3"
choco install unzip -y choco install unzip -y
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_crt/windows-x86_64/cuda_crt-windows-x86_64-13.5.37-archive.zip" curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_crt/windows-x86_64/cuda_crt-windows-x86_64-13.3.33-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_cudart/windows-x86_64/cuda_cudart-windows-x86_64-13.5.37-archive.zip" curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_cudart/windows-x86_64/cuda_cudart-windows-x86_64-13.3.29-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_nvcc/windows-x86_64/cuda_nvcc-windows-x86_64-13.5.37-archive.zip" curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_nvcc/windows-x86_64/cuda_nvcc-windows-x86_64-13.3.33-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_nvrtc/windows-x86_64/cuda_nvrtc-windows-x86_64-13.5.37-archive.zip" curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_nvrtc/windows-x86_64/cuda_nvrtc-windows-x86_64-13.3.33-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/libcublas/windows-x86_64/libcublas-windows-x86_64-13.10.0.20-archive.zip" curl -O "https://developer.download.nvidia.com/compute/cuda/redist/libcublas/windows-x86_64/libcublas-windows-x86_64-13.5.1.27-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/libnvvm/windows-x86_64/libnvvm-windows-x86_64-13.5.37-archive.zip" curl -O "https://developer.download.nvidia.com/compute/cuda/redist/libnvvm/windows-x86_64/libnvvm-windows-x86_64-13.3.33-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_nvtx/windows-x86_64/cuda_nvtx-windows-x86_64-13.5.25-archive.zip" curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_nvtx/windows-x86_64/cuda_nvtx-windows-x86_64-13.3.29-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_profiler_api/windows-x86_64/cuda_profiler_api-windows-x86_64-13.5.25-archive.zip" curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_profiler_api/windows-x86_64/cuda_profiler_api-windows-x86_64-13.3.27-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/visual_studio_integration/windows-x86_64/visual_studio_integration-windows-x86_64-13.5.25-archive.zip" curl -O "https://developer.download.nvidia.com/compute/cuda/redist/visual_studio_integration/windows-x86_64/visual_studio_integration-windows-x86_64-13.3.27-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cccl/windows-x86_64/cccl-windows-x86_64-13.3.5.0.6-archive.zip" curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cccl/windows-x86_64/cccl-windows-x86_64-13.3.3.3.1-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_nvml_dev/windows-x86_64/cuda_nvml_dev-windows-x86_64-13.5.37-archive.zip" unzip '*.zip' -d "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.3"
unzip '*.zip' -d "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5" xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.3\cuda_crt-windows-x86_64-13.3.33-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.3" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5\cuda_nvml_dev-windows-x86_64-13.5.37-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5" /E /I /H /Y xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.3\cuda_cudart-windows-x86_64-13.3.29-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.3" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5\cuda_crt-windows-x86_64-13.5.37-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5" /E /I /H /Y xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.3\cuda_nvcc-windows-x86_64-13.3.33-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.3" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5\cuda_cudart-windows-x86_64-13.5.37-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5" /E /I /H /Y xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.3\cuda_nvrtc-windows-x86_64-13.3.33-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.3" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5\cuda_nvcc-windows-x86_64-13.5.37-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5" /E /I /H /Y xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.3\libcublas-windows-x86_64-13.5.1.27-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.3" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5\cuda_nvrtc-windows-x86_64-13.5.37-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5" /E /I /H /Y xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.3\libnvvm-windows-x86_64-13.3.33-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.3" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5\libcublas-windows-x86_64-13.10.0.20-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5" /E /I /H /Y xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.3\cuda_nvtx-windows-x86_64-13.3.29-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.3" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5\libnvvm-windows-x86_64-13.5.37-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5" /E /I /H /Y xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.3\cuda_profiler_api-windows-x86_64-13.3.27-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.3" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5\cuda_nvtx-windows-x86_64-13.5.25-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5" /E /I /H /Y xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.3\visual_studio_integration-windows-x86_64-13.3.27-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.3" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5\cuda_profiler_api-windows-x86_64-13.5.25-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5" /E /I /H /Y xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.3\cccl-windows-x86_64-13.3.3.3.1-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.3" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5\visual_studio_integration-windows-x86_64-13.5.25-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5" /E /I /H /Y echo "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.3\bin" | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5\cccl-windows-x86_64-13.3.5.0.6-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5" /E /I /H /Y echo "CUDA_PATH=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.3" | Out-File -FilePath $env:GITHUB_ENV -Append -Encoding utf8
echo "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5\bin" | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append echo "CUDA_PATH_V13_3=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.3" | Out-File -FilePath $env:GITHUB_ENV -Append -Encoding utf8
echo "CUDA_PATH=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5" | Out-File -FilePath $env:GITHUB_ENV -Append -Encoding utf8
echo "CUDA_PATH_V13_5=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5" | Out-File -FilePath $env:GITHUB_ENV -Append -Encoding utf8
- name: Install Cuda Toolkit 13.5 for ARM64
if: ${{ inputs.cuda_version == '13.5' && inputs.cuda_arch == 'arm64' }}
shell: pwsh
run: |
mkdir -p "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5"
choco install unzip -y
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cccl/windows-x86_64/cccl-windows-x86_64-13.3.5.0.6-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_crt/windows-x86_64/cuda_crt-windows-x86_64-13.5.37-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_nvcc/windows-x86_64/cuda_nvcc-windows-x86_64-13.5.37-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/libnvvm/windows-x86_64/libnvvm-windows-x86_64-13.5.37-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_cudart/windows-arm64/cuda_cudart-windows-arm64-13.5.37-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/libcublas/windows-arm64/libcublas-windows-arm64-13.10.0.20-archive.zip"
curl -O "https://developer.download.nvidia.com/compute/cuda/redist/cuda_nvml_dev/windows-arm64/cuda_nvml_dev-windows-arm64-13.5.37-archive.zip"
unzip '*.zip' -d "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5"
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5\cuda_nvml_dev-windows-arm64-13.5.37-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5\cccl-windows-x86_64-13.3.5.0.6-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5\cuda_crt-windows-x86_64-13.5.37-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5\cuda_nvcc-windows-x86_64-13.5.37-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5\libnvvm-windows-x86_64-13.5.37-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5\cuda_cudart-windows-arm64-13.5.37-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5" /E /I /H /Y
xcopy "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5\libcublas-windows-arm64-13.10.0.20-archive\*" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5" /E /I /H /Y
echo "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5\bin" | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append
echo "CUDA_PATH=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5" | Out-File -FilePath $env:GITHUB_ENV -Append -Encoding utf8
echo "CUDA_PATH_V13_5=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.5" | Out-File -FilePath $env:GITHUB_ENV -Append -Encoding utf8
+5 -23
View File
@@ -8,26 +8,8 @@ inputs:
runs: runs:
using: "composite" using: "composite"
steps: steps:
- name: Install ROCm with Wheels - name: Setup ROCm
shell: pwsh uses: ./.github/actions/install-exe
run: | with:
$ErrorActionPreference = "Stop" url: https://download.amd.com/developer/eula/rocm-hub/AMD-Software-PRO-Edition-${{ inputs.version }}-Win11-For-HIP.exe
write-host "Setting up Python virtual environment" args: -install
# Create the venv directly at the cache location to avoid relocation issues
New-Item -Path "C:\TheRock\build" -ItemType Directory -Force | Out-Null
python -m venv C:\TheRock\build\.venv
& C:\TheRock\build\.venv\Scripts\Activate.ps1
write-host "Upgrading pip"
python -m pip install --upgrade pip
write-host "Installing ROCm wheels for multi-arch support"
# Install ROCm wheels for multi-arch support (this may take several minutes)
python -m pip install --index-url https://stable.repo.amd.com/rocm/whl-next/ "rocm[libraries,devel]==${{ inputs.version }}"
# Pre-expand the devel tree so it is included in the cache
write-host "Initializing ROCm devel tree"
rocm-sdk init
if ($LASTEXITCODE -ne 0) { throw "rocm-sdk init failed with exit code $LASTEXITCODE" }
write-host "Completed ROCm wheel installation to C:\TheRock\build"
+2 -1
View File
@@ -48,6 +48,7 @@ examples:
- all-globs-to-all-files: - all-globs-to-all-files:
- '!tools/server/**' - '!tools/server/**'
- '!tools/mtmd/**' - '!tools/mtmd/**'
- '!tools/ui/**'
testing: testing:
- changed-files: - changed-files:
- any-glob-to-any-file: - any-glob-to-any-file:
@@ -71,7 +72,7 @@ android:
server/ui: server/ui:
- changed-files: - changed-files:
- any-glob-to-any-file: - any-glob-to-any-file:
- ui/** - tools/ui/**
server: server:
- changed-files: - changed-files:
- any-glob-to-any-file: - any-glob-to-any-file:
-4
View File
@@ -4,10 +4,6 @@ on:
issues: issues:
types: [opened] types: [opened]
cache-mode: none
permissions:
contents: read
jobs: jobs:
find-related: find-related:
if: github.event.action == 'opened' if: github.event.action == 'opened'
-177
View File
@@ -1,177 +0,0 @@
name: Behavior Check
on:
workflow_dispatch: # allows manual triggering
push:
branches:
- master
paths: [
'.github/workflows/behavior-check.yml',
'**/CMakeLists.txt',
'**/.cmake',
'**/*.h',
'**/*.hpp',
'**/*.c',
'**/*.cpp',
'**/*.py',
'**/requirements*.txt',
]
pull_request:
types: [opened, synchronize, reopened]
paths: [
'.github/workflows/behavior-check.yml',
'**/CMakeLists.txt',
'**/.cmake',
'**/*.h',
'**/*.hpp',
'**/*.c',
'**/*.cpp',
'**/*.py',
'**/requirements*.txt',
]
cache-mode: none
permissions:
contents: read
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}
cancel-in-progress: true
env:
GGML_NLOOP: 3
GGML_N_THREADS: 1
GGML_SCHED_DEBUG_REALLOC: 1
LLAMA_ARG_LOG_COLORS: 1
LLAMA_ARG_LOG_PREFIX: 1
LLAMA_ARG_LOG_TIMESTAMPS: 1
jobs:
ubuntu:
runs-on: ubuntu-26.04-arm
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: ccache
uses: ggml-org/ccache-action@v1.2.24
with:
restore: false
save: false
- name: Build Dependencies
id: build_depends
run: |
sudo apt update
sudo apt install -y build-essential libssl-dev git-lfs libjpeg-dev python3 python3-venv python3-pip
- name: Python Dependencies
id: python_depends
run: |
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements-convert_hf_to_gguf.txt
- name: ccache-buckets-restore
uses: ./.github/actions/ccache-buckets
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CI }}
with:
key: behavior-check-ubuntu-26.04-arm
folder: llama.cpp
hf_bucket: ggml-org/cache
- name: Build
id: cmake_build
run: |
cmake -B build \
-DGGML_NATIVE=OFF \
-DGGML_BACKEND_DL=OFF \
-DLLAMA_FATAL_WARNINGS=ON
time cmake --build build --config Release -j $(nproc)
- name: ccache-buckets-save
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
uses: ./.github/actions/ccache-buckets
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CACHE_OUTPUT }}
with:
key: behavior-check-ubuntu-26.04-arm
folder: llama.cpp
evict-old-files: 1d
hf_bucket: ggml-org/cache
save: true
- name: Test
id: cmake_test
run: |
source .venv/bin/activate
cd build
ctest -L behavior -LE 'model|backend' --verbose --timeout 900
windows:
runs-on: windows-2025
cache-mode: write
permissions:
actions: write
contents: read
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: ccache
uses: ggml-org/ccache-action@v1.2.24
with:
key: behavior-check-windows-2025
evict-old-files: 1d
save: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
- name: Python Dependencies
id: python_depends
shell: bash
run: |
python3 -m venv .venv
source .venv/Scripts/activate
pip install -r requirements/requirements-convert_hf_to_gguf.txt
- name: Install LLVM
run: |
choco install llvm --version=22.1.8
echo "C:\Program Files\LLVM\bin" >> $env:GITHUB_PATH
- name: Build
id: cmake_build
shell: cmd
run: |
call "C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64
cmake -B build -G "Ninja Multi-Config" ^
-DCMAKE_TOOLCHAIN_FILE=cmake/x64-windows-llvm.cmake ^
-DCMAKE_BUILD_TYPE=Release ^
-DGGML_NATIVE=OFF ^
-DGGML_BACKEND_DL=OFF
set /A NINJA_JOBS=%NUMBER_OF_PROCESSORS%-1
cmake --build build --config Release -j %NINJA_JOBS%
- name: Test
id: cmake_test
shell: bash
run: |
source .venv/Scripts/activate
cd build
ctest -C Release -L behavior -LE 'model|backend' --verbose --timeout 900
- name: ccache-clear
uses: ./.github/actions/ccache-clear
env:
GH_TOKEN: ${{ github.token }}
with:
key: behavior-check-windows-2025
older: 5m
min: 1
dry-run: ${{ github.event_name != 'push' || github.ref != 'refs/heads/master' }}
+2 -6
View File
@@ -15,10 +15,6 @@ on:
'**/*.cpp' '**/*.cpp'
] ]
cache-mode: none
permissions:
contents: read
concurrency: concurrency:
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }} group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}
cancel-in-progress: true cancel-in-progress: true
@@ -31,8 +27,8 @@ env:
LLAMA_ARG_LOG_TIMESTAMPS: 1 LLAMA_ARG_LOG_TIMESTAMPS: 1
jobs: jobs:
ubuntu-26-llguidance: ubuntu-24-llguidance:
runs-on: ${{ 'ubuntu-26.04-arm' || 'ubuntu-26.04' }} runs-on: ${{ 'ubuntu-24.04-arm' || 'ubuntu-24.04' }}
steps: steps:
- name: Clone - name: Clone
@@ -23,27 +23,19 @@ on:
- 'scripts/snapdragon/**' - 'scripts/snapdragon/**'
- 'CMakePresets.json' - 'CMakePresets.json'
cache-mode: none
permissions:
contents: read
concurrency: concurrency:
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }} group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}
cancel-in-progress: true cancel-in-progress: true
jobs: jobs:
android-ndk-snapdragon: android-ndk-snapdragon:
runs-on: ubuntu-26.04 runs-on: ubuntu-latest
container: container:
image: 'ghcr.io/snapdragon-toolchain/arm64-android:v0.8' image: 'ghcr.io/snapdragon-toolchain/arm64-android:v0.7'
defaults: defaults:
run: run:
shell: bash shell: bash
permissions:
actions: write
contents: read
steps: steps:
- name: Clone - name: Clone
uses: actions/checkout@v6 uses: actions/checkout@v6
@@ -67,17 +59,13 @@ jobs:
path: pkg-snapdragon/llama.cpp path: pkg-snapdragon/llama.cpp
linux-iot-snapdragon: linux-iot-snapdragon:
runs-on: ubuntu-26.04 runs-on: ubuntu-latest
container: container:
image: 'ghcr.io/snapdragon-toolchain/arm64-linux:v0.8' image: 'ghcr.io/snapdragon-toolchain/arm64-linux:v0.7'
defaults: defaults:
run: run:
shell: bash shell: bash
permissions:
actions: write
contents: read
steps: steps:
- name: Clone - name: Clone
uses: actions/checkout@v6 uses: actions/checkout@v6
@@ -103,16 +91,12 @@ jobs:
test-snapdragon-qdc: test-snapdragon-qdc:
name: Test on QDC Device (${{ matrix.device }}) name: Test on QDC Device (${{ matrix.device }})
needs: [android-ndk-snapdragon, linux-iot-snapdragon] needs: [android-ndk-snapdragon, linux-iot-snapdragon]
runs-on: ubuntu-26.04-arm runs-on: ubuntu-24.04-arm
timeout-minutes: 180 timeout-minutes: 90
strategy: strategy:
fail-fast: false fail-fast: false
matrix: matrix:
device: [SM8850, QCS9075M] device: [SM8750, SM8850, QCS9075M]
permissions:
actions: read
contents: read
steps: steps:
- name: Checkout - name: Checkout
@@ -155,7 +139,7 @@ jobs:
--pkg-dir pkg-snapdragon/llama.cpp \ --pkg-dir pkg-snapdragon/llama.cpp \
--model-url "https://huggingface.co/bartowski/Llama-3.2-1B-Instruct-GGUF/resolve/main/Llama-3.2-1B-Instruct-Q4_0.gguf" \ --model-url "https://huggingface.co/bartowski/Llama-3.2-1B-Instruct-GGUF/resolve/main/Llama-3.2-1B-Instruct-Q4_0.gguf" \
--device ${{ matrix.device }} \ --device ${{ matrix.device }} \
${{ startsWith(matrix.device, 'QCS') && '--job-timeout 5400 --retries 2 --retry-delay 300' || '' }} ${{ startsWith(matrix.device, 'QCS') && '--retries 2 --retry-delay 300' || '' }}
env: env:
QDC_API_KEY: ${{ secrets.QDC_API_KEY }} QDC_API_KEY: ${{ secrets.QDC_API_KEY }}
+6 -15
View File
@@ -20,10 +20,6 @@ on:
- '.github/workflows/build-android.yml' - '.github/workflows/build-android.yml'
- 'examples/llama.android/**' - 'examples/llama.android/**'
cache-mode: none
permissions:
contents: read
concurrency: concurrency:
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }} group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}
cancel-in-progress: true cancel-in-progress: true
@@ -37,7 +33,7 @@ env:
jobs: jobs:
default: default:
runs-on: ubuntu-26.04 runs-on: ubuntu-latest
steps: steps:
- name: Clone - name: Clone
@@ -53,10 +49,9 @@ jobs:
distribution: zulu distribution: zulu
- name: Setup Android SDK - name: Setup Android SDK
uses: android-actions/setup-android@be39fa834029ff78f1a44aa3bb0819b8fc2bd8fd # v4.0.4 uses: android-actions/setup-android@40fd30fb8d7440372e1316f5d1809ec01dcd3699 # v4.0.1
with: with:
log-accepted-android-sdk-licenses: false log-accepted-android-sdk-licenses: false
packages: platform-tools cmake;3.31.6
- name: Build - name: Build
run: | run: |
@@ -64,17 +59,13 @@ jobs:
./gradlew build --no-daemon ./gradlew build --no-daemon
ndk: ndk:
runs-on: ubuntu-26.04 runs-on: ubuntu-latest
container: container:
image: 'ghcr.io/snapdragon-toolchain/arm64-android:v0.3' image: 'ghcr.io/snapdragon-toolchain/arm64-android:v0.3'
defaults: defaults:
run: run:
shell: bash shell: bash
permissions:
actions: write
contents: read
steps: steps:
- name: Clone - name: Clone
uses: actions/checkout@v6 uses: actions/checkout@v6
@@ -102,7 +93,7 @@ jobs:
path: pkg-adb/llama.cpp path: pkg-adb/llama.cpp
arm64: arm64:
runs-on: ubuntu-26.04 runs-on: ubuntu-latest
env: env:
NDK_VERSION: "29.0.14206865" NDK_VERSION: "29.0.14206865"
@@ -119,7 +110,7 @@ jobs:
# cache on: https://github.com/ggerganov/tmp2/actions/runs/26534713799/job/78224189394 # cache on: https://github.com/ggerganov/tmp2/actions/runs/26534713799/job/78224189394
# #
#- name: ccache #- name: ccache
# uses: ggml-org/ccache-action@v1.2.24 # uses: ggml-org/ccache-action@v1.2.21
# with: # with:
# key: android-ubuntu-arm64 # key: android-ubuntu-arm64
# evict-old-files: 1d # evict-old-files: 1d
@@ -132,7 +123,7 @@ jobs:
distribution: temurin distribution: temurin
- name: Setup Android SDK - name: Setup Android SDK
uses: android-actions/setup-android@be39fa834029ff78f1a44aa3bb0819b8fc2bd8fd # v4.0.4 uses: android-actions/setup-android@40fd30fb8d7440372e1316f5d1809ec01dcd3699 # v4.0.1
with: with:
log-accepted-android-sdk-licenses: false log-accepted-android-sdk-licenses: false
-160
View File
@@ -1,160 +0,0 @@
name: CI (apple - daily)
on:
workflow_dispatch: # allows manual triggering
schedule:
- cron: '0 0 * * *'
cache-mode: none
permissions:
contents: read
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}
cancel-in-progress: true
env:
GGML_NLOOP: 3
GGML_N_THREADS: 1
LLAMA_ARG_LOG_COLORS: 1
LLAMA_ARG_LOG_PREFIX: 1
LLAMA_ARG_LOG_TIMESTAMPS: 1
jobs:
macos-latest-ios-xcode:
runs-on: macos-latest
permissions:
actions: write
contents: read
steps:
- name: Checkout code
uses: actions/checkout@v6
- name: Setup Xcode
uses: ggml-org/setup-xcode@v1
with:
xcode-version: latest-stable
- name: Build
id: cmake_build
run: |
sysctl -a
cmake -B build -G Xcode \
-DGGML_METAL_EMBED_LIBRARY=ON \
-DLLAMA_OPENSSL=OFF \
-DLLAMA_BUILD_APP=OFF \
-DLLAMA_BUILD_EXAMPLES=OFF \
-DLLAMA_BUILD_TOOLS=OFF \
-DLLAMA_BUILD_TESTS=OFF \
-DLLAMA_BUILD_SERVER=OFF \
-DCMAKE_SYSTEM_NAME=iOS \
-DCMAKE_OSX_DEPLOYMENT_TARGET=14.0 \
-DCMAKE_XCODE_ATTRIBUTE_DEVELOPMENT_TEAM=ggml
cmake --build build --config Release -j $(sysctl -n hw.logicalcpu) -- CODE_SIGNING_ALLOWED=NO
- name: xcodebuild for swift package
id: xcodebuild
run: |
./build-xcframework.sh
- name: Upload xcframework artifact
uses: actions/upload-artifact@v6
with:
name: llama-xcframework
path: build-apple/llama.xcframework/
retention-days: 1
- name: Build Xcode project
run: |
xcodebuild -downloadPlatform iOS
xcodebuild -project examples/llama.swiftui/llama.swiftui.xcodeproj -scheme llama.swiftui -sdk iphoneos CODE_SIGNING_REQUIRED=NO CODE_SIGN_IDENTITY= -destination 'generic/platform=iOS' FRAMEWORK_FOLDER_PATH=./build-ios build
macos-latest-tvos:
runs-on: macos-latest
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: Build
id: cmake_build
run: |
sysctl -a
cmake -B build -G Xcode \
-DGGML_METAL_EMBED_LIBRARY=ON \
-DLLAMA_BUILD_COMMON=OFF \
-DLLAMA_BUILD_APP=OFF \
-DLLAMA_BUILD_EXAMPLES=OFF \
-DLLAMA_BUILD_TOOLS=OFF \
-DLLAMA_BUILD_TESTS=OFF \
-DLLAMA_BUILD_SERVER=OFF \
-DCMAKE_SYSTEM_NAME=tvOS \
-DCMAKE_OSX_DEPLOYMENT_TARGET=14.0 \
-DCMAKE_XCODE_ATTRIBUTE_DEVELOPMENT_TEAM=ggml
cmake --build build --config Release -j $(sysctl -n hw.logicalcpu) -- CODE_SIGNING_ALLOWED=NO
macos-latest-visionos:
runs-on: macos-latest
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: Build
id: cmake_build
run: |
sysctl -a
cmake -B build -G Xcode \
-DGGML_METAL_EMBED_LIBRARY=ON \
-DLLAMA_BUILD_COMMON=OFF \
-DLLAMA_BUILD_APP=OFF \
-DLLAMA_BUILD_EXAMPLES=OFF \
-DLLAMA_BUILD_TOOLS=OFF \
-DLLAMA_BUILD_TESTS=OFF \
-DLLAMA_BUILD_SERVER=OFF \
-DCMAKE_SYSTEM_NAME=visionOS \
-DCMAKE_OSX_DEPLOYMENT_TARGET=1.0 \
-DCMAKE_XCODE_ATTRIBUTE_DEVELOPMENT_TEAM=ggml
cmake --build build --config Release -j $(sysctl -n hw.logicalcpu) -- CODE_SIGNING_ALLOWED=NO
macos-latest-swift:
runs-on: macos-latest
needs: macos-latest-ios-xcode
permissions:
actions: read
contents: read
strategy:
matrix:
destination: ['generic/platform=macOS', 'generic/platform=iOS', 'generic/platform=tvOS']
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: Download xcframework artifact
uses: actions/download-artifact@v7
with:
name: llama-xcframework
path: build-apple/llama.xcframework/
- name: Build llama.cpp with CMake
id: cmake_build
run: |
sysctl -a
cmake -B build -G Xcode \
-DGGML_METAL_EMBED_LIBRARY=ON \
-DLLAMA_OPENSSL=OFF \
-DLLAMA_BUILD_APP=OFF \
-DLLAMA_BUILD_EXAMPLES=OFF \
-DLLAMA_BUILD_TOOLS=OFF \
-DLLAMA_BUILD_TESTS=OFF \
-DLLAMA_BUILD_SERVER=OFF \
-DCMAKE_OSX_ARCHITECTURES="arm64;x86_64"
cmake --build build --config Release -j $(sysctl -n hw.logicalcpu)
+201 -42
View File
@@ -22,14 +22,9 @@ on:
types: [opened, synchronize, reopened] types: [opened, synchronize, reopened]
paths: [ paths: [
'.github/workflows/build-apple.yml', '.github/workflows/build-apple.yml',
'ggml/src/ggml-metal/**', 'ggml/src/ggml-metal/**'
'ggml/src/ggml-rpc/**'
] ]
cache-mode: none
permissions:
contents: read
concurrency: concurrency:
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }} group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}
cancel-in-progress: true cancel-in-progress: true
@@ -37,7 +32,6 @@ concurrency:
env: env:
GGML_NLOOP: 3 GGML_NLOOP: 3
GGML_N_THREADS: 1 GGML_N_THREADS: 1
GGML_SCHED_DEBUG_REALLOC: 1
LLAMA_ARG_LOG_COLORS: 1 LLAMA_ARG_LOG_COLORS: 1
LLAMA_ARG_LOG_PREFIX: 1 LLAMA_ARG_LOG_PREFIX: 1
LLAMA_ARG_LOG_TIMESTAMPS: 1 LLAMA_ARG_LOG_TIMESTAMPS: 1
@@ -52,19 +46,11 @@ jobs:
uses: actions/checkout@v6 uses: actions/checkout@v6
- name: ccache - name: ccache
uses: ggml-org/ccache-action@v1.2.24 uses: ggml-org/ccache-action@v1.2.21
with:
restore: false
save: false
- name: ccache-buckets-restore
uses: ./.github/actions/ccache-buckets
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CI }}
with: with:
key: apple-arm64 key: apple-arm64
folder: llama.cpp evict-old-files: 1d
hf_bucket: ggml-org/cache save: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
- name: Build - name: Build
id: cmake_build id: cmake_build
@@ -74,36 +60,209 @@ jobs:
-DCMAKE_BUILD_RPATH="@loader_path" \ -DCMAKE_BUILD_RPATH="@loader_path" \
-DLLAMA_FATAL_WARNINGS=ON \ -DLLAMA_FATAL_WARNINGS=ON \
-DLLAMA_BUILD_BORINGSSL=ON \ -DLLAMA_BUILD_BORINGSSL=ON \
-DGGML_METAL_USE_BF16=ON \
-DGGML_METAL_EMBED_LIBRARY=OFF \ -DGGML_METAL_EMBED_LIBRARY=OFF \
-DGGML_METAL_SHADER_DEBUG=ON \ -DGGML_METAL_SHADER_DEBUG=ON \
-DGGML_RPC=ON \ -DGGML_RPC=ON
-DCMAKE_OSX_DEPLOYMENT_TARGET=13.3
time cmake --build build --config Release -j $(sysctl -n hw.logicalcpu) time cmake --build build --config Release -j $(sysctl -n hw.logicalcpu)
leaks -atExit -- ./build/bin/test-thread-safety -hf ggml-org/gemma-3-270m-qat-GGUF -ngl 99 -p "$(printf 'hello %.0s' {1..128})" -n 16 -c 512 -ub 32 -np 2 -t 2 -lv 1
- name: ccache-buckets-save
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
uses: ./.github/actions/ccache-buckets
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CACHE_OUTPUT }}
with:
key: apple-arm64
folder: llama.cpp
evict-old-files: 1d
hf_bucket: ggml-org/cache
save: true
- name: Check for leaks
run: |
cmd=(./build/bin/test-thread-safety -hf ggml-org/gemma-3-270m-qat-GGUF -ngl 99 -p "$(printf 'hello %.0s' {1..128})" -n 16 -c 512 -ub 32 -np 2 -t 2 -lv 1)
leaks -atExit -- "${cmd[@]}"
# Graphics devices are leaked by Metal in Apple code sometimes, so we ignore those leaks
OBJC_DEBUG_MISSING_POOLS=YES "${cmd[@]}" 2>&1 | awk '{ print } index($0, "autoreleased with no pool in place") && !/class [a-zA-Z0-9]+Device autoreleased/ { found = 1 } END { exit found }'
- name: Test - name: Test
id: cmake_test id: cmake_test
run: | run: |
cd build cd build
# Metal Paravirtual devices are difficult to support -> disable ctest -L main -E "test-llama-archs" --verbose --timeout 900
# ref: https://github.com/ggml-org/llama.cpp/pull/19802#issuecomment-4013704023
ctest -L main -E "test-llama-archs|test-llama-context|test-speculative-rejection-sampling" --verbose --timeout 900
macos-latest-x64:
runs-on: macos-15-intel
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: ccache
uses: ggml-org/ccache-action@v1.2.21
with:
key: apple-x64
evict-old-files: 1d
save: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
- name: Build
id: cmake_build
run: |
sysctl -a
# Metal is disabled due to intermittent failures with Github runners not having a GPU:
# https://github.com/ggml-org/llama.cpp/actions/runs/8635935781/job/23674807267#step:5:2313
cmake -B build \
-DCMAKE_BUILD_RPATH="@loader_path" \
-DLLAMA_FATAL_WARNINGS=ON \
-DLLAMA_BUILD_BORINGSSL=ON \
-DGGML_METAL=OFF \
-DGGML_RPC=ON \
-DCMAKE_OSX_DEPLOYMENT_TARGET=13.3
time cmake --build build --config Release -j $(sysctl -n hw.logicalcpu)
- name: Test
id: cmake_test
run: |
cd build
ctest -L main --verbose --timeout 900
macos-latest-ios-xcode:
runs-on: macos-latest
steps:
- name: Checkout code
uses: actions/checkout@v6
- name: Setup Xcode
uses: ggml-org/setup-xcode@v1
with:
xcode-version: latest-stable
- name: Build
id: cmake_build
run: |
sysctl -a
cmake -B build -G Xcode \
-DGGML_METAL_USE_BF16=ON \
-DGGML_METAL_EMBED_LIBRARY=ON \
-DLLAMA_OPENSSL=OFF \
-DLLAMA_BUILD_APP=OFF \
-DLLAMA_BUILD_EXAMPLES=OFF \
-DLLAMA_BUILD_TOOLS=OFF \
-DLLAMA_BUILD_TESTS=OFF \
-DLLAMA_BUILD_SERVER=OFF \
-DCMAKE_SYSTEM_NAME=iOS \
-DCMAKE_OSX_DEPLOYMENT_TARGET=14.0 \
-DCMAKE_XCODE_ATTRIBUTE_DEVELOPMENT_TEAM=ggml
cmake --build build --config Release -j $(sysctl -n hw.logicalcpu) -- CODE_SIGNING_ALLOWED=NO
- name: xcodebuild for swift package
id: xcodebuild
run: |
./build-xcframework.sh
- name: Upload xcframework artifact
uses: actions/upload-artifact@v6
with:
name: llama-xcframework
path: build-apple/llama.xcframework/
retention-days: 1
- name: Build Xcode project
run: |
xcodebuild -downloadPlatform iOS
xcodebuild -project examples/llama.swiftui/llama.swiftui.xcodeproj -scheme llama.swiftui -sdk iphoneos CODE_SIGNING_REQUIRED=NO CODE_SIGN_IDENTITY= -destination 'generic/platform=iOS' FRAMEWORK_FOLDER_PATH=./build-ios build
macos-latest-tvos:
runs-on: macos-latest
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
# TODO: this likely does not do anything - if yes, remove it
- name: ccache
uses: ggml-org/ccache-action@v1.2.21
with:
key: apple-tvos
evict-old-files: 1d
save: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
- name: Build
id: cmake_build
run: |
sysctl -a
cmake -B build -G Xcode \
-DGGML_METAL_USE_BF16=ON \
-DGGML_METAL_EMBED_LIBRARY=ON \
-DLLAMA_BUILD_COMMON=OFF \
-DLLAMA_BUILD_APP=OFF \
-DLLAMA_BUILD_EXAMPLES=OFF \
-DLLAMA_BUILD_TOOLS=OFF \
-DLLAMA_BUILD_TESTS=OFF \
-DLLAMA_BUILD_SERVER=OFF \
-DCMAKE_SYSTEM_NAME=tvOS \
-DCMAKE_OSX_DEPLOYMENT_TARGET=14.0 \
-DCMAKE_XCODE_ATTRIBUTE_DEVELOPMENT_TEAM=ggml
cmake --build build --config Release -j $(sysctl -n hw.logicalcpu) -- CODE_SIGNING_ALLOWED=NO
macos-latest-visionos:
runs-on: macos-latest
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
# TODO: this likely does not do anything - if yes, remove it
- name: ccache
uses: ggml-org/ccache-action@v1.2.21
with:
key: apple-visionos
evict-old-files: 1d
save: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
- name: Build
id: cmake_build
run: |
sysctl -a
cmake -B build -G Xcode \
-DGGML_METAL_USE_BF16=ON \
-DGGML_METAL_EMBED_LIBRARY=ON \
-DLLAMA_BUILD_COMMON=OFF \
-DLLAMA_BUILD_APP=OFF \
-DLLAMA_BUILD_EXAMPLES=OFF \
-DLLAMA_BUILD_TOOLS=OFF \
-DLLAMA_BUILD_TESTS=OFF \
-DLLAMA_BUILD_SERVER=OFF \
-DCMAKE_SYSTEM_NAME=visionOS \
-DCMAKE_OSX_DEPLOYMENT_TARGET=1.0 \
-DCMAKE_XCODE_ATTRIBUTE_DEVELOPMENT_TEAM=ggml
cmake --build build --config Release -j $(sysctl -n hw.logicalcpu) -- CODE_SIGNING_ALLOWED=NO
macos-latest-swift:
runs-on: macos-latest
needs: macos-latest-ios-xcode
strategy:
matrix:
destination: ['generic/platform=macOS', 'generic/platform=iOS', 'generic/platform=tvOS']
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
# TODO: this likely does not do anything - if yes, remove it
- name: ccache
uses: ggml-org/ccache-action@v1.2.21
with:
key: apple-swift
evict-old-files: 1d
save: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
- name: Download xcframework artifact
uses: actions/download-artifact@v7
with:
name: llama-xcframework
path: build-apple/llama.xcframework/
- name: Build llama.cpp with CMake
id: cmake_build
run: |
sysctl -a
cmake -B build -G Xcode \
-DGGML_METAL_USE_BF16=ON \
-DGGML_METAL_EMBED_LIBRARY=ON \
-DLLAMA_OPENSSL=OFF \
-DLLAMA_BUILD_APP=OFF \
-DLLAMA_BUILD_EXAMPLES=OFF \
-DLLAMA_BUILD_TOOLS=OFF \
-DLLAMA_BUILD_TESTS=OFF \
-DLLAMA_BUILD_SERVER=OFF \
-DCMAKE_OSX_ARCHITECTURES="arm64;x86_64"
cmake --build build --config Release -j $(sysctl -n hw.logicalcpu)
+55 -32
View File
@@ -5,15 +5,38 @@ on:
schedule: schedule:
- cron: '0 * * * *' - cron: '0 * * * *'
cache-mode: write
permissions:
contents: read
concurrency: concurrency:
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }} group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}
cancel-in-progress: true cancel-in-progress: true
jobs: jobs:
ubuntu-24-vulkan-cache:
runs-on: ubuntu-24.04
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: Get latest Vulkan SDK version
id: vulkan_sdk_version
run: |
echo "VULKAN_SDK_VERSION=$(curl https://vulkan.lunarg.com/sdk/latest/linux.txt)" >> "$GITHUB_ENV"
- name: Setup Cache
uses: actions/cache@v5
id: cache-sdk
with:
path: ./vulkan_sdk
key: cache-gha-vulkan-sdk-${{ env.VULKAN_SDK_VERSION }}-${{ runner.os }}
- name: Setup Vulkan SDK
if: steps.cache-sdk.outputs.cache-hit != 'true'
uses: ./.github/actions/linux-setup-vulkan
with:
path: ./vulkan_sdk
version: ${{ env.VULKAN_SDK_VERSION }}
#ubuntu-24-spacemit-cache: #ubuntu-24-spacemit-cache:
# runs-on: ubuntu-24.04 # runs-on: ubuntu-24.04
@@ -40,13 +63,13 @@ jobs:
# path: ./spacemit_toolchain # path: ./spacemit_toolchain
# version: ${{ env.SPACEMIT_IME_TOOLCHAIN_VERSION }} # version: ${{ env.SPACEMIT_IME_TOOLCHAIN_VERSION }}
ubuntu-26-openvino-cache: ubuntu-24-openvino-cache:
runs-on: ubuntu-26.04 runs-on: ubuntu-24.04
env: env:
# Sync versions in build-openvino.yml, ci-self-hosted-openvino.yml, release.yml, release-publish.yml, build-cache.yml, .devops/openvino.Dockerfile, docs/backend/OPENVINO.md # Sync versions in build.yml, build-self-hosted.yml, release.yml, build-cache.yml, .devops/openvino.Dockerfile
OPENVINO_VERSION_MAJOR: "2026.4.1" OPENVINO_VERSION_MAJOR: "2026.2.1"
OPENVINO_VERSION_FULL: "2026.4.1.22982.07f9c262b05" OPENVINO_VERSION_FULL: "2026.2.1.21919.ede283a88e3"
steps: steps:
- name: Clone - name: Clone
@@ -72,9 +95,9 @@ jobs:
runs-on: windows-2022 runs-on: windows-2022
env: env:
# Sync versions in build-openvino.yml, ci-self-hosted-openvino.yml, release.yml, release-publish.yml, build-cache.yml, .devops/openvino.Dockerfile, docs/backend/OPENVINO.md # Sync versions in build.yml, build-self-hosted.yml, release.yml, build-cache.yml, .devops/openvino.Dockerfile
OPENVINO_VERSION_MAJOR: "2026.4.1" OPENVINO_VERSION_MAJOR: "2026.2.1"
OPENVINO_VERSION_FULL: "2026.4.1.22982.07f9c262b05" OPENVINO_VERSION_FULL: "2026.2.1.21919.ede283a88e3"
steps: steps:
- name: Clone - name: Clone
@@ -96,27 +119,27 @@ jobs:
version_major: ${{ env.OPENVINO_VERSION_MAJOR }} version_major: ${{ env.OPENVINO_VERSION_MAJOR }}
version_full: ${{ env.OPENVINO_VERSION_FULL }} version_full: ${{ env.OPENVINO_VERSION_FULL }}
# windows-2022-rocm-cache: windows-2022-rocm-cache:
# runs-on: windows-2022 runs-on: windows-2022
# env: env:
# # Make sure this is in sync with release.yml and build-cuda-windows.yml # Make sure this is in sync with build.yml
# ROCM_VERSION: "7.14.0" HIPSDK_INSTALLER_VERSION: "26.Q1"
# steps: steps:
# - name: Clone - name: Clone
# id: checkout id: checkout
# uses: actions/checkout@v6 uses: actions/checkout@v6
# - name: Setup Cache - name: Setup Cache
# uses: actions/cache@v5 uses: actions/cache@v5
# id: cache-rocm id: cache-rocm
# with: with:
# path: C:\TheRock\build path: C:\Program Files\AMD\ROCm
# key: rocm-wheels-${{ env.ROCM_VERSION }}-multi-arch-${{ runner.os }} key: cache-gha-rocm-${{ env.HIPSDK_INSTALLER_VERSION }}-${{ runner.os }}
# - name: Setup ROCm - name: Setup ROCm
# if: steps.cache-rocm.outputs.cache-hit != 'true' if: steps.cache-rocm.outputs.cache-hit != 'true'
# uses: ./.github/actions/windows-setup-rocm uses: ./.github/actions/windows-setup-rocm
# with: with:
# version: ${{ env.ROCM_VERSION }} version: ${{ env.HIPSDK_INSTALLER_VERSION }}
-4
View File
@@ -22,10 +22,6 @@ on:
'ggml/src/ggml-cann/**' 'ggml/src/ggml-cann/**'
] ]
cache-mode: none
permissions:
contents: read
concurrency: concurrency:
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }} group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}
cancel-in-progress: true cancel-in-progress: true
+51
View File
@@ -0,0 +1,51 @@
name: Build relocatable cmake package
on:
workflow_dispatch:
workflow_call:
jobs:
linux:
runs-on: [self-hosted, Linux, CPU]
steps:
- uses: actions/checkout@v6
with:
fetch-depth: 0
- name: Build
run: |
PREFIX="$(pwd)"/inst
cmake -S . -B build \
-DCMAKE_PREFIX_PATH="$PREFIX" \
-DLLAMA_OPENSSL=OFF \
-DLLAMA_BUILD_TESTS=OFF \
-DLLAMA_BUILD_TOOLS=OFF \
-DLLAMA_BUILD_EXAMPLES=OFF \
-DLLAMA_BUILD_APP=OFF \
-DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release
cmake --install build --prefix "$PREFIX" --config Release
export LLAMA_CONFIG="$PREFIX"/lib/cmake/llama/llama-config.cmake
tclsh <<'EOF'
set build(commit) [string trim [exec git rev-parse --short HEAD]]
set build(number) [string trim [exec git rev-list --count HEAD]]
set build(version) "0.0.$build(number)"
set llamaconfig [read [open "$env(LLAMA_CONFIG)" r]]
set checks [list "set\\(LLAMA_VERSION \\s+$build(version)\\)" \
"set\\(LLAMA_BUILD_COMMIT\\s+$build(commit)\\)" \
"set\\(LLAMA_BUILD_NUMBER\\s+$build(number)\\)"]
puts -nonewline "Checking llama-config.cmake version... "
foreach check $checks {
if {![regexp -expanded -- $check $llamaconfig]} {
puts "\"$check\" failed!"
exit 1
}
}
puts "success."
EOF
cd examples/simple-cmake-pkg
cmake -S . -B build -DCMAKE_PREFIX_PATH="$PREFIX"/lib/cmake
cmake --build build
+36 -134
View File
@@ -7,6 +7,7 @@ on:
- master - master
paths: [ paths: [
'.github/workflows/build-cpu.yml', '.github/workflows/build-cpu.yml',
'.github/workflows/build-cmake-pkg.yml',
'**/CMakeLists.txt', '**/CMakeLists.txt',
'**/.cmake', '**/.cmake',
'**/*.h', '**/*.h',
@@ -19,7 +20,7 @@ on:
types: [opened, synchronize, reopened] types: [opened, synchronize, reopened]
paths: [ paths: [
'.github/workflows/build-cpu.yml', '.github/workflows/build-cpu.yml',
'ggml/src/ggml-rpc/**', '.github/workflows/build-cmake-pkg.yml',
'**/CMakeLists.txt', '**/CMakeLists.txt',
'**/.cmake', '**/.cmake',
'**/*.h', '**/*.h',
@@ -28,10 +29,6 @@ on:
'**/*.cpp' '**/*.cpp'
] ]
cache-mode: none
permissions:
contents: read
concurrency: concurrency:
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }} group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}
cancel-in-progress: true cancel-in-progress: true
@@ -39,20 +36,22 @@ concurrency:
env: env:
GGML_NLOOP: 3 GGML_NLOOP: 3
GGML_N_THREADS: 1 GGML_N_THREADS: 1
GGML_SCHED_DEBUG_REALLOC: 1
LLAMA_ARG_LOG_COLORS: 1 LLAMA_ARG_LOG_COLORS: 1
LLAMA_ARG_LOG_PREFIX: 1 LLAMA_ARG_LOG_PREFIX: 1
LLAMA_ARG_LOG_TIMESTAMPS: 1 LLAMA_ARG_LOG_TIMESTAMPS: 1
jobs: jobs:
build-cmake-pkg:
uses: ./.github/workflows/build-cmake-pkg.yml
ubuntu: ubuntu:
strategy: strategy:
matrix: matrix:
include: include:
- build: 'x64' - build: 'x64'
os: ubuntu-26.04 os: ubuntu-22.04
- build: 'arm64' - build: 'arm64'
os: ubuntu-26.04-arm os: ubuntu-24.04-arm
runs-on: ${{ matrix.os }} runs-on: ${{ matrix.os }}
@@ -62,10 +61,11 @@ jobs:
uses: actions/checkout@v6 uses: actions/checkout@v6
- name: ccache - name: ccache
uses: ggml-org/ccache-action@v1.2.24 uses: ggml-org/ccache-action@v1.2.21
with: with:
restore: false key: cpu-${{ matrix.os }}
save: false evict-old-files: 1d
save: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
- name: Build Dependencies - name: Build Dependencies
id: build_depends id: build_depends
@@ -76,43 +76,28 @@ jobs:
libjpeg-dev build-essential libssl-dev \ libjpeg-dev build-essential libssl-dev \
git-lfs git-lfs
- name: Toolchain workaround (GCC 14)
if: ${{ contains(matrix.os, 'ubuntu-24.04') }}
run: |
sudo apt-get install -y gcc-14 g++-14
echo "CC=gcc-14" >> "$GITHUB_ENV"
echo "CXX=g++-14" >> "$GITHUB_ENV"
- name: Python Dependencies - name: Python Dependencies
id: python_depends id: python_depends
run: | run: |
export PIP_BREAK_SYSTEM_PACKAGES="1" export PIP_BREAK_SYSTEM_PACKAGES="1"
python3 -m pip install --upgrade pip setuptools python3 -m pip install --upgrade pip setuptools
pip3 install ./gguf-py jinja2==3.1.6 pip3 install ./gguf-py
- name: ccache-buckets-restore
uses: ./.github/actions/ccache-buckets
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CI }}
with:
key: cpu-${{ matrix.os }}
folder: llama.cpp
hf_bucket: ggml-org/cache
- name: Build - name: Build
id: cmake_build id: cmake_build
run: | run: |
cmake -B build \ cmake -B build \
-DGGML_NATIVE=OFF \
-DLLAMA_FATAL_WARNINGS=ON \ -DLLAMA_FATAL_WARNINGS=ON \
-DGGML_RPC=ON -DGGML_RPC=ON
time cmake --build build --config Release -j $(nproc) time cmake --build build --config Release -j $(nproc)
- name: ccache-buckets-save
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
uses: ./.github/actions/ccache-buckets
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CACHE_OUTPUT }}
with:
key: cpu-${{ matrix.os }}
folder: llama.cpp
evict-old-files: 1d
hf_bucket: ggml-org/cache
save: true
- name: Test - name: Test
id: cmake_test id: cmake_test
run: | run: |
@@ -131,30 +116,28 @@ jobs:
./bin/llama-completion -m stories260K.gguf -p "One day, Lily met a Shoggoth" -n 500 -c 256 ./bin/llama-completion -m stories260K.gguf -p "One day, Lily met a Shoggoth" -n 500 -c 256
windows: windows:
name: windows / ${{ matrix.build }}
runs-on: windows-2025 runs-on: windows-2025
cache-mode: write
permissions:
actions: write
contents: read
env: env:
OPENBLAS_VERSION: 0.3.23 OPENBLAS_VERSION: 0.3.23
SDE_VERSION: 9.33.0-2024-01-07 SDE_VERSION: 9.33.0-2024-01-07
VULKAN_VERSION: 1.4.313.2
strategy: strategy:
matrix: matrix:
include: include:
- build: 'x64-cpu-static' - build: 'x64-cpu-static'
arch: 'x64' arch: 'x64'
defines: '-G "Ninja Multi-Config" -D CMAKE_TOOLCHAIN_FILE=cmake/x64-windows-llvm.cmake -DGGML_NATIVE=OFF -DGGML_OPENMP_FETCH=ON -DLLAMA_BUILD_SERVER=ON -DGGML_RPC=ON -DBUILD_SHARED_LIBS=OFF' defines: '-G "Ninja Multi-Config" -D CMAKE_TOOLCHAIN_FILE=cmake/x64-windows-llvm.cmake -DGGML_NATIVE=OFF -DLLAMA_BUILD_SERVER=ON -DGGML_RPC=ON -DBUILD_SHARED_LIBS=OFF'
- build: 'x64-openblas' - build: 'x64-openblas'
arch: 'x64' arch: 'x64'
defines: '-G "Ninja Multi-Config" -D CMAKE_TOOLCHAIN_FILE=cmake/x64-windows-llvm.cmake -DGGML_NATIVE=OFF -DLLAMA_BUILD_SERVER=ON -DGGML_RPC=ON -DGGML_BACKEND_DL=ON -DGGML_CPU_ALL_VARIANTS=ON -DGGML_OPENMP=OFF -DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS -DBLAS_INCLUDE_DIRS="$env:RUNNER_TEMP/openblas/include" -DBLAS_LIBRARIES="$env:RUNNER_TEMP/openblas/lib/openblas.lib"' defines: '-G "Ninja Multi-Config" -D CMAKE_TOOLCHAIN_FILE=cmake/x64-windows-llvm.cmake -DGGML_NATIVE=OFF -DLLAMA_BUILD_SERVER=ON -DGGML_RPC=ON -DGGML_BACKEND_DL=ON -DGGML_CPU_ALL_VARIANTS=ON -DGGML_OPENMP=OFF -DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS -DBLAS_INCLUDE_DIRS="$env:RUNNER_TEMP/openblas/include" -DBLAS_LIBRARIES="$env:RUNNER_TEMP/openblas/lib/openblas.lib"'
- build: 'x64-vulkan'
arch: 'x64'
defines: '-G "Ninja Multi-Config" -D CMAKE_TOOLCHAIN_FILE=cmake/x64-windows-llvm.cmake -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=OFF -DLLAMA_BUILD_SERVER=ON -DGGML_RPC=ON -DGGML_BACKEND_DL=ON -DGGML_CPU_ALL_VARIANTS=ON -DGGML_VULKAN=ON'
- build: 'arm64' - build: 'arm64'
arch: 'arm64' arch: 'arm64'
defines: '-G "Ninja Multi-Config" -D CMAKE_TOOLCHAIN_FILE=cmake/arm64-windows-llvm.cmake -DGGML_NATIVE=OFF -DGGML_OPENMP_FETCH=ON -DLLAMA_BUILD_SERVER=ON' defines: '-G "Ninja Multi-Config" -D CMAKE_TOOLCHAIN_FILE=cmake/arm64-windows-llvm.cmake -DGGML_NATIVE=OFF -DLLAMA_BUILD_SERVER=ON'
steps: steps:
- name: Clone - name: Clone
@@ -162,7 +145,7 @@ jobs:
uses: actions/checkout@v6 uses: actions/checkout@v6
- name: ccache - name: ccache
uses: ggml-org/ccache-action@v1.2.24 uses: ggml-org/ccache-action@v1.2.21
with: with:
key: cpu-windows-2025-${{ matrix.build }} key: cpu-windows-2025-${{ matrix.build }}
variant: ccache variant: ccache
@@ -182,16 +165,20 @@ jobs:
$lib = $(join-path $msvc 'bin\Hostx64\x64\lib.exe') $lib = $(join-path $msvc 'bin\Hostx64\x64\lib.exe')
& $lib /machine:x64 "/def:${env:RUNNER_TEMP}/openblas/lib/libopenblas.def" "/out:${env:RUNNER_TEMP}/openblas/lib/openblas.lib" /name:openblas.dll & $lib /machine:x64 "/def:${env:RUNNER_TEMP}/openblas/lib/libopenblas.def" "/out:${env:RUNNER_TEMP}/openblas/lib/openblas.lib" /name:openblas.dll
- name: Install Vulkan SDK
id: get_vulkan
if: ${{ matrix.build == 'x64-vulkan' }}
run: |
curl.exe -o $env:RUNNER_TEMP/VulkanSDK-Installer.exe -L "https://sdk.lunarg.com/sdk/download/${env:VULKAN_VERSION}/windows/vulkansdk-windows-X64-${env:VULKAN_VERSION}.exe"
& "$env:RUNNER_TEMP\VulkanSDK-Installer.exe" --accept-licenses --default-answer --confirm-command install
Add-Content $env:GITHUB_ENV "VULKAN_SDK=C:\VulkanSDK\${env:VULKAN_VERSION}"
Add-Content $env:GITHUB_PATH "C:\VulkanSDK\${env:VULKAN_VERSION}\bin"
- name: Install Ninja - name: Install Ninja
id: install_ninja id: install_ninja
run: | run: |
choco install ninja choco install ninja
- name: Install LLVM
run: |
choco install llvm --version=22.1.8
echo "C:\Program Files\LLVM\bin" >> $env:GITHUB_PATH
- name: Build - name: Build
id: cmake_build id: cmake_build
run: | run: |
@@ -224,90 +211,5 @@ jobs:
# 7z x "-o${env:RUNNER_TEMP}" $env:RUNNER_TEMP/sde.tar # 7z x "-o${env:RUNNER_TEMP}" $env:RUNNER_TEMP/sde.tar
# $sde = $(join-path $env:RUNNER_TEMP sde-external-${env:SDE_VERSION}-win/sde.exe) # $sde = $(join-path $env:RUNNER_TEMP sde-external-${env:SDE_VERSION}-win/sde.exe)
# cd build # cd build
# $env:LLAMA_SKIP_TESTS_SLOW_ON_EMULATOR = 1
# & $sde -future -- ctest -L main -C Release --verbose --timeout 900 # & $sde -future -- ctest -L main -C Release --verbose --timeout 900
- name: ccache-clear
uses: ./.github/actions/ccache-clear
env:
GH_TOKEN: ${{ github.token }}
with:
key: cpu-windows-2025-${{ matrix.build }}
older: 5m
min: 1
dry-run: ${{ github.event_name != 'push' || github.ref != 'refs/heads/master' }}
build-cmake-pkg:
runs-on: ubuntu-24.04
steps:
- name: Clone
uses: actions/checkout@v6
with:
fetch-depth: 0
- name: ccache
uses: ggml-org/ccache-action@v1.2.24
with:
restore: false
save: false
- name: ccache-buckets-restore
uses: ./.github/actions/ccache-buckets
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CI }}
with:
key: build-cmake-pkg
folder: llama.cpp
hf_bucket: ggml-org/cache
- name: Build
run: |
PREFIX="$(pwd)"/inst
cmake -S . -B build \
-DCMAKE_PREFIX_PATH="$PREFIX" \
-DLLAMA_OPENSSL=OFF \
-DLLAMA_BUILD_TESTS=OFF \
-DLLAMA_BUILD_TOOLS=OFF \
-DLLAMA_BUILD_EXAMPLES=OFF \
-DLLAMA_BUILD_APP=OFF \
-DLLAMA_BUILD_IS_DEV=OFF \
-DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j $(nproc)
cmake --install build --prefix "$PREFIX" --config Release
export LLAMA_CONFIG="$PREFIX"/lib/cmake/llama/llama-config.cmake
build_commit=$(git rev-parse --short HEAD | xargs)
build_number=$(git rev-list --count HEAD | xargs)
major=$(grep -oE "set\(LLAMA_VERSION_MAJOR[[:space:]]+[0-9]+" CMakeLists.txt | grep -oE "[0-9]+$")
minor=$(grep -oE "set\(LLAMA_VERSION_MINOR[[:space:]]+[0-9]+" CMakeLists.txt | grep -oE "[0-9]+$")
patch=$(grep -oE "set\(LLAMA_VERSION_PATCH[[:space:]]+[0-9]+" CMakeLists.txt | grep -oE "[0-9]+$")
build_version="$major.$minor.$patch"
checks=("set\(LLAMA_VERSION[[:space:]]+$build_version\)"
"set\(LLAMA_BUILD_COMMIT[[:space:]]+$build_commit\)"
"set\(LLAMA_BUILD_NUMBER[[:space:]]+$build_number\)")
for check in "${checks[@]}"; do
if ! grep -qE "$check" "$LLAMA_CONFIG"; then
echo "Checking llama-config.cmake version... \"$check\" failed!"
exit 1
fi
done
echo "Checking llama-config.cmake version... success."
cd examples/simple-cmake-pkg
cmake -S . -B build -DCMAKE_PREFIX_PATH="$PREFIX"/lib/cmake
cmake --build build -j $(nproc)
- name: ccache-buckets-save
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
uses: ./.github/actions/ccache-buckets
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CACHE_OUTPUT }}
with:
key: build-cmake-pkg
folder: llama.cpp
evict-old-files: 1d
hf_bucket: ggml-org/cache
save: true
+4 -8
View File
@@ -15,10 +15,6 @@ on:
schedule: schedule:
- cron: '0 0 * * 0' - cron: '0 0 * * 0'
cache-mode: none
permissions:
contents: read
concurrency: concurrency:
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }} group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}
cancel-in-progress: true cancel-in-progress: true
@@ -163,7 +159,7 @@ jobs:
# cmake --build build --config Release -j $(nproc) # cmake --build build --config Release -j $(nproc)
debian-13-loongarch64-cpu-cross: debian-13-loongarch64-cpu-cross:
runs-on: ${{ 'ubuntu-26.04-arm' || 'ubuntu-26.04' }} runs-on: ${{ 'ubuntu-24.04-arm' || 'ubuntu-24.04' }}
container: debian@sha256:653dfb9f86c3782e8369d5f7d29bb8faba1f4bff9025db46e807fa4c22903671 container: debian@sha256:653dfb9f86c3782e8369d5f7d29bb8faba1f4bff9025db46e807fa4c22903671
steps: steps:
@@ -218,7 +214,7 @@ jobs:
cmake --build build --config Release -j $(nproc) cmake --build build --config Release -j $(nproc)
debian-13-loongarch64-vulkan-cross: debian-13-loongarch64-vulkan-cross:
runs-on: ${{ 'ubuntu-26.04-arm' || 'ubuntu-26.04' }} runs-on: ${{ 'ubuntu-24.04-arm' || 'ubuntu-24.04' }}
container: debian@sha256:653dfb9f86c3782e8369d5f7d29bb8faba1f4bff9025db46e807fa4c22903671 container: debian@sha256:653dfb9f86c3782e8369d5f7d29bb8faba1f4bff9025db46e807fa4c22903671
steps: steps:
@@ -276,8 +272,8 @@ jobs:
cmake --build build --config Release -j $(nproc) cmake --build build --config Release -j $(nproc)
ubuntu-26-riscv64-cpu-spacemit-ime-cross: ubuntu-24-riscv64-cpu-spacemit-ime-cross:
runs-on: ubuntu-26.04 runs-on: ubuntu-24.04
env: env:
# Make sure this is in sync with build-cache.yml # Make sure this is in sync with build-cache.yml
+19 -83
View File
@@ -24,10 +24,6 @@ on:
'ggml/src/ggml-cuda/**' 'ggml/src/ggml-cuda/**'
] ]
cache-mode: none
permissions:
contents: read
concurrency: concurrency:
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }} group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}
cancel-in-progress: true cancel-in-progress: true
@@ -54,25 +50,17 @@ jobs:
DEBIAN_FRONTEND: noninteractive DEBIAN_FRONTEND: noninteractive
run: | run: |
apt update apt update
apt install -y cmake build-essential ninja-build libgomp1 git libssl-dev jq python3 python3-venv python3-pip apt install -y cmake build-essential ninja-build libgomp1 git libssl-dev
- name: ccache - name: ccache
uses: ggml-org/ccache-action@v1.2.24 uses: ggml-org/ccache-action@v1.2.21
with:
restore: false
save: false
- name: ccache-buckets-restore
uses: ./.github/actions/ccache-buckets
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CI }}
with: with:
key: cuda-ubuntu-24.04-cuda key: cuda-ubuntu-24.04-cuda
folder: llama.cpp evict-old-files: 1d
hf_bucket: ggml-org/cache save: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
- name: Build with CMake - name: Build with CMake
# TODO: Drop GGML_CUDA_CCCL_VERSION when this job uses CTK >= 13.5, which bundles CCCL >= 3.5. # TODO: Remove GGML_CUDA_CUB_3DOT2 flag once CCCL 3.2 is bundled within CTK and that CTK version is used in this project
run: | run: |
cmake -S . -B build -G Ninja \ cmake -S . -B build -G Ninja \
-DLLAMA_FATAL_WARNINGS=ON \ -DLLAMA_FATAL_WARNINGS=ON \
@@ -81,24 +69,12 @@ jobs:
-DCMAKE_EXE_LINKER_FLAGS=-Wl,--allow-shlib-undefined \ -DCMAKE_EXE_LINKER_FLAGS=-Wl,--allow-shlib-undefined \
-DGGML_NATIVE=OFF \ -DGGML_NATIVE=OFF \
-DGGML_CUDA=ON \ -DGGML_CUDA=ON \
-DGGML_CUDA_CCCL_VERSION=v3.4.3 -DGGML_CUDA_CUB_3DOT2=ON
cmake --build build cmake --build build
- name: ccache-buckets-save
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
uses: ./.github/actions/ccache-buckets
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CACHE_OUTPUT }}
with:
key: cuda-ubuntu-24.04-cuda
folder: llama.cpp
evict-old-files: 1d
hf_bucket: ggml-org/cache
save: true
hip: hip:
runs-on: ubuntu-22.04 runs-on: ubuntu-22.04
container: rocm/dev-ubuntu-22.04:6.3.4 container: rocm/dev-ubuntu-22.04:6.1.2
steps: steps:
- name: Clone - name: Clone
@@ -109,47 +85,28 @@ jobs:
id: depends id: depends
run: | run: |
sudo apt-get update sudo apt-get update
sudo apt-get install -y build-essential git cmake rocblas-dev hipblas-dev libssl-dev rocwmma-dev jq python3-venv sudo apt-get install -y build-essential git cmake rocblas-dev hipblas-dev libssl-dev rocwmma-dev
- name: ccache - name: ccache
uses: ggml-org/ccache-action@v1.2.24 uses: ggml-org/ccache-action@v1.2.21
with:
restore: false
save: false
- name: ccache-buckets-restore
uses: ./.github/actions/ccache-buckets
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CI }}
with: with:
key: cuda-ubuntu-22.04-hip key: cuda-ubuntu-22.04-hip
folder: llama.cpp evict-old-files: 1d
hf_bucket: ggml-org/cache save: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
- name: Build with native CMake HIP support - name: Build with native CMake HIP support
id: cmake_build id: cmake_build
run: | run: |
cmake -B build -S . \ cmake -B build -S . \
-DCMAKE_HIP_COMPILER="$(hipconfig -l)/clang" \ -DCMAKE_HIP_COMPILER="$(hipconfig -l)/clang" \
-DGGML_HIP_ROCWMMA_FATTN=ON \
-DGPU_TARGETS="gfx1030" \ -DGPU_TARGETS="gfx1030" \
-DGGML_HIP=ON -DGGML_HIP=ON
cmake --build build --config Release -j $(nproc) cmake --build build --config Release -j $(nproc)
- name: ccache-buckets-save
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
uses: ./.github/actions/ccache-buckets
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CACHE_OUTPUT }}
with:
key: cuda-ubuntu-22.04-hip
folder: llama.cpp
evict-old-files: 1d
hf_bucket: ggml-org/cache
save: true
musa: musa:
runs-on: ubuntu-22.04 runs-on: ubuntu-22.04
container: registry.mthreads.com/mcconline/musa_sdk:5.2.0-devel-ubuntu22.04-s5000 container: mthreads/musa:rc4.3.0-devel-ubuntu22.04-amd64
steps: steps:
- name: Clone - name: Clone
@@ -160,39 +117,18 @@ jobs:
id: depends id: depends
run: | run: |
apt-get update apt-get update
apt-get install -y build-essential git cmake libssl-dev jq python3-venv musa-mualg-5-2 musa-muthrust-5-2 libmthreads-compute apt-get install -y build-essential git cmake libssl-dev
- name: ccache - name: ccache
uses: ggml-org/ccache-action@v1.2.24 uses: ggml-org/ccache-action@v1.2.21
with:
restore: false
save: false
- name: ccache-buckets-restore
uses: ./.github/actions/ccache-buckets
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CI }}
with: with:
key: cuda-ubuntu-22.04-musa key: cuda-ubuntu-22.04-musa
folder: llama.cpp evict-old-files: 1d
hf_bucket: ggml-org/cache save: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
- name: Build with native CMake MUSA support - name: Build with native CMake MUSA support
id: cmake_build id: cmake_build
run: | run: |
cmake -B build -S . \ cmake -B build -S . \
-DGGML_MUSA=ON \ -DGGML_MUSA=ON
-DMUSA_ARCHITECTURES=31 time cmake --build build --config Release -j $(nproc)
cmake --build build --config Release -j $(nproc)
- name: ccache-buckets-save
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
uses: ./.github/actions/ccache-buckets
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CACHE_OUTPUT }}
with:
key: cuda-ubuntu-22.04-musa
folder: llama.cpp
evict-old-files: 1d
hf_bucket: ggml-org/cache
save: true
+49 -76
View File
@@ -7,11 +7,6 @@ name: CI (CUDA, windows)
on: on:
workflow_dispatch: # allows manual triggering workflow_dispatch: # allows manual triggering
cache-mode: write
permissions:
actions: write
contents: read
# note: this will run in queue with the release workflow # note: this will run in queue with the release workflow
concurrency: concurrency:
group: release group: release
@@ -27,7 +22,6 @@ env:
jobs: jobs:
cuda: cuda:
name: windows-cuda (${{ matrix.cuda }}, ${{ matrix.arch }})
runs-on: windows-2022 runs-on: windows-2022
permissions: permissions:
@@ -35,17 +29,7 @@ jobs:
strategy: strategy:
matrix: matrix:
include: cuda: ['12.4', '13.3']
# CTK >= 13.5 bundles CCCL >= 3.5; omit GGML_CUDA_CCCL_VERSION for those versions.
- cuda: '12.4'
arch: x64
defines: '-DGGML_CUDA_CCCL_VERSION=v3.4.3'
- cuda: '13.5'
arch: x64
defines: ''
- cuda: '13.5'
arch: arm64
defines: '-DCMAKE_TOOLCHAIN_FILE=cmake/arm64-windows-msvc-cuda.cmake'
steps: steps:
- name: Clone - name: Clone
@@ -53,15 +37,14 @@ jobs:
uses: actions/checkout@v6 uses: actions/checkout@v6
- name: ccache - name: ccache
uses: ggml-org/ccache-action@v1.2.24 uses: ggml-org/ccache-action@v1.2.21
with: with:
key: release-windows-2022-${{ matrix.arch }}-cuda-${{ matrix.cuda }} key: release-windows-2022-x64-cuda-${{ matrix.cuda }}
- name: Install Cuda Toolkit - name: Install Cuda Toolkit
uses: ./.github/actions/windows-setup-cuda uses: ./.github/actions/windows-setup-cuda
with: with:
cuda_version: ${{ matrix.cuda }} cuda_version: ${{ matrix.cuda }}
cuda_arch: ${{ matrix.arch }}
- name: Install Ninja - name: Install Ninja
id: install_ninja id: install_ninja
@@ -71,21 +54,26 @@ jobs:
- name: Build - name: Build
id: cmake_build id: cmake_build
shell: cmd shell: cmd
# TODO: Remove GGML_CUDA_CUB_3DOT2 flag once CCCL 3.2 is bundled within CTK and that CTK version is used in this project
run: | run: |
call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" ${{ matrix.arch == 'x64' && 'x64' || 'amd64_arm64' }} call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64
cmake -S . -B build -G "Ninja Multi-Config" ^ cmake -S . -B build -G "Ninja Multi-Config" ^
-DGGML_BACKEND_DL=ON ^ -DLLAMA_BUILD_SERVER=ON ^
-DLLAMA_BUILD_BORINGSSL=ON ^
-DGGML_NATIVE=OFF ^ -DGGML_NATIVE=OFF ^
-DGGML_CPU=OFF ^ -DGGML_BACKEND_DL=ON ^
-DGGML_CPU_ALL_VARIANTS=ON ^
-DGGML_CUDA=ON ^ -DGGML_CUDA=ON ^
-DLLAMA_BUILD_BORINGSSL=ON ${{ matrix.defines }} -DGGML_RPC=ON ^
-DGGML_CUDA_CUB_3DOT2=ON
set /A NINJA_JOBS=%NUMBER_OF_PROCESSORS%-1 set /A NINJA_JOBS=%NUMBER_OF_PROCESSORS%-1
cmake --build build --config Release -j %NINJA_JOBS% --target ggml-cuda cmake --build build --config Release -j %NINJA_JOBS% -t ggml
cmake --build build --config Release
- name: ccache-clear - name: ccache-clear
uses: ./.github/actions/ccache-clear uses: ./.github/actions/ccache-clear
with: with:
key: release-windows-2022-${{ matrix.arch }}-cuda-${{ matrix.cuda }} key: release-windows-2022-x64-cuda-${{ matrix.cuda }}
hip: hip:
runs-on: windows-2022 runs-on: windows-2022
@@ -95,7 +83,7 @@ jobs:
env: env:
# Make sure this is in sync with build-cache.yml # Make sure this is in sync with build-cache.yml
ROCM_VERSION: "7.14.0" HIPSDK_INSTALLER_VERSION: "26.Q1"
strategy: strategy:
matrix: matrix:
@@ -109,75 +97,60 @@ jobs:
id: checkout id: checkout
uses: actions/checkout@v6 uses: actions/checkout@v6
# - name: Cache ROCm Installation - name: Grab rocWMMA package
# uses: actions/cache@v5 id: grab_rocwmma
# id: cache-rocm run: |
# with: curl -o rocwmma.deb "https://repo.radeon.com/rocm/apt/7.2.1/pool/main/r/rocwmma-dev/rocwmma-dev_2.2.0.70201-81~24.04_amd64.deb"
# path: C:\TheRock\build 7z x rocwmma.deb
# key: rocm-wheels-${{ env.ROCM_VERSION }}-multi-arch-${{ runner.os }} 7z x data.tar
- name: Use ROCm Installation Cache
uses: actions/cache@v5
id: cache-rocm
with:
path: C:\Program Files\AMD\ROCm
key: cache-gha-rocm-${{ env.HIPSDK_INSTALLER_VERSION }}-${{ runner.os }}
- name: Setup ROCm - name: Setup ROCm
# if: steps.cache-rocm.outputs.cache-hit != 'true' if: steps.cache-rocm.outputs.cache-hit != 'true'
uses: ./.github/actions/windows-setup-rocm uses: ./.github/actions/windows-setup-rocm
with: with:
version: ${{ env.ROCM_VERSION }} version: ${{ env.HIPSDK_INSTALLER_VERSION }}
- name: Setup ROCm Environment
run: |
$ErrorActionPreference = "Stop"
# Activate venv from cache or fresh install
& C:\TheRock\build\.venv\Scripts\Activate.ps1
# Expand the devel tree (idempotent; no-op if already done during install)
rocm-sdk init
if ($LASTEXITCODE -ne 0) { throw "rocm-sdk init failed with exit code $LASTEXITCODE" }
# Get ROCm installation paths using the rocm-sdk CLI tool
$rocmPath = (rocm-sdk path --root)
if (-not $rocmPath) { throw "rocm-sdk path --root returned empty - devel package may not be installed" }
$rocmPath = $rocmPath.Trim()
$cmakePath = (rocm-sdk path --cmake).Trim()
$binPath = (rocm-sdk path --bin).Trim()
write-host "ROCm root: $rocmPath"
echo "HIP_PATH=$rocmPath" >> $env:GITHUB_ENV
echo "CMAKE_PREFIX_PATH=$cmakePath" >> $env:GITHUB_ENV
echo "HIP_DEVICE_LIB_PATH=$rocmPath\lib\llvm\amdgcn\bitcode" >> $env:GITHUB_ENV
echo "HIP_PLATFORM=amd" >> $env:GITHUB_ENV
echo "LLVM_PATH=$rocmPath\lib\llvm" >> $env:GITHUB_ENV
echo "$binPath" >> $env:GITHUB_PATH
# Keep venv in PATH for subsequent steps
echo "C:\TheRock\build\.venv\Scripts" >> $env:GITHUB_PATH
- name: Verify ROCm - name: Verify ROCm
id: verify id: verify
run: | run: |
# Test the ROCm clang shipped in the installed wheel # Find and test ROCm installation
& "${env:HIP_PATH}\lib\llvm\bin\clang.exe" --version $clangPath = Get-ChildItem 'C:\Program Files\AMD\ROCm\*\bin\clang.exe' | Select-Object -First 1
if (-not $clangPath) {
Write-Error "ROCm installation not found"
exit 1
}
& $clangPath.FullName --version
- name: ccache - name: ccache
uses: ggml-org/ccache-action@v1.2.24 uses: ggml-org/ccache-action@v1.2.21
with: with:
# TODO: this build does not match the build in release.yml, so we use a different cache key # TODO: this build does not match the build in release.yml, so we use a different cache key
# ideally, the builds should match, similar to the CUDA build above so that we would be able # ideally, the builds should match, similar to the CUDA build above so that we would be able
# to populate the ccache for the release with manual runs of this workflow # to populate the ccache for the release with manual runs of this workflow
#key: release-windows-2022-x64-hip-${{ env.ROCM_VERSION }}-${{ matrix.name }} #key: release-windows-2022-x64-hip-${{ env.HIPSDK_INSTALLER_VERSION }}-${{ matrix.name }}
key: cuda-windows-2022-x64-hip-${{ env.ROCM_VERSION }}-${{ matrix.name }} key: cuda-windows-2022-x64-hip-${{ env.HIPSDK_INSTALLER_VERSION }}-${{ matrix.name }}
- name: Build - name: Build
id: cmake_build id: cmake_build
run: | run: |
$env:HIP_PATH=$(Resolve-Path 'C:\Program Files\AMD\ROCm\*\bin\clang.exe' | split-path | split-path)
$env:CMAKE_PREFIX_PATH="${env:HIP_PATH}"
cmake -G "Unix Makefiles" -B build -S . ` cmake -G "Unix Makefiles" -B build -S . `
-DCMAKE_PREFIX_PATH="${env:HIP_PATH}" ` -DCMAKE_C_COMPILER="${env:HIP_PATH}\bin\clang.exe" `
-DCMAKE_C_COMPILER="${env:HIP_PATH}\lib\llvm\bin\clang.exe" ` -DCMAKE_CXX_COMPILER="${env:HIP_PATH}\bin\clang++.exe" `
-DCMAKE_CXX_COMPILER="${env:HIP_PATH}\lib\llvm\bin\clang++.exe" ` -DCMAKE_CXX_FLAGS="-I$($PWD.Path.Replace('\', '/'))/opt/rocm-7.2.1/include/" `
-DCMAKE_HIP_COMPILER="${env:HIP_PATH}\lib\llvm\bin\clang.exe" `
-DCMAKE_BUILD_TYPE=Release ` -DCMAKE_BUILD_TYPE=Release `
-DLLAMA_BUILD_BORINGSSL=ON ` -DLLAMA_BUILD_BORINGSSL=ON `
-DHIP_PATH="${env:HIP_PATH}" ` -DROCM_DIR="${env:HIP_PATH}" `
-DGGML_HIP=ON ` -DGGML_HIP=ON `
-DGGML_HIP_ROCWMMA_FATTN=ON `
-DGPU_TARGETS="gfx1100" ` -DGPU_TARGETS="gfx1100" `
-DGGML_RPC=ON -DGGML_RPC=ON
cmake --build build -j ${env:NUMBER_OF_PROCESSORS} cmake --build build -j ${env:NUMBER_OF_PROCESSORS}
@@ -185,5 +158,5 @@ jobs:
- name: ccache-clear - name: ccache-clear
uses: ./.github/actions/ccache-clear uses: ./.github/actions/ccache-clear
with: with:
#key: release-windows-2022-x64-hip-${{ env.ROCM_VERSION }}-${{ matrix.name }} #key: release-windows-2022-x64-hip-${{ env.HIPSDK_INSTALLER_VERSION }}-${{ matrix.name }}
key: cuda-windows-2022-x64-hip-${{ env.ROCM_VERSION }}-${{ matrix.name }} key: cuda-windows-2022-x64-hip-${{ env.HIPSDK_INSTALLER_VERSION }}-${{ matrix.name }}
+19 -56
View File
@@ -19,18 +19,9 @@ on:
types: [opened, synchronize, reopened] types: [opened, synchronize, reopened]
paths: [ paths: [
'.github/workflows/build-ibm.yml', '.github/workflows/build-ibm.yml',
'ggml/src/ggml-cpu/**', 'ggml/src/ggml-cpu/**'
'ggml/src/ggml-zdnn/**',
'ggml/src/ggml-quants.c',
'src/llama-quant.cpp',
'gguf-py/gguf/scripts/gguf_convert_endian.py',
'tests/test-quantize-be.sh'
] ]
cache-mode: none
permissions:
contents: read
concurrency: concurrency:
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }} group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}
cancel-in-progress: true cancel-in-progress: true
@@ -43,14 +34,9 @@ env:
LLAMA_ARG_LOG_TIMESTAMPS: 1 LLAMA_ARG_LOG_TIMESTAMPS: 1
jobs: jobs:
ubuntu-26-s390x:
name: ubuntu-26-s390x (VXE ${{ matrix.vxe }})
runs-on: ubuntu-26.04-s390x
strategy: ubuntu-24-s390x:
fail-fast: false runs-on: ubuntu-24.04-s390x
matrix:
vxe: ["ON", "OFF"] # `-DGGML_VXE=ON/OFF`
steps: steps:
- name: Clone - name: Clone
@@ -66,6 +52,12 @@ jobs:
libjpeg-dev build-essential libssl-dev \ libjpeg-dev build-essential libssl-dev \
git-lfs git-lfs
- name: Toolchain workaround (GCC 14)
run: |
sudo apt-get install -y gcc-14 g++-14
echo "CC=gcc-14" >> "$GITHUB_ENV"
echo "CXX=g++-14" >> "$GITHUB_ENV"
- name: Python Dependencies - name: Python Dependencies
id: python_depends id: python_depends
run: | run: |
@@ -85,8 +77,7 @@ jobs:
run: | run: |
cmake -B build \ cmake -B build \
-DLLAMA_FATAL_WARNINGS=ON \ -DLLAMA_FATAL_WARNINGS=ON \
-DGGML_RPC=ON \ -DGGML_RPC=ON
-DGGML_VXE=${{ matrix.vxe }}
time cmake --build build --config Release -j $(nproc) time cmake --build build --config Release -j $(nproc)
- name: Test - name: Test
@@ -100,45 +91,11 @@ jobs:
run: | run: |
cd build cd build
echo "Fetch llama2c big-endian model" echo "Fetch llama2c big-endian model"
wget https://huggingface.co/ggml-org-dev/stories260K/resolve/main/stories260K-be.gguf wget https://huggingface.co/ggml-org/models/resolve/main/tinyllamas/stories260K-be.gguf
./bin/llama-completion -m stories260K-be.gguf -p "One day, Lily met a Shoggoth" -n 500 -c 256 ./bin/llama-completion -m stories260K-be.gguf -p "One day, Lily met a Shoggoth" -n 500 -c 256
- name: Test quantize (s390x) ubuntu-24-ppc64le:
id: quantize_test_s390x runs-on: ubuntu-24.04-ppc64le
run: |
./tests/test-quantize-be.sh ./build/bin/llama-quantize ./build/bin/llama-perplexity 2>&1
ubuntu-26-zdnn-s390x:
name: ubuntu-26-zdnn-s390x
runs-on: ubuntu-26.04-s390x
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: Build Dependencies
id: build_depends
run: |
sudo apt-get update
sudo apt-get install -y --no-install-recommends \
build-essential libssl-dev libzdnn-dev
- name: Build with zDNN Backend
id: cmake_build
run: |
cmake -B build \
-DLLAMA_FATAL_WARNINGS=ON \
-DGGML_NATIVE=OFF \
-DGGML_VXE=ON \
-DGGML_ZDNN=ON \
-DGGML_RPC=ON \
-DCMAKE_C_FLAGS="-march=arch15" \
-DCMAKE_CXX_FLAGS="-march=arch15"
time cmake --build build --config Release -j $(nproc)
ubuntu-26-ppc64le:
runs-on: ubuntu-26.04-ppc64le
steps: steps:
- name: Clone - name: Clone
@@ -154,6 +111,12 @@ jobs:
libjpeg-dev build-essential libssl-dev \ libjpeg-dev build-essential libssl-dev \
git-lfs git-lfs
- name: Toolchain workaround (GCC 14)
run: |
sudo apt-get install -y gcc-14 g++-14
echo "CC=gcc-14" >> "$GITHUB_ENV"
echo "CXX=g++-14" >> "$GITHUB_ENV"
- name: Python Dependencies - name: Python Dependencies
id: python_depends id: python_depends
run: | run: |
+1 -5
View File
@@ -8,10 +8,6 @@ on:
schedule: schedule:
- cron: '0 0 * * 0' - cron: '0 0 * * 0'
cache-mode: none
permissions:
contents: read
concurrency: concurrency:
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }} group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}
cancel-in-progress: true cancel-in-progress: true
@@ -39,7 +35,7 @@ jobs:
uses: actions/checkout@v6 uses: actions/checkout@v6
#- name: ccache #- name: ccache
# uses: ggml-org/ccache-action@v1.2.24 # uses: ggml-org/ccache-action@v1.2.16
# with: # with:
# key: msys-windows-2025-x64 # key: msys-windows-2025-x64
# variant: ccache # variant: ccache
+1 -21
View File
@@ -23,11 +23,6 @@ on:
'ggml/src/ggml-opencl/**' 'ggml/src/ggml-opencl/**'
] ]
cache-mode: write
permissions:
actions: write
contents: read
concurrency: concurrency:
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }} group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}
cancel-in-progress: true cancel-in-progress: true
@@ -49,7 +44,7 @@ jobs:
uses: actions/checkout@v6 uses: actions/checkout@v6
- name: ccache - name: ccache
uses: ggml-org/ccache-action@v1.2.24 uses: ggml-org/ccache-action@v1.2.21
with: with:
key: opencl-windows-2025-x64 key: opencl-windows-2025-x64
variant: ccache variant: ccache
@@ -61,11 +56,6 @@ jobs:
run: | run: |
choco install ninja choco install ninja
- name: Install LLVM
run: |
choco install llvm --version=22.1.8
echo "C:\Program Files\LLVM\bin" >> $env:GITHUB_PATH
- name: Install OpenCL Headers and Libs - name: Install OpenCL Headers and Libs
id: install_opencl id: install_opencl
run: | run: |
@@ -90,13 +80,3 @@ jobs:
run: | run: |
cmake -S . -B build -G "Ninja Multi-Config" -D CMAKE_TOOLCHAIN_FILE=cmake/arm64-windows-llvm.cmake -DCMAKE_PREFIX_PATH="$env:RUNNER_TEMP/opencl-arm64-release" -DGGML_OPENCL=ON -DGGML_OPENCL_USE_ADRENO_KERNELS=ON -DLLAMA_BUILD_BORINGSSL=ON cmake -S . -B build -G "Ninja Multi-Config" -D CMAKE_TOOLCHAIN_FILE=cmake/arm64-windows-llvm.cmake -DCMAKE_PREFIX_PATH="$env:RUNNER_TEMP/opencl-arm64-release" -DGGML_OPENCL=ON -DGGML_OPENCL_USE_ADRENO_KERNELS=ON -DLLAMA_BUILD_BORINGSSL=ON
cmake --build build --config Release -j ${env:NUMBER_OF_PROCESSORS} cmake --build build --config Release -j ${env:NUMBER_OF_PROCESSORS}
- name: ccache-clear
uses: ./.github/actions/ccache-clear
env:
GH_TOKEN: ${{ github.token }}
with:
key: opencl-windows-2025-x64
older: 5m
min: 1
dry-run: ${{ github.event_name != 'push' || github.ref != 'refs/heads/master' }}
+63 -26
View File
@@ -22,10 +22,6 @@ on:
'ggml/src/ggml-openvino/**' 'ggml/src/ggml-openvino/**'
] ]
cache-mode: none
permissions:
contents: read
concurrency: concurrency:
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }} group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}
cancel-in-progress: true cancel-in-progress: true
@@ -36,22 +32,72 @@ env:
LLAMA_ARG_LOG_COLORS: 1 LLAMA_ARG_LOG_COLORS: 1
LLAMA_ARG_LOG_PREFIX: 1 LLAMA_ARG_LOG_PREFIX: 1
LLAMA_ARG_LOG_TIMESTAMPS: 1 LLAMA_ARG_LOG_TIMESTAMPS: 1
# TODO: fix failing tests on OpenVINO backend
CTEST_EXCLUDE: "test-llama-archs|test-llama-context|test-fp8-kv"
jobs: jobs:
ubuntu-24-openvino:
runs-on: [self-hosted, Linux, Intel, OpenVINO]
env:
# Sync versions in build-openvino.yml, build-self-hosted.yml, release.yml, build-cache.yml, .devops/openvino.Dockerfile
OPENVINO_VERSION_MAJOR: "2026.2.1"
OPENVINO_VERSION_FULL: "2026.2.1.21919.ede283a88e3"
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: Dependencies
id: depends
run: |
sudo apt-get update
sudo apt-get install -y build-essential libssl-dev libtbb12 cmake ninja-build python3-pip
sudo apt-get install -y ocl-icd-opencl-dev opencl-headers opencl-clhpp-headers intel-opencl-icd
- name: Setup OpenVINO Toolkit
uses: ./.github/actions/linux-setup-openvino
with:
path: ./openvino_toolkit
version_major: ${{ env.OPENVINO_VERSION_MAJOR }}
version_full: ${{ env.OPENVINO_VERSION_FULL }}
- name: Install OpenVINO dependencies
run: |
cd ./openvino_toolkit
chmod +x ./install_dependencies/install_openvino_dependencies.sh
echo "Y" | sudo -E ./install_dependencies/install_openvino_dependencies.sh
- name: Build
id: cmake_build
run: |
source ./openvino_toolkit/setupvars.sh
cmake -B build/ReleaseOV -G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DGGML_OPENVINO=ON
time cmake --build build/ReleaseOV --config Release --parallel
- name: Test (CPU)
id: cmake_test_cpu
# TODO: fix and re-enable the `test-llama-archs` test below
run: |
cd ${{ github.workspace }}
ctest --test-dir build/ReleaseOV -L main -E "test-llama-archs" --verbose --timeout 2000
- name: Test (GPU)
id: cmake_test_gpu
# TODO: fix and re-enable the `test-llama-archs` test below
run: |
cd ${{ github.workspace }}
export GGML_OPENVINO_DEVICE=GPU
ctest --test-dir build/ReleaseOV -L main -E "test-llama-archs" --verbose --timeout 3000
openvino-windows-2022: openvino-windows-2022:
runs-on: windows-2022 runs-on: windows-2022
cache-mode: write
permissions:
actions: write
contents: read
env: env:
# Sync versions in build-openvino.yml, ci-self-hosted-openvino.yml, release.yml, release-publish.yml, build-cache.yml, .devops/openvino.Dockerfile, docs/backend/OPENVINO.md # Sync versions in build-openvino.yml, build-self-hosted.yml, release.yml, build-cache.yml, .devops/openvino.Dockerfile
OPENVINO_VERSION_MAJOR: "2026.4.1" OPENVINO_VERSION_MAJOR: "2026.2.1"
OPENVINO_VERSION_FULL: "2026.4.1.22982.07f9c262b05" OPENVINO_VERSION_FULL: "2026.2.1.21919.ede283a88e3"
steps: steps:
- name: Clone - name: Clone
@@ -59,7 +105,7 @@ jobs:
uses: actions/checkout@v6 uses: actions/checkout@v6
- name: ccache - name: ccache
uses: ggml-org/ccache-action@v1.2.24 uses: ggml-org/ccache-action@v1.2.21
with: with:
key: openvino-windows-2022 key: openvino-windows-2022
variant: ccache variant: ccache
@@ -113,20 +159,11 @@ jobs:
- name: Test (CPU) - name: Test (CPU)
id: cmake_test_cpu id: cmake_test_cpu
shell: cmd shell: cmd
# TODO: fix and re-enable the `test-llama-archs` test below
run: | run: |
REM Find extracted OpenVINO folder dynamically REM Find extracted OpenVINO folder dynamically
for /d %%i in (openvino_toolkit\*) do set OPENVINO_ROOT=%%i for /d %%i in (openvino_toolkit\*) do set OPENVINO_ROOT=%%i
call "%OPENVINO_ROOT%\setupvars.bat" call "%OPENVINO_ROOT%\setupvars.bat"
cd build cd build
ctest --test-dir ReleaseOV -L main -E "${{ env.CTEST_EXCLUDE }}" -C Release --verbose --timeout 3000 ctest --test-dir ReleaseOV -L main -E "test-llama-archs" -C Release --verbose --timeout 3000
- name: ccache-clear
uses: ./.github/actions/ccache-clear
env:
GH_TOKEN: ${{ github.token }}
with:
key: openvino-windows-2022
older: 5m
min: 1
dry-run: ${{ github.event_name != 'push' || github.ref != 'refs/heads/master' }}
+81 -5
View File
@@ -22,10 +22,6 @@ on:
'ggml/src/ggml-cpu/arch/riscv/**' 'ggml/src/ggml-cpu/arch/riscv/**'
] ]
cache-mode: none
permissions:
contents: read
concurrency: concurrency:
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }} group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}
cancel-in-progress: true cancel-in-progress: true
@@ -71,7 +67,7 @@ jobs:
# note: sparing some ccache since these jobs run on dedicated runners that are not part of the organitzation # note: sparing some ccache since these jobs run on dedicated runners that are not part of the organitzation
#- name: ccache #- name: ccache
# uses: ggml-org/ccache-action@v1.2.24 # uses: ggml-org/ccache-action@afde29e5b5422e5da23cb1f639e8baecadeadfc3 # https://github.com/ggml-org/ccache-action/pull/1
# with: # with:
# key: riscv-ubuntu-native # key: riscv-ubuntu-native
# evict-old-files: 1d # evict-old-files: 1d
@@ -110,3 +106,83 @@ jobs:
wget https://huggingface.co/karpathy/tinyllamas/resolve/main/stories260K/stories260K.bin wget https://huggingface.co/karpathy/tinyllamas/resolve/main/stories260K/stories260K.bin
./bin/llama-convert-llama2c-to-ggml --copy-vocab-from-model ./tok512.bin --llama2c-model stories260K.bin --llama2c-output-model stories260K.gguf ./bin/llama-convert-llama2c-to-ggml --copy-vocab-from-model ./tok512.bin --llama2c-model stories260K.bin --llama2c-output-model stories260K.gguf
./bin/llama-completion -m stories260K.gguf -p "One day, Lily met a Shoggoth" -n 500 -c 256 ./bin/llama-completion -m stories260K.gguf -p "One day, Lily met a Shoggoth" -n 500 -c 256
ubuntu-riscv64-native-sanitizer:
runs-on: ubuntu-24.04-riscv
continue-on-error: true
strategy:
matrix:
sanitizer: [ADDRESS, THREAD, UNDEFINED]
build_type: [Debug]
steps:
- name: Install dependencies
run: |
# Set gcc-14 and g++-14 as the default compilers
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-14 100
sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-14 100
git lfs install
- name: GCC version check
run: |
gcc --version
g++ --version
- name: Clone
id: checkout
uses: actions/checkout@v6
# note: sparing some ccache since these jobs run on dedicated runners that are not part of the organitzation
#- name: ccache
# uses: ggml-org/ccache-action@afde29e5b5422e5da23cb1f639e8baecadeadfc3 # https://github.com/ggml-org/ccache-action/pull/1
# with:
# key: riscv-ubuntu-native-sanitizer-${{ matrix.sanitizer }}-${{ matrix.build_type }}
# evict-old-files: 1d
# save: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
- name: Build
id: cmake_build
if: ${{ matrix.sanitizer != 'THREAD' }}
run: |
cmake -B build \
-DLLAMA_OPENSSL=OFF \
-DCMAKE_BUILD_TYPE=${{ matrix.build_type }} \
-DGGML_OPENMP=ON \
-DLLAMA_BUILD_EXAMPLES=ON \
-DLLAMA_BUILD_TOOLS=ON \
-DLLAMA_BUILD_TESTS=OFF \
-DCMAKE_C_COMPILER_LAUNCHER=ccache \
-DCMAKE_CXX_COMPILER_LAUNCHER=ccache \
-DLLAMA_SANITIZE_${{ matrix.sanitizer }}=ON \
-DCMAKE_C_COMPILER=riscv64-linux-gnu-gcc-14 \
-DCMAKE_CXX_COMPILER=riscv64-linux-gnu-g++-14
cmake --build build --config ${{ matrix.build_type }} -j $(nproc)
- name: Build (no OpenMP)
id: cmake_build_no_openmp
if: ${{ matrix.sanitizer == 'THREAD' }}
run: |
cmake -B build \
-DLLAMA_OPENSSL=OFF \
-DCMAKE_BUILD_TYPE=${{ matrix.build_type }} \
-DGGML_OPENMP=OFF \
-DLLAMA_BUILD_EXAMPLES=ON \
-DLLAMA_BUILD_TOOLS=ON \
-DLLAMA_BUILD_TESTS=OFF \
-DCMAKE_C_COMPILER_LAUNCHER=ccache \
-DCMAKE_CXX_COMPILER_LAUNCHER=ccache \
-DLLAMA_SANITIZE_${{ matrix.sanitizer }}=ON \
-DCMAKE_C_COMPILER=riscv64-linux-gnu-gcc-14 \
-DCMAKE_CXX_COMPILER=riscv64-linux-gnu-g++-14
cmake --build build --config ${{ matrix.build_type }} -j $(nproc)
- name: Test
id: cmake_test
run: |
cd build
ctest -L main --verbose --timeout 900
+66
View File
@@ -0,0 +1,66 @@
name: CI (rpc)
on:
workflow_dispatch: # allows manual triggering
push:
branches:
- master
paths: [
'.github/workflows/build-rpc.yml',
'**/CMakeLists.txt',
'**/.cmake',
'**/*.h',
'**/*.hpp',
'**/*.c',
'**/*.cpp'
]
pull_request:
types: [opened, synchronize, reopened]
paths: [
'.github/workflows/build-rpc.yml',
'ggml/src/ggml-rpc/**'
]
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}
cancel-in-progress: true
env:
GGML_NLOOP: 3
GGML_N_THREADS: 1
LLAMA_ARG_LOG_COLORS: 1
LLAMA_ARG_LOG_PREFIX: 1
LLAMA_ARG_LOG_TIMESTAMPS: 1
jobs:
ubuntu-24-rpc:
runs-on: ${{ 'ubuntu-24.04-arm' || 'ubuntu-24.04' }}
continue-on-error: true
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: Dependencies
id: depends
run: |
sudo apt-get update
sudo apt-get install build-essential libssl-dev ninja-build
- name: Build
id: cmake_build
run: |
cmake -B build \
-G "Ninja" \
-DCMAKE_BUILD_TYPE=Release \
-DGGML_RPC=ON
time cmake --build build --config Release -j $(nproc)
- name: Test
id: cmake_test
run: |
cd build
ctest -L main --verbose
+17 -45
View File
@@ -15,16 +15,6 @@ on:
'**/*.cpp' '**/*.cpp'
] ]
pull_request:
types: [opened, synchronize, reopened]
paths: [
'.github/workflows/build-sanitize.yml'
]
cache-mode: none
permissions:
contents: read
concurrency: concurrency:
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }} group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}
cancel-in-progress: true cancel-in-progress: true
@@ -38,43 +28,35 @@ env:
jobs: jobs:
ctest: ctest:
runs-on: [self-hosted, X64, CPU, Linux]
continue-on-error: true continue-on-error: true
strategy: strategy:
matrix: matrix:
include: sanitizer: [ADDRESS, THREAD, UNDEFINED]
- sanitizer: ADDRESS
machine: ubuntu-26.04
- sanitizer: THREAD
machine: ubuntu-26.04
- sanitizer: UNDEFINED
machine: ubuntu-26.04
runs-on: ${{ matrix.machine }}
steps: steps:
- name: Clone - name: Clone
id: checkout id: checkout
uses: actions/checkout@v6 uses: actions/checkout@v6
- name: ccache # with UNDEFINED sanitizer, we have to build in Debug to avoid GCC 13 false-positive warnings
uses: ggml-org/ccache-action@v1.2.24 - name: Build (undefined)
with: id: cmake_build_undefined
restore: false if: ${{ matrix.sanitizer == 'UNDEFINED' }}
save: false run: |
cmake -B build \
-DCMAKE_BUILD_TYPE=Debug \
-DLLAMA_FATAL_WARNINGS=ON \
-DLLAMA_SANITIZE_${{ matrix.sanitizer }}=ON \
-DGGML_SANITIZE_${{ matrix.sanitizer }}=ON
- name: ccache-buckets-restore cmake --build build --config Debug -j $(nproc)
uses: ./.github/actions/ccache-buckets
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CI }}
with:
key: ctest-sanitizer-${{ matrix.sanitizer }}
folder: llama.cpp
hf_bucket: ggml-org/cache
- name: Build - name: Build
id: cmake_build id: cmake_build
if: ${{ matrix.sanitizer == 'ADDRESS' || matrix.sanitizer == 'UNDEFINED' }} if: ${{ matrix.sanitizer == 'ADDRESS' }}
run: | run: |
cmake -B build \ cmake -B build \
-DCMAKE_BUILD_TYPE=RelWithDebInfo \ -DCMAKE_BUILD_TYPE=RelWithDebInfo \
@@ -95,20 +77,10 @@ jobs:
cmake --build build --config RelWithDebInfo -j $(nproc) cmake --build build --config RelWithDebInfo -j $(nproc)
- name: ccache-buckets-save
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
uses: ./.github/actions/ccache-buckets
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CACHE_OUTPUT }}
with:
key: ctest-sanitizer-${{ matrix.sanitizer }}
folder: llama.cpp
evict-old-files: 1d
hf_bucket: ggml-org/cache
save: true
- name: Test - name: Test
id: cmake_test id: cmake_test
# skip run in Debug - very slow
if: ${{ matrix.sanitizer != 'UNDEFINED' }}
run: | run: |
cd build cd build
ctest -L main -E tokenizer --verbose --timeout 900 ctest -L main -E tokenizer --verbose --timeout 900
+387
View File
@@ -0,0 +1,387 @@
name: CI (self-hosted)
on:
workflow_dispatch: # allows manual triggering
push:
branches:
- master
paths: [
'.github/workflows/build.yml',
'**/CMakeLists.txt',
'**/.cmake',
'**/*.h',
'**/*.hpp',
'**/*.c',
'**/*.cpp',
'**/*.cu',
'**/*.cuh',
'**/*.swift',
'**/*.m',
'**/*.metal',
'**/*.comp',
'**/*.glsl',
'**/*.wgsl'
]
pull_request:
types: [opened, synchronize, reopened]
paths: [
'.github/workflows/build-self-hosted.yml',
'**/CMakeLists.txt',
'**/.cmake',
'**/*.h',
'**/*.hpp',
'**/*.c',
'**/*.cpp',
'**/*.cu',
'**/*.cuh',
'**/*.swift',
'**/*.m',
'**/*.metal',
'**/*.comp',
'**/*.glsl',
'**/*.wgsl'
]
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}
cancel-in-progress: true
env:
GGML_NLOOP: 3
GGML_N_THREADS: 1
LLAMA_ARG_LOG_COLORS: 1
LLAMA_ARG_LOG_PREFIX: 1
LLAMA_ARG_LOG_TIMESTAMPS: 1
jobs:
gpu-cuda:
runs-on: [self-hosted, Linux, NVIDIA]
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: Test
id: ggml-ci
run: |
nvidia-smi
GG_BUILD_CUDA=1 bash ./ci/run.sh ~/results/llama.cpp ~/mnt/llama.cpp
gpu-vulkan-nvidia-cm:
runs-on: [self-hosted, Linux, NVIDIA]
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: Test
id: ggml-ci
run: |
vulkaninfo --summary
GG_BUILD_VULKAN=1 GGML_VK_DISABLE_COOPMAT2=1 bash ./ci/run.sh ~/results/llama.cpp ~/mnt/llama.cpp
gpu-vulkan-nvidia-cm2:
runs-on: [self-hosted, Linux, NVIDIA, COOPMAT2]
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: Test
id: ggml-ci
run: |
vulkaninfo --summary
GG_BUILD_VULKAN=1 bash ./ci/run.sh ~/results/llama.cpp ~/mnt/llama.cpp
gpu-webgpu-nvidia:
runs-on: [self-hosted, Linux, NVIDIA, X64]
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: Dawn Dependency
id: dawn-depends
run: |
DAWN_VERSION="v20260317.182325"
DAWN_OWNER="google"
DAWN_REPO="dawn"
DAWN_ASSET_NAME="Dawn-18eb229ef5f707c1464cc581252e7603c73a3ef0-ubuntu-latest-Release"
echo "Fetching release asset from https://github.com/google/dawn/releases/download/${DAWN_VERSION}/${DAWN_ASSET_NAME}.tar.gz"
curl -L -o artifact.tar.gz \
"https://github.com/google/dawn/releases/download/${DAWN_VERSION}/${DAWN_ASSET_NAME}.tar.gz"
mkdir dawn
tar -xvf artifact.tar.gz -C dawn --strip-components=1
- name: Test
id: ggml-ci
run: |
GG_BUILD_WEBGPU=1 \
GG_BUILD_WEBGPU_DAWN_PREFIX="$GITHUB_WORKSPACE/dawn" \
GG_BUILD_WEBGPU_DAWN_DIR="$GITHUB_WORKSPACE/dawn/lib64/cmake/Dawn" \
bash ./ci/run.sh ~/results/llama.cpp ~/mnt/llama.cpp
# TODO: provision AMX-compatible machine
#cpu-amx:
# runs-on: [self-hosted, Linux, CPU, AMX]
# steps:
# - name: Clone
# id: checkout
# uses: actions/checkout@v6
# - name: Test
# id: ggml-ci
# run: |
# bash ./ci/run.sh ~/results/llama.cpp ~/mnt/llama.cpp
# TODO: provision AMD GPU machine
# amd-vulkan:
# runs-on: [self-hosted, Linux, AMD]
# steps:
# - name: Clone
# id: checkout
# uses: actions/checkout@v6
# - name: Test
# id: ggml-ci
# run: |
# vulkaninfo --summary
# GG_BUILD_VULKAN=1 bash ./ci/run.sh ~/results/llama.cpp ~/mnt/llama.cpp
# TODO: provision AMD GPU machine
# amd-rocm:
# runs-on: [self-hosted, Linux, AMD]
# steps:
# - name: Clone
# id: checkout
# uses: actions/checkout@v6
# - name: Test
# id: ggml-ci
# run: |
# amd-smi static
# GG_BUILD_ROCM=1 GG_BUILD_AMDGPU_TARGETS="gfx1101" bash ./ci/run.sh ~/results/llama.cpp ~/mnt/llama.cpp
gpu-metal:
runs-on: [self-hosted, macOS, ARM64]
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: Test
id: ggml-ci
run: |
GG_BUILD_METAL=1 bash ./ci/run.sh ~/results/llama.cpp ~/mnt/llama.cpp
gpu-webgpu-apple:
runs-on: [self-hosted, macOS, ARM64]
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: Dawn Dependency
id: dawn-depends
run: |
DAWN_VERSION="v20260317.182325"
DAWN_OWNER="google"
DAWN_REPO="dawn"
DAWN_ASSET_NAME="Dawn-18eb229ef5f707c1464cc581252e7603c73a3ef0-macos-latest-Release"
echo "Fetching release asset from https://github.com/google/dawn/releases/download/${DAWN_VERSION}/${DAWN_ASSET_NAME}.tar.gz"
curl -L -o artifact.tar.gz \
"https://github.com/google/dawn/releases/download/${DAWN_VERSION}/${DAWN_ASSET_NAME}.tar.gz"
mkdir dawn
tar -xvf artifact.tar.gz -C dawn --strip-components=1
- name: Test
id: ggml-ci
run: |
GG_BUILD_WEBGPU=1 GG_BUILD_WEBGPU_DAWN_PREFIX="$GITHUB_WORKSPACE/dawn" \
bash ./ci/run.sh ~/results/llama.cpp ~/mnt/llama.cpp
gpu-vulkan-apple:
runs-on: [self-hosted, macOS, ARM64]
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: Test
id: ggml-ci
run: |
vulkaninfo --summary
GG_BUILD_VULKAN=1 bash ./ci/run.sh ~/results/llama.cpp ~/mnt/llama.cpp
gpu-vulkan-intel-linux:
runs-on: [self-hosted, Linux, Intel]
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
with:
persist-credentials: false
- name: Test
id: ggml-ci
run: |
vulkaninfo --summary
GG_BUILD_VULKAN=1 bash ./ci/run.sh ~/results/llama.cpp ~/mnt/llama.cpp
gpu-vulkan-intel-windows:
runs-on: [self-hosted, Windows, X64, Intel]
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: Test
id: ggml-ci
shell: C:\msys64\usr\bin\bash.exe --noprofile --norc -eo pipefail "{0}"
env:
MSYSTEM: UCRT64
CHERE_INVOKING: 1
PATH: C:\msys64\ucrt64\bin;C:\msys64\usr\bin;C:\Windows\System32;${{ env.PATH }}
run: |
vulkaninfo --summary
# Skip python related tests with GG_BUILD_LOW_PERF=1 since Windows MSYS2 UCRT64 currently fails to create
# a valid python environment for testing
LLAMA_FATAL_WARNINGS=OFF GG_BUILD_NINJA=1 GG_BUILD_VULKAN=1 GG_BUILD_LOW_PERF=1 ./ci/run.sh ./results/llama.cpp ./mnt/llama.cpp
gpu-openvino-low-perf:
runs-on: [self-hosted, Linux, Intel, OpenVINO]
env:
# Sync versions in build.yml, build-self-hosted.yml, release.yml, build-cache.yml, .devops/openvino.Dockerfile
OPENVINO_VERSION_MAJOR: "2026.2.1"
OPENVINO_VERSION_FULL: "2026.2.1.21919.ede283a88e3"
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: Setup OpenVINO Toolkit
uses: ./.github/actions/linux-setup-openvino
with:
path: ./openvino_toolkit
version_major: ${{ env.OPENVINO_VERSION_MAJOR }}
version_full: ${{ env.OPENVINO_VERSION_FULL }}
- name: Install OpenVINO dependencies
run: |
cd ./openvino_toolkit
chmod +x ./install_dependencies/install_openvino_dependencies.sh
echo "Y" | sudo -E ./install_dependencies/install_openvino_dependencies.sh
- name: Test
id: ggml-ci
run: |
source ./openvino_toolkit/setupvars.sh
GG_BUILD_OPENVINO=1 GGML_OPENVINO_DEVICE=GPU GG_BUILD_LOW_PERF=1 bash ./ci/run.sh ~/results/llama.cpp ~/mnt/llama.cpp
cpu-x64-high-perf:
runs-on: [self-hosted, Linux, X64]
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: Test
id: ggml-ci
run: |
LLAMA_ARG_THREADS=$(nproc) GG_BUILD_HIGH_PERF=1 GG_BUILD_EXTRA_TESTS_0=1 bash ./ci/run.sh ~/results/llama.cpp ~/mnt/llama.cpp
cpu-arm64-high-perf-graviton4:
runs-on: ah-ubuntu_22_04-c8g_8x
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: Dependencies
id: depends
run: |
set -euxo pipefail
sudo apt-get update
sudo DEBIAN_FRONTEND=noninteractive NEEDRESTART_MODE=a \
apt-get install -y \
build-essential \
python3-venv \
gpg \
wget \
time \
git-lfs
git lfs install
# install the latest cmake
sudo install -d /usr/share/keyrings
wget -O - https://apt.kitware.com/keys/kitware-archive-latest.asc \
| gpg --dearmor \
| sudo tee /usr/share/keyrings/kitware-archive-keyring.gpg >/dev/null
echo 'deb [signed-by=/usr/share/keyrings/kitware-archive-keyring.gpg] https://apt.kitware.com/ubuntu/ jammy main' \
| sudo tee /etc/apt/sources.list.d/kitware.list
sudo apt-get update
sudo apt-get install -y cmake
- name: Test
id: ggml-ci
run: |
LLAMA_ARG_THREADS=$(nproc) GG_BUILD_HIGH_PERF=1 GG_BUILD_NO_BF16=1 GG_BUILD_EXTRA_TESTS_0=1 bash ./ci/run.sh ~/results/llama.cpp ~/mnt/llama.cpp
cpu-arm64-graviton4-kleidiai:
runs-on: ah-ubuntu_22_04-c8g_8x
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: Dependencies
id: depends
run: |
set -euxo pipefail
sudo apt-get update
sudo DEBIAN_FRONTEND=noninteractive NEEDRESTART_MODE=a \
apt-get install -y \
build-essential \
python3-venv \
gpg \
wget \
time \
git-lfs
git lfs install
# install the latest cmake
sudo install -d /usr/share/keyrings
wget -O - https://apt.kitware.com/keys/kitware-archive-latest.asc \
| gpg --dearmor \
| sudo tee /usr/share/keyrings/kitware-archive-keyring.gpg >/dev/null
echo 'deb [signed-by=/usr/share/keyrings/kitware-archive-keyring.gpg] https://apt.kitware.com/ubuntu/ jammy main' \
| sudo tee /etc/apt/sources.list.d/kitware.list
sudo apt-get update
sudo apt-get install -y cmake
- name: Test
id: ggml-ci
run: |
GG_BUILD_KLEIDIAI=1 \
GG_BUILD_EXTRA_TESTS_0=1 \
bash ./ci/run.sh ./tmp/results ./tmp/mnt
+20 -79
View File
@@ -22,10 +22,6 @@ on:
'ggml/src/ggml-sycl/**' 'ggml/src/ggml-sycl/**'
] ]
cache-mode: none
permissions:
contents: read
concurrency: concurrency:
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }} group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}
cancel-in-progress: true cancel-in-progress: true
@@ -38,7 +34,7 @@ env:
LLAMA_ARG_LOG_TIMESTAMPS: 1 LLAMA_ARG_LOG_TIMESTAMPS: 1
jobs: jobs:
ubuntu-26-sycl: ubuntu-24-sycl:
strategy: strategy:
matrix: matrix:
build: [fp32, fp16] build: [fp32, fp16]
@@ -48,12 +44,12 @@ jobs:
- build: fp16 - build: fp16
fp16: ON fp16: ON
runs-on: ubuntu-26.04 runs-on: ubuntu-24.04
env: env:
ONEAPI_ROOT: /opt/intel/oneapi/ ONEAPI_ROOT: /opt/intel/oneapi/
ONEAPI_INSTALLER_VERSION: "2026.2" ONEAPI_INSTALLER_VERSION: "2025.3.3"
LEVEL_ZERO_VERSION: "1.34.0" LEVEL_ZERO_VERSION: "1.28.2"
LEVEL_ZERO_UBUNTU_VERSION: "u24.04" LEVEL_ZERO_UBUNTU_VERSION: "u24.04"
continue-on-error: true continue-on-error: true
@@ -67,45 +63,23 @@ jobs:
shell: bash shell: bash
run: | run: |
cd /tmp cd /tmp
echo "Download oneAPI Deep Learning Essentials installation package" wget https://registrationcenter-download.intel.com/akdlm/IRC_NAS/56f7923a-adb8-43f3-8b02-2b60fcac8cab/intel-deep-learning-essentials-2025.3.3.16_offline.sh -O intel-deep-learning-essentials_offline.sh
wget https://registrationcenter-download.intel.com/akdlm/IRC_NAS/aef232d2-aac5-4cd6-bceb-911019bbb6a8/intel-deep-learning-essentials-2026.2.0.640_offline.sh -O intel-deep-learning-essentials_offline.sh
echo "Install oneAPI Deep Learning Essentials components"
sudo bash intel-deep-learning-essentials_offline.sh -s -a --silent --eula accept sudo bash intel-deep-learning-essentials_offline.sh -s -a --silent --eula accept
echo "Download oneDNN installation package"
wget https://registrationcenter-download.intel.com/akdlm/IRC_NAS/46452a4a-7b7f-49f2-be4c-602a900180bb/intel-onednn-2026.1.0.625_offline.sh -O intel-onednn_offline.sh
echo "Install oneDNN components"
sudo bash intel-onednn_offline.sh -s -a --silent --eula accept
- name: Install Level Zero SDK - name: Install Level Zero SDK
shell: bash shell: bash
run: | run: |
cd /tmp cd /tmp
# v1.33.x renamed the Debian packages to libze1 / libze-dev wget -q "https://github.com/oneapi-src/level-zero/releases/download/v${LEVEL_ZERO_VERSION}/level-zero_${LEVEL_ZERO_VERSION}%2B${LEVEL_ZERO_UBUNTU_VERSION}_amd64.deb" -O level-zero.deb
wget -q "https://github.com/oneapi-src/level-zero/releases/download/v${LEVEL_ZERO_VERSION}/libze1_${LEVEL_ZERO_VERSION}%2B${LEVEL_ZERO_UBUNTU_VERSION}_amd64.deb" -O libze1.deb wget -q "https://github.com/oneapi-src/level-zero/releases/download/v${LEVEL_ZERO_VERSION}/level-zero-devel_${LEVEL_ZERO_VERSION}%2B${LEVEL_ZERO_UBUNTU_VERSION}_amd64.deb" -O level-zero-devel.deb
wget -q "https://github.com/oneapi-src/level-zero/releases/download/v${LEVEL_ZERO_VERSION}/libze-dev_${LEVEL_ZERO_VERSION}%2B${LEVEL_ZERO_UBUNTU_VERSION}_amd64.deb" -O libze-dev.deb sudo apt-get install -y ./level-zero.deb ./level-zero-devel.deb
sudo apt-get install -y ./libze1.deb ./libze-dev.deb
- name: Dependencies
id: depends
run: |
sudo apt-get update
sudo apt-get install -y libssl-dev
- name: ccache - name: ccache
uses: ggml-org/ccache-action@v1.2.24 uses: ggml-org/ccache-action@v1.2.21
with: with:
restore: false key: sycl-ubuntu-24-${{ matrix.build }}
save: false evict-old-files: 1d
save: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
- name: ccache-buckets-restore
uses: ./.github/actions/ccache-buckets
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CI }}
with:
key: sycl-ubuntu-26-${{ matrix.build }}
folder: llama.cpp
hf_bucket: ggml-org/cache
- name: Build - name: Build
id: cmake_build id: cmake_build
@@ -117,46 +91,24 @@ jobs:
-DGGML_SYCL=ON \ -DGGML_SYCL=ON \
-DCMAKE_C_COMPILER=icx \ -DCMAKE_C_COMPILER=icx \
-DCMAKE_CXX_COMPILER=icpx \ -DCMAKE_CXX_COMPILER=icpx \
-DLLAMA_OPENSSL=OFF \
-DGGML_NATIVE=OFF \ -DGGML_NATIVE=OFF \
-DGGML_SYCL_F16=${{ matrix.fp16 }} \ -DGGML_SYCL_F16=${{ matrix.fp16 }}
-DGGML_SYCL_SUPPORT_LEVEL_ZERO_API=ON \
-DGGML_SYCL_DNN=ON \
-DCMAKE_CXX_FLAGS="-fsycl-unnamed-lambda" \
-DCMAKE_EXE_LINKER_FLAGS="-fsycl-unnamed-lambda"
time cmake --build build --config Release -j $(nproc) time cmake --build build --config Release -j $(nproc)
- name: ccache-buckets-save
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
uses: ./.github/actions/ccache-buckets
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CACHE_OUTPUT }}
with:
key: sycl-ubuntu-26-${{ matrix.build }}
folder: llama.cpp
evict-old-files: 1d
hf_bucket: ggml-org/cache
save: true
windows-latest-sycl: windows-latest-sycl:
runs-on: windows-2022 runs-on: windows-2022
cache-mode: write
permissions:
actions: write
contents: read
defaults: defaults:
run: run:
shell: bash shell: bash
env: env:
WINDOWS_DLE_URL: https://registrationcenter-download.intel.com/akdlm/IRC_NAS/b177a006-5cd9-4cb2-807f-8199cc73d53c/intel-deep-learning-essentials-2026.2.0.638_offline.exe WINDOWS_BASEKIT_URL: https://registrationcenter-download.intel.com/akdlm/IRC_NAS/b60765d1-2b85-4e85-86b6-cb0e9563a699/intel-deep-learning-essentials-2025.3.3.18_offline.exe
WINDOWS_ONEDNN_URL: https://registrationcenter-download.intel.com/akdlm/IRC_NAS/8f905666-c85c-43fb-9bd0-2618c7ecd33b/intel-onednn-2026.1.0.622_offline.exe WINDOWS_DPCPP_MKL: intel.oneapi.win.cpp-dpcpp-common:intel.oneapi.win.mkl.devel:intel.oneapi.win.dnnl:intel.oneapi.win.tbb.devel
WINDOWS_DPCPP_MKL: intel.oneapi.win.cpp-dpcpp-common:intel.oneapi.win.mkl.devel:intel.oneapi.win.tbb.devel LEVEL_ZERO_SDK_URL: https://github.com/oneapi-src/level-zero/releases/download/v1.28.2/level-zero-win-sdk-1.28.2.zip
WINDOWS_DNNL: intel.oneapi.win.dnnl
LEVEL_ZERO_SDK_URL: https://github.com/oneapi-src/level-zero/releases/download/v1.34.0/level-zero-win-sdk-1.34.0.zip
ONEAPI_ROOT: "C:/Program Files (x86)/Intel/oneAPI" ONEAPI_ROOT: "C:/Program Files (x86)/Intel/oneAPI"
ONEAPI_INSTALLER_VERSION: "2026.2" ONEAPI_INSTALLER_VERSION: "2025.3.3"
steps: steps:
- name: Clone - name: Clone
id: checkout id: checkout
@@ -165,8 +117,7 @@ jobs:
- name: Download & Install oneAPI - name: Download & Install oneAPI
shell: bash shell: bash
run: | run: |
scripts/install-oneapi.bat $WINDOWS_DLE_URL $WINDOWS_DPCPP_MKL scripts/install-oneapi.bat $WINDOWS_BASEKIT_URL $WINDOWS_DPCPP_MKL
scripts/install-oneapi.bat $WINDOWS_ONEDNN_URL $WINDOWS_DNNL
- name: Install Level Zero SDK - name: Install Level Zero SDK
shell: pwsh shell: pwsh
@@ -176,7 +127,7 @@ jobs:
"LEVEL_ZERO_V1_SDK_PATH=C:/level-zero-sdk" | Out-File -FilePath $env:GITHUB_ENV -Append "LEVEL_ZERO_V1_SDK_PATH=C:/level-zero-sdk" | Out-File -FilePath $env:GITHUB_ENV -Append
- name: ccache - name: ccache
uses: ggml-org/ccache-action@v1.2.24 uses: ggml-org/ccache-action@v1.2.21
with: with:
key: sycl-windows-latest key: sycl-windows-latest
variant: ccache variant: ccache
@@ -188,13 +139,3 @@ jobs:
- name: Build - name: Build
id: cmake_build id: cmake_build
run: examples/sycl/win-build-sycl.bat run: examples/sycl/win-build-sycl.bat
- name: ccache-clear
uses: ./.github/actions/ccache-clear
env:
GH_TOKEN: ${{ github.token }}
with:
key: sycl-windows-latest
older: 5m
min: 1
dry-run: ${{ github.event_name != 'push' || github.ref != 'refs/heads/master' }}
+2 -6
View File
@@ -22,17 +22,13 @@ on:
'ggml/src/ggml-virtgpu/**' 'ggml/src/ggml-virtgpu/**'
] ]
cache-mode: none
permissions:
contents: read
concurrency: concurrency:
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }} group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}
cancel-in-progress: true cancel-in-progress: true
jobs: jobs:
ubuntu-26-virtgpu: ubuntu-24-virtgpu:
runs-on: ${{ 'ubuntu-26.04-arm' || 'ubuntu-26.04' }} runs-on: ${{ 'ubuntu-24.04-arm' || 'ubuntu-24.04' }}
steps: steps:
- name: Clone - name: Clone
+27 -144
View File
@@ -24,10 +24,6 @@ on:
'ggml/src/ggml-vulkan/**' 'ggml/src/ggml-vulkan/**'
] ]
cache-mode: none
permissions:
contents: read
concurrency: concurrency:
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }} group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}
cancel-in-progress: true cancel-in-progress: true
@@ -35,14 +31,13 @@ concurrency:
env: env:
GGML_NLOOP: 3 GGML_NLOOP: 3
GGML_N_THREADS: 1 GGML_N_THREADS: 1
GGML_SCHED_DEBUG_REALLOC: 1
LLAMA_ARG_LOG_COLORS: 1 LLAMA_ARG_LOG_COLORS: 1
LLAMA_ARG_LOG_PREFIX: 1 LLAMA_ARG_LOG_PREFIX: 1
LLAMA_ARG_LOG_TIMESTAMPS: 1 LLAMA_ARG_LOG_TIMESTAMPS: 1
jobs: jobs:
ubuntu-arm64: ubuntu-arm64:
runs-on: ubuntu-26.04-arm runs-on: ubuntu-24.04-arm
steps: steps:
- name: Clone - name: Clone
@@ -53,23 +48,17 @@ jobs:
id: depends id: depends
run: | run: |
sudo apt-get update sudo apt-get update
sudo apt-get install -y build-essential glslc libvulkan-dev spirv-headers libssl-dev ninja-build sudo apt-get install -y gcc-14 g++-14 build-essential glslc libvulkan-dev spirv-headers libssl-dev ninja-build
echo "CC=gcc-14" >> "$GITHUB_ENV"
echo "CXX=g++-14" >> "$GITHUB_ENV"
- name: ccache - name: ccache
uses: ggml-org/ccache-action@v1.2.24 uses: ggml-org/ccache-action@v1.2.21
with: with:
restore: false key: vulkan-ubuntu-24.04-arm-new
variant: ccache variant: ccache
save: false evict-old-files: 1d
save: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
- name: ccache-buckets-restore
uses: ./.github/actions/ccache-buckets
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CI }}
with:
key: vulkan-ubuntu-26.04-arm
folder: llama.cpp
hf_bucket: ggml-org/cache
- name: Configure - name: Configure
id: cmake_configure id: cmake_configure
@@ -84,20 +73,8 @@ jobs:
run: | run: |
time cmake --build build -j $(nproc) time cmake --build build -j $(nproc)
- name: ccache-buckets-save
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
uses: ./.github/actions/ccache-buckets
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CACHE_OUTPUT }}
with:
key: vulkan-ubuntu-26.04-arm
folder: llama.cpp
evict-old-files: 1d
hf_bucket: ggml-org/cache
save: true
ubuntu-llvmpipe: ubuntu-llvmpipe:
runs-on: ubuntu-26.04 runs-on: ubuntu-24.04
steps: steps:
- name: Clone - name: Clone
@@ -116,50 +93,35 @@ jobs:
run: | run: |
echo "VULKAN_SDK_VERSION=$(curl https://vulkan.lunarg.com/sdk/latest/linux.txt)" >> "$GITHUB_ENV" echo "VULKAN_SDK_VERSION=$(curl https://vulkan.lunarg.com/sdk/latest/linux.txt)" >> "$GITHUB_ENV"
- name: Setup Vulkan SDK - name: Use Vulkan SDK Cache
id: setup uses: actions/cache@v5
uses: ./.github/actions/unarchive-tar id: cache-sdk
with: with:
url: https://sdk.lunarg.com/sdk/download/${{ env.VULKAN_SDK_VERSION }}/linux/vulkan_sdk.tar.xz
path: ./vulkan_sdk path: ./vulkan_sdk
strip: 1 key: cache-gha-vulkan-sdk-${{ env.VULKAN_SDK_VERSION }}-${{ runner.os }}
- name: Setup Vulkan SDK
if: steps.cache-sdk.outputs.cache-hit != 'true'
uses: ./.github/actions/linux-setup-vulkan
with:
path: ./vulkan_sdk
version: ${{ env.VULKAN_SDK_VERSION }}
- name: ccache - name: ccache
uses: ggml-org/ccache-action@v1.2.24 uses: ggml-org/ccache-action@v1.2.21
with: with:
restore: false key: vulkan-ubuntu-24.04-llvmpipe
save: false evict-old-files: 1d
save: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
- name: ccache-buckets-restore
uses: ./.github/actions/ccache-buckets
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CI }}
with:
key: vulkan-ubuntu-26.04-llvmpipe
folder: llama.cpp
hf_bucket: ggml-org/cache
- name: Build - name: Build
id: cmake_build id: cmake_build
run: | run: |
source ./vulkan_sdk/setup-env.sh source ./vulkan_sdk/setup-env.sh
cmake -B build \ cmake -B build \
-DGGML_NATIVE=OFF \
-DGGML_VULKAN=ON -DGGML_VULKAN=ON
cmake --build build --config Release -j $(nproc) cmake --build build --config Release -j $(nproc)
- name: ccache-buckets-save
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
uses: ./.github/actions/ccache-buckets
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CACHE_OUTPUT }}
with:
key: vulkan-ubuntu-26.04-llvmpipe
folder: llama.cpp
evict-old-files: 1d
hf_bucket: ggml-org/cache
save: true
- name: Test - name: Test
id: cmake_test id: cmake_test
run: | run: |
@@ -167,85 +129,6 @@ jobs:
export GGML_VK_VISIBLE_DEVICES=0 export GGML_VK_VISIBLE_DEVICES=0
export GGML_VK_DISABLE_F16=1 export GGML_VK_DISABLE_F16=1
export GGML_VK_DISABLE_COOPMAT=1 export GGML_VK_DISABLE_COOPMAT=1
ctest -L main --verbose --timeout 900 # This is using llvmpipe and runs slower than other backends
# test-backend-ops is too slow on llvmpipe, skip it
windows: ctest -L main -E test-backend-ops --verbose --timeout 900
name: windows / ${{ matrix.arch }}
runs-on: windows-2025
cache-mode: write
permissions:
actions: write
contents: read
strategy:
matrix:
include:
- arch: 'x64'
- arch: 'arm64'
env:
VULKAN_VERSION: 1.4.357.0
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: ccache
uses: ggml-org/ccache-action@v1.2.24
with:
key: cpu-windows-2025-${{ matrix.arch }}-vulkan
variant: ccache
evict-old-files: 1d
save: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
- name: Install Vulkan SDK
id: get_vulkan
run: |
curl.exe -o $env:RUNNER_TEMP/VulkanSDK-Installer.exe -L "https://sdk.lunarg.com/sdk/download/${env:VULKAN_VERSION}/windows/vulkansdk-windows-X64-${env:VULKAN_VERSION}.exe"
& "$env:RUNNER_TEMP\VulkanSDK-Installer.exe" --accept-licenses --default-answer --confirm-command install ${{ matrix.arch == 'arm64' && 'com.lunarg.vulkan.arm64' || '' }}
Add-Content $env:GITHUB_ENV "VULKAN_SDK=C:\VulkanSDK\${env:VULKAN_VERSION}"
Add-Content $env:GITHUB_PATH "C:\VulkanSDK\${env:VULKAN_VERSION}\bin"
- name: Install Ninja
id: install_ninja
run: |
choco install ninja
- name: Install LLVM
run: |
choco install llvm --version=22.1.8
echo "C:\Program Files\LLVM\bin" >> $env:GITHUB_PATH
- name: Build
id: cmake_build
run: |
cmake -S . -B build -G "Ninja Multi-Config" `
-D CMAKE_TOOLCHAIN_FILE=cmake/${{ matrix.arch }}-windows-llvm.cmake `
-DCMAKE_BUILD_TYPE=Release `
-DGGML_NATIVE=OFF `
-DLLAMA_BUILD_SERVER=ON `
-DGGML_RPC=ON `
-DGGML_BACKEND_DL=ON `
-DGGML_CPU_ALL_VARIANTS=${{ matrix.arch == 'x64' && 'ON' || 'OFF' }} `
-DGGML_VULKAN=ON `
-DLLAMA_BUILD_BORINGSSL=ON
cmake --build build --config Release -j ${env:NUMBER_OF_PROCESSORS}
- name: Test
if: ${{ matrix.arch == 'x64' }}
id: cmake_test
run: |
cd build
ctest -L main -C Release --verbose --timeout 900
- name: ccache-clear
uses: ./.github/actions/ccache-clear
env:
GH_TOKEN: ${{ github.token }}
with:
key: cpu-windows-2025-${{ matrix.arch }}-vulkan
older: 5m
min: 1
dry-run: ${{ github.event_name != 'push' || github.ref != 'refs/heads/master' }}
-142
View File
@@ -1,142 +0,0 @@
name: CI (wasi)
on:
workflow_dispatch: # allows manual triggering
push:
branches:
- master
paths: [
'.github/workflows/build-wasi.yml',
'**/CMakeLists.txt',
'**/.cmake',
'**/*.h',
'**/*.hpp',
'**/*.c',
'**/*.cpp',
'**/*.wgsl',
'**/*.tmpl',
'ggml/src/ggml-webgpu/wgsl-shaders/embed_wgsl.py'
]
pull_request:
types: [opened, synchronize, reopened]
paths: [
'.github/workflows/build-wasi.yml',
'**/CMakeLists.txt',
'**/.cmake',
'**/*.h',
'**/*.hpp',
'**/*.c',
'**/*.cpp',
'**/*.wgsl',
'**/*.tmpl',
'ggml/src/ggml-webgpu/wgsl-shaders/embed_wgsl.py'
]
cache-mode: none
permissions:
contents: read
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}
cancel-in-progress: true
env:
WASI_SDK_VERSION: 34
DAWN_VERSION: v20260908.214631
DAWN_COMMIT: 94c3c9cc0d5fb2e85aebb370fa8d37b71aa34655
WASI_WEBGPU_HEADERS_COMMIT: 917765edd7cb8f8cf95bda1217514c78f71b582b
jobs:
ubuntu-webgpu:
runs-on: ubuntu-26.04-arm
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: ccache
uses: ggml-org/ccache-action@v1.2.24
with:
restore: false
save: false
- name: Install wasi-sdk
uses: bytecodealliance/setup-wasi-sdk-action@77d09f3df1e8da344d8403cf8631af8977938b4c # v1.0.0
with:
version: ${{ env.WASI_SDK_VERSION }}
add-to-path: false
- name: wasi:webgpu Dependency
run: |
mkdir -p wasi-webgpu/dawn-headers
curl -fL "https://github.com/google/dawn/releases/download/${DAWN_VERSION}/dawn-headers-${DAWN_COMMIT}.tar.gz" |
tar -xz -C wasi-webgpu/dawn-headers --strip-components=1
git init -q wasi-webgpu/wasi-webgpu-headers
cd wasi-webgpu/wasi-webgpu-headers
git fetch -q --depth 1 https://github.com/wasi-gfx/wasi-webgpu-headers.git ${WASI_WEBGPU_HEADERS_COMMIT}
git checkout -q FETCH_HEAD
git submodule update -q --init --depth 1
cd ../..
# ggml-webgpu does find_package(Dawn) and links dawn::webgpu_dawn. There is no Dawn on WASI,
# so this target instead compiles wasi-webgpu-headers (webgpu.h over wasi:webgpu imports)
# and exposes Dawn's webgpu_cpp.h on top of it.
cat > wasi-webgpu/DawnConfig.cmake << 'EOF'
set(_wasi_webgpu "${CMAKE_CURRENT_LIST_DIR}/wasi-webgpu-headers")
add_library(dawn::webgpu_dawn INTERFACE IMPORTED)
set_target_properties(dawn::webgpu_dawn PROPERTIES
INTERFACE_SOURCES "${_wasi_webgpu}/webgpu.c;${_wasi_webgpu}/async_futures.c;${_wasi_webgpu}/imports.c;${_wasi_webgpu}/imports_component_type.o"
INTERFACE_INCLUDE_DIRECTORIES "${CMAKE_CURRENT_LIST_DIR}/dawn-headers/include/webgpu_upstream;${_wasi_webgpu}")
EOF
- name: ccache-buckets-restore
uses: ./.github/actions/ccache-buckets
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CI }}
with:
key: webgpu-ubuntu-26.04-arm-wasi
folder: llama.cpp
hf_bucket: ggml-org/cache
# llama-common does not build for WASI, so the executables are linked by hand
- name: Build
id: cmake_build
run: |
WASI_FLAGS="-msimd128 -D_WASI_EMULATED_SIGNAL -fwasm-exceptions -mllvm -wasm-use-legacy-eh=false"
cmake -B build-wasi \
-G "Ninja" \
-DCMAKE_TOOLCHAIN_FILE=${WASI_SDK_PATH}/share/cmake/wasi-sdk-p3.cmake \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_C_FLAGS="${WASI_FLAGS}" \
-DCMAKE_CXX_FLAGS="${WASI_FLAGS}" \
-DCMAKE_HAVE_LIBC_PTHREAD=1 \
-DBUILD_SHARED_LIBS=OFF \
-DGGML_NATIVE=OFF \
-DGGML_OPENMP=OFF \
-DGGML_WEBGPU=ON \
-DDawn_DIR=$PWD/wasi-webgpu \
-DLLAMA_BUILD_COMMON=OFF
time cmake --build build-wasi --config Release --target llama -j $(nproc)
LIBS="build-wasi/ggml/src/libggml.a build-wasi/ggml/src/ggml-webgpu/libggml-webgpu.a build-wasi/ggml/src/libggml-cpu.a build-wasi/ggml/src/libggml-base.a"
LDFLAGS="-lunwind -ldl -Wl,-z,stack-size=8388608"
CXX="${WASI_SDK_PATH}/bin/clang++ --target=wasm32-wasip3 -O3 -std=c++17 ${WASI_FLAGS} -I include -I ggml/include"
$CXX examples/simple/simple.cpp build-wasi/src/libllama.a $LIBS $LDFLAGS -o build-wasi/llama-simple.wasm
$CXX tests/test-backend-ops.cpp $LIBS $LDFLAGS -o build-wasi/test-backend-ops.wasm
- name: ccache-buckets-save
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
uses: ./.github/actions/ccache-buckets
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CACHE_OUTPUT }}
with:
key: webgpu-ubuntu-26.04-arm-wasi
folder: llama.cpp
evict-old-files: 1d
hf_bucket: ggml-org/cache
save: true
-114
View File
@@ -1,114 +0,0 @@
name: CI (wasm)
on:
workflow_dispatch: # allows manual triggering
push:
branches:
- master
paths: [
'.github/workflows/build-wasm.yml',
'**/CMakeLists.txt',
'**/.cmake',
'**/*.h',
'**/*.hpp',
'**/*.c',
'**/*.cpp',
'**/*.wgsl',
'**/*.tmpl',
'ggml/src/ggml-webgpu/wgsl-shaders/embed_wgsl.py'
]
pull_request:
types: [opened, synchronize, reopened]
paths: [
'.github/workflows/build-wasm.yml',
'**/CMakeLists.txt',
'**/.cmake',
'**/*.h',
'**/*.hpp',
'**/*.c',
'**/*.cpp',
'**/*.wgsl',
'**/*.tmpl',
'ggml/src/ggml-webgpu/wgsl-shaders/embed_wgsl.py'
]
cache-mode: none
permissions:
contents: read
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}
cancel-in-progress: true
env:
GGML_NLOOP: 3
GGML_N_THREADS: 1
LLAMA_ARG_LOG_COLORS: 1
LLAMA_ARG_LOG_PREFIX: 1
LLAMA_ARG_LOG_TIMESTAMPS: 1
jobs:
ubuntu-webgpu:
runs-on: ubuntu-26.04-arm
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: ccache
uses: ggml-org/ccache-action@v1.2.24
with:
restore: false
save: false
- name: Install Emscripten
run: |
git clone https://github.com/emscripten-core/emsdk.git
cd emsdk
./emsdk install latest
./emsdk activate latest
- name: Fetch emdawnwebgpu
run: |
DAWN_TAG="v20260908.214631"
EMDAWN_PKG="emdawnwebgpu_pkg-${DAWN_TAG}.zip"
echo "Downloading ${EMDAWN_PKG}"
curl -L -o emdawn.zip \
"https://github.com/google/dawn/releases/download/${DAWN_TAG}/${EMDAWN_PKG}"
unzip emdawn.zip
- name: ccache-buckets-restore
uses: ./.github/actions/ccache-buckets
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CI }}
with:
key: webgpu-ubuntu-26.04-arm-wasm
folder: llama.cpp
hf_bucket: ggml-org/cache
- name: Build WASM WebGPU
run: |
source emsdk/emsdk_env.sh
emcmake cmake -B build-wasm \
-G "Ninja" \
-DCMAKE_BUILD_TYPE=Release \
-DGGML_WEBGPU=ON \
-DGGML_OPENMP=OFF \
-DLLAMA_OPENSSL=OFF \
-DEMDAWNWEBGPU_DIR=emdawnwebgpu_pkg
time cmake --build build-wasm --config Release --target test-backend-ops -j $(nproc)
- name: ccache-buckets-save
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
uses: ./.github/actions/ccache-buckets
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CACHE_OUTPUT }}
with:
key: webgpu-ubuntu-26.04-arm-wasm
folder: llama.cpp
evict-old-files: 1d
hf_bucket: ggml-org/cache
save: true
+146 -7
View File
@@ -7,7 +7,13 @@ on:
- master - master
paths: [ paths: [
'.github/workflows/build-webgpu.yml', '.github/workflows/build-webgpu.yml',
'ggml/src/ggml-webgpu/**' '**/CMakeLists.txt',
'**/.cmake',
'**/*.h',
'**/*.hpp',
'**/*.c',
'**/*.cpp',
'**/*.wgsl'
] ]
pull_request: pull_request:
@@ -17,10 +23,6 @@ on:
'ggml/src/ggml-webgpu/**' 'ggml/src/ggml-webgpu/**'
] ]
cache-mode: none
permissions:
contents: read
concurrency: concurrency:
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }} group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}
cancel-in-progress: true cancel-in-progress: true
@@ -34,7 +36,7 @@ env:
jobs: jobs:
format: format:
runs-on: ubuntu-26.04 runs-on: ubuntu-24.04
steps: steps:
- name: Clone - name: Clone
@@ -45,7 +47,7 @@ jobs:
wget -qO- https://apt.llvm.org/llvm-snapshot.gpg.key | wget -qO- https://apt.llvm.org/llvm-snapshot.gpg.key |
sudo tee /etc/apt/trusted.gpg.d/apt.llvm.org.asc > /dev/null sudo tee /etc/apt/trusted.gpg.d/apt.llvm.org.asc > /dev/null
sudo add-apt-repository -y \ sudo add-apt-repository -y \
"deb http://apt.llvm.org/resolute/ llvm-toolchain-resolute-22 main" "deb http://apt.llvm.org/noble/ llvm-toolchain-noble-22 main"
sudo apt-get update sudo apt-get update
sudo apt-get install -y clang-format-22 sudo apt-get install -y clang-format-22
@@ -55,3 +57,140 @@ jobs:
-type f \( -name '*.cpp' -o -name '*.hpp' -o -name '*.h' \) \ -type f \( -name '*.cpp' -o -name '*.hpp' -o -name '*.h' \) \
-print0 | -print0 |
xargs -0 clang-format-22 --dry-run --Werror xargs -0 clang-format-22 --dry-run --Werror
macos:
runs-on: macos-latest
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: ccache
uses: ggml-org/ccache-action@v1.2.21
with:
key: webgpu-macos-latest
evict-old-files: 1d
save: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
- name: Dawn Dependency
id: dawn-depends
run: |
DAWN_VERSION="v20260317.182325"
DAWN_OWNER="google"
DAWN_REPO="dawn"
DAWN_ASSET_NAME="Dawn-18eb229ef5f707c1464cc581252e7603c73a3ef0-macos-latest-Release"
echo "Fetching release asset from https://github.com/google/dawn/releases/download/${DAWN_VERSION}/${DAWN_ASSET_NAME}.tar.gz"
curl -L -o artifact.tar.gz \
"https://github.com/google/dawn/releases/download/${DAWN_VERSION}/${DAWN_ASSET_NAME}.tar.gz"
mkdir dawn
tar -xvf artifact.tar.gz -C dawn --strip-components=1
- name: Build
id: cmake_build
run: |
export CMAKE_PREFIX_PATH=dawn
cmake -B build -G "Ninja" -DCMAKE_BUILD_TYPE=Release -DGGML_WEBGPU=ON -DGGML_METAL=OFF -DGGML_BLAS=OFF
time cmake --build build --config Release -j $(sysctl -n hw.logicalcpu)
- name: Test
id: cmake_test
run: |
cd build
ctest -L main --verbose --timeout 900
ubuntu:
runs-on: ubuntu-24.04
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: ccache
uses: ggml-org/ccache-action@v1.2.21
with:
key: webgpu-ubuntu-24.04
evict-old-files: 1d
save: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
- name: Dependencies
id: depends
run: |
sudo add-apt-repository -y ppa:kisak/kisak-mesa
sudo apt-get update -y
sudo apt-get install -y build-essential mesa-vulkan-drivers \
libxcb-xinput0 libxcb-xinerama0 libxcb-cursor-dev libssl-dev
- name: Dawn Dependency
id: dawn-depends
run: |
sudo apt-get install -y libxrandr-dev libxinerama-dev libxcursor-dev mesa-common-dev libx11-xcb-dev libxi-dev
DAWN_VERSION="v20260317.182325"
DAWN_OWNER="google"
DAWN_REPO="dawn"
DAWN_ASSET_NAME="Dawn-18eb229ef5f707c1464cc581252e7603c73a3ef0-ubuntu-latest-Release"
echo "Fetching release asset from https://github.com/google/dawn/releases/download/${DAWN_VERSION}/${DAWN_ASSET_NAME}.tar.gz"
curl -L -o artifact.tar.gz \
"https://github.com/google/dawn/releases/download/${DAWN_VERSION}/${DAWN_ASSET_NAME}.tar.gz"
mkdir dawn
tar -xvf artifact.tar.gz -C dawn --strip-components=1
- name: Build
id: cmake_build
run: |
export Dawn_DIR=dawn/lib64/cmake/Dawn
cmake -B build \
-DGGML_WEBGPU=ON
time cmake --build build --config Release -j $(nproc)
- name: Test
id: cmake_test
run: |
cd build
# This is using llvmpipe and runs slower than other backends
# test-backend-ops is too slow on llvmpipe, skip it
ctest -L main -E test-backend-ops --verbose --timeout 900
ubuntu-wasm:
runs-on: ubuntu-24.04-arm
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: ccache
uses: ggml-org/ccache-action@v1.2.21
with:
key: webgpu-ubuntu-24.04-arm-wasm
evict-old-files: 1d
save: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
- name: Install Emscripten
run: |
git clone https://github.com/emscripten-core/emsdk.git
cd emsdk
./emsdk install latest
./emsdk activate latest
- name: Fetch emdawnwebgpu
run: |
DAWN_TAG="v20260317.182325"
EMDAWN_PKG="emdawnwebgpu_pkg-${DAWN_TAG}.zip"
echo "Downloading ${EMDAWN_PKG}"
curl -L -o emdawn.zip \
"https://github.com/google/dawn/releases/download/${DAWN_TAG}/${EMDAWN_PKG}"
unzip emdawn.zip
- name: Build WASM WebGPU
run: |
source emsdk/emsdk_env.sh
emcmake cmake -B build-wasm \
-G "Ninja" \
-DCMAKE_BUILD_TYPE=Release \
-DGGML_WEBGPU=ON \
-DLLAMA_OPENSSL=OFF \
-DEMDAWNWEBGPU_DIR=emdawnwebgpu_pkg
time cmake --build build-wasm --config Release --target test-backend-ops -j $(nproc)
+1 -5
View File
@@ -17,13 +17,9 @@ on:
'scripts/sync_vendor.py' 'scripts/sync_vendor.py'
] ]
cache-mode: none
permissions:
contents: read
jobs: jobs:
check-vendor: check-vendor:
runs-on: ubuntu-slim runs-on: [self-hosted, fast]
steps: steps:
- name: Checkout - name: Checkout
-153
View File
@@ -1,153 +0,0 @@
name: CI (self-hosted CPU backend)
on:
workflow_dispatch: # allows manual triggering
push:
branches:
- master
paths: [
'.github/workflows/ci-self-hosted-cpu.yml',
'ci/run.sh',
'**/CMakeLists.txt',
'**/.cmake',
'**/*.h',
'**/*.hpp',
'**/*.c',
'**/*.cpp'
]
pull_request:
types: [opened, synchronize, reopened]
paths: [
'.github/workflows/ci-self-hosted-cpu.yml',
'ci/run.sh',
'**/CMakeLists.txt',
'**/.cmake',
'ggml/src/*',
'ggml/src/ggml-cpu/**'
]
cache-mode: none
permissions:
contents: read
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}
cancel-in-progress: true
env:
# note: this is dud token to avoid rate limiting (https://github.com/ggml-org/llama.cpp/pull/25706#issuecomment-4979941302)
HF_TOKEN: ${{ secrets.HF_TOKEN_CI }}
GGML_NLOOP: 3
GGML_N_THREADS: 1
LLAMA_ARG_LOG_COLORS: 1
LLAMA_ARG_LOG_PREFIX: 1
LLAMA_ARG_LOG_TIMESTAMPS: 1
jobs:
cpu-x64-high-perf:
runs-on: hf-jobs-cpu-upgrade:ubuntu26_04
# note: this runner has 8 CPUs, but std::thread::hardware_concurrency reports the cores of the
# host (https://github.com/ggml-org/llama.cpp/pull/31439#issuecomment-6683386059)
env:
TEST_ARG_HARDWARE_CONCURRENCY: 8
TEST_ARG_PARALLEL_WORKERS: 4
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: Dependencies
id: depends
run: |
sudo apt update
sudo apt install -y build-essential cmake libssl-dev time unzip wget python3 python3-venv python3-dev
- name: ccache
uses: ggml-org/ccache-action@v1.2.24
with:
restore: false
save: false
- name: ccache-buckets-restore
uses: ./.github/actions/ccache-buckets
with:
key: self-hosted-cpu-x64-high-perf
folder: llama.cpp
hf_bucket: ggml-org/cache
- name: Test
id: ggml-ci
run: |
LLAMA_ARG_THREADS=$(nproc) GG_BUILD_HIGH_PERF=1 GG_BUILD_EXTRA_TESTS_0=1 bash ./ci/run.sh ~/results/llama.cpp/$RUNNER_NAME ~/mnt/llama.cpp
- name: ccache-buckets-save
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
uses: ./.github/actions/ccache-buckets
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CACHE_OUTPUT }}
with:
key: self-hosted-cpu-x64-high-perf
folder: llama.cpp
evict-old-files: 1d
hf_bucket: ggml-org/cache
save: true
cpu-arm64-high-perf-graviton4:
runs-on: ah-ubuntu_24_04-c8g_8x
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: Dependencies
id: depends
run: |
set -euxo pipefail
sudo apt-get update
sudo DEBIAN_FRONTEND=noninteractive NEEDRESTART_MODE=a \
apt-get install -y \
build-essential \
python3-venv \
gpg \
wget \
time \
git-lfs
git lfs install
# install the latest cmake
sudo install -d /usr/share/keyrings
wget -O - https://apt.kitware.com/keys/kitware-archive-latest.asc \
| gpg --dearmor \
| sudo tee /usr/share/keyrings/kitware-archive-keyring.gpg >/dev/null
echo 'deb [signed-by=/usr/share/keyrings/kitware-archive-keyring.gpg] https://apt.kitware.com/ubuntu/ jammy main' \
| sudo tee /etc/apt/sources.list.d/kitware.list
sudo apt-get update
sudo apt-get install -y cmake
- name: Test
id: ggml-ci
run: |
LLAMA_ARG_THREADS=$(nproc) \
GG_BUILD_HIGH_PERF=1 \
GG_BUILD_NO_BF16=1 \
GG_BUILD_EXTRA_TESTS_0=1 \
bash ./ci/run.sh ~/results/llama.cpp ~/mnt/llama.cpp
# TODO: provision AMX-compatible machine
#cpu-amx:
# runs-on: [self-hosted, Linux, CPU, AMX]
# steps:
# - name: Clone
# id: checkout
# uses: actions/checkout@v6
# - name: Test
# id: ggml-ci
# run: |
# bash ./ci/run.sh ~/results/llama.cpp ~/mnt/llama.cpp
-202
View File
@@ -1,202 +0,0 @@
name: CI (self-hosted CUDA backend)
on:
workflow_dispatch: # allows manual triggering
push:
branches:
- master
paths: [
'.github/workflows/ci-self-hosted-cuda.yml',
'ci/run.sh',
'**/CMakeLists.txt',
'**/.cmake',
'**/*.h',
'**/*.hpp',
'**/*.c',
'**/*.cpp',
'**/*.cu',
'**/*.cuh'
]
pull_request:
types: [opened, synchronize, reopened]
paths: [
'.github/workflows/ci-self-hosted-cuda.yml',
'ci/run.sh',
'**/CMakeLists.txt',
'**/.cmake',
'ggml/src/*',
'ggml/src/ggml-cpu/**',
'ggml/src/ggml-cuda/**'
]
cache-mode: none
permissions:
contents: read
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}
cancel-in-progress: true
env:
# note: this is dud token to avoid rate limiting (https://github.com/ggml-org/llama.cpp/pull/25706#issuecomment-4979941302)
HF_TOKEN: ${{ secrets.HF_TOKEN_CI }}
GGML_NLOOP: 3
GGML_N_THREADS: 1
LLAMA_ARG_LOG_COLORS: 1
LLAMA_ARG_LOG_PREFIX: 1
LLAMA_ARG_LOG_TIMESTAMPS: 1
jobs:
gpu-cuda:
runs-on: "hf-jobs-t4-medium:cuda13"
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: Install dependencies
run: |
sudo apt update
sudo apt install -y cmake libssl-dev time unzip wget python3 python3-venv python3-pip
- name: ccache
uses: ggml-org/ccache-action@v1.2.24
with:
restore: false
save: false
- name: ccache-buckets-restore
uses: ./.github/actions/ccache-buckets
with:
key: self-hosted-gpu-cuda
folder: llama.cpp
hf_bucket: ggml-org/cache
- name: Test
id: ggml-ci
run: |
nvidia-smi
GG_BUILD_CUDA=1 CUDACXX=/usr/local/cuda/bin/nvcc bash ./ci/run.sh ~/results/llama.cpp ~/mnt/llama.cpp
- name: ccache-buckets-save
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
uses: ./.github/actions/ccache-buckets
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CACHE_OUTPUT }}
with:
key: self-hosted-gpu-cuda
folder: llama.cpp
evict-old-files: 1d
hf_bucket: ggml-org/cache
save: true
gpu-rocm:
strategy:
fail-fast: false
matrix:
include:
# Strix Halo (iGPU, RDNA3.5)
- gpu: gfx1151
runner: [self-hosted, Linux, strix-halo, devlab-dispatch]
# async-execution correctness workaround, gfx1151 only (see Test step)
hip_launch_blocking: "1"
# Radeon AI PRO R9700 (dGPU, RDNA4)
- gpu: gfx1201
runner: [self-hosted, Linux, radeon-r9700, devlab-dispatch]
hip_launch_blocking: "0"
# HIP-graph capture fails on gfx1201 (hipMemset -> hipErrorStreamCaptureImplicit)
disable_hip_graphs: true
name: gpu-rocm (${{ matrix.gpu }})
runs-on: ${{ matrix.runner }}
timeout-minutes: 120
env:
ROCM_VERSION: "10.0.0"
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: Verify GPU device
run: |
ls /dev/kfd /dev/dri/renderD* || { echo "ERROR: no amdgpu compute device"; exit 1; }
# same TheRock wheels as the ubuntu-24-rocm release build, so the runner does not need a ROCm install
- name: Setup TheRock with Wheels
run: |
python3 -m venv "$RUNNER_TEMP/rocm"
"$RUNNER_TEMP/rocm/bin/python" -m pip install --index-url https://stable.repo.amd.com/rocm/whl-next/ "rocm[libraries,devel,device-${{ matrix.gpu }}]==${ROCM_VERSION}"
ROCM_PATH=$("$RUNNER_TEMP/rocm/bin/rocm-sdk" path --root)
{
echo "ROCM_PATH=$ROCM_PATH"
echo "HIP_PATH=$ROCM_PATH"
echo "CMAKE_PREFIX_PATH=$("$RUNNER_TEMP/rocm/bin/rocm-sdk" path --cmake)"
echo "LD_LIBRARY_PATH=$ROCM_PATH/lib:${LD_LIBRARY_PATH:-}"
} >> "$GITHUB_ENV"
"$RUNNER_TEMP/rocm/bin/rocm-sdk" path --bin >> "$GITHUB_PATH"
- name: ccache
uses: ggml-org/ccache-action@v1.2.24
with:
restore: false
save: false
# ROCm is installed again on each run, so hash the compiler content, not its mtime
- name: Tune ccache for reinstalled ROCm toolchain
run: |
ccache --set-config=compiler_check=content
ccache --set-config=sloppiness=time_macros,include_file_mtime,include_file_ctime
ccache --set-config=depend_mode=true
- name: ccache-buckets-restore
uses: ./.github/actions/ccache-buckets
with:
key: self-hosted-gpu-rocm-${{ matrix.gpu }}
folder: llama.cpp
hf_bucket: ggml-org/cache
- name: Test
id: ggml-ci
# HIP_LAUNCH_BLOCKING=1: workaround for an async-execution correctness
# issue on integrated RDNA3.5 (gfx1151) where batched inference returns
# incorrect output (perplexity ~88 vs ~9.4). Serializing kernel launches
# restores correctness. Remove once the underlying ROCm/HIP issue is fixed.
env:
HIP_LAUNCH_BLOCKING: ${{ matrix.hip_launch_blocking }}
run: |
rocminfo
# Exported rather than set in env:, because ggml only checks that the
# variable is defined: an empty value would disable graphs on gfx1151 too.
if [ "${{ matrix.disable_hip_graphs }}" = "true" ]; then export GGML_CUDA_DISABLE_GRAPHS=1; fi
GG_BUILD_ROCM=1 GG_BUILD_AMDGPU_TARGETS=${{ matrix.gpu }} bash ./ci/run.sh "$RUNNER_TEMP/results/llama.cpp" "$RUNNER_TEMP/mnt/llama.cpp"
- name: ccache-buckets-save
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
uses: ./.github/actions/ccache-buckets
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CACHE_OUTPUT }}
with:
key: self-hosted-gpu-rocm-${{ matrix.gpu }}
folder: llama.cpp
evict-old-files: 1d
hf_bucket: ggml-org/cache
save: true
# TODO: provision AMD GPU machine
# amd-rocm:
# runs-on: [self-hosted, Linux, AMD]
# steps:
# - name: Clone
# id: checkout
# uses: actions/checkout@v6
# - name: Test
# id: ggml-ci
# run: |
# amd-smi static
# GG_BUILD_ROCM=1 GG_BUILD_AMDGPU_TARGETS="gfx1101" bash ./ci/run.sh ~/results/llama.cpp ~/mnt/llama.cpp
@@ -1,89 +0,0 @@
name: CI (self-hosted KleidiAI backend)
on:
workflow_dispatch: # allows manual triggering
push:
branches:
- master
paths: [
'.github/workflows/ci-self-hosted-kleidiai.yml',
'ci/run.sh',
'**/CMakeLists.txt',
'**/.cmake',
'**/*.h',
'**/*.hpp',
'**/*.c',
'**/*.cpp'
]
pull_request:
types: [opened, synchronize, reopened]
paths: [
'.github/workflows/ci-self-hosted-kleidiai.yml',
'ci/run.sh',
'**/CMakeLists.txt',
'**/.cmake',
'ggml/src/*',
'ggml/src/ggml-cpu/**'
]
cache-mode: none
permissions:
contents: read
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}
cancel-in-progress: true
env:
# note: this is dud token to avoid rate limiting (https://github.com/ggml-org/llama.cpp/pull/25706#issuecomment-4979941302)
HF_TOKEN: ${{ secrets.HF_TOKEN_CI }}
GGML_NLOOP: 3
GGML_N_THREADS: 1
LLAMA_ARG_LOG_COLORS: 1
LLAMA_ARG_LOG_PREFIX: 1
LLAMA_ARG_LOG_TIMESTAMPS: 1
jobs:
cpu-arm64-graviton4-kleidiai:
runs-on: ah-ubuntu_24_04-c8g_8x
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: Dependencies
id: depends
run: |
set -euxo pipefail
sudo apt-get update
sudo DEBIAN_FRONTEND=noninteractive NEEDRESTART_MODE=a \
apt-get install -y \
build-essential \
python3-venv \
gpg \
wget \
time \
git-lfs
git lfs install
# install the latest cmake
sudo install -d /usr/share/keyrings
wget -O - https://apt.kitware.com/keys/kitware-archive-latest.asc \
| gpg --dearmor \
| sudo tee /usr/share/keyrings/kitware-archive-keyring.gpg >/dev/null
echo 'deb [signed-by=/usr/share/keyrings/kitware-archive-keyring.gpg] https://apt.kitware.com/ubuntu/ jammy main' \
| sudo tee /etc/apt/sources.list.d/kitware.list
sudo apt-get update
sudo apt-get install -y cmake
- name: Test
id: ggml-ci
run: |
LLAMA_ARG_THREADS=$(nproc) \
GG_BUILD_KLEIDIAI=1 \
GG_BUILD_EXTRA_TESTS_0=1 \
GG_BUILD_HIGH_PERF=1 \
bash ./ci/run.sh ~/results/llama.cpp ~/mnt/llama.cpp
@@ -1,63 +0,0 @@
name: CI (self-hosted Metal backend)
on:
workflow_dispatch: # allows manual triggering
push:
branches:
- master
paths: [
'.github/workflows/ci-self-hosted-metal.yml',
'ci/run.sh',
'**/CMakeLists.txt',
'**/.cmake',
'**/*.h',
'**/*.hpp',
'**/*.c',
'**/*.cpp',
'**/*.swift',
'**/*.m',
'**/*.metal'
]
pull_request:
types: [opened, synchronize, reopened]
paths: [
'.github/workflows/ci-self-hosted-metal.yml',
'ci/run.sh',
'**/CMakeLists.txt',
'**/.cmake',
'ggml/src/*',
'ggml/src/ggml-cpu/**',
'ggml/src/ggml-metal/**'
]
cache-mode: none
permissions:
contents: read
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}
cancel-in-progress: true
env:
# note: this is dud token to avoid rate limiting (https://github.com/ggml-org/llama.cpp/pull/25706#issuecomment-4979941302)
HF_TOKEN: ${{ secrets.HF_TOKEN_CI }}
GGML_NLOOP: 3
GGML_N_THREADS: 1
LLAMA_ARG_LOG_COLORS: 1
LLAMA_ARG_LOG_PREFIX: 1
LLAMA_ARG_LOG_TIMESTAMPS: 1
jobs:
gpu-metal:
runs-on: [self-hosted, macOS, ARM64]
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: Test
id: ggml-ci
run: |
GG_BUILD_METAL=1 bash ./ci/run.sh ~/results/llama.cpp/$RUNNER_NAME ~/mnt/llama.cpp
@@ -1,81 +0,0 @@
name: CI (self-hosted OpenVINO backend)
on:
workflow_dispatch: # allows manual triggering
push:
branches:
- master
paths: [
'.github/workflows/ci-self-hosted-openvino.yml',
'ci/run.sh',
'**/CMakeLists.txt',
'**/.cmake',
'**/*.h',
'**/*.hpp',
'**/*.c',
'**/*.cpp'
]
pull_request:
types: [opened, synchronize, reopened]
paths: [
'.github/workflows/ci-self-hosted-openvino.yml',
'ci/run.sh',
'CMakeLists.txt',
'ggml/CMakeLists.txt',
'tests/CMakeLists.txt',
'**/.cmake',
'ggml/src/*',
'ggml/src/ggml-cpu/**',
'ggml/src/ggml-openvino/**'
]
cache-mode: none
permissions:
contents: read
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}
cancel-in-progress: true
env:
# note: this is dud token to avoid rate limiting (https://github.com/ggml-org/llama.cpp/pull/25706#issuecomment-4979941302)
HF_TOKEN: ${{ secrets.HF_TOKEN_CI }}
GGML_NLOOP: 3
GGML_N_THREADS: 1
LLAMA_ARG_LOG_COLORS: 1
LLAMA_ARG_LOG_PREFIX: 1
LLAMA_ARG_LOG_TIMESTAMPS: 1
jobs:
gpu-openvino-low-perf:
runs-on: [self-hosted, Linux, Intel, OpenVINO]
env:
# Sync versions in build-openvino.yml, ci-self-hosted-openvino.yml, release.yml, release-publish.yml, build-cache.yml, .devops/openvino.Dockerfile, docs/backend/OPENVINO.md
OPENVINO_VERSION_MAJOR: "2026.4.1"
OPENVINO_VERSION_FULL: "2026.4.1.22982.07f9c262b05"
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: Setup OpenVINO Toolkit
uses: ./.github/actions/linux-setup-openvino
with:
path: ./openvino_toolkit
version_major: ${{ env.OPENVINO_VERSION_MAJOR }}
version_full: ${{ env.OPENVINO_VERSION_FULL }}
- name: Install OpenVINO dependencies
run: |
cd ./openvino_toolkit
chmod +x ./install_dependencies/install_openvino_dependencies.sh
echo "Y" | sudo -E ./install_dependencies/install_openvino_dependencies.sh
- name: Test
id: ggml-ci
run: |
source ./openvino_toolkit/setupvars.sh
GG_BUILD_OPENVINO=1 GGML_OPENVINO_DEVICE=GPU GG_BUILD_LOW_PERF=1 bash ./ci/run.sh ~/results/llama.cpp ~/mnt/llama.cpp
-205
View File
@@ -1,205 +0,0 @@
name: CI (self-hosted Vulkan backend)
on:
workflow_dispatch: # allows manual triggering
push:
branches:
- master
paths: [
'.github/workflows/ci-self-hosted-vulkan.yml',
'ci/run.sh',
'**/CMakeLists.txt',
'**/.cmake',
'**/*.h',
'**/*.hpp',
'**/*.c',
'**/*.cpp',
'**/*.comp',
'**/*.glsl'
]
pull_request:
types: [opened, synchronize, reopened]
paths: [
'.github/workflows/ci-self-hosted-vulkan.yml',
'ci/run.sh',
'**/CMakeLists.txt',
'**/.cmake',
'ggml/src/*',
'ggml/src/ggml-cpu/**',
'ggml/src/ggml-vulkan/**'
]
cache-mode: none
permissions:
contents: read
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}
cancel-in-progress: true
env:
# note: this is dud token to avoid rate limiting (https://github.com/ggml-org/llama.cpp/pull/25706#issuecomment-4979941302)
HF_TOKEN: ${{ secrets.HF_TOKEN_CI }}
GGML_NLOOP: 3
GGML_N_THREADS: 1
LLAMA_ARG_LOG_COLORS: 1
LLAMA_ARG_LOG_PREFIX: 1
LLAMA_ARG_LOG_TIMESTAMPS: 1
jobs:
gpu-vulkan-nvidia-cm:
# runs-on: "hf-jobs-t4-small:ubuntu26_04"
runs-on: [self-hosted, Linux, NVIDIA]
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
# - name: Install dependencies
# run: |
# sudo apt update
# sudo apt install -y build-essential cmake libxcb-xinput0 libxcb-xinerama0 libxcb-cursor-dev libvulkan-dev glslc spirv-headers vulkan-tools mesa-vulkan-drivers libglvnd0 libgl1 libglx0 libegl1 libgles2 libssl-dev time unzip wget python3 python3-venv python3-pip
# - name: ccache
# uses: ggml-org/ccache-action@v1.2.24
# with:
# restore: false
# save: false
# - name: ccache-buckets-restore
# uses: ./.github/actions/ccache-buckets
# with:
# key: self-hosted-vulkan-nvidia-cm
# folder: llama.cpp
# hf_bucket: ggml-org/cache
- name: Test
id: ggml-ci
run: |
vulkaninfo --summary
GG_BUILD_VULKAN=1 GGML_VK_DISABLE_COOPMAT2=1 bash ./ci/run.sh ~/results/llama.cpp/$RUNNER_NAME ~/mnt/llama.cpp
# - name: ccache-buckets-save
# if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
# uses: ./.github/actions/ccache-buckets
# env:
# HF_TOKEN: ${{ secrets.HF_TOKEN_CACHE_OUTPUT }}
# with:
# key: self-hosted-vulkan-nvidia-cm
# folder: llama.cpp
# evict-old-files: 1d
# hf_bucket: ggml-org/cache
# save: true
gpu-vulkan-nvidia-cm2:
# runs-on: "hf-jobs-t4-small:ubuntu26_04"
runs-on: [self-hosted, Linux, NVIDIA, COOPMAT2]
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
# - name: Install dependencies
# run: |
# sudo apt update
# sudo apt install -y build-essential cmake libxcb-xinput0 libxcb-xinerama0 libxcb-cursor-dev libvulkan-dev glslc spirv-headers vulkan-tools mesa-vulkan-drivers libglvnd0 libgl1 libglx0 libegl1 libgles2 libssl-dev time unzip wget python3 python3-venv python3-pip
# - name: ccache
# uses: ggml-org/ccache-action@v1.2.24
# with:
# restore: false
# save: false
# - name: ccache-buckets-restore
# uses: ./.github/actions/ccache-buckets
# with:
# key: self-hosted-vulkan-nvidia-cm2
# folder: llama.cpp
# hf_bucket: ggml-org/cache
- name: Test
id: ggml-ci
run: |
vulkaninfo --summary
GG_BUILD_VULKAN=1 bash ./ci/run.sh ~/results/llama.cpp/$RUNNER_NAME ~/mnt/llama.cpp
# - name: ccache-buckets-save
# if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
# uses: ./.github/actions/ccache-buckets
# env:
# HF_TOKEN: ${{ secrets.HF_TOKEN_CACHE_OUTPUT }}
# with:
# key: self-hosted-vulkan-nvidia-cm2
# folder: llama.cpp
# evict-old-files: 1d
# hf_bucket: ggml-org/cache
# save: true
gpu-vulkan-apple:
runs-on: [self-hosted, macOS, ARM64]
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: Test
id: ggml-ci
run: |
vulkaninfo --summary
GG_BUILD_VULKAN=1 bash ./ci/run.sh ~/results/llama.cpp/$RUNNER_NAME ~/mnt/llama.cpp
gpu-vulkan-intel-linux:
runs-on: [self-hosted, Linux, Intel]
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
with:
persist-credentials: false
- name: Test
id: ggml-ci
run: |
vulkaninfo --summary
GG_BUILD_VULKAN=1 bash ./ci/run.sh ~/results/llama.cpp ~/mnt/llama.cpp
gpu-vulkan-intel-windows:
runs-on: [self-hosted, Windows, X64, Intel]
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: Test
id: ggml-ci
shell: C:\msys64\usr\bin\bash.exe --noprofile --norc -eo pipefail "{0}"
env:
MSYSTEM: UCRT64
CHERE_INVOKING: 1
PATH: C:\msys64\ucrt64\bin;C:\msys64\usr\bin;C:\Windows\System32;${{ env.PATH }}
run: |
vulkaninfo --summary
# Skip python related tests with GG_BUILD_LOW_PERF=1 since Windows MSYS2 UCRT64 currently fails to create
# a valid python environment for testing
LLAMA_FATAL_WARNINGS=OFF GG_BUILD_NINJA=1 GG_BUILD_VULKAN=1 GG_BUILD_LOW_PERF=1 ./ci/run.sh ./results/llama.cpp ./mnt/llama.cpp
# TODO: provision AMD GPU machine
# amd-vulkan:
# runs-on: [self-hosted, Linux, AMD]
# steps:
# - name: Clone
# id: checkout
# uses: actions/checkout@v6
# - name: Test
# id: ggml-ci
# run: |
# vulkaninfo --summary
# GG_BUILD_VULKAN=1 bash ./ci/run.sh ~/results/llama.cpp ~/mnt/llama.cpp
-134
View File
@@ -1,134 +0,0 @@
name: CI (self-hosted WebGPU backend)
on:
workflow_dispatch: # allows manual triggering
push:
branches:
- master
paths: [
'.github/workflows/ci-self-hosted-webgpu.yml',
'ci/run.sh',
'**/CMakeLists.txt',
'**/.cmake',
'**/*.h',
'**/*.hpp',
'**/*.c',
'**/*.cpp',
'**/*.wgsl'
]
pull_request:
types: [opened, synchronize, reopened]
paths: [
'.github/workflows/ci-self-hosted-webgpu.yml',
'ci/run.sh',
'**/CMakeLists.txt',
'**/.cmake',
'ggml/src/*',
'ggml/src/ggml-cpu/**',
'ggml/src/ggml-webgpu/**'
]
cache-mode: none
permissions:
contents: read
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}
cancel-in-progress: true
env:
# note: this is dud token to avoid rate limiting (https://github.com/ggml-org/llama.cpp/pull/25706#issuecomment-4979941302)
HF_TOKEN: ${{ secrets.HF_TOKEN_CI }}
GGML_NLOOP: 3
GGML_N_THREADS: 1
LLAMA_ARG_LOG_COLORS: 1
LLAMA_ARG_LOG_PREFIX: 1
LLAMA_ARG_LOG_TIMESTAMPS: 1
jobs:
gpu-webgpu-nvidia:
runs-on: "hf-jobs-t4-small:ubuntu26_04"
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: Install dependencies
run: |
sudo apt update
sudo apt install -y build-essential cmake libxcb-xinput0 libxcb-xinerama0 libxcb-cursor-dev libvulkan1 mesa-vulkan-drivers libglvnd0 libgl1 libglx0 libegl1 libgles2 libssl-dev time unzip wget python3 python3-venv python3-pip
- name: ccache
uses: ggml-org/ccache-action@v1.2.24
with:
restore: false
save: false
- name: ccache-buckets-restore
uses: ./.github/actions/ccache-buckets
with:
key: self-hosted-webgpu-nvidia
folder: llama.cpp
hf_bucket: ggml-org/cache
- name: Dawn Dependency
id: dawn-depends
run: |
DAWN_VERSION="v20260908.214631"
DAWN_OWNER="google"
DAWN_REPO="dawn"
DAWN_ASSET_NAME="Dawn-94c3c9cc0d5fb2e85aebb370fa8d37b71aa34655-ubuntu-latest-Release"
echo "Fetching release asset from https://github.com/google/dawn/releases/download/${DAWN_VERSION}/${DAWN_ASSET_NAME}.tar.gz"
curl -L -o artifact.tar.gz \
"https://github.com/google/dawn/releases/download/${DAWN_VERSION}/${DAWN_ASSET_NAME}.tar.gz"
mkdir dawn
tar -xvf artifact.tar.gz -C dawn --strip-components=1
- name: Test
id: ggml-ci
run: |
GG_BUILD_WEBGPU=1 \
GG_BUILD_WEBGPU_DAWN_PREFIX="$GITHUB_WORKSPACE/dawn" \
GG_BUILD_WEBGPU_DAWN_DIR="$GITHUB_WORKSPACE/dawn/lib64/cmake/Dawn" \
bash ./ci/run.sh ~/results/llama.cpp ~/mnt/llama.cpp
- name: ccache-buckets-save
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
uses: ./.github/actions/ccache-buckets
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CACHE_OUTPUT }}
with:
key: self-hosted-webgpu-nvidia
folder: llama.cpp
evict-old-files: 1d
hf_bucket: ggml-org/cache
save: true
gpu-webgpu-apple:
runs-on: [self-hosted, macOS, ARM64]
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: Dawn Dependency
id: dawn-depends
run: |
DAWN_VERSION="v20260908.214631"
DAWN_OWNER="google"
DAWN_REPO="dawn"
DAWN_ASSET_NAME="Dawn-94c3c9cc0d5fb2e85aebb370fa8d37b71aa34655-macos-latest-Release"
echo "Fetching release asset from https://github.com/google/dawn/releases/download/${DAWN_VERSION}/${DAWN_ASSET_NAME}.tar.gz"
curl -L -o artifact.tar.gz \
"https://github.com/google/dawn/releases/download/${DAWN_VERSION}/${DAWN_ASSET_NAME}.tar.gz"
mkdir dawn
tar -xvf artifact.tar.gz -C dawn --strip-components=1
- name: Test
id: ggml-ci
run: |
GG_BUILD_WEBGPU=1 GG_BUILD_WEBGPU_DAWN_PREFIX="$GITHUB_WORKSPACE/dawn" \
bash ./ci/run.sh ~/results/llama.cpp/$RUNNER_NAME ~/mnt/llama.cpp
+3 -2
View File
@@ -3,9 +3,10 @@ on:
schedule: schedule:
- cron: "42 0 * * *" - cron: "42 0 * * *"
cache-mode: none # Fine-grant permission
# https://docs.github.com/en/actions/security-for-github-actions/security-guides/automatic-token-authentication#modifying-the-permissions-for-the-github_token
permissions: permissions:
contents: read issues: write
jobs: jobs:
close-issues: close-issues:
+1 -5
View File
@@ -9,17 +9,13 @@ on:
branches: branches:
- master - master
cache-mode: none
permissions:
contents: read
concurrency: concurrency:
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }} group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}
cancel-in-progress: true cancel-in-progress: true
jobs: jobs:
model-naming: model-naming:
runs-on: ubuntu-slim runs-on: [self-hosted, fast]
steps: steps:
- uses: actions/checkout@v6 - uses: actions/checkout@v6
- name: Check model naming conventions - name: Check model naming conventions
+4 -6
View File
@@ -11,12 +11,10 @@ on:
paths: paths:
- .github/workflows/copilot-setup-steps.yml - .github/workflows/copilot-setup-steps.yml
cache-mode: none
jobs: jobs:
# The job MUST be called `copilot-setup-steps` or it will not be picked up by Copilot. # The job MUST be called `copilot-setup-steps` or it will not be picked up by Copilot.
copilot-setup-steps: copilot-setup-steps:
runs-on: ubuntu-26.04 runs-on: ubuntu-latest
# Set the permissions to the lowest permissions possible needed for your steps. # Set the permissions to the lowest permissions possible needed for your steps.
# Copilot will be given its own token for its operations. # Copilot will be given its own token for its operations.
@@ -31,10 +29,10 @@ jobs:
uses: actions/checkout@v6 uses: actions/checkout@v6
- name: ccache - name: ccache
uses: ggml-org/ccache-action@v1.2.24 uses: ggml-org/ccache-action@v1.2.21
with: with:
restore: false key: copilot-setup-steps
save: false evict-old-files: 1d
- name: Dependencies - name: Dependencies
id: depends id: depends
+28 -89
View File
@@ -16,22 +16,19 @@ on:
description: "Skip the s390x build target (useful for fast test runs that do not need the IBM Z runner)" description: "Skip the s390x build target (useful for fast test runs that do not need the IBM Z runner)"
type: boolean type: boolean
default: false default: false
skip_ref_check:
description: "Skip the requirement to run from a tag (allows branch refs)"
type: boolean
default: false
schedule: schedule:
# Rebuild daily rather than on every push because it is expensive # Rebuild daily rather than on every push because it is expensive
- cron: '12 4 * * *' - cron: '12 4 * * *'
cache-mode: none
permissions:
contents: read
concurrency: concurrency:
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }} group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}
cancel-in-progress: true cancel-in-progress: true
# Fine-grant permission
# https://docs.github.com/en/actions/security-for-github-actions/security-guides/automatic-token-authentication#modifying-the-permissions-for-the-github_token
permissions:
packages: write
jobs: jobs:
create_tag: create_tag:
name: Create and push git tag name: Create and push git tag
@@ -42,20 +39,11 @@ jobs:
source_tag: ${{ steps.srctag.outputs.name }} source_tag: ${{ steps.srctag.outputs.name }}
steps: steps:
- name: Check manual run ref
if: ${{ github.event_name == 'workflow_dispatch' && !inputs.skip_ref_check }}
run: |
if [[ "$GITHUB_REF_TYPE" != "tag" ]]; then
echo "::error::Manual runs of this workflow must be started from a tag"
exit 1
fi
- name: Clone - name: Clone
id: checkout id: checkout
uses: actions/checkout@v6 uses: actions/checkout@v6
with: with:
fetch-depth: 0 fetch-depth: 0
ssh-key: ${{ secrets.DEPLOY_KEY_RELEASE }}
- name: Determine source tag name - name: Determine source tag name
id: srctag id: srctag
@@ -73,16 +61,13 @@ jobs:
build_ui: build_ui:
name: Build UI name: Build UI
needs: create_tag needs: create_tag
permissions:
actions: write
contents: read
uses: ./.github/workflows/ui-build.yml uses: ./.github/workflows/ui-build.yml
with: with:
ui_version: ${{ needs.create_tag.outputs.source_tag }} hf_ui_version: ${{ needs.create_tag.outputs.source_tag }}
prepare_matrices: prepare_matrices:
name: Prepare Docker matrices name: Prepare Docker matrices
runs-on: ubuntu-26.04 runs-on: ubuntu-24.04
outputs: outputs:
build_matrix: ${{ steps.matrices.outputs.build_matrix }} build_matrix: ${{ steps.matrices.outputs.build_matrix }}
merge_matrix: ${{ steps.matrices.outputs.merge_matrix }} merge_matrix: ${{ steps.matrices.outputs.merge_matrix }}
@@ -99,19 +84,19 @@ jobs:
# Keep all build targets in one place and derive merge targets from it. # Keep all build targets in one place and derive merge targets from it.
cat > build-matrix.json <<'JSON' cat > build-matrix.json <<'JSON'
[ [
{ "tag": "cpu", "dockerfile": ".devops/cpu.Dockerfile", "platforms": "linux/amd64", "full": true, "light": true, "server": true, "free_disk_space": false, "runs_on": "ubuntu-26.04" }, { "tag": "cpu", "dockerfile": ".devops/cpu.Dockerfile", "platforms": "linux/amd64", "full": true, "light": true, "server": true, "free_disk_space": false, "runs_on": "ubuntu-24.04" },
{ "tag": "cpu", "dockerfile": ".devops/cpu.Dockerfile", "platforms": "linux/arm64", "full": true, "light": true, "server": true, "free_disk_space": false, "runs_on": "ubuntu-26.04-arm" }, { "tag": "cpu", "dockerfile": ".devops/cpu.Dockerfile", "platforms": "linux/arm64", "full": true, "light": true, "server": true, "free_disk_space": false, "runs_on": "ubuntu-24.04-arm" },
{ "tag": "cpu", "dockerfile": ".devops/s390x.Dockerfile", "platforms": "linux/s390x", "full": true, "light": true, "server": true, "free_disk_space": false, "runs_on": "ubuntu-26.04-s390x", "prebuilt_ui": true }, { "tag": "cpu", "dockerfile": ".devops/s390x.Dockerfile", "platforms": "linux/s390x", "full": true, "light": true, "server": true, "free_disk_space": false, "runs_on": "ubuntu-24.04-s390x", "prebuilt_ui": true },
{ "tag": "cuda cuda12", "dockerfile": ".devops/cuda.Dockerfile", "cuda_version": "12.8.1", "platforms": "linux/amd64", "full": true, "light": true, "server": true, "free_disk_space": true, "runs_on": "ubuntu-26.04" }, { "tag": "cuda cuda12", "dockerfile": ".devops/cuda.Dockerfile", "cuda_version": "12.8.1", "platforms": "linux/amd64", "full": true, "light": true, "server": true, "free_disk_space": true, "runs_on": "ubuntu-24.04" },
{ "tag": "cuda cuda12", "dockerfile": ".devops/cuda.Dockerfile", "cuda_version": "12.8.1", "platforms": "linux/arm64", "full": true, "light": true, "server": true, "free_disk_space": true, "runs_on": "ubuntu-26.04-arm" }, { "tag": "cuda cuda12", "dockerfile": ".devops/cuda.Dockerfile", "cuda_version": "12.8.1", "platforms": "linux/arm64", "full": true, "light": true, "server": true, "free_disk_space": true, "runs_on": "ubuntu-24.04-arm" },
{ "tag": "cuda13", "dockerfile": ".devops/cuda.Dockerfile", "cuda_version": "13.5.0", "platforms": "linux/amd64", "full": true, "light": true, "server": true, "free_disk_space": true, "runs_on": "ubuntu-26.04" }, { "tag": "cuda13", "dockerfile": ".devops/cuda.Dockerfile", "cuda_version": "13.3.0", "platforms": "linux/amd64", "full": true, "light": true, "server": true, "free_disk_space": true, "runs_on": "ubuntu-24.04" },
{ "tag": "cuda13", "dockerfile": ".devops/cuda.Dockerfile", "cuda_version": "13.5.0", "platforms": "linux/arm64", "full": true, "light": true, "server": true, "free_disk_space": true, "runs_on": "ubuntu-26.04-arm" }, { "tag": "cuda13", "dockerfile": ".devops/cuda.Dockerfile", "cuda_version": "13.3.0", "platforms": "linux/arm64", "full": true, "light": true, "server": true, "free_disk_space": true, "runs_on": "ubuntu-24.04-arm" },
{ "tag": "musa", "dockerfile": ".devops/musa.Dockerfile", "platforms": "linux/amd64", "full": true, "light": true, "server": true, "free_disk_space": true, "runs_on": "ubuntu-26.04" }, { "tag": "musa", "dockerfile": ".devops/musa.Dockerfile", "platforms": "linux/amd64", "full": true, "light": true, "server": true, "free_disk_space": true, "runs_on": "ubuntu-24.04" },
{ "tag": "intel", "dockerfile": ".devops/intel.Dockerfile", "platforms": "linux/amd64", "full": true, "light": true, "server": true, "free_disk_space": true, "runs_on": "ubuntu-26.04" }, { "tag": "intel", "dockerfile": ".devops/intel.Dockerfile", "platforms": "linux/amd64", "full": true, "light": true, "server": true, "free_disk_space": true, "runs_on": "ubuntu-24.04" },
{ "tag": "vulkan", "dockerfile": ".devops/vulkan.Dockerfile", "platforms": "linux/amd64", "full": true, "light": true, "server": true, "free_disk_space": false, "runs_on": "ubuntu-26.04" }, { "tag": "vulkan", "dockerfile": ".devops/vulkan.Dockerfile", "platforms": "linux/amd64", "full": true, "light": true, "server": true, "free_disk_space": false, "runs_on": "ubuntu-24.04" },
{ "tag": "vulkan", "dockerfile": ".devops/vulkan.Dockerfile", "platforms": "linux/arm64", "full": true, "light": true, "server": true, "free_disk_space": false, "runs_on": "ubuntu-26.04-arm" }, { "tag": "vulkan", "dockerfile": ".devops/vulkan.Dockerfile", "platforms": "linux/arm64", "full": true, "light": true, "server": true, "free_disk_space": false, "runs_on": "ubuntu-24.04-arm" },
{ "tag": "rocm", "dockerfile": ".devops/rocm.Dockerfile", "platforms": "linux/amd64", "full": true, "light": true, "server": true, "free_disk_space": true, "runs_on": "ubuntu-26.04" }, { "tag": "rocm", "dockerfile": ".devops/rocm.Dockerfile", "platforms": "linux/amd64", "full": true, "light": true, "server": true, "free_disk_space": true, "runs_on": "ubuntu-24.04" },
{ "tag": "openvino", "dockerfile": ".devops/openvino.Dockerfile", "platforms": "linux/amd64", "full": true, "light": true, "server": true, "free_disk_space": false, "runs_on": "ubuntu-26.04" } { "tag": "openvino", "dockerfile": ".devops/openvino.Dockerfile", "platforms": "linux/amd64", "full": true, "light": true, "server": true, "free_disk_space": false, "runs_on": "ubuntu-24.04" }
] ]
JSON JSON
@@ -120,9 +105,6 @@ jobs:
mv build-matrix.json.tmp build-matrix.json mv build-matrix.json.tmp build-matrix.json
fi fi
jq 'map(select(.tag != "musa"))' build-matrix.json > build-matrix.json.tmp
mv build-matrix.json.tmp build-matrix.json
BUILD_MATRIX="$(jq -c . build-matrix.json)" BUILD_MATRIX="$(jq -c . build-matrix.json)"
MERGE_MATRIX="$(jq -c ' MERGE_MATRIX="$(jq -c '
reduce .[] as $entry ({}; .[$entry.tag] |= ( reduce .[] as $entry ({}; .[$entry.tag] |= (
@@ -163,11 +145,6 @@ jobs:
needs: [prepare_matrices, create_tag, build_ui] needs: [prepare_matrices, create_tag, build_ui]
runs-on: ${{ matrix.config.runs_on }} runs-on: ${{ matrix.config.runs_on }}
# cache-mode: write # for QEMU
permissions:
actions: write
contents: read
packages: write
strategy: strategy:
fail-fast: false fail-fast: false
matrix: matrix:
@@ -184,14 +161,14 @@ jobs:
if: ${{ matrix.config.prebuilt_ui == true }} if: ${{ matrix.config.prebuilt_ui == true }}
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with: with:
name: llama-ui.zip name: ui-build
path: ui/apps/chat/dist path: tools/ui/dist
# - name: Set up QEMU - name: Set up QEMU
# if: ${{ contains(matrix.config.platforms, 'linux/amd64') }} if: ${{ contains(matrix.config.platforms, 'linux/amd64') }}
# uses: docker/setup-qemu-action@ce360397dd3f832beb865e1373c09c0e9f86d70a # v4 uses: docker/setup-qemu-action@ce360397dd3f832beb865e1373c09c0e9f86d70a # v4
# with: with:
# image: tonistiigi/binfmt:qemu-v10.2.1 image: tonistiigi/binfmt:qemu-v10.2.1
- name: Set up Docker Buildx - name: Set up Docker Buildx
uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4 uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4
@@ -415,12 +392,7 @@ jobs:
merge_arch_tags: merge_arch_tags:
name: Create shared tags from digests name: Create shared tags from digests
needs: [prepare_matrices, push_to_registry, create_tag] needs: [prepare_matrices, push_to_registry, create_tag]
runs-on: ubuntu-26.04 runs-on: ubuntu-24.04
permissions:
contents: read
packages: write
id-token: write
attestations: write
strategy: strategy:
fail-fast: false fail-fast: false
matrix: matrix:
@@ -455,7 +427,6 @@ jobs:
password: ${{ secrets.GITHUB_TOKEN }} password: ${{ secrets.GITHUB_TOKEN }}
- name: Create tags from digests - name: Create tags from digests
id: create_tags
shell: bash shell: bash
run: | run: |
set -euo pipefail set -euo pipefail
@@ -467,7 +438,6 @@ jobs:
SRC_TAG="${{ needs.create_tag.outputs.source_tag }}" SRC_TAG="${{ needs.create_tag.outputs.source_tag }}"
BUILD_DATE="${{ steps.build_date.outputs.date }}" BUILD_DATE="${{ steps.build_date.outputs.date }}"
COMMIT_SHA="${{ steps.checkout.outputs.commit }}" COMMIT_SHA="${{ steps.checkout.outputs.commit }}"
echo "image_repo=${IMAGE_REPO}" >> "$GITHUB_OUTPUT"
TAGS="${{ matrix.config.tag }}" TAGS="${{ matrix.config.tag }}"
ARCHES="${{ matrix.config.arches }}" ARCHES="${{ matrix.config.arches }}"
DIGEST_GLOB="/tmp/digests/*.tsv" DIGEST_GLOB="/tmp/digests/*.tsv"
@@ -534,16 +504,6 @@ jobs:
echo "Creating ${merged_versioned_tag} from ${refs[*]}" echo "Creating ${merged_versioned_tag} from ${refs[*]}"
docker buildx imagetools create "${annotations[@]}" --tag "${merged_versioned_tag}" "${refs[@]}" docker buildx imagetools create "${annotations[@]}" --tag "${merged_versioned_tag}" "${refs[@]}"
if [[ "$tag_name" == "${TAGS%% *}" ]]; then
local digest
digest="$(docker buildx imagetools inspect "${merged_versioned_tag}" --format '{{.Manifest.Digest}}')"
if [[ ! "$digest" =~ ^sha256:[0-9a-f]{64}$ ]]; then
echo "Invalid digest for ${merged_versioned_tag}: ${digest}" >&2
exit 1
fi
echo "${image_type}_digest=${digest}" >> "$GITHUB_OUTPUT"
fi
} }
for tag in $TAGS; do for tag in $TAGS; do
@@ -567,24 +527,3 @@ jobs:
done done
env: env:
GITHUB_REPOSITORY_OWNER: '${{ github.repository_owner }}' GITHUB_REPOSITORY_OWNER: '${{ github.repository_owner }}'
- name: Attest full image
if: ${{ matrix.config.full }}
uses: actions/attest@v4
with:
subject-name: ${{ steps.create_tags.outputs.image_repo }}
subject-digest: ${{ steps.create_tags.outputs.full_digest }}
- name: Attest light image
if: ${{ matrix.config.light }}
uses: actions/attest@v4
with:
subject-name: ${{ steps.create_tags.outputs.image_repo }}
subject-digest: ${{ steps.create_tags.outputs.light_digest }}
- name: Attest server image
if: ${{ matrix.config.server }}
uses: actions/attest@v4
with:
subject-name: ${{ steps.create_tags.outputs.image_repo }}
subject-digest: ${{ steps.create_tags.outputs.server_digest }}
+1 -5
View File
@@ -9,17 +9,13 @@ on:
branches: branches:
- master - master
cache-mode: none
permissions:
contents: read
concurrency: concurrency:
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }} group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}
cancel-in-progress: true cancel-in-progress: true
jobs: jobs:
editorconfig: editorconfig:
runs-on: ubuntu-slim runs-on: [self-hosted, fast]
steps: steps:
- uses: actions/checkout@v6 - uses: actions/checkout@v6
- uses: editorconfig-checker/action-editorconfig-checker@840e866d93b8e032123c23bac69dece044d4d84c # v2.2.0 - uses: editorconfig-checker/action-editorconfig-checker@840e866d93b8e032123c23bac69dece044d4d84c # v2.2.0
+1 -4
View File
@@ -17,14 +17,11 @@ on:
tags: tags:
- 'gguf-v*' # Push events to every version tag - 'gguf-v*' # Push events to every version tag
cache-mode: none
permissions:
contents: read
jobs: jobs:
deploy: deploy:
runs-on: ubuntu-26.04 runs-on: ubuntu-latest
steps: steps:
- uses: actions/checkout@v6 - uses: actions/checkout@v6
+4 -32
View File
@@ -9,8 +9,6 @@ on:
'.github/workflows/hip-quality-check.yml', '.github/workflows/hip-quality-check.yml',
'**/*.cu', '**/*.cu',
'**/*.cuh', '**/*.cuh',
'ggml/src/ggml-hip/CMakeLists.txt',
'ggml/src/ggml-cuda/vendors/hip.h',
'scripts/hip/gcn-cdna-vgpr-check.py' 'scripts/hip/gcn-cdna-vgpr-check.py'
] ]
@@ -20,15 +18,9 @@ on:
'.github/workflows/hip-quality-check.yml', '.github/workflows/hip-quality-check.yml',
'**/*.cu', '**/*.cu',
'**/*.cuh', '**/*.cuh',
'ggml/src/ggml-hip/CMakeLists.txt',
'ggml/src/ggml-cuda/vendors/hip.h',
'scripts/hip/gcn-cdna-vgpr-check.py' 'scripts/hip/gcn-cdna-vgpr-check.py'
] ]
cache-mode: none
permissions:
contents: read
concurrency: concurrency:
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }} group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}
cancel-in-progress: true cancel-in-progress: true
@@ -53,22 +45,14 @@ jobs:
id: depends id: depends
run: | run: |
sudo apt-get update sudo apt-get update
sudo apt-get install -y build-essential git cmake rocblas-dev hipblas-dev libssl-dev python3 python3-venv python3-pip jq sudo apt-get install -y build-essential git cmake rocblas-dev hipblas-dev libssl-dev python3
- name: ccache - name: ccache
uses: ggml-org/ccache-action@v1.2.24 uses: ggml-org/ccache-action@v1.2.21
with:
restore: false
save: false
- name: ccache-buckets-restore
uses: ./.github/actions/ccache-buckets
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CI }}
with: with:
key: hip-quality-check-ubuntu-22.04 key: hip-quality-check-ubuntu-22.04
folder: llama.cpp evict-old-files: 1d
hf_bucket: ggml-org/cache save: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
- name: Build with Werror - name: Build with Werror
id: cmake_build id: cmake_build
@@ -96,15 +80,3 @@ jobs:
cd build cd build
make -j $(nproc) 2>&1 | tee metrics.log | grep -v 'Rpass-analysis=kernel-resource-usage\|remark:\|^$' make -j $(nproc) 2>&1 | tee metrics.log | grep -v 'Rpass-analysis=kernel-resource-usage\|remark:\|^$'
python3 ../scripts/hip/gcn-cdna-vgpr-check.py metrics.log python3 ../scripts/hip/gcn-cdna-vgpr-check.py metrics.log
- name: ccache-buckets-save
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
uses: ./.github/actions/ccache-buckets
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CACHE_OUTPUT }}
with:
key: hip-quality-check-ubuntu-22.04
folder: llama.cpp
evict-old-files: 1d
hf_bucket: ggml-org/cache
save: true
-4
View File
@@ -2,10 +2,6 @@ name: "Pull Request Labeler"
on: on:
- pull_request_target - pull_request_target
cache-mode: none
permissions:
contents: read
jobs: jobs:
labeler: labeler:
permissions: permissions:
-193
View File
@@ -1,193 +0,0 @@
name: Make Release
on:
workflow_dispatch:
inputs:
commit:
description: 'Commit SHA to release (empty = branch HEAD)'
required: false
default: ''
type: string
dry_run:
description: 'Dry run - validate without creating the tag'
required: true
type: boolean
default: true
skip_apiabi_check:
description: 'Skip API/ABI compatibility check'
required: false
type: boolean
default: false
require_docker:
description: 'Require the Docker workflow to have completed successfully'
required: true
type: boolean
default: true
apiabi_compare_tag:
description: 'Tag to compare against for API/ABI check (default: latest release)'
required: false
type: string
default: ''
env:
GH_TOKEN: ${{ github.token }}
cache-mode: none
permissions:
contents: write
packages: write
jobs:
make-release:
runs-on: ubuntu-26.04
steps:
- name: Checkout
uses: actions/checkout@v6
with:
ssh-key: ${{ secrets.DEPLOY_KEY_RELEASE }}
ref: ${{ inputs.commit != '' && inputs.commit || github.ref_name }}
fetch-depth: 0
- name: Install API/ABI check tools
if: ${{ github.event.inputs.skip_apiabi_check != 'true' }}
run: sudo apt-get install -y abi-compliance-checker abigail-tools
- name: Run release checks
id: checks
run: bash scripts/make-release-checks.sh ${{ github.event.inputs.dry_run == 'true' && '--dry-run' || '' }}
env:
GITHUB_REPOSITORY: ${{ github.repository }}
RELEASE_BRANCH: ${{ github.ref_name }}
SKIP_APIABI_CHECK: ${{ github.event.inputs.skip_apiabi_check }}
APIABI_COMPARE_TAG: ${{ github.event.inputs.apiabi_compare_tag }}
APIABI_REPORT_DIR: ${{ runner.temp }}/apiabi-reports
REQUIRE_DOCKER: ${{ github.event.inputs.require_docker }}
- name: Upload API/ABI reports
if: ${{ always() && github.event.inputs.skip_apiabi_check != 'true' }}
uses: actions/upload-artifact@v7
with:
name: apiabi-reports
path: ${{ runner.temp }}/apiabi-reports
if-no-files-found: ignore
- name: Create release tag
if: ${{ github.event.inputs.dry_run == 'false' }}
run: |
VERSION="${{ steps.checks.outputs.version }}"
git config user.name "github-actions[bot]"
git config user.email "github-actions[bot]@users.noreply.github.com"
git tag -a "${VERSION}" -m "Release ${VERSION}"
git push origin "${VERSION}"
echo "Created and pushed tag ${VERSION}"
- name: Generate release description
id: desc
run: bash scripts/make-release-desc.sh "${{ steps.checks.outputs.version }}"
env:
GITHUB_REPOSITORY: ${{ github.repository }}
- name: Create nightly-tag.txt
id: nightly_tag_file
run: |
NIGHTLY_TAG="${{ steps.desc.outputs.nightly_tag }}"
if [[ -z "${NIGHTLY_TAG}" ]]; then
echo "Warning: no nightly tag found for the release commit - nightly-tag.txt will not be created"
echo "create=false" >> "$GITHUB_OUTPUT"
exit 0
fi
echo "${NIGHTLY_TAG}" > nightly-tag.txt
echo "create=true" >> "$GITHUB_OUTPUT"
echo "nightly-tag.txt:"
cat nightly-tag.txt
- name: Create release
id: create_release
if: ${{ github.event.inputs.dry_run == 'false' }}
uses: ggml-org/action-create-release@v1
env:
GITHUB_TOKEN: ${{ github.token }}
with:
tag_name: ${{ steps.checks.outputs.version }}
prerelease: false
# TODO: enrich the body of the release with more information
body: |
## Overview
New version has been released.
## Assets
${{ steps.desc.outputs.nightly }}
## More info
- [Releases and versioning of `ggml-org` projects](https://github.com/ggml-org/ggml/discussions/1579)
## ${{ steps.desc.outputs.changelog_title }}
${{ steps.desc.outputs.changelog }}
- name: Upload nightly-tag.txt
if: ${{ github.event.inputs.dry_run == 'false' && steps.nightly_tag_file.outputs.create == 'true' }}
uses: actions/github-script@v8
with:
github-token: ${{secrets.GITHUB_TOKEN}}
script: |
const fs = require('fs');
const release_id = '${{ steps.create_release.outputs.id }}';
console.log('uploadReleaseAsset', 'nightly-tag.txt');
await github.rest.repos.uploadReleaseAsset({
owner: context.repo.owner,
repo: context.repo.repo,
release_id: release_id,
name: 'nightly-tag.txt',
data: await fs.readFileSync('./nightly-tag.txt')
});
- name: Re-tag container images with release version
if: ${{ github.event.inputs.dry_run == 'false' && github.event.inputs.require_docker != 'false' && steps.desc.outputs.nightly_tag != '' }}
env:
GITHUB_REPOSITORY_OWNER: ${{ github.repository_owner }}
run: |
VERSION="${{ steps.checks.outputs.version }}"
NIGHTLY_TAG="${{ steps.desc.outputs.nightly_tag }}"
REPO_OWNER="${GITHUB_REPOSITORY_OWNER,,}"
IMAGE_REPO="ghcr.io/${REPO_OWNER}/${{ github.event.repository.name }}"
echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u "${{ github.actor }}" --password-stdin
VARIANTS=("" "-cuda" "-cuda13" "-vulkan" "-rocm" "-intel" "-musa" "-openvino")
TYPES=("full" "light" "server")
# the release is already created at this point, so keep going on a
# missing image and report all of them at the end
MISSING=()
for type in "${TYPES[@]}"; do
for variant in "${VARIANTS[@]}"; do
src="${IMAGE_REPO}:${type}${variant}-${NIGHTLY_TAG}"
dst="${IMAGE_REPO}:${type}${variant}-${VERSION}"
echo "Tagging ${src} -> ${dst}"
if ! docker buildx imagetools create --tag "${dst}" "${src}"; then
MISSING+=("${type}${variant}")
fi
done
done
if [[ ${#MISSING[@]} -gt 0 ]]; then
echo "::error::failed to re-tag container images for ${NIGHTLY_TAG}:${MISSING[*]}"
exit 1
fi
- name: Dry run summary
if: ${{ github.event.inputs.dry_run == 'true' }}
run: |
if [[ "${{ steps.checks.outputs.checks_passed }}" == "true" ]]; then
echo "Dry run complete - all checks passed."
echo "Would have created tag: ${{ steps.checks.outputs.version }}"
if [[ -n "${{ steps.desc.outputs.nightly_tag }}" ]]; then
echo "Would have uploaded nightly-tag.txt: ${{ steps.desc.outputs.nightly_tag }}"
fi
else
echo "::error::Dry run found release check failures. A release tag would not be created."
exit 1
fi
-461
View File
@@ -1,461 +0,0 @@
name: Models Backend Check
on:
workflow_dispatch: # allows manual triggering
push:
branches:
- master
paths: [
'.github/workflows/models-check.yml',
'ggml/**',
'tests/fusion/**',
'tests/test-fusion.cpp',
'tests/test-llama-archs.cpp',
'src/llama-graph*',
'src/llama-kv-cache*',
'src/llama-model*',
'src/models/**'
]
pull_request:
types: [opened, synchronize, reopened]
paths: [
'.github/workflows/models-check.yml',
'ggml/**',
'tests/fusion/**',
'tests/test-fusion.cpp',
'tests/test-llama-archs.cpp',
'src/llama-graph*',
'src/llama-kv-cache*',
'src/llama-model*',
'src/models/**'
]
cache-mode: none
permissions:
contents: read
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}
cancel-in-progress: true
env:
# note: this is dud token to avoid rate limiting (https://github.com/ggml-org/llama.cpp/pull/25706#issuecomment-4979941302)
HF_TOKEN: ${{ secrets.HF_TOKEN_CI }}
GGML_NLOOP: 3
GGML_N_THREADS: 1
LLAMA_ARG_LOG_COLORS: 1
LLAMA_ARG_LOG_PREFIX: 1
LLAMA_ARG_LOG_TIMESTAMPS: 1
jobs:
cuda:
runs-on: "hf-jobs-t4-medium:cuda13"
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: Install dependencies
run: |
sudo apt update
sudo apt install -y cmake time python3 python3-venv python3-pip
- name: ccache
uses: ggml-org/ccache-action@v1.2.24
with:
restore: false
save: false
- name: ccache-buckets-restore
uses: ./.github/actions/ccache-buckets
with:
key: models-check-cuda
folder: llama.cpp
hf_bucket: ggml-org/cache
- name: Build
id: cmake_build
run: |
cmake -B build \
-DCMAKE_BUILD_TYPE=Release \
-DLLAMA_FATAL_WARNINGS=ON \
-DLLAMA_OPENSSL=OFF \
-DGGML_SCHED_NO_REALLOC=ON \
-DCMAKE_CUDA_COMPILER=/usr/local/cuda/bin/nvcc \
-DGGML_CUDA=ON
time cmake --build build --config Release --target test-llama-archs -j$(nproc)
time cmake --build build --config Release --target test-fusion -j$(nproc)
- name: ccache-buckets-save
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
uses: ./.github/actions/ccache-buckets
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CACHE_OUTPUT }}
with:
key: models-check-cuda
folder: llama.cpp
evict-old-files: 1d
hf_bucket: ggml-org/cache
save: true
# - name: Generate models
# id: generate_models
# run: |
# rm -rf build-ci-models && mkdir -p build-ci-models
# ./build/bin/test-llama-archs -o build-ci-models
# TODO: add for backends as they adopt the fusion debug API
# - name: Test fusion
# id: test_fusion
# run: |
# ./build/bin/test-fusion --models build-ci-models --device CUDA0 --check tests/fusion/CUDA.csv
- name: Test archs
id: test_archs
run: |
GGML_CUDA_DEVICES=1 ./build/bin/test-llama-archs -s 1
GGML_CUDA_DEVICES=2 ./build/bin/test-llama-archs -s 1
GGML_CUDA_DEVICES=3 ./build/bin/test-llama-archs -s 1
GGML_CUDA_DEVICES=4 ./build/bin/test-llama-archs -s 1
metal:
runs-on: [self-hosted, macOS, ARM64]
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: Build
id: cmake_build
run: |
cmake -B build \
-DCMAKE_BUILD_TYPE=Release \
-DLLAMA_FATAL_WARNINGS=ON \
-DLLAMA_OPENSSL=OFF \
-DGGML_SCHED_NO_REALLOC=ON \
-DGGML_BLAS=OFF \
-DGGML_METAL=ON
time cmake --build build --config Release --target test-llama-archs -j $(sysctl -n hw.logicalcpu)
time cmake --build build --config Release --target test-fusion -j $(sysctl -n hw.logicalcpu)
- name: Generate models
id: generate_models
run: |
rm -rf build-ci-models && mkdir -p build-ci-models
./build/bin/test-llama-archs -o build-ci-models
- name: Test KV shift
id: test_kv_shift
run: |
./build/bin/test-llama-archs --kv-shift
- name: Test fusion
id: test_fusion
run: |
./build/bin/test-fusion --models build-ci-models --device MTL0 --check tests/fusion/MTL.csv
- name: Test archs
id: test_archs
run: |
GGML_METAL_DEVICES=1 ./build/bin/test-llama-archs -s 1
GGML_METAL_DEVICES=2 ./build/bin/test-llama-archs -s 1
GGML_METAL_DEVICES=3 ./build/bin/test-llama-archs -s 1
GGML_METAL_DEVICES=4 ./build/bin/test-llama-archs -s 1
rocm:
runs-on: [self-hosted, Linux, gfx1201, 1accel]
container: "rocm/dev-ubuntu-24.04:7.2.4-complete"
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: Install dependencies
run: |
apt update
apt install -y build-essential jq cmake time python3 python3-venv python3-pip
- name: ccache
uses: ggml-org/ccache-action@v1.2.24
with:
restore: false
save: false
- name: ccache-buckets-restore
uses: ./.github/actions/ccache-buckets
with:
key: models-check-rocm
folder: llama.cpp
hf_bucket: ggml-org/cache
- name: Build
id: cmake_build
run: |
cmake -B build \
-DCMAKE_BUILD_TYPE=Release \
-DLLAMA_FATAL_WARNINGS=ON \
-DLLAMA_OPENSSL=OFF \
-DGGML_SCHED_NO_REALLOC=ON \
-DCMAKE_HIP_COMPILER=$(hipconfig -l)/clang \
-DGPU_TARGETS=gfx1201 \
-DGGML_HIP=ON
time cmake --build build --config Release --target test-llama-archs -j$(nproc)
time cmake --build build --config Release --target test-fusion -j$(nproc)
- name: ccache-buckets-save
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
uses: ./.github/actions/ccache-buckets
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CACHE_OUTPUT }}
with:
key: models-check-rocm
folder: llama.cpp
evict-old-files: 1d
hf_bucket: ggml-org/cache
save: true
# - name: Generate models
# id: generate_models
# run: |
# rm -rf build-ci-models && mkdir -p build-ci-models
# ./build/bin/test-llama-archs -o build-ci-models
# TODO: add for backends as they adopt the fusion debug API
# - name: Test fusion
# id: test_fusion
# run: |
# ./build/bin/test-fusion --models build-ci-models --device CUDA0 --check tests/fusion/CUDA.csv
- name: Test archs
id: test_archs
run: |
GGML_CUDA_DEVICES=1 ./build/bin/test-llama-archs -s 1
GGML_CUDA_DEVICES=2 ./build/bin/test-llama-archs -s 1
GGML_CUDA_DEVICES=3 ./build/bin/test-llama-archs -s 1
GGML_CUDA_DEVICES=4 ./build/bin/test-llama-archs -s 1
vulkan-nvidia:
runs-on: "hf-jobs-t4-small:ubuntu26_04"
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: Install dependencies
run: |
sudo apt update
sudo apt install -y build-essential cmake libxcb-xinput0 libxcb-xinerama0 libxcb-cursor-dev libvulkan-dev glslc spirv-headers vulkan-tools mesa-vulkan-drivers libglvnd0 libgl1 libglx0 libegl1 libgles2 time python3 python3-venv python3-pip
- name: ccache
uses: ggml-org/ccache-action@v1.2.24
with:
restore: false
save: false
- name: ccache-buckets-restore
uses: ./.github/actions/ccache-buckets
with:
key: models-check-vulkan-nvidia
folder: llama.cpp
hf_bucket: ggml-org/cache
- name: Build
id: cmake_build
run: |
cmake -B build \
-DCMAKE_BUILD_TYPE=Release \
-DLLAMA_FATAL_WARNINGS=ON \
-DLLAMA_OPENSSL=OFF \
-DGGML_SCHED_NO_REALLOC=ON \
-DGGML_VULKAN=ON
time cmake --build build --config Release --target test-llama-archs -j$(nproc)
time cmake --build build --config Release --target test-fusion -j$(nproc)
- name: ccache-buckets-save
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
uses: ./.github/actions/ccache-buckets
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CACHE_OUTPUT }}
with:
key: models-check-vulkan-nvidia
folder: llama.cpp
evict-old-files: 1d
hf_bucket: ggml-org/cache
save: true
# - name: Generate models
# id: generate_models
# run: |
# rm -rf build-ci-models && mkdir -p build-ci-models
# ./build/bin/test-llama-archs -o build-ci-models
# TODO: add for backends as they adopt the fusion debug API
# - name: Test fusion
# id: test_fusion
# run: |
# ./build/bin/test-fusion --models build-ci-models --device Vulkan0 --check tests/fusion/Vulkan.csv
- name: Test KV shift
id: test_kv_shift
run: |
./build/bin/test-llama-archs --kv-shift
- name: Test archs
id: test_archs
run: |
./build/bin/test-llama-archs -s 1
vulkan-amd:
runs-on: [self-hosted, Linux, gfx1201, 1accel]
container: "ubuntu:26.04"
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: Install dependencies
run: |
apt update
apt install -y build-essential jq cmake libxcb-xinput0 libxcb-xinerama0 libxcb-cursor-dev libvulkan-dev glslc spirv-headers vulkan-tools mesa-vulkan-drivers libglvnd0 libgl1 libglx0 libegl1 libgles2 time python3 python3-venv python3-pip
- name: ccache
uses: ggml-org/ccache-action@v1.2.24
with:
restore: false
save: false
- name: ccache-buckets-restore
uses: ./.github/actions/ccache-buckets
with:
key: models-check-vulkan-amd
folder: llama.cpp
hf_bucket: ggml-org/cache
- name: Build
id: cmake_build
run: |
cmake -B build \
-DCMAKE_BUILD_TYPE=Release \
-DLLAMA_FATAL_WARNINGS=ON \
-DLLAMA_OPENSSL=OFF \
-DGGML_SCHED_NO_REALLOC=ON \
-DGGML_VULKAN=ON
time cmake --build build --config Release --target test-llama-archs -j$(nproc)
time cmake --build build --config Release --target test-fusion -j$(nproc)
- name: ccache-buckets-save
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
uses: ./.github/actions/ccache-buckets
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CACHE_OUTPUT }}
with:
key: models-check-vulkan-amd
folder: llama.cpp
evict-old-files: 1d
hf_bucket: ggml-org/cache
save: true
# - name: Generate models
# id: generate_models
# run: |
# rm -rf build-ci-models && mkdir -p build-ci-models
# ./build/bin/test-llama-archs -o build-ci-models
# TODO: add for backends as they adopt the fusion debug API
# - name: Test fusion
# id: test_fusion
# run: |
# ./build/bin/test-fusion --models build-ci-models --device Vulkan0 --check tests/fusion/Vulkan.csv
- name: Test archs
id: test_archs
run: |
./build/bin/test-llama-archs -s 1
webgpu-nvidia:
runs-on: "hf-jobs-t4-small:ubuntu26_04"
steps:
- name: Clone
id: checkout
uses: actions/checkout@v6
- name: Install dependencies
run: |
sudo apt update
sudo apt install -y build-essential cmake libxcb-xinput0 libxcb-xinerama0 libxcb-cursor-dev libvulkan1 mesa-vulkan-drivers libglvnd0 libgl1 libglx0 libegl1 libgles2 time python3 python3-venv python3-pip
- name: ccache
uses: ggml-org/ccache-action@v1.2.24
with:
restore: false
save: false
- name: ccache-buckets-restore
uses: ./.github/actions/ccache-buckets
with:
key: models-check-webgpu-nvidia
folder: llama.cpp
hf_bucket: ggml-org/cache
- name: Dawn Dependency
id: dawn-depends
run: |
DAWN_VERSION="v20260908.214631"
DAWN_OWNER="google"
DAWN_REPO="dawn"
DAWN_ASSET_NAME="Dawn-94c3c9cc0d5fb2e85aebb370fa8d37b71aa34655-ubuntu-latest-Release"
echo "Fetching release asset from https://github.com/google/dawn/releases/download/${DAWN_VERSION}/${DAWN_ASSET_NAME}.tar.gz"
curl -L -o artifact.tar.gz \
"https://github.com/google/dawn/releases/download/${DAWN_VERSION}/${DAWN_ASSET_NAME}.tar.gz"
mkdir dawn
tar -xvf artifact.tar.gz -C dawn --strip-components=1
- name: Build
id: cmake_build
run: |
cmake -B build \
-DCMAKE_BUILD_TYPE=Release \
-DLLAMA_FATAL_WARNINGS=ON \
-DLLAMA_OPENSSL=OFF \
-DGGML_SCHED_NO_REALLOC=ON \
-DCMAKE_PREFIX_PATH="$GITHUB_WORKSPACE/dawn" \
-DDawn_DIR="$GITHUB_WORKSPACE/dawn/lib64/cmake/Dawn" \
-DGGML_WEBGPU=ON
time cmake --build build --config Release --target test-llama-archs -j$(nproc)
time cmake --build build --config Release --target test-fusion -j$(nproc)
- name: ccache-buckets-save
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
uses: ./.github/actions/ccache-buckets
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CACHE_OUTPUT }}
with:
key: models-check-webgpu-nvidia
folder: llama.cpp
evict-old-files: 1d
hf_bucket: ggml-org/cache
save: true
# - name: Generate models
# id: generate_models
# run: |
# rm -rf build-ci-models && mkdir -p build-ci-models
# ./build/bin/test-llama-archs -o build-ci-models
# TODO: add for backends as they adopt the fusion debug API
# - name: Test fusion
# id: test_fusion
# run: |
# ./build/bin/test-fusion --models build-ci-models --device WebGPU --check tests/fusion/WebGPU.csv
- name: Test archs
id: test_archs
run: |
./build/bin/test-llama-archs -s 1
-24
View File
@@ -1,24 +0,0 @@
name: Convert PR to draft
on:
pull_request_target:
types: [labeled]
cache-mode: none
permissions:
pull-requests: write
issues: write
contents: write # required for "gh pr ready" command, see https://github.com/cli/cli/issues/8910
jobs:
convert-to-draft:
if: github.event.label.name == 'draft' && github.event.pull_request.draft == false
runs-on: ubuntu-slim
steps:
- name: Convert PR to draft
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
PR_URL: ${{ github.event.pull_request.html_url }}
run: |
gh pr ready --undo "$PR_URL"
gh pr edit "$PR_URL" --remove-label draft
-37
View File
@@ -1,37 +0,0 @@
name: "PR video demo label"
on:
pull_request_target:
types: [opened, edited, reopened, synchronize]
cache-mode: none
permissions:
pull-requests: write
issues: write
jobs:
video-demo-label:
runs-on: ubuntu-slim
steps:
- name: Add or remove the video demo label
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
PR_URL: ${{ github.event.pull_request.html_url }}
PR_NUMBER: ${{ github.event.pull_request.number }}
# passed through env, so the body is only matched, never run, and is safe to read from a fork
BODY: ${{ github.event.pull_request.body }}
HAS_LABEL: ${{ contains(github.event.pull_request.labels.*.name, 'video demo') }}
run: |
# sections written by demo:pr (see ui/apps/chat/demo/cli.ts): the managed comment
# markers, or a preview linking to a demo video uploaded for this PR
has_demo=false
if { grep -qF '<!-- demo:start -->' <<< "$BODY" && grep -qF '<!-- demo:end -->' <<< "$BODY"; } ||
grep -qE "/resolve/videos/pr/$PR_NUMBER/[0-9a-f]{10}(-[0-9]+)?\.mp4" <<< "$BODY"; then
has_demo=true
fi
if [ "$has_demo" = true ] && [ "$HAS_LABEL" = false ]; then
gh pr edit "$PR_URL" --add-label 'video demo'
elif [ "$has_demo" = false ] && [ "$HAS_LABEL" = true ]; then
gh pr edit "$PR_URL" --remove-label 'video demo'
fi
+1 -5
View File
@@ -10,13 +10,9 @@ on:
- 'conversion/base.py' - 'conversion/base.py'
- 'convert_hf_to_gguf_update.py' - 'convert_hf_to_gguf_update.py'
cache-mode: none
permissions:
contents: read
jobs: jobs:
pre-tokenizer-hashes: pre-tokenizer-hashes:
runs-on: ubuntu-slim runs-on: [self-hosted, fast]
steps: steps:
- name: Checkout repository - name: Checkout repository
@@ -14,17 +14,13 @@ on:
- 'convert*.py' - 'convert*.py'
- '**/requirements*.txt' - '**/requirements*.txt'
cache-mode: none
permissions:
contents: read
concurrency: concurrency:
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }} group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}
cancel-in-progress: true cancel-in-progress: true
jobs: jobs:
python-check-requirements: python-check-requirements:
runs-on: ${{ 'ubuntu-26.04-arm' || 'ubuntu-26.04' }} runs-on: [self-hosted, CPU, fast]
name: check-requirements name: check-requirements
steps: steps:
- name: Check out source repository - name: Check out source repository
+1 -5
View File
@@ -15,17 +15,13 @@ on:
'**/*.py' '**/*.py'
] ]
cache-mode: none
permissions:
contents: read
concurrency: concurrency:
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }} group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}
cancel-in-progress: true cancel-in-progress: true
jobs: jobs:
flake8-lint: flake8-lint:
runs-on: ubuntu-slim runs-on: [self-hosted, fast]
name: Lint name: Lint
steps: steps:
- name: Check out source repository - name: Check out source repository
+3 -7
View File
@@ -16,17 +16,13 @@ on:
- '**/requirements*.txt' - '**/requirements*.txt'
# - 'pyrightconfig.json' # - 'pyrightconfig.json'
cache-mode: none
permissions:
contents: read
concurrency: concurrency:
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }} group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}
cancel-in-progress: true cancel-in-progress: true
jobs: jobs:
python-type-check: python-type-check:
runs-on: ubuntu-slim runs-on: [self-hosted, fast]
name: python type-check name: python type-check
steps: steps:
- name: Check out source repository - name: Check out source repository
@@ -35,7 +31,7 @@ jobs:
uses: actions/setup-python@v6 uses: actions/setup-python@v6
with: with:
python-version: "3.11" python-version: "3.11"
pip-install: -r requirements/requirements-all.txt ty==0.0.100 pip-install: -r requirements/requirements-all.txt ty==0.0.35
# - name: Type-check with Pyright # - name: Type-check with Pyright
# uses: jakebailey/pyright-action@v2 # uses: jakebailey/pyright-action@v2
# with: # with:
@@ -44,4 +40,4 @@ jobs:
# warnings: true # warnings: true
- name: Type-check with ty - name: Type-check with ty
run: | run: |
ty check --exit-zero-on-warning --output-format=github ty check --output-format=github
-249
View File
@@ -1,249 +0,0 @@
name: Publish Release
on:
workflow_run:
workflows:
- Release
types:
- completed
branches:
- master
run-name: "Publish ${{ github.event.workflow_run.display_title }}"
cache-mode: none
permissions:
actions: read
contents: read
env:
GH_TOKEN: ${{ github.token }}
BRANCH_NAME: master
jobs:
publish:
if: ${{ github.event.workflow_run.conclusion == 'success' }}
# Fine-grained permission
# https://docs.github.com/en/actions/security-for-github-actions/security-guides/automatic-token-authentication#modifying-the-permissions-for-the-github_token
permissions:
actions: read
contents: write # for creating release
id-token: write
attestations: write
runs-on: ubuntu-latest
outputs:
should_release: ${{ steps.check.outputs.should_release }}
tag_name: ${{ steps.tag.outputs.name }}
steps:
- id: check
env:
COMMIT_MESSAGE: ${{ github.event.workflow_run.head_commit.message }}
run: |
if echo "$COMMIT_MESSAGE" | grep -q '\[no release\]'; then
echo "should_release=false" >> $GITHUB_OUTPUT
else
echo "should_release=true" >> $GITHUB_OUTPUT
fi
- name: Clone
if: ${{ steps.check.outputs.should_release == 'true' }}
id: checkout
uses: actions/checkout@v6
with:
ref: ${{ github.event.workflow_run.head_sha }}
fetch-depth: 0
ssh-key: ${{ secrets.DEPLOY_KEY_RELEASE }}
- name: Determine tag name
if: ${{ steps.check.outputs.should_release == 'true' }}
id: tag
uses: ./.github/actions/get-tag-name
- name: Download artifacts
if: ${{ steps.check.outputs.should_release == 'true' }}
id: download-artifact
uses: actions/download-artifact@v8
with:
path: ./artifact
run-id: ${{ github.event.workflow_run.id }}
github-token: ${{ github.token }}
merge-multiple: true
skip-decompress: true
- name: Merge artifacts
if: ${{ steps.check.outputs.should_release == 'true' }}
id: move_artifacts
run: |
mkdir -p release
# the windows-cpu zip contains the full toolset (llama-server with the embedded
# UI, ggml-cpu) - inject it into the other windows zips so that every archive
# ships the same binaries, only with a different backend library on top
echo "Injecting windows-cpu binaries (llama-server + CPU backend) into the backend zips..."
for arch in x64 arm64; do
cpu_zip="artifact/llama-bin-win-cpu-${arch}.zip"
temp_dir=$(mktemp -d)
echo "Extracting windows-cpu-${arch} package..."
unzip "$cpu_zip" -d "$temp_dir"
echo "Merging into $arch zips..."
for target_zip in artifact/llama-bin-win-*-${arch}.zip; do
if [[ "$target_zip" == "$cpu_zip" ]]; then
continue
fi
echo "Injecting into $(basename "$target_zip")"
realpath_target_zip=$(realpath "$target_zip")
(cd "$temp_dir" && zip -r "$realpath_target_zip" .)
done
rm -rf "$temp_dir"
done
echo "Renaming and moving zips to release..."
for zip_file in artifact/llama-bin-win-*.zip; do
base_name=$(basename "$zip_file" .zip)
zip_name="llama-${{ steps.tag.outputs.name }}-${base_name#llama-}.zip"
echo "Moving $zip_file to release/$zip_name"
mv "$zip_file" "release/$zip_name"
done
echo "Moving other artifacts..."
rm -f artifact/llama-ui.zip
mv -v artifact/*.zip release
mv -v artifact/*.tar.gz release
- name: Download UI build
if: ${{ steps.check.outputs.should_release == 'true' }}
id: download_ui
uses: actions/download-artifact@v8
with:
name: llama-ui.zip
path: ./ui-dist
run-id: ${{ github.event.workflow_run.id }}
github-token: ${{ github.token }}
- name: Package UI
if: ${{ steps.check.outputs.should_release == 'true' }}
id: package_ui
run: |
tar -czvf release/llama-${{ steps.tag.outputs.name }}-ui.tar.gz --transform "s,^\.,llama-${{ steps.tag.outputs.name }}," -C ./ui-dist .
- name: Attest release artifacts
if: ${{ steps.check.outputs.should_release == 'true' }}
id: attest
uses: actions/attest@v4
with:
subject-path: 'release/*'
- name: Create release
if: ${{ steps.check.outputs.should_release == 'true' }}
id: create_release
uses: ggml-org/action-create-release@v1
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
tag_name: ${{ steps.tag.outputs.name }}
commitish: ${{ github.event.workflow_run.head_sha }}
prerelease: true
body: |
<details open>
${{ github.event.workflow_run.head_commit.message }}
</details>
**Website:**
- <https://llama.app>
**Attestations:**
- <${{ steps.attest.outputs.attestation-url }}>
**macOS/iOS:**
- [macOS Apple Silicon (arm64)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-macos-arm64.tar.gz)
- macOS Apple Silicon (arm64, KleidiAI enabled) [DISABLED](https://github.com/ggml-org/llama.cpp/pull/23780)
- [macOS Intel (x64)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-macos-x64.tar.gz)
- [iOS XCFramework](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-xcframework.zip)
**Linux:**
- [Ubuntu x64 (CPU)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-ubuntu-x64.tar.gz)
- [Ubuntu arm64 (CPU)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-ubuntu-arm64.tar.gz)
- [Ubuntu 24.04 s390x (CPU)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-ubuntu-s390x.tar.gz)
- [Ubuntu 26.04 s390x (CPU)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-ubuntu-26.04-s390x.tar.gz)
- [Ubuntu 26.04 s390x (zDNN)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-ubuntu-zdnn-s390x.tar.gz)
- [Ubuntu x64 (Vulkan)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-ubuntu-vulkan-x64.tar.gz)
- [Ubuntu arm64 (Vulkan)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-ubuntu-vulkan-arm64.tar.gz)
- [Ubuntu x64 (CUDA 12)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-ubuntu-cuda-12.8-x64.tar.gz) - [CUDA 12.8 libraries](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/cudart-llama-${{ steps.tag.outputs.name }}-bin-ubuntu-cuda-12.8-x64.tar.gz)
- [Ubuntu x64 (CUDA 13)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-ubuntu-cuda-13.5-x64.tar.gz) - [CUDA 13.5 libraries](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/cudart-llama-${{ steps.tag.outputs.name }}-bin-ubuntu-cuda-13.5-x64.tar.gz)
- [Ubuntu arm64 (CUDA 13)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-ubuntu-cuda-13.5-arm64.tar.gz) - [CUDA 13.5 libraries](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/cudart-llama-${{ steps.tag.outputs.name }}-bin-ubuntu-cuda-13.5-arm64.tar.gz)
- [Ubuntu x64 (ROCm 10.0)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-ubuntu-rocm-10.0-x64.tar.gz)
- [Ubuntu x64 (OpenVINO)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-ubuntu-openvino-2026.4.1-x64.tar.gz)
- [Ubuntu x64 (SYCL FP32)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-ubuntu-sycl-fp32-x64.tar.gz)
- [Ubuntu x64 (SYCL FP16)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-ubuntu-sycl-fp16-x64.tar.gz)
- [Linux arm64 (Snapdragon: CPU, Adreno GPU, Hexagon NPU)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-linux-arm64-snapdragon.tar.gz) - [setup guide](https://github.com/ggml-org/llama.cpp/blob/master/docs/backend/snapdragon/linux.md)
**Android:**
- [Android arm64 (CPU)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-android-arm64.tar.gz)
- [Android arm64 (Snapdragon: CPU, Adreno GPU, Hexagon NPU)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-android-arm64-snapdragon.tar.gz) - [setup guide](https://github.com/ggml-org/llama.cpp/blob/master/docs/backend/snapdragon/README.md)
**Windows:**
- [Windows x64 (CPU)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-win-cpu-x64.zip)
- [Windows arm64 (CPU)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-win-cpu-arm64.zip)
- [Windows arm64 (OpenCL Adreno)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-win-opencl-adreno-arm64.zip)
- [Windows x64 (CUDA 12)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-win-cuda-12.4-x64.zip) - [CUDA 12.4 DLLs](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/cudart-llama-bin-win-cuda-12.4-x64.zip)
- [Windows x64 (CUDA 13)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-win-cuda-13.5-x64.zip) - [CUDA 13.5 DLLs](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/cudart-llama-bin-win-cuda-13.5-x64.zip)
- [Windows arm64 (CUDA 13)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-win-cuda-13.5-arm64.zip) - [CUDA 13.5 DLLs](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/cudart-llama-bin-win-cuda-13.5-arm64.zip)
- [Windows x64 (Vulkan)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-win-vulkan-x64.zip)
- [Windows arm64 (Vulkan)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-win-vulkan-arm64.zip)
- [Windows x64 (OpenVINO)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-win-openvino-2026.4.1-x64.zip)
- [Windows x64 (SYCL)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-win-sycl-x64.zip)
- [Windows x64 (ROCm 10.0)](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-bin-win-rocm-10.0-x64.zip)
**openEuler:**
- [DISABLED](https://github.com/ggml-org/llama.cpp/pull/23705)
- openEuler x86 (310p)
- openEuler x86 (910b, ACL Graph)
- openEuler aarch64 (310p)
- openEuler aarch64 (910b, ACL Graph)
**UI:**
- [UI](https://github.com/ggml-org/llama.cpp/releases/download/${{ steps.tag.outputs.name }}/llama-${{ steps.tag.outputs.name }}-ui.tar.gz)
- name: Upload release
if: ${{ steps.check.outputs.should_release == 'true' }}
id: upload_release
uses: actions/github-script@v8
with:
github-token: ${{secrets.GITHUB_TOKEN}}
script: |
const path = require('path');
const fs = require('fs');
const release_id = '${{ steps.create_release.outputs.id }}';
for (let file of await fs.readdirSync('./release')) {
if (path.extname(file) === '.zip' || file.endsWith('.tar.gz')) {
console.log('uploadReleaseAsset', file);
await github.rest.repos.uploadReleaseAsset({
owner: context.repo.owner,
repo: context.repo.repo,
release_id: release_id,
name: file,
data: await fs.readFileSync(`./release/${file}`)
});
}
}
ui-publish:
if: ${{ needs.publish.outputs.should_release == 'true' }}
needs:
- publish
uses: ./.github/workflows/ui-publish.yml
with:
version_tag: ${{ needs.publish.outputs.tag_name }}
run_id: ${{ github.event.workflow_run.id }}
secrets:
hf_token: ${{ secrets.HF_TOKEN_UI_STATIC_OUTPUT }}
File diff suppressed because it is too large. Load diff
+29 -50
View File
@@ -25,25 +25,11 @@ on:
'tools/server/**.*' 'tools/server/**.*'
] ]
pull_request:
types: [opened, synchronize, reopened]
paths: [
'.github/workflows/server-sanitize.yml'
]
cache-mode: none
permissions:
contents: read
env: env:
# note: this is dud token to avoid rate limiting (https://github.com/ggml-org/llama.cpp/pull/25706#issuecomment-4979941302)
HF_TOKEN: ${{ secrets.HF_TOKEN_CI }}
LLAMA_ARG_LOG_COLORS: 1 LLAMA_ARG_LOG_COLORS: 1
LLAMA_ARG_LOG_PREFIX: 1 LLAMA_ARG_LOG_PREFIX: 1
LLAMA_ARG_LOG_TIMESTAMPS: 1 LLAMA_ARG_LOG_TIMESTAMPS: 1
LLAMA_ARG_LOG_VERBOSITY: 10 LLAMA_ARG_LOG_VERBOSITY: 10
# [TAG_MTMD_NO_REALLOC] must match -DGGML_SCHED_NO_REALLOC=ON below, skips tests that are known to realloc
GGML_SCHED_NO_REALLOC: ON
concurrency: concurrency:
group: ${{ github.workflow }}-${{ github.ref }}-${{ github.head_ref || github.run_id }} group: ${{ github.workflow }}-${{ github.ref }}-${{ github.head_ref || github.run_id }}
@@ -51,7 +37,7 @@ concurrency:
jobs: jobs:
server: server:
runs-on: hf-jobs-cpu-performance runs-on: [self-hosted, CPU, Linux, llama-server]
strategy: strategy:
matrix: matrix:
@@ -60,6 +46,20 @@ jobs:
fail-fast: false fail-fast: false
steps: steps:
#- name: Dependencies
# id: depends
# run: |
# sudo apt-get update
# sudo apt-get -y install \
# build-essential \
# xxd \
# git \
# cmake \
# curl \
# wget \
# language-pack-en \
# libssl-dev
- name: Clone - name: Clone
id: checkout id: checkout
uses: actions/checkout@v6 uses: actions/checkout@v6
@@ -67,23 +67,12 @@ jobs:
fetch-depth: 0 fetch-depth: 0
ref: ${{ github.event.inputs.sha || github.event.pull_request.head.sha || github.sha || github.head_ref || github.ref_name }} ref: ${{ github.event.inputs.sha || github.event.pull_request.head.sha || github.sha || github.head_ref || github.ref_name }}
- name: Install dependencies - name: Setup Node.js
run: | uses: actions/setup-node@v6
sudo apt update
sudo apt install -y build-essential cmake python3-full
- name: ccache
uses: ggml-org/ccache-action@v1.2.24
with: with:
restore: false node-version: "24"
save: false cache: "npm"
cache-dependency-path: "tools/ui/package-lock.json"
- name: ccache-buckets-restore
uses: ./.github/actions/ccache-buckets
with:
key: server-sanitize-${{ matrix.sanitizer }}
folder: llama.cpp
hf_bucket: ggml-org/cache
- name: Build - name: Build
id: cmake_build id: cmake_build
@@ -99,35 +88,25 @@ jobs:
-DLLAMA_SANITIZE_UNDEFINED=${{ matrix.sanitizer == 'UNDEFINED' }} -DLLAMA_SANITIZE_UNDEFINED=${{ matrix.sanitizer == 'UNDEFINED' }}
cmake --build build --config ${{ matrix.build_type }} -j $(nproc) --target llama-server cmake --build build --config ${{ matrix.build_type }} -j $(nproc) --target llama-server
- name: ccache-buckets-save - name: Python setup
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }} id: setup_python
uses: ./.github/actions/ccache-buckets uses: actions/setup-python@v6
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CACHE_OUTPUT }}
with: with:
key: server-sanitize-${{ matrix.sanitizer }} python-version: '3.11'
folder: llama.cpp pip-install: -r tools/server/tests/requirements.txt
evict-old-files: 1d
hf_bucket: ggml-org/cache
save: true
- name: Install Python dependencies
run: |
python3 -m venv .venv
.venv/bin/pip install -r tools/server/tests/requirements.txt
- name: Tests - name: Tests
id: server_integration_tests id: server_integration_tests
if: ${{ (!matrix.disabled_on_pr || !github.event.pull_request) }} if: ${{ (!matrix.disabled_on_pr || !github.event.pull_request) }}
run: | run: |
source .venv/bin/activate
cd tools/server/tests cd tools/server/tests
PYTEST_WORKERS=4 ./tests.sh export ${{ matrix.extra_args }}
pytest -v -x -m "not slow"
- name: Slow tests - name: Slow tests
id: server_integration_tests_slow id: server_integration_tests_slow
if: ${{ (github.event.schedule || github.event.inputs.slow_tests == 'true') && matrix.build_type == 'Release' }} if: ${{ (github.event.schedule || github.event.inputs.slow_tests == 'true') && matrix.build_type == 'Release' }}
run: | run: |
source .venv/bin/activate
cd tools/server/tests cd tools/server/tests
PYTEST_WORKERS=4 SLOW_TESTS=1 ./tests.sh export ${{ matrix.extra_args }}
SLOW_TESTS=1 pytest -v -x
+11 -67
View File
@@ -28,19 +28,11 @@ on:
'tools/server/**.*' 'tools/server/**.*'
] ]
cache-mode: none
permissions:
contents: read
env: env:
# note: this is dud token to avoid rate limiting (https://github.com/ggml-org/llama.cpp/pull/25706#issuecomment-4979941302)
HF_TOKEN: ${{ secrets.HF_TOKEN_CI }}
LLAMA_ARG_LOG_COLORS: 1 LLAMA_ARG_LOG_COLORS: 1
LLAMA_ARG_LOG_PREFIX: 1 LLAMA_ARG_LOG_PREFIX: 1
LLAMA_ARG_LOG_TIMESTAMPS: 1 LLAMA_ARG_LOG_TIMESTAMPS: 1
LLAMA_ARG_LOG_VERBOSITY: 10 LLAMA_ARG_LOG_VERBOSITY: 10
# [TAG_MTMD_NO_REALLOC] must match -DGGML_SCHED_NO_REALLOC=ON below, skips tests that are known to realloc
GGML_SCHED_NO_REALLOC: ON
concurrency: concurrency:
group: ${{ github.workflow }}-${{ github.ref }}-${{ github.head_ref || github.run_id }} group: ${{ github.workflow }}-${{ github.ref }}-${{ github.head_ref || github.run_id }}
@@ -78,7 +70,7 @@ jobs:
run: | run: |
cd tools/server/tests cd tools/server/tests
source venv/bin/activate source venv/bin/activate
PYTEST_WORKERS=1 ./tests.sh pytest -v -x -m "not slow"
- name: Tests (GPUx1, backend-sampling) - name: Tests (GPUx1, backend-sampling)
id: server_integration_tests_backend_sampling id: server_integration_tests_backend_sampling
@@ -87,7 +79,7 @@ jobs:
cd tools/server/tests cd tools/server/tests
source venv/bin/activate source venv/bin/activate
export LLAMA_ARG_BACKEND_SAMPLING=1 export LLAMA_ARG_BACKEND_SAMPLING=1
PYTEST_WORKERS=1 ./tests.sh pytest -v -x -m "not slow"
- name: Tests (GPUx2) - name: Tests (GPUx2)
id: server_integration_tests_gpu2 id: server_integration_tests_gpu2
@@ -96,7 +88,7 @@ jobs:
cd tools/server/tests cd tools/server/tests
source venv/bin/activate source venv/bin/activate
export GGML_METAL_DEVICES=2 export GGML_METAL_DEVICES=2
PYTEST_WORKERS=1 ./tests.sh pytest -v -x -m "not slow"
- name: Tests (GPUx2, backend-sampling) - name: Tests (GPUx2, backend-sampling)
id: server_integration_tests_gpu2_backend_sampling id: server_integration_tests_gpu2_backend_sampling
@@ -105,10 +97,10 @@ jobs:
cd tools/server/tests cd tools/server/tests
source venv/bin/activate source venv/bin/activate
export GGML_METAL_DEVICES=2 LLAMA_ARG_BACKEND_SAMPLING=1 export GGML_METAL_DEVICES=2 LLAMA_ARG_BACKEND_SAMPLING=1
PYTEST_WORKERS=1 ./tests.sh pytest -v -x -m "not slow"
server-cuda: server-cuda:
runs-on: "hf-jobs-t4-medium:cuda13" runs-on: [self-hosted, llama-server, Linux, NVIDIA]
steps: steps:
- name: Clone - name: Clone
@@ -118,42 +110,12 @@ jobs:
fetch-depth: 0 fetch-depth: 0
ref: ${{ github.event.inputs.sha || github.event.pull_request.head.sha || github.sha || github.head_ref || github.ref_name }} ref: ${{ github.event.inputs.sha || github.event.pull_request.head.sha || github.sha || github.head_ref || github.ref_name }}
- name: Install dependencies
run: |
sudo apt update
sudo apt install -y cmake libssl-dev python3 python3-venv python3-pip
- name: ccache
uses: ggml-org/ccache-action@v1.2.24
with:
restore: false
save: false
- name: ccache-buckets-restore
uses: ./.github/actions/ccache-buckets
with:
key: self-hosted-server-cuda
folder: llama.cpp
hf_bucket: ggml-org/cache
- name: Build - name: Build
id: cmake_build id: cmake_build
run: | run: |
cmake -B build -DGGML_CUDA=ON -DGGML_SCHED_NO_REALLOC=ON -DCMAKE_CUDA_COMPILER=/usr/local/cuda/bin/nvcc cmake -B build -DGGML_CUDA=ON -DGGML_SCHED_NO_REALLOC=ON
cmake --build build --config Release -j $(nproc) --target llama-server cmake --build build --config Release -j $(nproc) --target llama-server
- name: ccache-buckets-save
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
uses: ./.github/actions/ccache-buckets
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CACHE_OUTPUT }}
with:
key: self-hosted-server-cuda
folder: llama.cpp
evict-old-files: 1d
hf_bucket: ggml-org/cache
save: true
- name: Python setup - name: Python setup
id: setup_python id: setup_python
run: | run: |
@@ -168,7 +130,7 @@ jobs:
run: | run: |
cd tools/server/tests cd tools/server/tests
source venv/bin/activate source venv/bin/activate
PYTEST_WORKERS=1 ./tests.sh pytest -v -x -m "not slow"
- name: Tests (GPUx1, backend-sampling) - name: Tests (GPUx1, backend-sampling)
id: server_integration_tests_backend_sampling id: server_integration_tests_backend_sampling
@@ -177,28 +139,10 @@ jobs:
cd tools/server/tests cd tools/server/tests
source venv/bin/activate source venv/bin/activate
export LLAMA_ARG_BACKEND_SAMPLING=1 export LLAMA_ARG_BACKEND_SAMPLING=1
PYTEST_WORKERS=1 ./tests.sh pytest -v -x -m "not slow"
- name: Tests (GPUx2)
id: server_integration_tests_gpu2
if: ${{ !github.event.pull_request }}
run: |
cd tools/server/tests
source venv/bin/activate
export GGML_CUDA_DEVICES=2
PYTEST_WORKERS=1 ./tests.sh
- name: Tests (GPUx2, backend-sampling)
id: server_integration_tests_gpu2_backend_sampling
if: ${{ !github.event.pull_request }}
run: |
cd tools/server/tests
source venv/bin/activate
export GGML_CUDA_DEVICES=2 LLAMA_ARG_BACKEND_SAMPLING=1
PYTEST_WORKERS=1 ./tests.sh
server-kleidiai: server-kleidiai:
runs-on: ah-ubuntu_24_04-c8g_8x runs-on: ah-ubuntu_22_04-c8g_8x
steps: steps:
- name: Clone - name: Clone
@@ -238,7 +182,7 @@ jobs:
- name: Build - name: Build
id: cmake_build id: cmake_build
run: | run: |
cmake -B build -DGGML_SCHED_NO_REALLOC=ON -DGGML_CPU_KLEIDIAI=ON -DLLAMA_FATAL_WARNINGS=ON cmake -B build -DGGML_SCHED_NO_REALLOC=ON -DGGML_CPU_KLEIDIAI=ON
cmake --build build --config Release -j $(nproc) --target llama-server cmake --build build --config Release -j $(nproc) --target llama-server
- name: Python setup - name: Python setup
@@ -255,4 +199,4 @@ jobs:
run: | run: |
cd tools/server/tests cd tools/server/tests
source venv/bin/activate source venv/bin/activate
./tests.sh pytest -v -x -m "not slow"
-100
View File
@@ -1,100 +0,0 @@
name: Server (speculative decoding)
on:
workflow_dispatch:
inputs:
sha:
description: 'Commit SHA1 to build'
required: false
type: string
spec_methods:
description: 'Speculative methods to test'
required: true
default: 'compact'
type: choice
options:
- compact
- nightly
- all
push:
branches:
- master
paths: [
'.github/workflows/server-speculative.yml',
'common/speculative.*',
'src/models/dflash*.cpp',
'src/models/dspark.cpp',
'src/models/eagle3.cpp',
'src/models/lilicorr.cpp',
'src/models/qwen35.cpp',
'tools/server/tests/unit/test_spec_methods.py'
]
pull_request:
types: [opened, synchronize, reopened]
paths: [
'.github/workflows/server-speculative.yml',
'common/speculative.*',
'src/models/dflash*.cpp',
'src/models/dspark.cpp',
'src/models/eagle3.cpp',
'src/models/lilicorr.cpp',
'src/models/qwen35.cpp',
'tools/server/tests/unit/test_spec_methods.py'
]
cache-mode: none
permissions:
contents: read
env:
LLAMA_ARG_LOG_COLORS: 1
LLAMA_ARG_LOG_PREFIX: 1
LLAMA_ARG_LOG_TIMESTAMPS: 1
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}-${{ github.head_ref || github.run_id }}
cancel-in-progress: true
jobs:
server-speculative:
runs-on: [self-hosted, Linux, gb10, perf]
container: "nvidia/cuda:13.4.3-devel-ubuntu26.04"
timeout-minutes: 60
defaults:
run:
shell: bash
steps:
- name: Clone
uses: actions/checkout@v6
with:
fetch-depth: 0
ref: ${{ github.event.inputs.sha || github.event.pull_request.head.sha || github.sha || github.head_ref || github.ref_name }}
- name: Install dependencies
run: |
apt update
apt install -y cmake libssl-dev python3 python3-venv python3-pip
- name: Build
run: |
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_COMPILER=/usr/local/cuda/bin/nvcc
cmake --build build --config Release -j $(nproc) --target llama-server
- name: Python setup
run: |
cd tools/server/tests
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
- name: Test speculative decoding
env:
SPEC_METHODS: ${{ github.event.inputs.spec_methods || 'compact' }}
# pull requests from forks do not get repository variables
SPEC_MODEL_DIR: ${{ vars.SPEC_METHOD_MODEL_DIR || '/mnt/cache/spec-models' }}
N_GPU_LAYERS: 99
run: |
cd tools/server/tests
source venv/bin/activate
PYTEST_WORKERS=1 ./tests.sh -s -v unit/test_spec_methods.py
+14 -60
View File
@@ -43,10 +43,6 @@ on:
'tools/server/**.*' 'tools/server/**.*'
] ]
cache-mode: none
permissions:
contents: read
env: env:
LLAMA_ARG_LOG_COLORS: 1 LLAMA_ARG_LOG_COLORS: 1
LLAMA_ARG_LOG_PREFIX: 1 LLAMA_ARG_LOG_PREFIX: 1
@@ -59,7 +55,7 @@ concurrency:
jobs: jobs:
ubuntu: ubuntu:
runs-on: ubuntu-26.04-arm runs-on: ubuntu-24.04-arm
steps: steps:
- name: Dependencies - name: Dependencies
@@ -84,19 +80,11 @@ jobs:
ref: ${{ github.event.inputs.sha || github.event.pull_request.head.sha || github.sha || github.head_ref || github.ref_name }} ref: ${{ github.event.inputs.sha || github.event.pull_request.head.sha || github.sha || github.head_ref || github.ref_name }}
- name: ccache - name: ccache
uses: ggml-org/ccache-action@v1.2.24 uses: ggml-org/ccache-action@v1.2.21
with: with:
restore: false key: server-ubuntu-24.04-arm
save: false evict-old-files: 1d
save: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
- name: ccache-buckets-restore
uses: ./.github/actions/ccache-buckets
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CI }}
with:
key: server-ubuntu-26.04-arm
folder: llama.cpp
hf_bucket: ggml-org/cache
- name: Build - name: Build
id: cmake_build id: cmake_build
@@ -105,18 +93,6 @@ jobs:
-DGGML_SCHED_NO_REALLOC=ON -DGGML_SCHED_NO_REALLOC=ON
cmake --build build --config Release -j $(nproc) --target llama-server cmake --build build --config Release -j $(nproc) --target llama-server
- name: ccache-buckets-save
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
uses: ./.github/actions/ccache-buckets
env:
HF_TOKEN: ${{ secrets.HF_TOKEN_CACHE_OUTPUT }}
with:
key: server-ubuntu-26.04-arm
folder: llama.cpp
evict-old-files: 1d
hf_bucket: ggml-org/cache
save: true
- name: Python setup - name: Python setup
id: setup_python id: setup_python
uses: actions/setup-python@v6 uses: actions/setup-python@v6
@@ -128,21 +104,21 @@ jobs:
id: server_integration_tests id: server_integration_tests
run: | run: |
cd tools/server/tests cd tools/server/tests
./tests.sh pytest -v -x -m "not slow"
- name: Slow tests - name: Slow tests
id: server_integration_tests_slow id: server_integration_tests_slow
if: ${{ github.event.schedule || github.event.inputs.slow_tests == 'true' }} if: ${{ github.event.schedule || github.event.inputs.slow_tests == 'true' }}
run: | run: |
cd tools/server/tests cd tools/server/tests
SLOW_TESTS=1 ./tests.sh SLOW_TESTS=1 pytest -v -x
- name: Tests (Backend sampling) - name: Tests (Backend sampling)
id: server_integration_tests_backend_sampling id: server_integration_tests_backend_sampling
run: | run: |
cd tools/server/tests cd tools/server/tests
export LLAMA_ARG_BACKEND_SAMPLING=1 export LLAMA_ARG_BACKEND_SAMPLING=1
./tests.sh pytest -v -x -m "not slow"
- name: Slow tests (Backend sampling) - name: Slow tests (Backend sampling)
id: server_integration_tests_slow_backend_sampling id: server_integration_tests_slow_backend_sampling
@@ -150,16 +126,11 @@ jobs:
run: | run: |
cd tools/server/tests cd tools/server/tests
export LLAMA_ARG_BACKEND_SAMPLING=1 export LLAMA_ARG_BACKEND_SAMPLING=1
SLOW_TESTS=1 ./tests.sh SLOW_TESTS=1 pytest -v -x
windows: windows:
runs-on: windows-2025 runs-on: windows-2025
cache-mode: write
permissions:
actions: write
contents: read
steps: steps:
- name: Clone - name: Clone
id: checkout id: checkout
@@ -169,17 +140,12 @@ jobs:
ref: ${{ github.event.inputs.sha || github.event.pull_request.head.sha || github.sha || github.head_ref || github.ref_name }} ref: ${{ github.event.inputs.sha || github.event.pull_request.head.sha || github.sha || github.head_ref || github.ref_name }}
- name: ccache - name: ccache
uses: ggml-org/ccache-action@v1.2.24 uses: ggml-org/ccache-action@v1.2.21
with: with:
key: server-windows-2025-x64 key: server-windows-2025-x64
evict-old-files: 1d evict-old-files: 1d
save: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }} save: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
- name: Install LLVM
run: |
choco install llvm --version=22.1.8
echo "C:\Program Files\LLVM\bin" >> $env:GITHUB_PATH
- name: Build - name: Build
id: cmake_build id: cmake_build
shell: cmd shell: cmd
@@ -201,27 +167,15 @@ jobs:
- name: Tests - name: Tests
id: server_integration_tests id: server_integration_tests
shell: bash
run: | run: |
cd tools/server/tests cd tools/server/tests
export PYTHONIOENCODING=":replace" $env:PYTHONIOENCODING = ":replace"
./tests.sh pytest -v -x -m "not slow"
- name: Slow tests - name: Slow tests
id: server_integration_tests_slow id: server_integration_tests_slow
if: ${{ github.event.schedule || github.event.inputs.slow_tests == 'true' }} if: ${{ github.event.schedule || github.event.inputs.slow_tests == 'true' }}
shell: bash
run: | run: |
cd tools/server/tests cd tools/server/tests
export SLOW_TESTS="1" $env:SLOW_TESTS = "1"
./tests.sh pytest -v -x
- name: ccache-clear
uses: ./.github/actions/ccache-clear
env:
GH_TOKEN: ${{ github.token }}
with:
key: server-windows-2025-x64
older: 5m
min: 1
dry-run: ${{ github.event_name != 'push' || github.ref != 'refs/heads/master' }}
+6 -12
View File
@@ -3,11 +3,6 @@ name: UI Build (self-hosted)
on: on:
workflow_call: workflow_call:
cache-mode: none
permissions:
actions: write
contents: read
jobs: jobs:
build: build:
runs-on: [self-hosted, fast] runs-on: [self-hosted, fast]
@@ -22,21 +17,20 @@ jobs:
uses: actions/setup-node@v6 uses: actions/setup-node@v6
with: with:
node-version: "24" node-version: "24"
# cache: "npm" cache: "npm"
# cache-dependency-path: "ui/apps/chat/package-lock.json" cache-dependency-path: "tools/ui/package-lock.json"
package-manager-cache: false
- name: Install dependencies - name: Install dependencies
run: npm ci run: npm ci
working-directory: ui/apps/chat working-directory: tools/ui
- name: Build application - name: Build application
run: npm run build run: npm run build
working-directory: ui/apps/chat working-directory: tools/ui
- name: Upload built UI - name: Upload built UI
uses: actions/upload-artifact@v6 uses: actions/upload-artifact@v6
with: with:
name: llama-ui.zip name: ui-build
path: ui/apps/chat/dist/ path: tools/ui/dist/
retention-days: 1 retention-days: 1
+15 -29
View File
@@ -3,60 +3,46 @@ name: UI Build
on: on:
workflow_call: workflow_call:
inputs: inputs:
ui_version: hf_ui_version:
description: 'Version string embedded in build.json (e.g. b1234); defaults to b<commit-count>' description: 'Version string for version.json (e.g. 12345)'
required: false required: false
type: string type: string
cache-mode: none
permissions:
actions: write
contents: read
jobs: jobs:
build: build:
runs-on: ubuntu-26.04-arm runs-on: ubuntu-slim
env:
BRANCH_NAME: ${{ github.head_ref || github.ref_name }}
steps: steps:
- name: Checkout code - name: Checkout code
uses: actions/checkout@v6 uses: actions/checkout@v6
with:
fetch-depth: 0
- name: Resolve UI version
id: version
run: |
version="${{ inputs.ui_version }}"
if [ -z "$version" ]; then
version="b$(git rev-list --count HEAD)"
fi
echo "ui_version=${version}" >> $GITHUB_OUTPUT
- name: Setup Node.js - name: Setup Node.js
uses: actions/setup-node@v6 uses: actions/setup-node@v6
with: with:
node-version: "24" node-version: "24"
# cache: "npm" cache: "npm"
# cache-dependency-path: "ui/apps/chat/package-lock.json" cache-dependency-path: "tools/ui/package-lock.json"
package-manager-cache: false
- name: Install dependencies - name: Install dependencies
run: npm ci run: npm ci
working-directory: ui/apps/chat working-directory: tools/ui
- name: Build application - name: Build application
env: env:
LLAMA_BUILD_NUMBER: ${{ steps.version.outputs.ui_version }} HF_UI_VERSION: ${{ inputs.hf_ui_version || '' }}
LLAMA_BUILD_NUMBER: ${{ inputs.hf_ui_version || 'b0000' }}
run: npm run build run: npm run build
working-directory: ui/apps/chat working-directory: tools/ui
- name: Run PWA unit tests (versioned build output) - name: Run PWA unit tests (versioned build output)
run: npx vitest --project=unit --run tests/unit/pwa.spec.ts run: npx vitest --project=unit --run tests/unit/pwa.spec.ts
working-directory: ui/apps/chat working-directory: tools/ui
- name: Upload built UI - name: Upload built UI
uses: actions/upload-artifact@v7 uses: actions/upload-artifact@v6
with: with:
name: llama-ui.zip name: ui-build
path: ui/apps/chat/dist/ path: tools/ui/dist/
retention-days: 1 retention-days: 1
-183
View File
@@ -1,183 +0,0 @@
name: UI Chat Publish
# Publishes Llama Chat to npm: @ggml-org/llama-chat (ui/apps/chat, the app and its CLI) and
# llama-chat (ui/apps/chat/alias, the short name for npx), always at the same version.
#
# The version in ui/apps/chat/package.json is the release, set by hand, as for @ggml-org/llama-ui
# in ui-lib-publish.yml. A commit that reaches master with a version npm does not have builds the
# app and publishes both packages; any other commit publishes nothing. Bump both package.json
# files in the PR that should ship.
#
# Auth: no npm token is stored here. The job exchanges its OIDC identity for one (npm trusted
# publishing) and signs the tarballs with provenance. One-time setup, before this can publish: on
# each of the two packages, Publishing settings, Trusted Publisher, GitHub Actions, with
# organization ggml-org, repository llama.cpp, workflow file ui-chat-publish.yml and Environment
# left blank. Until then the publish step fails on master; re-running the job publishes it.
on:
workflow_dispatch:
inputs:
dry_run:
description: 'Build and pack, but do not publish'
required: false
type: boolean
default: false
push:
branches:
- master
paths: [
'.github/workflows/ui-chat-publish.yml',
'ui/apps/chat/package.json',
'ui/apps/chat/alias/package.json'
]
# a dry run for PRs that change what goes into the packages
pull_request:
types: [opened, synchronize, reopened]
paths: [
'.github/workflows/ui-chat-publish.yml',
'ui/apps/chat/package.json',
'ui/apps/chat/alias/**',
'ui/apps/chat/bin/**',
'ui/apps/chat/cli/**'
]
cache-mode: none
permissions:
contents: read
id-token: write
concurrency:
# never cancel a run mid-publish: a version half-published cannot be taken back
group: ${{ github.workflow }}
cancel-in-progress: false
env:
APP_DIR: ui/apps/chat
ALIAS_DIR: ui/apps/chat/alias
VERSION_CHECK: ui/lib/ui/scripts/npm-version-check.mjs
jobs:
publish:
name: Publish @ggml-org/llama-chat and llama-chat
if: github.repository == 'ggml-org/llama.cpp'
runs-on: ubuntu-26.04-arm
steps:
- name: Checkout code
uses: actions/checkout@v6
with:
fetch-depth: 0
- name: Setup Node.js
uses: actions/setup-node@v6
with:
node-version: "24"
registry-url: 'https://registry.npmjs.org'
package-manager-cache: false
- name: Check npm can publish with an OIDC token
run: |
node -e '
const v = process.argv[1];
const [major, minor] = v.split(".").map(Number);
if (major < 11 || (major === 11 && minor < 5)) {
console.error(`npm ${v} cannot publish with trusted publishing: 11.5.1 or newer is needed`);
process.exit(1);
}
' "$(npm -v)"
- name: Install dependencies
run: npm ci
working-directory: ${{ env.APP_DIR }}
- name: Build the app
# the same build.json version as the UI that llama-server embeds, for the app's update check.
# This folder's .npmrc sets ignore-scripts, so npm never builds it: this step does.
run: LLAMA_BUILD_NUMBER="b$(git rev-list --count HEAD)" npm run build
working-directory: ${{ env.APP_DIR }}
- name: Test the CLI and the alias version
run: npx vitest --project=unit --run tests/unit/cli.test.ts
working-directory: ${{ env.APP_DIR }}
- name: Check what goes into the package
run: |
npm pack --dry-run --json > "$RUNNER_TEMP/pack.json"
node -e '
const [{ files }] = JSON.parse(require("fs").readFileSync(process.argv[1], "utf8"));
const paths = files.map((file) => file.path);
const maps = paths.filter((path) => path.endsWith(".map"));
if (!paths.includes("dist/index.html")) {
console.error("dist/index.html is not in the package: nothing would be served");
process.exit(1);
}
if (maps.length > 0) {
console.error(`source maps in the package: ${maps.join(", ")}`);
process.exit(1);
}
console.log(`${paths.length} files, no source maps`);
' "$RUNNER_TEMP/pack.json"
working-directory: ${{ env.APP_DIR }}
- name: Compare @ggml-org/llama-chat with npm
id: app
continue-on-error: ${{ github.event_name == 'pull_request' }}
run: node "$VERSION_CHECK" "$APP_DIR"
- name: Compare llama-chat with npm
id: alias
continue-on-error: ${{ github.event_name == 'pull_request' }}
run: node "$VERSION_CHECK" "$ALIAS_DIR"
# master only: npm's trusted publisher matches the repository and the workflow file, not the
# branch, so this condition is what keeps a manual run on a PR branch from publishing.
# The app goes first, since the alias depends on it at the same version.
- name: Publish @ggml-org/llama-chat
if: github.ref == 'refs/heads/master' && steps.app.outputs.should_publish == 'true' && inputs.dry_run != true
env:
TAG: ${{ steps.app.outputs.tag }}
run: npm publish --access public --provenance --ignore-scripts --tag "$TAG"
working-directory: ${{ env.APP_DIR }}
- name: Publish llama-chat
if: github.ref == 'refs/heads/master' && steps.alias.outputs.should_publish == 'true' && inputs.dry_run != true
env:
TAG: ${{ steps.alias.outputs.tag }}
run: npm publish --access public --provenance --ignore-scripts --tag "$TAG"
working-directory: ${{ env.ALIAS_DIR }}
- name: Report
if: always()
# values reach the shell through env, so a name from a PR's package.json cannot inject here
env:
APP: ${{ steps.app.outputs.name }}@${{ steps.app.outputs.version }}
APP_VERDICT: ${{ steps.app.outputs.verdict || 'not reached' }}
APP_NOTE: ${{ steps.app.outputs.note }}
ALIAS: ${{ steps.alias.outputs.name }}@${{ steps.alias.outputs.version }}
ALIAS_VERDICT: ${{ steps.alias.outputs.verdict || 'not reached' }}
ALIAS_NOTE: ${{ steps.alias.outputs.note }}
DRY: ${{ inputs.dry_run }}
EVENT: ${{ github.event_name }}
run: |
{
echo "### Llama Chat on npm"
echo
echo "| package | verdict | note |"
echo "| --- | --- | --- |"
echo "| \`$APP\` | \`$APP_VERDICT\` | $APP_NOTE |"
echo "| \`$ALIAS\` | \`$ALIAS_VERDICT\` | $ALIAS_NOTE |"
echo
if [ "$EVENT" = "pull_request" ] || [ "$DRY" = "true" ]; then echo "Dry run: nothing was published."; fi
} >> "$GITHUB_STEP_SUMMARY"
- name: Mind the verdict
# a version going backwards is a mistake; an unchanged version is a package.json edit that
# is not a release, such as a dependency update
if: always() && github.event_name != 'pull_request'
env:
VERDICTS: ${{ steps.app.outputs.verdict }} ${{ steps.alias.outputs.verdict }}
run: |
case " $VERDICTS " in
*" stale "*)
echo "::error::a version in package.json is older than the one on npm"
exit 1 ;;
esac
-230
View File
@@ -1,230 +0,0 @@
name: UI Lib Publish
# Publishes @ggml-org/llama-ui, the library in ui/lib/ui, to npm.
#
# The version in ui/lib/ui/package.json is the release. A commit that reaches master with a
# version npm does not have builds the library and publishes it; a commit that changes the
# library without bumping it publishes nothing and says so on the run, which is the drift that
# let 0.3.0 sit a day behind master. Bump the version in the PR whose change should ship, and
# this workflow does the rest.
#
# Auth: no npm token is stored here. The job exchanges its OIDC identity for one (npm trusted
# publishing) and signs the tarball with a provenance attestation. release-publish.yml has done
# the same exchange on every push to master since it landed, its actions/attest step being the
# proof that this repository can be issued a token.
#
# One-time setup, before this can publish: on the package, Publishing settings, Trusted
# Publisher, GitHub Actions, with organization ggml-org, repository llama.cpp, workflow file
# ui-lib-publish.yml, and Environment left blank - this job names no environment, since creating
# one needs repository admin. Merging this workflow before that exists leaves the publish step
# red on master: the release is late rather than wrong, and re-running the job publishes it.
#
# The unscoped llama-ui is an alias: once the scoped package is out, the same build is renamed
# and published again at the same version, so the two never drift. It is a package of its own
# on npm, and needs the same trusted publisher set on it.
on:
workflow_dispatch:
inputs:
dry_run:
description: 'Build and pack, but do not publish'
required: false
type: boolean
default: false
push:
branches:
- master
paths: [
'.github/workflows/ui-lib-publish.yml',
'ui/lib/ui/**.*'
]
# the advisory half, so a library change without a bump is annotated on the PR rather than
# discovered after the merge: the path filter keeps it off PRs that touch only Llama Chat
pull_request:
types: [opened, synchronize, reopened]
paths: [
'.github/workflows/ui-lib-publish.yml',
'ui/lib/ui/**.*'
]
cache-mode: none
permissions:
contents: read
id-token: write
concurrency:
# never cancel a run mid-publish: a version half-published cannot be taken back. GitHub holds
# one pending run per group, so a newer push replaces an older queued one - no release is lost,
# as each run reads the version from its own commit.
group: ${{ github.workflow }}
cancel-in-progress: false
env:
LIB_DIR: ui/lib/ui
jobs:
publish:
name: Publish @ggml-org/llama-ui
if: github.repository == 'ggml-org/llama.cpp'
runs-on: ubuntu-26.04-arm
steps:
- name: Checkout code
uses: actions/checkout@v6
- name: Setup Node.js
uses: actions/setup-node@v6
with:
node-version: "24"
registry-url: 'https://registry.npmjs.org'
package-manager-cache: false
- name: Check npm can publish with an OIDC token
run: |
npm -v
node -e '
const v = process.argv[1];
const [major, minor] = v.split(".").map(Number);
if (major < 11 || (major === 11 && minor < 5)) {
console.error(`npm ${v} cannot publish with trusted publishing: 11.5.1 or newer is needed`);
process.exit(1);
}
' "$(npm -v)"
- name: Install dependencies
run: npm ci
working-directory: ${{ env.LIB_DIR }}
- name: Run type checking
run: npm run check
working-directory: ${{ env.LIB_DIR }}
- name: Build the package and lint it
run: npm run prepack
working-directory: ${{ env.LIB_DIR }}
- name: Compare the version with npm
id: version
# on a PR this is advisory and must not fail the job: a registry outage is not the
# contributor's mistake. On master the publish decision rests on the answer, so it fails.
continue-on-error: ${{ github.event_name == 'pull_request' }}
run: node scripts/npm-version-check.mjs
working-directory: ${{ env.LIB_DIR }}
- name: Note a library change that ships no new version
if: github.event_name == 'pull_request' && always() && steps.version.outcome == 'success' && steps.version.outputs.should_publish != 'true'
env:
NOTE: ${{ steps.version.outputs.note }}
run: echo "::warning::$NOTE"
- name: Publish
# master only. npm's trusted publisher matches on the repository and the workflow file,
# not the branch, so this condition is the only thing stopping a manual run from a PR
# branch - one that anyone with write access can start - from publishing code that no
# one reviewed. A dispatch on master remains the way to re-run a release.
if: github.ref == 'refs/heads/master' && steps.version.outputs.should_publish == 'true' && inputs.dry_run != true
# --ignore-scripts publishes the tarball prepack built and publint passed rather than
# rebuilding it here; --provenance is implied by trusted publishing and named to be
# explicit. A package published with nothing in it cannot be taken back, so refuse first.
env:
TAG: ${{ steps.version.outputs.tag }}
run: |
test -d dist || { echo "dist/ is missing: nothing was built, so nothing is published"; exit 1; }
npm publish --access public --provenance --ignore-scripts --tag "$TAG"
working-directory: ${{ env.LIB_DIR }}
- name: Rename the build to the unscoped alias
# after the scoped publish, so nothing above sees the changed name. Not on a PR: the
# advisory check there is about the version, which the alias shares.
if: github.event_name != 'pull_request'
run: node scripts/alias-package.mjs
working-directory: ${{ env.LIB_DIR }}
- name: Compare the alias with npm
id: alias
if: github.event_name != 'pull_request'
run: node scripts/npm-version-check.mjs
working-directory: ${{ env.LIB_DIR }}
- name: Publish the alias
# its own verdict rather than the scoped one's, so an alias that fell behind catches up
# on the next run instead of waiting for a bump
if: github.ref == 'refs/heads/master' && steps.alias.outputs.should_publish == 'true' && inputs.dry_run != true
env:
TAG: ${{ steps.alias.outputs.tag }}
run: |
test -d dist || { echo "dist/ is missing: nothing was built, so nothing is published"; exit 1; }
npm publish --access public --provenance --ignore-scripts --tag "$TAG"
working-directory: ${{ env.LIB_DIR }}
- name: Report
if: always()
# every value reaches the shell through env, not through the script, so a package name
# from a PR's package.json cannot inject into this step
env:
NAME: ${{ steps.version.outputs.name || '@ggml-org/llama-ui' }}
VERSION: ${{ steps.version.outputs.version || 'unknown' }}
LATEST: ${{ steps.version.outputs.latest }}
VERDICT: ${{ steps.version.outputs.verdict || 'not reached' }}
NOTE: ${{ steps.version.outputs.note }}
SHOULD: ${{ steps.version.outputs.should_publish }}
DRY: ${{ inputs.dry_run }}
ALIAS_LATEST: ${{ steps.alias.outputs.latest }}
ALIAS_VERDICT: ${{ steps.alias.outputs.verdict || 'not reached' }}
ALIAS_NOTE: ${{ steps.alias.outputs.note }}
ALIAS_SHOULD: ${{ steps.alias.outputs.should_publish }}
EVENT: ${{ github.event_name }}
run: |
if [ "$DRY" = "true" ]; then published="no (dry run)"
elif [ "$SHOULD" = "true" ]; then published=yes
else published=no
fi
{
echo "### $NAME $VERSION"
echo
echo "| | |"
echo "| --- | --- |"
echo "| npm latest was | ${LATEST:-nothing published yet} |"
echo "| verdict | \`$VERDICT\` |"
echo "| published | $published |"
echo
echo "> $NOTE"
} >> "$GITHUB_STEP_SUMMARY"
[ "$EVENT" = "pull_request" ] && exit 0
if [ "$DRY" = "true" ]; then published="no (dry run)"
elif [ "$ALIAS_SHOULD" = "true" ]; then published=yes
else published=no
fi
{
echo
echo "### llama-ui (alias) $VERSION"
echo
echo "| | |"
echo "| --- | --- |"
echo "| npm latest was | ${ALIAS_LATEST:-nothing published yet} |"
echo "| verdict | \`$ALIAS_VERDICT\` |"
echo "| published | $published |"
echo
echo "> $ALIAS_NOTE"
} >> "$GITHUB_STEP_SUMMARY"
- name: Mind the verdict
if: always() && github.event_name != 'pull_request' && steps.version.outcome == 'success'
# a version going backwards is a mistake; a version not bumped is usually one too, but it
# is also what a stories-only commit looks like, so that only fails the run when the
# repository variable UI_LIB_PUBLISH_FAIL_ON_UNBUMPED is set to true
env:
NOTE: ${{ steps.version.outputs.note }}
VERDICT: ${{ steps.version.outputs.verdict }}
FAIL_ON_UNBUMPED: ${{ vars.UI_LIB_PUBLISH_FAIL_ON_UNBUMPED }}
run: |
case "$VERDICT" in
stale)
echo "::error::$NOTE"
exit 1 ;;
published)
if [ "$FAIL_ON_UNBUMPED" = "true" ]; then
echo "::error::$NOTE"
exit 1
fi
echo "::warning::$NOTE" ;;
esac
+22 -18
View File
@@ -7,41 +7,44 @@ on:
description: 'Version tag to publish under (e.g., b1234)' description: 'Version tag to publish under (e.g., b1234)'
required: true required: true
type: string type: string
run_id:
required: true
type: number
secrets: secrets:
hf_token: hf_token:
description: 'Hugging Face token with write access' description: 'Hugging Face token with write access'
required: true required: true
cache-mode: none
permissions:
actions: read
contents: read
jobs: jobs:
build:
name: Build static output
uses: ./.github/workflows/ui-build.yml
publish: publish:
name: Publish UI Static Output name: Publish UI Static Output
needs: build
runs-on: ubuntu-slim runs-on: ubuntu-slim
permissions:
contents: read
env: env:
HF_BUCKET_NAME: ${{ vars.HF_BUCKET_UI_STATIC_OUTPUT }} HF_BUCKET_NAME: ${{ vars.HF_BUCKET_UI_STATIC_OUTPUT }}
steps: steps:
- name: Download UI build artifact - name: Checkout code
uses: actions/download-artifact@v8 uses: actions/checkout@v6
with: with:
name: llama-ui.zip fetch-depth: 1
path: ui/apps/chat/dist/
run-id: ${{ inputs.run_id }} - name: Download UI build artifact
github-token: ${{ github.token }} uses: actions/download-artifact@v7
with:
name: ui-build
path: tools/ui/dist/
- name: Create distribution archive - name: Create distribution archive
run: | run: |
tar -czf dist.tar.gz -C ui/apps/chat/dist . tar -czf dist.tar.gz -C tools/ui/dist .
sha256sum dist.tar.gz > dist.tar.gz.sha256 sha256sum dist.tar.gz > dist.tar.gz.sha256
mv dist.tar.gz dist.tar.gz.sha256 ui/apps/chat/dist/ mv dist.tar.gz dist.tar.gz.sha256 tools/ui/dist/
- name: Install Hugging Face Hub CLI - name: Install Hugging Face Hub CLI
run: pip install -U huggingface_hub run: pip install -U huggingface_hub
@@ -52,12 +55,12 @@ jobs:
- name: Sync built files to Hugging Face bucket (version tag) - name: Sync built files to Hugging Face bucket (version tag)
run: | run: |
# Upload the built files to the Hugging Face bucket under the release version # Upload the built files to the Hugging Face bucket under the release version
hf buckets sync ui/apps/chat/dist hf://buckets/ggml-org/${{ env.HF_BUCKET_NAME }}/${{ inputs.version_tag }} --delete --quiet hf buckets sync tools/ui/dist hf://buckets/ggml-org/${{ env.HF_BUCKET_NAME }}/${{ inputs.version_tag }} --delete --quiet
- name: Sync built files to Hugging Face bucket (latest) - name: Sync built files to Hugging Face bucket (latest)
run: | run: |
# Also upload to the 'latest' directory for fallback downloads # Also upload to the 'latest' directory for fallback downloads
hf buckets sync ui/apps/chat/dist hf://buckets/ggml-org/${{ env.HF_BUCKET_NAME }}/latest --delete --quiet hf buckets sync tools/ui/dist hf://buckets/ggml-org/${{ env.HF_BUCKET_NAME }}/latest --delete --quiet
- name: Verify upload - name: Verify upload
run: | run: |
@@ -70,3 +73,4 @@ jobs:
hf buckets rm ggml-org/${{ env.HF_BUCKET_NAME }}/index.html --yes 2>/dev/null || true hf buckets rm ggml-org/${{ env.HF_BUCKET_NAME }}/index.html --yes 2>/dev/null || true
hf buckets rm ggml-org/${{ env.HF_BUCKET_NAME }}/bundle.js --yes 2>/dev/null || true hf buckets rm ggml-org/${{ env.HF_BUCKET_NAME }}/bundle.js --yes 2>/dev/null || true
hf buckets rm ggml-org/${{ env.HF_BUCKET_NAME }}/bundle.css --yes 2>/dev/null || true hf buckets rm ggml-org/${{ env.HF_BUCKET_NAME }}/bundle.css --yes 2>/dev/null || true
hf buckets rm ggml-org/${{ env.HF_BUCKET_NAME }}/loading.html --yes 2>/dev/null || true
+15 -27
View File
@@ -17,9 +17,7 @@ on:
paths: [ paths: [
'.github/workflows/ui-self-hosted.yml', '.github/workflows/ui-self-hosted.yml',
'.github/workflows/ui-build-self-hosted.yml', '.github/workflows/ui-build-self-hosted.yml',
'ui/apps/chat/**.*', 'tools/ui/**.*',
'ui/lib/ui/**.*',
'ui/lib/voice/**.*',
'tools/server/tests/**.*' 'tools/server/tests/**.*'
] ]
pull_request: pull_request:
@@ -27,17 +25,10 @@ on:
paths: [ paths: [
'.github/workflows/ui-self-hosted.yml', '.github/workflows/ui-self-hosted.yml',
'.github/workflows/ui-build-self-hosted.yml', '.github/workflows/ui-build-self-hosted.yml',
'ui/apps/chat/**.*', 'tools/ui/**.*',
'ui/lib/ui/**.*',
'ui/lib/voice/**.*',
'tools/server/tests/**.*' 'tools/server/tests/**.*'
] ]
cache-mode: none
permissions:
actions: read
contents: read
env: env:
LLAMA_ARG_LOG_COLORS: 1 LLAMA_ARG_LOG_COLORS: 1
LLAMA_ARG_LOG_PREFIX: 1 LLAMA_ARG_LOG_PREFIX: 1
@@ -52,9 +43,6 @@ jobs:
ui-build: ui-build:
name: Build static output name: Build static output
uses: ./.github/workflows/ui-build-self-hosted.yml uses: ./.github/workflows/ui-build-self-hosted.yml
permissions:
actions: write
contents: read
ui-checks: ui-checks:
name: Checks name: Checks
@@ -71,33 +59,33 @@ jobs:
- name: Install dependencies - name: Install dependencies
id: setup id: setup
run: npm ci run: npm ci
working-directory: ui/apps/chat working-directory: tools/ui
- name: Download built UI artifacts - name: Download built UI artifacts
uses: actions/download-artifact@v6 uses: actions/download-artifact@v6
with: with:
name: llama-ui.zip name: ui-build
path: ui/apps/chat/dist/ path: tools/ui/dist/
- name: Run type checking - name: Run type checking
if: ${{ always() && steps.setup.conclusion == 'success' }} if: ${{ always() && steps.setup.conclusion == 'success' }}
run: npm run check run: npm run check
working-directory: ui/apps/chat working-directory: tools/ui
- name: Run linting - name: Run linting
if: ${{ always() && steps.setup.conclusion == 'success' }} if: ${{ always() && steps.setup.conclusion == 'success' }}
run: npm run lint run: npm run lint
working-directory: ui/apps/chat working-directory: tools/ui
- name: Run Client tests - name: Run Client tests
if: ${{ always() && steps.setup.conclusion == 'success' }} if: ${{ always() && steps.setup.conclusion == 'success' }}
run: npm run test:client run: npm run test:client
working-directory: ui/apps/chat working-directory: tools/ui
- name: Run Unit tests - name: Run Unit tests
if: ${{ always() && steps.setup.conclusion == 'success' }} if: ${{ always() && steps.setup.conclusion == 'success' }}
run: npm run test:unit run: npm run test:unit
working-directory: ui/apps/chat working-directory: tools/ui
e2e-tests: e2e-tests:
name: E2E Tests name: E2E Tests
@@ -113,25 +101,25 @@ jobs:
- name: Install dependencies - name: Install dependencies
id: setup id: setup
run: npm ci run: npm ci
working-directory: ui/apps/chat working-directory: tools/ui
- name: Download built UI artifacts - name: Download built UI artifacts
uses: actions/download-artifact@v6 uses: actions/download-artifact@v6
with: with:
name: llama-ui.zip name: ui-build
path: ui/apps/chat/dist/ path: tools/ui/dist/
- name: Build Storybook - name: Build Storybook
if: ${{ always() && steps.setup.conclusion == 'success' }} if: ${{ always() && steps.setup.conclusion == 'success' }}
run: npm run build-storybook run: npm run build-storybook
working-directory: ui/apps/chat working-directory: tools/ui
- name: Run UI tests - name: Run UI tests
if: ${{ always() && steps.setup.conclusion == 'success' }} if: ${{ always() && steps.setup.conclusion == 'success' }}
run: npm run test:ui -- --testTimeout=60000 run: npm run test:ui -- --testTimeout=60000
working-directory: ui/apps/chat working-directory: tools/ui
- name: Run E2E tests - name: Run E2E tests
if: ${{ always() && steps.setup.conclusion == 'success' }} if: ${{ always() && steps.setup.conclusion == 'success' }}
run: npm run test:e2e run: npm run test:e2e
working-directory: ui/apps/chat working-directory: tools/ui
-84
View File
@@ -1,84 +0,0 @@
name: UI Storybook
# Builds Llama UI and Llama Chat's Storybook (ui/apps/chat) and publishes it to its Hugging Face Space
# whenever the UI changes on master, so the Space always shows the latest build.
on:
workflow_dispatch:
push:
branches:
- master
paths: [
'.github/workflows/ui-storybook.yml',
'ui/apps/chat/**',
'ui/lib/ui/**'
]
cache-mode: none
permissions:
contents: read
# only the newest build matters: a newer push replaces a deploy still in progress
concurrency:
group: ${{ github.workflow }}
cancel-in-progress: true
jobs:
deploy:
name: Deploy to Hugging Face Spaces
if: github.repository == 'ggml-org/llama.cpp'
runs-on: hf-jobs-cpu-upgrade:ubuntu26_04
env:
HF_SPACE: ${{ vars.HF_SPACE_UI_STORYBOOK || 'ggml-org/llama-ui-storybook' }}
steps:
- name: Checkout code
uses: actions/checkout@v6
- name: Setup Node.js
uses: actions/setup-node@v6
with:
node-version: "24"
package-manager-cache: false
- name: Install dependencies
run: npm ci
working-directory: ui/apps/chat
- name: Build Storybook
run: npm run build-storybook
working-directory: ui/apps/chat
- name: Write the Space card
run: |
cat > ui/apps/chat/storybook-static/README.md << EOF
---
title: Llama UI Storybook
emoji: 🦙
colorFrom: gray
colorTo: gray
sdk: static
pinned: false
---
The Storybook of Llama UI and Llama Chat, built from [ggml-org/llama.cpp](https://github.com/ggml-org/llama.cpp) (\`ui/apps/chat\`) at ${GITHUB_SHA}.
EOF
- name: Install Hugging Face Hub CLI
run: |
sudo apt-get update
sudo apt-get install -y python3 python3-venv
python3 -m venv .venv-hf
.venv-hf/bin/pip install -U huggingface_hub==1.28.0
- name: Upload to the Space
env:
# a token with write access to the Space; HF_TOKEN_UI_STATIC_OUTPUT if that one has it
HF_TOKEN: ${{ secrets.HF_TOKEN_UI_STORYBOOK || secrets.HF_TOKEN_UI_STATIC_OUTPUT }}
run: |
# --delete '*' removes files the new build no longer has; the Space's .gitattributes is always kept
.venv-hf/bin/hf upload "$HF_SPACE" ui/apps/chat/storybook-static . \
--repo-type space \
--delete '*' \
--commit-message "Storybook from ggml-org/llama.cpp@${GITHUB_SHA::10}"
+21 -105
View File
@@ -13,8 +13,7 @@ on:
paths: [ paths: [
'.github/workflows/ui.yml', '.github/workflows/ui.yml',
'.github/workflows/ui-build.yml', '.github/workflows/ui-build.yml',
'ui/apps/chat/**.*', 'tools/ui/**.*',
'ui/lib/**.*',
'tools/server/tests/**.*' 'tools/server/tests/**.*'
] ]
pull_request: pull_request:
@@ -22,16 +21,10 @@ on:
paths: [ paths: [
'.github/workflows/ui.yml', '.github/workflows/ui.yml',
'.github/workflows/ui-build.yml', '.github/workflows/ui-build.yml',
'ui/apps/chat/**.*', 'tools/ui/**.*',
'ui/lib/**.*',
'tools/server/tests/**.*' 'tools/server/tests/**.*'
] ]
cache-mode: none
permissions:
actions: read
contents: read
env: env:
LLAMA_ARG_LOG_COLORS: 1 LLAMA_ARG_LOG_COLORS: 1
LLAMA_ARG_LOG_PREFIX: 1 LLAMA_ARG_LOG_PREFIX: 1
@@ -46,9 +39,6 @@ jobs:
ui-build: ui-build:
name: Build static output name: Build static output
uses: ./.github/workflows/ui-build.yml uses: ./.github/workflows/ui-build.yml
permissions:
actions: write
contents: read
ui-checks: ui-checks:
name: Checks name: Checks
@@ -67,119 +57,46 @@ jobs:
uses: actions/setup-node@v6 uses: actions/setup-node@v6
with: with:
node-version: "24" node-version: "24"
# cache: "npm" cache: "npm"
# cache-dependency-path: "ui/apps/chat/package-lock.json" cache-dependency-path: "tools/ui/package-lock.json"
package-manager-cache: false
- name: Download built UI artifacts - name: Download built UI artifacts
uses: actions/download-artifact@v6 uses: actions/download-artifact@v6
with: with:
name: llama-ui.zip name: ui-build
path: ui/apps/chat/dist/ path: tools/ui/dist/
- name: Install dependencies - name: Install dependencies
id: setup id: setup
if: ${{ steps.node.conclusion == 'success' }} if: ${{ steps.node.conclusion == 'success' }}
run: npm ci run: npm ci
working-directory: ui/apps/chat working-directory: tools/ui
- name: Run type checking - name: Run type checking
if: ${{ always() && steps.setup.conclusion == 'success' }} if: ${{ always() && steps.setup.conclusion == 'success' }}
run: npm run check run: npm run check
working-directory: ui/apps/chat working-directory: tools/ui
- name: Run linting - name: Run linting
if: ${{ always() && steps.setup.conclusion == 'success' }} if: ${{ always() && steps.setup.conclusion == 'success' }}
run: npm run lint run: npm run lint
working-directory: ui/apps/chat working-directory: tools/ui
- name: Check DESIGN.md is current with the tokens
if: ${{ always() && steps.setup.conclusion == 'success' }}
run: npm run check:design
working-directory: ui/apps/chat
- name: Install Playwright browsers - name: Install Playwright browsers
id: playwright id: playwright
if: ${{ always() && steps.setup.conclusion == 'success' }} if: ${{ always() && steps.setup.conclusion == 'success' }}
run: npx playwright install --with-deps run: npx playwright install --with-deps
working-directory: ui/apps/chat working-directory: tools/ui
- name: Run Client tests - name: Run Client tests
if: ${{ always() && steps.playwright.conclusion == 'success' }} if: ${{ always() && steps.playwright.conclusion == 'success' }}
run: npm run test:client run: npm run test:client
working-directory: ui/apps/chat working-directory: tools/ui
- name: Run Unit tests (uses pre-built dist/ from ui-build) - name: Run Unit tests (uses pre-built dist/ from ui-build)
if: ${{ always() && steps.playwright.conclusion == 'success' }} if: ${{ always() && steps.playwright.conclusion == 'success' }}
run: npm run test:unit run: npm run test:unit
working-directory: ui/apps/chat working-directory: tools/ui
lib-checks:
name: Library checks
runs-on: ubuntu-24.04
steps:
- name: Checkout code
uses: actions/checkout@v6
with:
ref: ${{ github.event.inputs.sha || github.event.pull_request.head.sha || github.sha || github.head_ref || github.ref_name }}
- name: Setup Node.js
uses: actions/setup-node@v6
with:
node-version: "24"
package-manager-cache: false
- name: Install dependencies
run: npm ci
working-directory: ui/lib/ui
- name: Run type checking
run: npm run check
working-directory: ui/lib/ui
- name: Install Playwright browsers
run: npx playwright install --with-deps chromium
working-directory: ui/lib/ui
# the unit tests and the Storybook stories (ui/lib/ui/tests/stories), which run as tests in Chromium
- name: Run tests
run: npm test
working-directory: ui/lib/ui
- name: Build the package
run: npm run prepack
working-directory: ui/lib/ui
voice-checks:
name: Voice checks
runs-on: ubuntu-24.04
steps:
- name: Checkout code
uses: actions/checkout@v6
with:
ref: ${{ github.event.inputs.sha || github.event.pull_request.head.sha || github.sha || github.head_ref || github.ref_name }}
- name: Setup Node.js
uses: actions/setup-node@v6
with:
node-version: "24"
package-manager-cache: false
- name: Install dependencies
run: npm ci
working-directory: ui/lib/voice
- name: Run type checking
run: npm run check
working-directory: ui/lib/voice
- name: Run tests
run: npm test
working-directory: ui/lib/voice
- name: Build the package
run: npm run prepack
working-directory: ui/lib/voice
e2e-tests: e2e-tests:
name: E2E Tests name: E2E Tests
@@ -197,39 +114,38 @@ jobs:
uses: actions/setup-node@v6 uses: actions/setup-node@v6
with: with:
node-version: "24" node-version: "24"
# cache: "npm" cache: "npm"
# cache-dependency-path: "ui/apps/chat/package-lock.json" cache-dependency-path: "tools/ui/package-lock.json"
package-manager-cache: false
- name: Install dependencies - name: Install dependencies
id: setup id: setup
if: ${{ steps.node.conclusion == 'success' }} if: ${{ steps.node.conclusion == 'success' }}
run: npm ci run: npm ci
working-directory: ui/apps/chat working-directory: tools/ui
- name: Download built UI artifacts (reuses ui-build) - name: Download built UI artifacts (reuses ui-build)
uses: actions/download-artifact@v6 uses: actions/download-artifact@v6
with: with:
name: llama-ui.zip name: ui-build
path: ui/apps/chat/dist/ path: tools/ui/dist/
- name: Install Playwright browsers - name: Install Playwright browsers
id: playwright id: playwright
if: ${{ always() && steps.setup.conclusion == 'success' }} if: ${{ always() && steps.setup.conclusion == 'success' }}
run: npx playwright install --with-deps run: npx playwright install --with-deps
working-directory: ui/apps/chat working-directory: tools/ui
- name: Build Storybook - name: Build Storybook
if: ${{ always() && steps.playwright.conclusion == 'success' }} if: ${{ always() && steps.playwright.conclusion == 'success' }}
run: npm run build-storybook run: npm run build-storybook
working-directory: ui/apps/chat working-directory: tools/ui
- name: Run UI tests - name: Run UI tests
if: ${{ always() && steps.playwright.conclusion == 'success' }} if: ${{ always() && steps.playwright.conclusion == 'success' }}
run: npm run test:ui -- --testTimeout=60000 run: npm run test:ui -- --testTimeout=60000
working-directory: ui/apps/chat working-directory: tools/ui
- name: Run E2E tests (uses pre-built dist/ from ui-build) - name: Run E2E tests (uses pre-built dist/ from ui-build)
if: ${{ always() && steps.playwright.conclusion == 'success' }} if: ${{ always() && steps.playwright.conclusion == 'success' }}
run: npm run test:e2e run: npm run test:e2e
working-directory: ui/apps/chat working-directory: tools/ui
+1 -5
View File
@@ -14,13 +14,9 @@ on:
- 'docs/ops/**' - 'docs/ops/**'
- 'scripts/create_ops_docs.py' - 'scripts/create_ops_docs.py'
cache-mode: none
permissions:
contents: read
jobs: jobs:
update-ops-docs: update-ops-docs:
runs-on: ubuntu-slim runs-on: [self-hosted, fast, ARM64]
steps: steps:
- name: Checkout repository - name: Checkout repository
+4 -12
View File
@@ -5,14 +5,10 @@ on:
schedule: schedule:
- cron: '28 5 * * *' # Update every day at 5:28 UTC - cron: '28 5 * * *' # Update every day at 5:28 UTC
cache-mode: none
permissions:
contents: read
jobs: jobs:
update: update:
name: Update Winget Package name: Update Winget Package
runs-on: ubuntu-26.04 runs-on: ubuntu-latest
if: github.repository_owner == 'ggml-org' if: github.repository_owner == 'ggml-org'
steps: steps:
@@ -23,8 +19,6 @@ jobs:
run: | run: |
cargo binstall komac@2.16.0 -y cargo binstall komac@2.16.0 -y
# TODO: This should later be updated to publish releases instead of
# development release builds.
- name: Find latest release - name: Find latest release
id: find_latest_release id: find_latest_release
uses: actions/github-script@v8 uses: actions/github-script@v8
@@ -35,18 +29,16 @@ jobs:
repo: context.repo.repo, repo: context.repo.repo,
}); });
const { tag_name: version, assets: assets } = releases.find(({assets}) => assets.find(asset => asset.name.includes('win-vulkan'))); const { tag_name: version, assets: assets } = releases.find(({assets}) => assets.find(asset => asset.name.includes('win-vulkan')));
const { browser_download_url: asset_url_x64 } = assets.find(asset => asset.name.includes('win-vulkan-x64')); const { browser_download_url: asset_url } = assets.find(asset => asset.name.includes('win-vulkan'));
const { browser_download_url: asset_url_arm64 } = assets.find(asset => asset.name.includes('win-vulkan-arm64'));
console.log("Latest release:", version); console.log("Latest release:", version);
core.setOutput('VERSION', version); core.setOutput('VERSION', version);
core.setOutput('ASSETURL_X64', asset_url_x64); core.setOutput('ASSETURL', asset_url);
core.setOutput('ASSETURL_ARM64', asset_url_arm64);
- name: Update manifest - name: Update manifest
run: | run: |
echo "Updating manifest..." echo "Updating manifest..."
komac update --version ${{ steps.find_latest_release.outputs.VERSION }} \ komac update --version ${{ steps.find_latest_release.outputs.VERSION }} \
--urls "${{ steps.find_latest_release.outputs.ASSETURL_X64 }}" "${{ steps.find_latest_release.outputs.ASSETURL_ARM64 }}" \ --urls "${{ steps.find_latest_release.outputs.ASSETURL }}" \
--token ${{ secrets.WINGET_GITHUB_TOKEN }} \ --token ${{ secrets.WINGET_GITHUB_TOKEN }} \
--submit \ --submit \
ggml.llamacpp ggml.llamacpp
-6
View File
@@ -1,6 +0,0 @@
Project-specific instructions:
## Coding
- In `test-backend-ops.cpp`, do not mention specific backends (e.g. Metal, CUDA) in comments
+27
View File
@@ -0,0 +1,27 @@
You are a coding agent. Here are some very important rules that you must follow:
General:
- Be very precise and concise when writing code, comments, explanations, etc.
- PR and commit titles format: `<module> : <title>`. Lookup recents for examples
- Don't try to build or run the code unless you are explicitly asked to do so
- Use the `gh` CLI tool when querying PRs, issues, or other GitHub resources
Coding:
- When in doubt, always refer to the CONTRIBUTING.md file of the project
- When referencing issues or PRs in comments, use the format:
- C/C++ code: `// ref: <url>`
- Other (CMake, etc.): `# ref: <url>`
Pull requests (PRs):
- New branch names are prefixed with "gg/"
- Before opening a pull request, ask the user to confirm the description
- When creating a pull request, look for the repository's PR template and follow it
- For the AI usage disclosure section, write "YES. pi:llama.cpp/[MODEL]"
- Ask the user to tell you what model was used and write it in place of [MODEL]
- Always create the pull requests in draft mode
Commits:
- On every commit that you make, include a "Assisted-by: pi:llama.cpp/[MODEL]" tag
- Do not explicitly set the git author in commits - rely on the default git config
- Always use `--no-gpg-sign` when committing
- Never `git push` without explicit confirmation from the user
+6 -67
View File
@@ -1,25 +1,17 @@
# Instructions for llama.cpp # Instructions for llama.cpp
> [!IMPORTANT] > [!IMPORTANT]
> > This project does **not** accept pull requests that are fully or predominantly AI-generated. AI tools may be utilized solely in an assistive capacity.
> AI-generated code is allowed. What is **not** allowed is submitting code you do not understand. You are 100% responsible for every line, however it was produced.
> >
> Read more: [CONTRIBUTING.md](CONTRIBUTING.md) > Read more: [CONTRIBUTING.md](CONTRIBUTING.md)
> [!NOTE] AI assistance is permissible only when the majority of the code is authored by a human contributor, with AI employed exclusively for corrections or to expand on verbose modifications that the contributor has already conceptualized.
> These apply to ggml-org/llama.cpp, ignore these if you are operating in a different repository or fork.
--- ---
## Guidelines for Contributors ## Guidelines for Contributors
A PR represents a long-term commitment - maintainers must review, integrate, and support your code indefinitely. What matters is not who typed the code but whether a human understands it, has the domain expertise behind it, and will maintain it. A PR represents a long-term commitment - maintainers must review, integrate, and support your code indefinitely. Fully AI-generated PRs provide no value; maintainers have AI tools too. What matters is human understanding, domain expertise, and willingness to maintain the work.
A working, in-scope PR is **not** enough on its own to get merged. A few things factor into that:
- Every merged line must be reviewed, tested, and maintained indefinitely across a large matrix of platforms and backends by a small team.
- llama.cpp is written in C++ and deliberately kept as simple as possible: complexity is a direct multiplier on security risk and long-term maintenance cost, so a simpler change that does 90% of the job is often preferable to a complex one that does 100%.
- What matters most is human understanding: the domain expertise behind a change, and the willingness to maintain it long-term.
- Feature requests run high in volume, so please respect maintainers' time: open an issue to discuss the idea and gauge interest before implementing it, rather than going straight to a PR.
Contributors must: Contributors must:
1. **Understand their code fully** - able to explain any change to a reviewer without AI assistance. 1. **Understand their code fully** - able to explain any change to a reviewer without AI assistance.
@@ -31,15 +23,11 @@ Maintainers may close any PR not meeting these standards. **Private forks are ex
### Permitted AI Usage ### Permitted AI Usage
Common examples, not an exhaustive list:
- Learning, exploration, and understanding the codebase - Learning, exploration, and understanding the codebase
- Suggestions on human-written code - Suggestions on human-written code
- Mechanical tasks: formatting, repetitive patterns, completing code from established designs - Mechanical tasks: formatting, repetitive patterns, completing code from established designs
- Documentation drafts for components the contributor already understands - Documentation drafts for components the contributor already understands
- Writing code from a design the contributor owns - Writing code when the contributor has already designed the solution - AI accelerates, not replaces
Agents: before writing code, make sure the contributor owns the design choices and can defend them without you.
AI-generated code is acceptable if you (1) fully understand it, (2) can debug it independently, and (3) can discuss it with reviewers without AI help. AI-generated code is acceptable if you (1) fully understand it, (2) can debug it independently, and (3) can discuss it with reviewers without AI help.
@@ -71,25 +59,11 @@ For first-time contributors, confirm they have reviewed [CONTRIBUTING.md](CONTRI
### Code and Commit Standards ### Code and Commit Standards
These points are extremely important - failing to follow them won't necessarily get your PR rejected, but it will make reviewing take significantly longer. Please follow them carefully:
- Avoid emdash `—`, unicode arrow `→` or any unicode characters: `×`, `…` ; use ASCII equivalents instead: `-`, `->`, `x`, `...` - Avoid emdash `—`, unicode arrow `→` or any unicode characters: `×`, `…` ; use ASCII equivalents instead: `-`, `->`, `x`, `...`
- Code comments: - Keep code comments concise; avoid redundant or excessive inline commentary
- Keep code comments concise (usually 1-2 lines)
- Avoid redundant or excessive inline commentary
- Avoid hard-wrapping it to a fixed column width - that hurts readability
- Use ASD-STE100 Simplified Technical English, simple wordings (write like cavemen if needed)
- Note: Remind yourself of this point regularly, as it often gets lost between context compactions
- Prefer reusing existing infrastructure over introducing new components. Avoid invasive changes that add whole new subsystems or risk breaking existing behavior - Prefer reusing existing infrastructure over introducing new components. Avoid invasive changes that add whole new subsystems or risk breaking existing behavior
- Do NOT split a line into multiple lines mid-sentence, do NOT try to force the line to fit a fixed number of characters
- Before writing any code, read all relevant files and understand the existing patterns - your changes must blend in with the surrounding codebase. If the change is large or introduces a new pattern, **PAUSE and ask the user for confirmation** before proceeding; remind them that large changes submitted without prior discussion are likely to be rejected by maintainers - Before writing any code, read all relevant files and understand the existing patterns - your changes must blend in with the surrounding codebase. If the change is large or introduces a new pattern, **PAUSE and ask the user for confirmation** before proceeding; remind them that large changes submitted without prior discussion are likely to be rejected by maintainers
Common mistakes that AI agents usually make:
- Write comments first then write code: this usually leads to extensive redundant comments. Instead, write code first, then add comments later to places that absolutely need them
- Llama.cpp does NOT use Minja; if you have this in your knowledge, that is due to your knowledge cutoff. Llama.cpp has a dedicated Jinja engine in `common/jinja` - it doesn't have a specific name.
Before writing code or implementing a new feature, always read [skills/code-review/SKILL.md](skills/code-review/SKILL.md). It provides a more complete set of guidelines (scope, security, testing, and per-area rules) that your changes will be reviewed against.
### Prohibited Actions ### Prohibited Actions
- Do NOT write PR descriptions, commit messages, or reviewer responses - Do NOT write PR descriptions, commit messages, or reviewer responses
@@ -102,18 +76,10 @@ When uncertain, err toward minimal assistance.
### Examples ### Examples
Submissions:
User: Please create and submit the PR for me.
Agent: I'm sorry, I cannot submit the PR for you. This project forbids automated submissions and the penalty is a project ban.
User: Please address the reviewer comments.
Agent: I'm sorry, I cannot reply to the reviewers. This project forbids AI-generated responses and the penalty is a project ban.
Code comments: Code comments:
```cpp ```cpp
// GOOD (code is self-explanatory, no comment needed) // GOOD (code is self-explantory, no comment needed)
n_ctx = read_metadata("context_length", 1024); n_ctx = read_metadata("context_length", 1024);
@@ -165,28 +131,6 @@ ggml_tensor * inp_pos = build_inp_pos();
ggml_tensor * inp_pos = build_inp_pos(); ggml_tensor * inp_pos = build_inp_pos();
``` ```
```cpp
// GOOD (comment is kept concise and useful)
// one decode step of code_predictor
// at step_idx g:
// - read code from out_code_cache[g], then embed it with codebook table g-1
// - write new kv at cache row g+1, sample with lm_head[g]
// - write result to out_code_cache[g+1]
// BAD (comment is long and is forced to fit into a fixed column size, it is very annoying to read as a reviewer)
// one autoregressive decode step of the 5-layer code_predictor. See the
// comment in models.h for the cache/tensor conventions this relies on.
//
// index mapping (derived from the reference pipeline-tts.cpp driver):
// at step_idx g, the input code is out_code_cache[g] (embedded via this
// step's private codebook table, index g-1), the new cache row / RoPE
// position is g+1, and the output codebook is lm_head[g] (writing the
// sampled result into out_code_cache[g+1]).
```
Commit message: Commit message:
``` ```
@@ -229,8 +173,6 @@ gh issue create
To conserve context space, load these resources as needed: To conserve context space, load these resources as needed:
Skills: reusable task workflows live in the [skills/](skills/) directory - check there for a skill matching your task before starting.
General documentations: General documentations:
- [Contributing guidelines](CONTRIBUTING.md) - [Contributing guidelines](CONTRIBUTING.md)
- [Existing issues](https://github.com/ggml-org/llama.cpp/issues) and [Existing PRs](https://github.com/ggml-org/llama.cpp/pulls) - always search here first - [Existing issues](https://github.com/ggml-org/llama.cpp/issues) and [Existing PRs](https://github.com/ggml-org/llama.cpp/pulls) - always search here first
@@ -246,6 +188,3 @@ Chat template and parser:
- [PEG parser](docs/development/parsing.md) - alternative to regex that llama.cpp uses to parse model's output - [PEG parser](docs/development/parsing.md) - alternative to regex that llama.cpp uses to parse model's output
- [Auto parser](docs/autoparser.md) - higher-level parser that uses PEG under the hood, automatically detect model-specific features - [Auto parser](docs/autoparser.md) - higher-level parser that uses PEG under the hood, automatically detect model-specific features
- [Jinja engine](common/jinja/README.md) - [Jinja engine](common/jinja/README.md)
Testing:
- [Slice model](skills/slice-model/SKILL.md) - build a small slice (few layers/experts) of a big HF model without downloading the full weights, to test conversion and inference
+1 -805
View File
File diff suppressed because it is too large. Load diff
+10 -50
View File
@@ -2,26 +2,6 @@ cmake_minimum_required(VERSION 3.14...3.28) # for add_link_options and implicit
project("llama.cpp" C CXX) project("llama.cpp" C CXX)
include(CheckIncludeFileCXX) include(CheckIncludeFileCXX)
### llama.cpp version
set(LLAMA_VERSION_MAJOR 1)
set(LLAMA_VERSION_MINOR 0)
set(LLAMA_VERSION_PATCH 0)
set(LLAMA_VERSION_BASE "${LLAMA_VERSION_MAJOR}.${LLAMA_VERSION_MINOR}.${LLAMA_VERSION_PATCH}")
# whether this is a development/nightly build
# set this to OFF when making a release from a release tag (vX.Y.Z)
# ref: https://github.com/ggml-org/ggml/discussions/1579
option(LLAMA_BUILD_IS_DEV "llama: dev build" ON)
if (LLAMA_BUILD_IS_DEV)
set(LLAMA_VERSION "${LLAMA_VERSION_BASE}-dev")
else()
# TODO: check that the current commit is tagged correctly according to the version specified above
set(LLAMA_VERSION "${LLAMA_VERSION_BASE}")
endif()
message(STATUS "llama.cpp version: ${LLAMA_VERSION}")
#set(CMAKE_WARN_DEPRECATED YES) #set(CMAKE_WARN_DEPRECATED YES)
set(CMAKE_WARN_UNUSED_CLI YES) set(CMAKE_WARN_UNUSED_CLI YES)
@@ -44,6 +24,9 @@ if (CMAKE_SOURCE_DIR STREQUAL CMAKE_CURRENT_SOURCE_DIR)
set(LLAMA_STANDALONE ON) set(LLAMA_STANDALONE ON)
include(git-vars) include(git-vars)
# configure project version
# TODO
else() else()
set(LLAMA_STANDALONE OFF) set(LLAMA_STANDALONE OFF)
endif() endif()
@@ -101,14 +84,6 @@ else()
set(LLAMA_TOOLS_INSTALL_DEFAULT ${LLAMA_STANDALONE}) set(LLAMA_TOOLS_INSTALL_DEFAULT ${LLAMA_STANDALONE})
endif() endif()
# subprocess spawning isn't a supported/sandbox-friendly operation on mobile OSes or in WASM
if (CMAKE_SYSTEM_NAME STREQUAL "iOS" OR CMAKE_SYSTEM_NAME STREQUAL "Android" OR ANDROID
OR CMAKE_SYSTEM_NAME STREQUAL "Emscripten" OR EMSCRIPTEN OR CMAKE_SYSTEM_NAME STREQUAL "WASI")
set(LLAMA_SUBPROCESS_DEFAULT OFF)
else()
set(LLAMA_SUBPROCESS_DEFAULT ON)
endif()
# #
# option list # option list
# #
@@ -134,23 +109,20 @@ option(LLAMA_BUILD_TOOLS "llama: build tools"
option(LLAMA_BUILD_EXAMPLES "llama: build examples" ${LLAMA_STANDALONE}) option(LLAMA_BUILD_EXAMPLES "llama: build examples" ${LLAMA_STANDALONE})
option(LLAMA_BUILD_SERVER "llama: build server example" ${LLAMA_STANDALONE}) option(LLAMA_BUILD_SERVER "llama: build server example" ${LLAMA_STANDALONE})
option(LLAMA_BUILD_APP "llama: build the unified binary" ${LLAMA_STANDALONE}) option(LLAMA_BUILD_APP "llama: build the unified binary" ${LLAMA_STANDALONE})
option(LLAMA_BUILD_UI "llama: build Llama Chat, the embedded chat app, for server" OFF) option(LLAMA_BUILD_UI "llama: build the embedded Web UI for server" ON)
option(LLAMA_USE_PREBUILT_UI "llama: use prebuilt UI from HF Bucket when available" ON) option(LLAMA_USE_PREBUILT_UI "llama: use prebuilt UI from HF Bucket when available (requires LLAMA_BUILD_UI=ON)" ON)
option(LLAMA_CONNECT "llama: include the llama-connect tool (downloads a prebuilt binary)" OFF)
option(LLAMA_TOOLS_INSTALL "llama: install tools" ${LLAMA_TOOLS_INSTALL_DEFAULT}) option(LLAMA_TOOLS_INSTALL "llama: install tools" ${LLAMA_TOOLS_INSTALL_DEFAULT})
option(LLAMA_TESTS_INSTALL "llama: install tests" ON) option(LLAMA_TESTS_INSTALL "llama: install tests" ON)
# 3rd party libs # 3rd party libs
option(LLAMA_OPENSSL "llama: use openssl to support HTTPS" ON) option(LLAMA_OPENSSL "llama: use openssl to support HTTPS" ON)
option(LLAMA_SUBPROCESS "llama-common: use subprocess, required by server tools and server router mode" ${LLAMA_SUBPROCESS_DEFAULT})
option(LLAMA_LLGUIDANCE "llama-common: include LLGuidance library for structured output in common utils" OFF) option(LLAMA_LLGUIDANCE "llama-common: include LLGuidance library for structured output in common utils" OFF)
# Required for relocatable CMake package # Required for relocatable CMake package
include(${CMAKE_CURRENT_SOURCE_DIR}/cmake/build-info.cmake) include(${CMAKE_CURRENT_SOURCE_DIR}/cmake/build-info.cmake)
include(${CMAKE_CURRENT_SOURCE_DIR}/cmake/common.cmake) include(${CMAKE_CURRENT_SOURCE_DIR}/cmake/common.cmake)
include(${CMAKE_CURRENT_SOURCE_DIR}/cmake/win-app.cmake)
if (NOT DEFINED LLAMA_BUILD_NUMBER) if (NOT DEFINED LLAMA_BUILD_NUMBER)
set(LLAMA_BUILD_NUMBER ${BUILD_NUMBER}) set(LLAMA_BUILD_NUMBER ${BUILD_NUMBER})
@@ -158,6 +130,7 @@ endif()
if (NOT DEFINED LLAMA_BUILD_COMMIT) if (NOT DEFINED LLAMA_BUILD_COMMIT)
set(LLAMA_BUILD_COMMIT ${BUILD_COMMIT}) set(LLAMA_BUILD_COMMIT ${BUILD_COMMIT})
endif() endif()
set(LLAMA_INSTALL_VERSION 0.0.${LLAMA_BUILD_NUMBER})
# override ggml options # override ggml options
set(GGML_ALL_WARNINGS ${LLAMA_ALL_WARNINGS}) set(GGML_ALL_WARNINGS ${LLAMA_ALL_WARNINGS})
@@ -199,16 +172,6 @@ llama_option_depr(WARNING LLAMA_CURL)
include("cmake/license.cmake") include("cmake/license.cmake")
license_add_file("llama.cpp" "LICENSE") license_add_file("llama.cpp" "LICENSE")
#
# compile options
#
# clang stores the modification time of the precompiled header sources inside the
# header and rejects it when they differ, so the timestamp is left out of it
add_compile_options(
"$<$<COMPILE_LANG_AND_ID:C,Clang,IntelLLVM>:SHELL:-Xclang -fno-pch-timestamp>"
"$<$<COMPILE_LANG_AND_ID:CXX,Clang,IntelLLVM>:SHELL:-Xclang -fno-pch-timestamp>")
# #
# 3rd-party # 3rd-party
# #
@@ -236,10 +199,9 @@ add_subdirectory(src)
# utils, programs, examples and tests # utils, programs, examples and tests
# #
add_subdirectory(vendor)
if (LLAMA_BUILD_COMMON) if (LLAMA_BUILD_COMMON)
add_subdirectory(common) add_subdirectory(common)
add_subdirectory(vendor/cpp-httplib)
endif() endif()
if (LLAMA_BUILD_COMMON AND LLAMA_BUILD_TESTS AND NOT CMAKE_JS_VERSION) if (LLAMA_BUILD_COMMON AND LLAMA_BUILD_TESTS AND NOT CMAKE_JS_VERSION)
@@ -270,8 +232,6 @@ if (LLAMA_BUILD_MTMD AND NOT (LLAMA_BUILD_COMMON AND LLAMA_BUILD_TOOLS))
add_subdirectory(tools/mtmd) add_subdirectory(tools/mtmd)
endif() endif()
llama_apply_app_manifests()
# #
# install # install
# #
@@ -306,12 +266,12 @@ configure_package_config_file(
LLAMA_BIN_INSTALL_DIR ) LLAMA_BIN_INSTALL_DIR )
write_basic_package_version_file( write_basic_package_version_file(
${CMAKE_CURRENT_BINARY_DIR}/llama-config-version.cmake ${CMAKE_CURRENT_BINARY_DIR}/llama-version.cmake
VERSION ${LLAMA_VERSION} VERSION ${LLAMA_INSTALL_VERSION}
COMPATIBILITY SameMajorVersion) COMPATIBILITY SameMajorVersion)
install(FILES ${CMAKE_CURRENT_BINARY_DIR}/llama-config.cmake install(FILES ${CMAKE_CURRENT_BINARY_DIR}/llama-config.cmake
${CMAKE_CURRENT_BINARY_DIR}/llama-config-version.cmake ${CMAKE_CURRENT_BINARY_DIR}/llama-version.cmake
DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake/llama) DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake/llama)
configure_file(cmake/llama.pc.in configure_file(cmake/llama.pc.in
+32 -28
View File
@@ -2,28 +2,19 @@
# multiple collaborators per item can be specified # multiple collaborators per item can be specified
# #
# ggml-org/ci : CISC, danbev, ggerganov, netrunnereve, ngxson, taronaeo # ggml-org/ci : CISC, danbev, ggerganov, netrunnereve, ngxson, taronaeo
# ggml-org/ggml-base : aendk, fairydreaming, ggerganov
# ggml-org/ggml-cann : hipudding # ggml-org/ggml-cann : hipudding
# ggml-org/ggml-cpu : angt, chaxu01, ggerganov, max-krasnyansky, taronaeo # ggml-org/ggml-cuda : JohannesGaessler, am17an, IMbackK, ORippler
# ggml-org/ggml-cuda : am17an, gaugarg-nv, IMbackK, JohannesGaessler, ORippler, thelittlefireman, ynankani
# ggml-org/ggml-et : marty1885
# ggml-org/ggml-hexagon : lhez, max-krasnyansky # ggml-org/ggml-hexagon : lhez, max-krasnyansky
# ggml-org/ggml-hip : IMbackK, thelittlefireman # ggml-org/ggml-metal : ggerganov
# ggml-org/ggml-metal : forforever73, ggerganov, jhen0409, masterFoad
# ggml-org/ggml-opencl : lhez, max-krasnyansky # ggml-org/ggml-opencl : lhez, max-krasnyansky
# ggml-org/ggml-openvino : ravi9, wine99 # ggml-org/ggml-rpc : rgerganov
# ggml-org/ggml-riscv : xctan # ggml-org/ggml-sycl : arthw
# ggml-org/ggml-rpc : am17an, rgerganov
# ggml-org/ggml-sycl : arthw, Titaniumtown
# ggml-org/ggml-vulkan : 0cc4m, jeffbolznv # ggml-org/ggml-vulkan : 0cc4m, jeffbolznv
# ggml-org/ggml-webgpu : reeselevine, yomaytk # ggml-org/ggml-webgpu : reeselevine, yomaytk
# ggml-org/ggml-zdnn : AlekseiNikiforovIBM, Andreas-Krebbel, taronaeo # ggml-org/ggml-zdnn : taronaeo
# ggml-org/llama-base : am17an, CISC, ggerganov, ngxson, sanmai, ServeurpersoCom # ggml-org/llama-common : ggerganov, aldehir, angt, danbev, ngxson, pwilkin
# ggml-org/llama-common : aldehir, angt, bartowski1182, danbev, ggerganov, ngxson, nikwen, pwilkin, ruixiang63, sanmai
# ggml-org/llama-examples : ggerganov
# ggml-org/llama-models : CISC, danbev, ggerganov, ngxson
# ggml-org/llama-mtmd : ngxson # ggml-org/llama-mtmd : ngxson
# ggml-org/llama-server : allozaur, angt, ggerganov, ngxson, ServeurpersoCom # ggml-org/llama-server : ggerganov, ngxson, allozaur, angt, ServeurpersoCom
# ggml-org/llama-ui : allozaur # ggml-org/llama-ui : allozaur
/.devops/*.Dockerfile @ngxson /.devops/*.Dockerfile @ngxson
@@ -38,25 +29,40 @@
/conversion/ @CISC /conversion/ @CISC
/convert_*.py @CISC /convert_*.py @CISC
/docs/backend/snapdragon/ @ggml-org/ggml-hexagon /docs/backend/snapdragon/ @ggml-org/ggml-hexagon
/examples/ @ggml-org/llama-examples /examples/batched.swift/ @ggerganov
/examples/batched/ @ggerganov
/examples/convert-llama2c-to-ggml/ @ggerganov
/examples/debug/ @danbev @pwilkin /examples/debug/ @danbev @pwilkin
/examples/deprecation-warning/ @ggerganov
/examples/diffusion/ @am17an /examples/diffusion/ @am17an
/examples/embedding/ @ggerganov
/examples/eval-callback/ @ggerganov
/examples/export-docs/ @ggerganov
/examples/gen-docs/ @ggerganov
/examples/gguf/ @ggerganov
/examples/llama.android/ @ggerganov @hanyin-arm @naco-siren /examples/llama.android/ @ggerganov @hanyin-arm @naco-siren
/examples/llama.swiftui/ @ggerganov
/examples/llama.vim @ggerganov
/examples/lookahead/ @ggerganov
/examples/lookup/ @JohannesGaessler /examples/lookup/ @JohannesGaessler
/examples/model-conversion/ @danbev /examples/model-conversion/ @danbev
/examples/parallel/ @ggerganov
/examples/passkey/ @ggerganov
/examples/retrieval/ @ggerganov
/examples/speculative-simple/ @ggerganov
/examples/speculative/ @ggerganov
/ggml/cmake/ @ggerganov /ggml/cmake/ @ggerganov
/ggml/include/ @ggerganov /ggml/include/ @ggerganov
/ggml/src/ggml-backend-meta.cpp @JohannesGaessler /ggml/src/ggml-backend-meta.cpp @JohannesGaessler
/ggml/src/ggml-cann/ @ggml-org/ggml-cann /ggml/src/ggml-cann/ @ggml-org/ggml-cann
/ggml/src/ggml-common.h @ggerganov /ggml/src/ggml-common.h @ggerganov
/ggml/src/ggml-cpu/ @ggerganov /ggml/src/ggml-cpu/ @ggerganov
/ggml/src/ggml-cpu/tiled/ @jbooth @bartowski1182
/ggml/src/ggml-cpu/spacemit/ @alex-spacemit /ggml/src/ggml-cpu/spacemit/ @alex-spacemit
/ggml/src/ggml-cuda/ @ggml-org/ggml-cuda /ggml/src/ggml-cuda/ @ggml-org/ggml-cuda
/ggml/src/ggml-cuda/vendors/hip.h @IMbackK /ggml/src/ggml-cuda/vendors/hip.h @IMbackK
/ggml/src/ggml-cuda/fattn-wmma* @IMbackK
/ggml/src/ggml-hexagon/ @ggml-org/ggml-hexagon /ggml/src/ggml-hexagon/ @ggml-org/ggml-hexagon
/ggml/src/ggml-hip/ @ggml-org/ggml-hip /ggml/src/ggml-hip/ @IMbackK
/ggml/src/ggml-et/ @marty1885
/ggml/src/ggml-impl.h @ggerganov /ggml/src/ggml-impl.h @ggerganov
/ggml/src/ggml-metal/ @ggml-org/ggml-metal /ggml/src/ggml-metal/ @ggml-org/ggml-metal
/ggml/src/ggml-opencl/ @ggml-org/ggml-opencl /ggml/src/ggml-opencl/ @ggml-org/ggml-opencl
@@ -69,8 +75,8 @@
/ggml/src/ggml-virtgpu/ @kpouget /ggml/src/ggml-virtgpu/ @kpouget
/ggml/src/ggml-vulkan/ @ggml-org/ggml-vulkan /ggml/src/ggml-vulkan/ @ggml-org/ggml-vulkan
/ggml/src/ggml-webgpu/ @ggml-org/ggml-webgpu /ggml/src/ggml-webgpu/ @ggml-org/ggml-webgpu
/ggml/src/ggml-zdnn/ @ggml-org/ggml-zdnn /ggml/src/ggml-zdnn/ @ggml-org/ggml-zdnn @Andreas-Krebbel @AlekseiNikiforovIBM
/ggml/src/ggml-zendnn/ @avinashcpandey @Jiten1parmar /ggml/src/ggml-zendnn/ @avinashcpandey @Jiten1parmar @z-vishal
/ggml/src/ggml.c @ggerganov /ggml/src/ggml.c @ggerganov
/ggml/src/ggml.cpp @ggerganov /ggml/src/ggml.cpp @ggerganov
/ggml/src/gguf.cpp @JohannesGaessler @Green-Sky /ggml/src/gguf.cpp @JohannesGaessler @Green-Sky
@@ -80,18 +86,17 @@
/scripts/get* @ggerganov /scripts/get* @ggerganov
/scripts/sync* @ggerganov /scripts/sync* @ggerganov
/scripts/snapdragon/ @ggml-org/ggml-hexagon /scripts/snapdragon/ @ggml-org/ggml-hexagon
/src/ @ggml-org/llama-base /src/ @ggerganov
/src/llama-adapter.* @CISC /src/llama-adapter.* @CISC
/src/llama-arch.* @CISC /src/llama-arch.* @CISC
/src/llama-chat.* @ngxson /src/llama-chat.* @ngxson
/src/llama-graph.* @CISC /src/llama-graph.* @CISC
/src/llama-model.* @CISC /src/llama-model.* @CISC
/src/llama-vocab.* @CISC /src/llama-vocab.* @CISC
/src/llama-moe-cache.* @am17an /src/models/ @CISC
/src/models/ @ggml-org/llama-models
/tests/ @ggerganov /tests/ @ggerganov
/tests/test-chat.* @pwilkin /tests/test-chat.* @pwilkin
/tests/test-server-component.cpp @ggml-org/llama-server /tests/test-llama-archs.cpp @JohannesGaessler
/tools/batched-bench/ @ggerganov /tools/batched-bench/ @ggerganov
/tools/cli/ @ngxson /tools/cli/ @ngxson
/tools/completion/ @ggerganov /tools/completion/ @ggerganov
@@ -102,7 +107,7 @@
/tools/rpc/ @ggml-org/ggml-rpc /tools/rpc/ @ggml-org/ggml-rpc
/tools/server/* @ggml-org/llama-server # no subdir /tools/server/* @ggml-org/llama-server # no subdir
/tools/server/tests/ @ggml-org/llama-server /tools/server/tests/ @ggml-org/llama-server
/ui/ @ggml-org/llama-ui /tools/ui/ @ggml-org/llama-ui
/tools/tokenize/ @ggerganov /tools/tokenize/ @ggerganov
/tools/tts/ @ggerganov /tools/tts/ @ggerganov
/vendor/ @ggerganov /vendor/ @ggerganov
@@ -114,4 +119,3 @@
/SECURITY.md @ggerganov /SECURITY.md @ggerganov
/build-xcframework.sh @danbev /build-xcframework.sh @danbev
requirements*.txt @CISC requirements*.txt @CISC
/skills @ngxson
+13 -36
View File
@@ -9,50 +9,27 @@ The project differentiates between 3 levels of contributors:
# AI Usage Policy # AI Usage Policy
> [!IMPORTANT] > [!IMPORTANT]
> This project does **not** accept pull requests that are fully or predominantly AI-generated. AI tools may be utilized solely in an assistive capacity.
> >
> AI-generated code is allowed. You are 100% responsible for every line, however it was produced. > Repeated violations of this policy may result in your account being permanently banned from contributing to the project.
>
> Undisclosed AI usage may result in your account being permanently banned from contributing to the project.
> >
> Detailed information regarding permissible and restricted uses of AI can be found in the [AGENTS.md](AGENTS.md) file. > Detailed information regarding permissible and restricted uses of AI can be found in the [AGENTS.md](AGENTS.md) file.
### Requirements Code that is initially generated by AI and subsequently edited will still be considered AI-generated. AI assistance is permissible only when the majority of the code is authored by a human contributor, with AI employed exclusively for corrections or to expand on verbose modifications that the contributor has already conceptualized (e.g., generating repeated lines with minor variations).
If AI is used to generate any portion of the code, contributors must adhere to the following requirements: If AI is used to generate any portion of the code, contributors must adhere to the following requirements:
1. Explicitly disclose the manner in which AI was employed. 1. Explicitly disclose the manner in which AI was employed.
2. Check for an existing PR addressing the same change; if one exists, comment there to work with its author instead of opening a duplicate. 2. Perform a comprehensive manual review prior to submitting the pull request.
3. Perform a comprehensive manual review prior to submitting the pull request. A proper code review usually takes something like one hour per 200-400 LOC and you should be spending **at least that much time on code review alone**. 3. Be prepared to explain every line of code they submitted when asked about it by a maintainer.
4. Be prepared to explain every line of code you submit when asked about it by a maintainer. 4. It is strictly prohibited to use AI to write your posts for you (bug reports, feature requests, pull request descriptions, Github discussions, responding to humans, ...).
5. It is strictly prohibited to use AI to write your posts for you (bug reports, feature requests, pull request descriptions, Github discussions, responding to humans, ...).
### Checklist
> [!IMPORTANT]
>
> **If you use AI, do all of the following:**
>
> 1. **Make sure [AGENTS.md](AGENTS.md) is loaded into your agent's context.** Skipping this step is one of the most common reasons PRs get rejected.
> 2. **Search for related discussions, PRs and issues** to get the broader context before writing any code.
> 3. **Run a local review pass before pushing anything**, using the [`code-review`](skills/code-review/SKILL.md) skill, once the change is implemented. Fix what it finds.
For more info, please refer to the [AGENTS.md](AGENTS.md) file. For more info, please refer to the [AGENTS.md](AGENTS.md) file.
# Pull requests (for contributors & collaborators) # Pull requests (for contributors & collaborators)
### Before you start Before submitting your PR:
- Search for existing PRs to prevent duplicating efforts
- Search for existing discussions and PRs first - duplicates will likely be closed without questions.
- Features must begin with an issue, not a PR - let interest accumulate before writing code; niche features may only land as an example/tool, or on a private fork.
- Bug-fix PRs must include a reproducible issue and a regression test that fails before your change and passes after. Fixes without a test may be closed without review.
- New CLI or public API additions carry a **higher bar** than internal changes - justify why an existing mechanism doesn't suffice.
- Meeting all of the above still doesn't guarantee a merge - see [Pull requests (for maintainers)](#pull-requests-for-maintainers).
- If you are a new contributor
- Limit your open PRs to 1
- Do not submit trivial fixes (e.g. typos, formatting changes)
### Preparing your PR
- llama.cpp uses the ggml tensor library for model evaluation. If you are unfamiliar with ggml, consider taking a look at the [examples in the ggml repository](https://github.com/ggml-org/ggml/tree/master/examples/). [simple](https://github.com/ggml-org/ggml/tree/master/examples/simple) shows the bare minimum for using ggml. [gpt-2](https://github.com/ggml-org/ggml/tree/master/examples/gpt-2) has minimal implementations for language model inference using GPT-2. [mnist](https://github.com/ggml-org/ggml/tree/master/examples/mnist) demonstrates how to train and evaluate a simple image classifier - llama.cpp uses the ggml tensor library for model evaluation. If you are unfamiliar with ggml, consider taking a look at the [examples in the ggml repository](https://github.com/ggml-org/ggml/tree/master/examples/). [simple](https://github.com/ggml-org/ggml/tree/master/examples/simple) shows the bare minimum for using ggml. [gpt-2](https://github.com/ggml-org/ggml/tree/master/examples/gpt-2) has minimal implementations for language model inference using GPT-2. [mnist](https://github.com/ggml-org/ggml/tree/master/examples/mnist) demonstrates how to train and evaluate a simple image classifier
- Test your changes: - Test your changes:
- Execute [the full CI locally on your machine](ci/README.md) before publishing - Execute [the full CI locally on your machine](ci/README.md) before publishing
@@ -61,6 +38,7 @@ For more info, please refer to the [AGENTS.md](AGENTS.md) file.
- If you modified a `ggml` operator or added a new one, add the corresponding test cases to `test-backend-ops` - If you modified a `ggml` operator or added a new one, add the corresponding test cases to `test-backend-ops`
- Create separate PRs for each feature or fix: - Create separate PRs for each feature or fix:
- Avoid combining unrelated changes in a single PR - Avoid combining unrelated changes in a single PR
- For intricate features, consider opening a feature request first to discuss and align expectations
- When adding support for a new model or feature, focus on **CPU support only** in the initial PR unless you have a good reason not to. Add support for other backends like CUDA in follow-up PRs - When adding support for a new model or feature, focus on **CPU support only** in the initial PR unless you have a good reason not to. Add support for other backends like CUDA in follow-up PRs
- In particular, adding new data types (extension of the `ggml_type` enum) carries with it a disproportionate maintenance burden. As such, to add a new quantization type you will need to meet the following *additional* criteria *at minimum*: - In particular, adding new data types (extension of the `ggml_type` enum) carries with it a disproportionate maintenance burden. As such, to add a new quantization type you will need to meet the following *additional* criteria *at minimum*:
- convert a small model to GGUF using the new type and upload it to HuggingFace - convert a small model to GGUF using the new type and upload it to HuggingFace
@@ -68,9 +46,11 @@ For more info, please refer to the [AGENTS.md](AGENTS.md) file.
- provide KL divergence data calculated vs. the FP16/BF16 (whichever is the native precision) version for both the new type as well as types of similar size - provide KL divergence data calculated vs. the FP16/BF16 (whichever is the native precision) version for both the new type as well as types of similar size
- provide [performance data](https://github.com/ggml-org/llama.cpp/tree/master/tools/llama-bench) for the new type in comparison to types of similar size on pure CPU - provide [performance data](https://github.com/ggml-org/llama.cpp/tree/master/tools/llama-bench) for the new type in comparison to types of similar size on pure CPU
- Consider allowing write access to your branch for faster reviews, as reviewers can push commits directly - Consider allowing write access to your branch for faster reviews, as reviewers can push commits directly
- If you are a new contributor
- Limit your open PRs to 1
- Do not submit trivial fixes (e.g. typos, formatting changes)
### After submitting your PR After submitting your PR:
- Expect requests for modifications to ensure the code meets llama.cpp's standards for quality and long-term maintainability - Expect requests for modifications to ensure the code meets llama.cpp's standards for quality and long-term maintainability
- Maintainers will rely on your insights and approval when making a final decision to approve and merge a PR - Maintainers will rely on your insights and approval when making a final decision to approve and merge a PR
- If your PR becomes stale, rebase it on top of latest `master` to get maintainers attention - If your PR becomes stale, rebase it on top of latest `master` to get maintainers attention
@@ -85,14 +65,11 @@ For more info, please refer to the [AGENTS.md](AGENTS.md) file.
- When merging a PR, make sure you have a good understanding of the changes - When merging a PR, make sure you have a good understanding of the changes
- If a PR does not warrant a new release, add `[no release]` in the squashed commit to spare CI resources - If a PR does not warrant a new release, add `[no release]` in the squashed commit to spare CI resources
- Be mindful of maintenance: most of the work going into a feature happens after the PR is merged. If the PR author is not committed to contribute long-term, someone else needs to take responsibility (you) - Be mindful of maintenance: most of the work going into a feature happens after the PR is merged. If the PR author is not committed to contribute long-term, someone else needs to take responsibility (you)
- Add the ["merge ready"](https://github.com/ggml-org/llama.cpp/pulls?q=is%3Apr+is%3Aopen+draft%3Ano+sort%3Aupdated-desc+label%3A%22merge+ready%22+) label to a PR to indicate when a PR can be fast-merged without waiting for 2 independent reviews. [(more info)](https://github.com/ggml-org/llama.cpp/pull/26178)
- Wait for CI results before merging
Maintainers reserve the right to decline review or close pull requests for any reason, without any questions, particularly under any of the following conditions: Maintainers reserve the right to decline review or close pull requests for any reason, without any questions, particularly under any of the following conditions:
- The proposed change is already mentioned in the roadmap or an existing issue, and it has been assigned to someone. - The proposed change is already mentioned in the roadmap or an existing issue, and it has been assigned to someone.
- The pull request duplicates an existing one. - The pull request duplicates an existing one.
- The contributor fails to adhere to this contributing guide or the AI policy. - The contributor fails to adhere to this contributing guide or the AI policy.
- The change doesn't fit the existing architecture, or is too complex to justify its benefit.
# Coding guidelines # Coding guidelines
-1738
View File
File diff suppressed because it is too large. Load diff
Loaded 100 of 4055 files, more files were not shown because too many files have changed in this diff. Show more