* Add FP8 KV-cache support * Add tests for scaled, FP8 KV-cache to test-backend-ops * Fix get_*_scale mapping and dequnatize FP8 also before rope shift * Normalizes cache-scale names independently of projection layout. While fused KQV is currently impractival for non-shared scales, this is an improvement for future use regardless * Enable Gemma 4 Assistant with FP8 by storing and fetching scale ptrs * Update comment wording * Move E2E FP8 KV cache tests to test-llama-archs They belong there as they are not just a test for ggml's backend ops * Increase support surface, remove seed * make scaled kv generic * Fix CI failings for test-kv-cache https://github.com/ggml-org/llama.cpp/actions/runs/38718763447/job/116367842578?pr=28392 * CI: execute kv-cache-scale tests * exclude fp8-kv tests from OpenVINO * Fix linting errors of test_kv_cache_scales * Disable kv-ffload=false + multi-GPU tests for FP8 See https://github.com/ggml-org/llama.cpp/actions/runs/38952722084/job/117148353699?pr=28413 * Adjust test label for test-kv-cache.cpp * Fix conflict-resolution mistakes in test-kv-cache.cpp https://github.com/ggml-org/llama.cpp/actions/runs/40554674463/job/122367426461?pr=28392 * Move KV tests to test-fp8.cpp https://github.com/ggml-org/llama.cpp/pull/27512#discussion_r4410518158 * Remove tmp file for test-kv-cache-scales.py on Windows as well * Fix fp8 kv cache test by population kv_source_impl * Exclude models that consume KV cache directly * Remove obsolete comment * Allow BF16 scales as well for KV cache * Rename *_cache_scale to *_cache_s https://github.com/ggml-org/llama.cpp/pull/28392#discussion_r4517194999 * Move `is_type_scaled()` to `llama-kv-cache.cpp`
CI
This CI implements heavy-duty workflows that run on self-hosted runners. Typically the purpose of these workflows is to cover hardware configurations that are not available from Github-hosted runners and/or require more computational resource than normally available.
It is a good practice, before publishing changes to execute the full CI locally on your machine. For example:
mkdir tmp
# CPU-only build
bash ./ci/run.sh ./tmp/results ./tmp/mnt
# with CUDA support
GG_BUILD_CUDA=1 bash ./ci/run.sh ./tmp/results ./tmp/mnt
# with SYCL support
source /opt/intel/oneapi/setvars.sh
GG_BUILD_SYCL=1 bash ./ci/run.sh ./tmp/results ./tmp/mnt
# with MUSA support
GG_BUILD_MUSA=1 bash ./ci/run.sh ./tmp/results ./tmp/mnt
# etc.
Adding self-hosted runners
- Add a self-hosted
ggml-ciworkflow to .github/workflows/build.yml with an appropriate label - Request a runner token from
ggml-org(for example, via a comment in the PR or email) - Set-up a machine using the received token (docs)
- Optionally update ci/run.sh to build and run on the target platform by gating the implementation with a
GG_BUILD_...env