Files
llama.cpp/tools/server/tests
Xuan-Son NguyenandPascal 52975342ff server: slot_sched: fix lora and recurrent (#32434)
* server: fix lora and recurrent

* tests: run the hybrid SWA checkpoint test with one token per ubatch

With and without the cache, the prompt was not split into the same
ubatches, and the generated tokens are decoded one at a time while the
uncached run processes them in a batch. On CUDA this moves the top
logprobs of the tiny model by up to 0.03 and swaps near ties, so the
test failed while the cache behaved as expected.

With one token per ubatch both runs go through the same kernels and
the output is compared exactly.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-18 00:12:20 +01:00
..

Server tests

Python based server tests scenario using pytest.

Tests target GitHub workflows job runners with 4 vCPU.

Note: If the host architecture inference speed is faster than GitHub runners one, parallel scenario may randomly fail. To mitigate it, you can increase values in n_predict, kv_size.

Install dependencies

pip install -r requirements.txt

Run tests

  1. Build the server
cd ../../..
cmake -B build
cmake --build build --target llama-server
  1. Start the test: ./tests.sh

It's possible to override some scenario steps values with environment variables:

variable description
PORT context.server_port to set the listening port of the server during scenario, default: 8080
LLAMA_SERVER_BIN_PATH to change the server binary path, default: ../../../build/bin/llama-server
DEBUG to enable steps and server verbose mode --verbose
N_GPU_LAYERS number of model layers to offload to VRAM -ngl --n-gpu-layers
LLAMA_CACHE by default server tests re-download models to the tmp subfolder. Set this to your cache (e.g. $HOME/Library/Caches/llama.cpp on Mac or $HOME/.cache/llama.cpp on Unix) to avoid this

To run slow tests (will download many models, make sure to set LLAMA_CACHE if needed):

SLOW_TESTS=1 ./tests.sh

The speculative-decoding method tests are opt-in and marked heavy. The compact suite covers MTP, DFlash, DSpark, and EAGLE-3 with five pinned files totaling 6.271 GiB:

SPEC_METHODS=compact SPEC_MODEL_DIR=/path/to/model-cache PYTEST_WORKERS=1 ./tests.sh -s -v unit/test_spec_methods.py
  • MTP target: Qwen3.5-4B-MTP-Q4_K_M.gguf, 2.904 GiB.
  • Shared DFlash/DSpark/EAGLE-3 target: Qwen3-4B-Q4_K_M.gguf, 2.326 GiB.
  • DFlash draft: dflash-Qwen3-4B-Q4_K_M.gguf, 294.065 MiB.
  • DSpark draft: dspark-Qwen3-4B-Q4_K_M.gguf, 544.453 MiB.
  • EAGLE-3 draft: eagle3-Qwen3-4B-Q8_0.gguf, 227.075 MiB.

All files come from ggml-org/spec-for-testing-GGUF, pinned to revision a0736c8e98d3fe580352dacffa3fbd5e9c22a437 in unit/test_spec_methods.py. The consolidated repository retains the licenses of its source models; review its model card and the source model cards before use.

Use SPEC_METHODS=nightly for DFlash2 and LiLiCorr, or a comma-separated subset such as SPEC_METHODS=mtp,dflash. The nightly pair requires 41.422 GiB and remains separate because no verified compact DFlash2 GGUF or compact LiLiCorr checkpoint is available. SPEC_METHODS=all runs both tiers and downloads about 47.693 GiB.

The Server (speculative decoding) workflow runs the compact suite on the dedicated DGX Spark runner for pull requests and pushes to master that change the speculative decoding code, the model files of the speculative methods, the test or the workflow. A manual dispatch runs the suite selected by its spec_methods input: compact (default), nightly or all. It does not use the GitHub Actions cache: the models are kept in the directory of the SPEC_METHOD_MODEL_DIR repository variable. Pull requests from forks do not get repository variables, so their runs use /mnt/cache/spec-models, the persistent model directory of the runner. Heavyweight DFlash2 and LiLiCorr measurements belong in the internal nightly performance/quality runs.

The acceptance floors use one-sided regression checks: improvements do not fail. Three identical runs on the GB10 runner against the PR merged with current master produced MTP 0.6640 / 2.9766, DFlash 0.2004 / 2.3687, DSpark 0.3367 / 3.3246, and EAGLE-3 0.2606 / 1.7757 for acceptance rate and length. The compact floors subtract six percentage points and 0.2 accepted tokens per verification step. The six-point rate margin is wider than the largest observed cross-environment shift of 0.0506. Warm compact runs completed in about 40 seconds. The DFlash2 and LiLiCorr nightly floors retain their earlier three-point rate and 0.2 length margins.

DFlash's token-acceptance rate is lower because the denominator counts every token proposed in each seven-token draft block. It accepted 219 of 1,093 proposed tokens over 160 verification steps, equivalent to 1.369 accepted draft tokens per step and a mean acceptance length of 2.3687. The floor is intended to detect regressions from that deterministic baseline, not to define a quality target.

To run with stdout/stderr display in real time (verbose output, but useful for debugging):

DEBUG=1 ./tests.sh -s -v -x

To run all the tests in a file:

./tests.sh unit/test_chat_completion.py -v -x

To run a single test:

./tests.sh unit/test_chat_completion.py::test_invalid_chat_completion_req

Hint: You can compile and run test in single command, useful for local development:

cmake --build build -j --target llama-server && ./tools/server/tests/tests.sh

To see all available arguments, please refer to pytest documentation

Debugging external llama-server

It can sometimes be useful to run the server in a debugger when invesigating test failures. To do this, the environment variable DEBUG_EXTERNAL=1 can be set which will cause the test to skip starting a llama-server itself. Instead, the server can be started in a debugger.

Example using gdb:

$ gdb --args ../../../build/bin/llama-server \
    --host 127.0.0.1 --port 8080 \
    --temp 0.8 --seed 42 \
    --hf-repo ggml-org-dev/stories260K --hf-file stories260K.gguf \
    --batch-size 32 --no-slots --alias tinyllama-2 --ctx-size 512 \
    --parallel 2 --n-predict 64

And a break point can be set in before running:

(gdb) br server.cpp:4604
(gdb) r
main: server is listening on http://127.0.0.1:8080 - starting the main loop
srv  update_slots: all slots are idle

And then the test in question can be run in another terminal:

(venv) $ env DEBUG_EXTERNAL=1 ./tests.sh unit/test_chat_completion.py -v -x

And this should trigger the breakpoint and allow inspection of the server state in the debugger terminal.