Lumpiasty/llama.cpp
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

server: refactor sleep handling, allow access /metrics during sleep (#27376)

Browse Source 
* add cached responses

* refactor on_sleeping_state

* allow accessing metrics during sleep

* metrics task should not reset timer

* updated docs

* fix

* fix get_res_model_info

* add test

* fix a race condition

* split metrics and slots tasks / results

* should_reset_buckets
This commit is contained in:
Xuan-Son Nguyen
2026-08-19 20:48:09 +02:00
committed by GitHub
parent ee0ea03adf
commit 947fd9bb2b
9 changed files with 422 additions and 181 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tools/server/README-dev.md
+30
View File
@@ -291,6 +291,36 @@ The flow for downloading a new model:
- If a stop request comes in, the router asks the child process to stop (same mechanism as running a model in child process)
- Otherwise, upon completion, we call `load_models()` to refresh the list of models
### Sleep mode
Sleep mode was initially introduced in PR [#18228](https://github.com/ggml-org/llama.cpp/pull/18228). The main idea is to have:
- `server_queue` keeping track of the idle timeout
- When the timeout is detected, `server_queue` signals to `server_context_impl` that it should go into sleep
- `server_context_impl` frees all `llama_context` and `mtmd_context`
Compared to simply exiting the whole process, this approach allows accessing some read-only endpoints during sleep, while also handling wakeup-on-request. Any inference request will wake the server up.
Call stack on entering sleeping:
- `server_queue::start_loop` (main thread) sees no task for `idle_sleep_ms` --> `sleeping = true`
- `cb0(true)` --> `server_routes::update_cached_responses`
- snapshots `/props`, `/models` and metrics; the model is still alive here
- `cb1(true)` --> `server_context_impl::handle_sleeping_state`
- `callback_state(SERVER_STATE_SLEEPING)` --> reported to router in child mode
- `destroy()` --> frees `llama_context` and `mtmd_context`
- `condition_tasks.wait` until `req_stop_sleeping`
Call stack on waking up:
- `server_res_generator` constructor (HTTP thread) --> `server_queue::wait_until_no_sleep`
- sets `req_stop_sleeping = true`, then waits until `sleeping == false`
- `server_queue::start_loop` (main thread) wakes up
- `cb1(false)` --> `server_context_impl::handle_sleeping_state`
- `load_model()`, which then emits `callback_state(SERVER_STATE_READY)`
- `cb0(false)` --> `server_routes::update_cached_responses`
- nothing to do, the cache is only read during sleep
- `sleeping = false` --> `notify_all` unblocks the HTTP thread, the request is handled as usual
Endpoints created with `create_response(true)` (`/health`, `/props`, `/models`, `/metrics`) skip `wait_until_no_sleep`, so they answer from the cached responses instead of waking the server.
### Notable Related PRs
- Initial server implementation: https://github.com/ggml-org/llama.cpp/pull/1443