server: keep the server awake while a request uses its state (#32133)

With --sleep-idle-seconds, a handler passes wait_until_no_sleep()
while the server is awake, then keeps using the server state, for
example to parse the body and tokenize. If the idle timer expires in
that window, the main loop enters sleep and the callbacks destroy the
model: the handler reads freed vocab memory, and a task posted after
that stays in the queue with nobody awake to process it.

Count the requests that passed wait_until_no_sleep() and are not done
yet, and do not enter sleep while the count is positive. The response
generator releases its count in its destructor, so every return path
of a handler gives it back.

Co-authored-by: Venkata Vineel <vyalamar@users.noreply.github.com>
This commit is contained in:
PascalandVenkata Vineel authored and GitHub committed 2026-11-12 12:16:16 +01:00
1 parent cfb9d4b754
commit 09452cdedd
3 files changed
+21 -4

No files matched your search

+7
View File
@@ -5302,12 +5302,19 @@ server_context_meta server_context::get_meta() const {
// may have bypass_sleep = true if the task does not use ctx_server
struct server_res_generator : server_res_spipe {
server_response_reader rd;
server_queue * queue_awake = nullptr; // set while this response keeps the server awake
server_res_generator(server_queue & queue_tasks, server_response & queue_results, int sleep_idle_seconds, bool bypass_sleep = false)
: rd(queue_tasks, queue_results, HTTP_POLLING_SECONDS) {
// fast path in case sleeping is disabled
bypass_sleep |= sleep_idle_seconds < 0;
if (!bypass_sleep) {
queue_tasks.wait_until_no_sleep();
queue_awake = &queue_tasks;
}
}
~server_res_generator() {
if (queue_awake) {
queue_awake->release_inflight();
}
}
void ok(const json & response_data) {