server: release the GPU token promptly on shutdown

Two ways a shutting-down server kept the VRAM arbiter's token, stalling every
other model:

- SIGTERM with a request in flight: terminate() stopped only the task queue.
  The HTTP handler kept waiting for results the loop would never send, the main
  thread blocked joining it, and the token was released only in the destructor
  after that join - i.e. never, until the client went away. Terminate the
  result queue too (readers now get "server is shutting down": a 503, or an SSE
  error event), and let wait_until_no_sleep give up once stopping.

- SIGTERM while waiting for the token: the loop thread sat in a blocking flock
  inside the wake callback, holding mutex_tasks, so terminate() waited for the
  current holder to finish and then for a pointless restore of our weights.
  Poll the lock non-blocking and abort the wait on shutdown; the aborted wake
  stays cold, skips the restore, and the loop exits before update_slots.

Also release the token right after start_loop() returns rather than in the
destructor, and only after the device buffers are actually freed: the old
order unlocked before destroy(), so the next model could start uploading into
VRAM that was still occupied. Freeing at exit uses a new
llama_context_discard_device(), which drops weights, KV, MoE cache and scratch
without capturing host shadows first.

Off production: SIGTERM mid-generation or mid-wait now exits in ~0.2 s with
exit 0 and frees the token within 5-70 ms (previously never / after the
holder's whole generation). Handoff latency and greedy output are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
LumpiastyandClaude Opus 5.5 committed 2026-10-24 18:33:19 +02:00
1 parent 11044b35e3
commit 6c480f7e70
17 files changed
+155 -52

No files matched your search

+16 -4
View File
@@ -116,7 +116,8 @@ void server_queue::wait_until_no_sleep() {
std::unique_lock<std::mutex> lock(mutex_tasks);
// re-ask on every wake: the loop clears req_stop_sleeping on the way into sleep, so a loop that
// goes back to sleep before we run would strand us here forever
while (sleeping) {
// a loop that terminates while asleep never wakes: give up then, the reader reports the shutdown
while (sleeping && !stopping) {
if (!req_stop_sleeping) {
QUE_DBG("%s", "requesting to stop sleeping\n");
req_stop_sleeping = true;
@@ -128,6 +129,7 @@ void server_queue::wait_until_no_sleep() {
}
void server_queue::terminate() {
stopping = true; // before the lock: the loop may hold it while it waits for the VRAM token
std::unique_lock<std::mutex> lock(mutex_tasks);
running = false;
condition_tasks.notify_all();
@@ -136,7 +138,7 @@ void server_queue::terminate() {
bool server_queue::process_new_tasks(bool is_yielding) {
while (true) {
std::unique_lock<std::mutex> lock(mutex_tasks);
if (!running) {
if (!running || stopping) {
QUE_DBG("%s", "terminate\n");
return true;
}
@@ -473,11 +475,12 @@ server_task_result_ptr server_response::recv_with_timeout(const std::unordered_s
}
}
std::cv_status cr_res = condition_results.wait_until(lock, deadline);
if (!running) {
RES_DBG("%s : queue result stop\n", __func__);
std::terminate(); // we cannot return here since the caller is HTTP code
return nullptr; // the caller checks is_running()
}
std::cv_status cr_res = condition_results.wait_until(lock, deadline);
if (cr_res == std::cv_status::timeout) {
return nullptr;
}
@@ -518,6 +521,7 @@ void server_response::broadcast(server_task_result_ptr && result) {
}
void server_response::terminate() {
std::unique_lock<std::mutex> lock(mutex_results);
running = false;
condition_results.notify_all();
}
@@ -565,6 +569,14 @@ server_task_result_ptr server_response_reader::next(const std::function<bool()>
while (true) {
server_task_result_ptr result = queue_results.recv_with_timeout(id_tasks, polling_interval_seconds);
if (result == nullptr) {
if (!queue_results.is_running()) {
// the server is shutting down: no result will ever come
stop();
auto res = std::make_unique<server_task_result_error>();
res->err_type = ERROR_TYPE_UNAVAILABLE;
res->err_msg = "server is shutting down";
return res;
}
// timeout, check stop condition
if (should_stop()) {
return nullptr;