server: release the GPU token promptly on shutdown
Two ways a shutting-down server kept the VRAM arbiter's token, stalling every other model: - SIGTERM with a request in flight: terminate() stopped only the task queue. The HTTP handler kept waiting for results the loop would never send, the main thread blocked joining it, and the token was released only in the destructor after that join - i.e. never, until the client went away. Terminate the result queue too (readers now get "server is shutting down": a 503, or an SSE error event), and let wait_until_no_sleep give up once stopping. - SIGTERM while waiting for the token: the loop thread sat in a blocking flock inside the wake callback, holding mutex_tasks, so terminate() waited for the current holder to finish and then for a pointless restore of our weights. Poll the lock non-blocking and abort the wait on shutdown; the aborted wake stays cold, skips the restore, and the loop exits before update_slots. Also release the token right after start_loop() returns rather than in the destructor, and only after the device buffers are actually freed: the old order unlocked before destroy(), so the next model could start uploading into VRAM that was still occupied. Freeing at exit uses a new llama_context_discard_device(), which drops weights, KV, MoE cache and scratch without capturing host shadows first. Off production: SIGTERM mid-generation or mid-wait now exits in ~0.2 s with exit 0 and frees the token within 5-70 ms (previously never / after the holder's whole generation). Handoff latency and greedy output are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
64543a435b
commit
5751c977b7
17 files changed
+155
-52
No files matched your search
@@ -116,7 +116,8 @@ void server_queue::wait_until_no_sleep() {
|
||||
std::unique_lock<std::mutex> lock(mutex_tasks);
|
||||
// re-ask on every wake: the loop clears req_stop_sleeping on the way into sleep, so a loop that
|
||||
// goes back to sleep before we run would strand us here forever
|
||||
while (sleeping) {
|
||||
// a loop that terminates while asleep never wakes: give up then, the reader reports the shutdown
|
||||
while (sleeping && !stopping) {
|
||||
if (!req_stop_sleeping) {
|
||||
QUE_DBG("%s", "requesting to stop sleeping\n");
|
||||
req_stop_sleeping = true;
|
||||
@@ -128,6 +129,7 @@ void server_queue::wait_until_no_sleep() {
|
||||
}
|
||||
|
||||
void server_queue::terminate() {
|
||||
stopping = true; // before the lock: the loop may hold it while it waits for the VRAM token
|
||||
std::unique_lock<std::mutex> lock(mutex_tasks);
|
||||
running = false;
|
||||
condition_tasks.notify_all();
|
||||
@@ -136,7 +138,7 @@ void server_queue::terminate() {
|
||||
bool server_queue::process_new_tasks(bool is_yielding) {
|
||||
while (true) {
|
||||
std::unique_lock<std::mutex> lock(mutex_tasks);
|
||||
if (!running) {
|
||||
if (!running || stopping) {
|
||||
QUE_DBG("%s", "terminate\n");
|
||||
return true;
|
||||
}
|
||||
@@ -472,11 +474,12 @@ server_task_result_ptr server_response::recv_with_timeout(const std::unordered_s
|
||||
}
|
||||
}
|
||||
|
||||
std::cv_status cr_res = condition_results.wait_for(lock, std::chrono::seconds(timeout));
|
||||
if (!running) {
|
||||
RES_DBG("%s : queue result stop\n", __func__);
|
||||
std::terminate(); // we cannot return here since the caller is HTTP code
|
||||
return nullptr; // the caller checks is_running()
|
||||
}
|
||||
|
||||
std::cv_status cr_res = condition_results.wait_for(lock, std::chrono::seconds(timeout));
|
||||
if (cr_res == std::cv_status::timeout) {
|
||||
return nullptr;
|
||||
}
|
||||
@@ -517,6 +520,7 @@ void server_response::broadcast(server_task_result_ptr && result) {
|
||||
}
|
||||
|
||||
void server_response::terminate() {
|
||||
std::unique_lock<std::mutex> lock(mutex_results);
|
||||
running = false;
|
||||
condition_results.notify_all();
|
||||
}
|
||||
@@ -564,6 +568,14 @@ server_task_result_ptr server_response_reader::next(const std::function<bool()>
|
||||
while (true) {
|
||||
server_task_result_ptr result = queue_results.recv_with_timeout(id_tasks, polling_interval_seconds);
|
||||
if (result == nullptr) {
|
||||
if (!queue_results.is_running()) {
|
||||
// the server is shutting down: no result will ever come
|
||||
stop();
|
||||
auto res = std::make_unique<server_task_result_error>();
|
||||
res->err_type = ERROR_TYPE_UNAVAILABLE;
|
||||
res->err_msg = "server is shutting down";
|
||||
return res;
|
||||
}
|
||||
// timeout, check stop condition
|
||||
if (should_stop()) {
|
||||
return nullptr;
|
||||
|
||||
Reference in new issue
Block a user