server: release the GPU token promptly on shutdown
Two ways a shutting-down server kept the VRAM arbiter's token, stalling every other model: - SIGTERM with a request in flight: terminate() stopped only the task queue. The HTTP handler kept waiting for results the loop would never send, the main thread blocked joining it, and the token was released only in the destructor after that join - i.e. never, until the client went away. Terminate the result queue too (readers now get "server is shutting down": a 503, or an SSE error event), and let wait_until_no_sleep give up once stopping. - SIGTERM while waiting for the token: the loop thread sat in a blocking flock inside the wake callback, holding mutex_tasks, so terminate() waited for the current holder to finish and then for a pointless restore of our weights. Poll the lock non-blocking and abort the wait on shutdown; the aborted wake stays cold, skips the restore, and the loop exits before update_slots. Also release the token right after start_loop() returns rather than in the destructor, and only after the device buffers are actually freed: the old order unlocked before destroy(), so the next model could start uploading into VRAM that was still occupied. Freeing at exit uses a new llama_context_discard_device(), which drops weights, KV, MoE cache and scratch without capturing host shadows first. Off production: SIGTERM mid-generation or mid-wait now exits in ~0.2 s with exit 0 and frees the token within 5-70 ms (previously never / after the holder's whole generation). Handoff latency and greedy output are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
33f8b89f9b
commit
83cc51c9da
17 files changed
+155
-52
No files matched your search
@@ -1114,13 +1114,13 @@ public:
|
||||
}
|
||||
|
||||
~server_context_impl() {
|
||||
// release the VRAM token before tearing anything down
|
||||
vram_share_shutdown();
|
||||
if (!sleeping) {
|
||||
// destroy() is already called when entering sleeping state
|
||||
// we don't call it again here to avoid double free
|
||||
destroy();
|
||||
}
|
||||
// only once our VRAM is freed (normally vram_share_exit() has done both already)
|
||||
vram_share_shutdown();
|
||||
}
|
||||
|
||||
server_metrics get_metrics() const {
|
||||
@@ -1283,7 +1283,8 @@ private:
|
||||
// take the shared VRAM token: register on want.lock, then wait for token.lock. want.lock is
|
||||
// level-triggered state, so the holder still learns of us if it is loading or restoring when we
|
||||
// start to wait. No timeout: the holder can be mid-generation, and decoding without the token
|
||||
// risks an OOM that kills both processes.
|
||||
// risks an OOM that kills both processes. Returns false without the token if the server shuts
|
||||
// down meanwhile (the caller checks queue_tasks.is_stopping()).
|
||||
bool vram_take_token() {
|
||||
#if !defined(_WIN32)
|
||||
if (!vram_flock) {
|
||||
@@ -1297,8 +1298,15 @@ private:
|
||||
SRV_INF("%s", "VRAM arbiter: waiting for the GPU token\n");
|
||||
logged = true;
|
||||
}
|
||||
flock(vram_lock_fd, LOCK_EX); // blocking: the kernel grants it the moment the holder unlocks
|
||||
continue; // re-check with LOCK_NB, which also succeeds on a lock we already own
|
||||
// poll, not a blocking flock: terminate() must be able to stop this wait, and it can
|
||||
// only signal us through an atomic (we hold mutex_tasks here)
|
||||
if (queue_tasks.is_stopping()) {
|
||||
SRV_INF("%s", "VRAM arbiter: shutting down, no longer waiting for the GPU token\n");
|
||||
vram_want_release();
|
||||
return false;
|
||||
}
|
||||
std::this_thread::sleep_for(std::chrono::milliseconds(2));
|
||||
continue;
|
||||
}
|
||||
if (errno == EINTR) {
|
||||
continue; // our signal handlers run without SA_RESTART
|
||||
@@ -1315,12 +1323,19 @@ private:
|
||||
}
|
||||
|
||||
// acquire the VRAM token (blocking) and bring weights back to the device. Called on the loop
|
||||
// thread right before a decode, so it never races compute.
|
||||
void vram_ensure_warm() {
|
||||
// thread right before a decode, so it never races compute. Returns false, still cold, if the
|
||||
// server shuts down first: the caller must not touch the device then.
|
||||
bool vram_ensure_warm() {
|
||||
if (!vram_only || !vram_cold) {
|
||||
return;
|
||||
return true;
|
||||
}
|
||||
if (queue_tasks.is_stopping()) {
|
||||
return false; // no restore just to exit
|
||||
}
|
||||
if (!vram_take_token()) {
|
||||
if (queue_tasks.is_stopping()) {
|
||||
return false;
|
||||
}
|
||||
SRV_WRN("%s", "VRAM arbiter: restoring weights without the GPU token\n");
|
||||
}
|
||||
llama_context_restore_device(ctx_tgt);
|
||||
@@ -1336,6 +1351,25 @@ private:
|
||||
// bring the weights up. We dropped want.lock when we took the token, so the next probe
|
||||
// reports only the waiters that are still there.
|
||||
vram_cold = false;
|
||||
return true;
|
||||
}
|
||||
|
||||
// the loop has stopped: free our VRAM and hand the token over now, without waiting for the HTTP
|
||||
// threads and the teardown. Discarded, not released: no host copy, and the context never decodes
|
||||
// again. Runs on the loop thread after the loop returned, so nothing else uses the device.
|
||||
void vram_share_exit() {
|
||||
if (vram_only && !vram_cold && ctx_tgt != nullptr) {
|
||||
llama_context_discard_device(ctx_tgt);
|
||||
if (ctx_dft != nullptr) {
|
||||
llama_context_discard_device(ctx_dft);
|
||||
}
|
||||
if (mctx != nullptr) {
|
||||
mtmd_release_device(mctx);
|
||||
}
|
||||
vram_cold = true;
|
||||
SRV_INF("%s", "VRAM arbiter: loop stopped, device memory freed, releasing the GPU token\n");
|
||||
}
|
||||
vram_share_shutdown();
|
||||
}
|
||||
|
||||
void vram_share_shutdown() {
|
||||
@@ -1454,6 +1488,7 @@ private:
|
||||
// (e.g. SWA checkpoint creation reads it via ggml_backend_tensor_get) before the
|
||||
// decode-time vram_ensure_warm(), so with KV eviction the KV must already be resident
|
||||
// here or those reads hit a freed (null) buffer.
|
||||
// false (still cold) only on shutdown: the loop then exits before update_slots
|
||||
vram_ensure_warm();
|
||||
}
|
||||
sleeping = new_state;
|
||||
@@ -4944,7 +4979,9 @@ private:
|
||||
// VRAM arbiter: make sure our weights are resident (acquiring the shared GPU token first)
|
||||
// before we decode. Runs on the loop thread before entering yield_to_queue(), so it never
|
||||
// races an in-flight decode.
|
||||
vram_ensure_warm();
|
||||
if (!vram_ensure_warm()) {
|
||||
throw std::runtime_error("server is shutting down");
|
||||
}
|
||||
|
||||
// yield to the queue, so we can still handle metrics tasks while decoding
|
||||
// note: the sync is done here too, so that the wait is also covered by the yield
|
||||
@@ -5463,9 +5500,12 @@ bool server_context::load_model(common_params & params) {
|
||||
void server_context::start_loop() {
|
||||
auto & params = impl->params_base;
|
||||
impl->queue_tasks.start_loop(params.sleep_idle_seconds * 1000);
|
||||
impl->vram_share_exit();
|
||||
}
|
||||
|
||||
void server_context::terminate() {
|
||||
// wake the HTTP threads waiting for results first: the loop will not produce them any more
|
||||
impl->queue_results.terminate();
|
||||
impl->queue_tasks.terminate();
|
||||
}
|
||||
|
||||
|
||||
Reference in new issue
Block a user