The doorbell is edge-triggered and only honoured by a process that is already
warm and already listening, so a ring is silently lost in three windows: while
the holder is still loading its model (the inotify watch does not exist yet, and
the kernel does not queue events for a watch that is not there), while it is
inside restore_device (vram_cold is still set, so its own warden discards the
ring), and when a cold waiter's warden consumes a third process's ring and then
wins the flock. Nothing re-sends it, flock(LOCK_EX) never times out, and with no
--sleep-idle-seconds the holder has no other reason to release - so a lost ring
wedges the waiter permanently. It blocks under mutex_tasks, so the whole server
stops answering while /health still returns 200.
Take the token through one helper that retries flock with LOCK_NB and re-rings
about once a second, warning every five. That makes every one of the three
windows self-healing: a dropped ring is simply re-sent once the holder is warm.
Also retry on EINTR: the return value was previously discarded and the signal
handlers run without SA_RESTART, so a signal made the caller restore weights
believing it held the token - two models uploading into 8 GB, the OOM the
arbiter exists to prevent.
Leave vram_cold where it is. Clearing it before the restore looks like the
obvious companion fix but is actively harmful: the warden then honours rings the
ringer has already satisfied, and the server hands back the token it just took
without serving, stalling every handoff for minutes.
Also unlink our own doorbell file on shutdown so the arena stops growing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
LumpiastyandClaude Opus 5
f0dda4d0e0