Two ways a shutting-down server kept the VRAM arbiter's token, stalling every
other model:
- SIGTERM with a request in flight: terminate() stopped only the task queue.
The HTTP handler kept waiting for results the loop would never send, the main
thread blocked joining it, and the token was released only in the destructor
after that join - i.e. never, until the client went away. Terminate the
result queue too (readers now get "server is shutting down": a 503, or an SSE
error event), and let wait_until_no_sleep give up once stopping.
- SIGTERM while waiting for the token: the loop thread sat in a blocking flock
inside the wake callback, holding mutex_tasks, so terminate() waited for the
current holder to finish and then for a pointless restore of our weights.
Poll the lock non-blocking and abort the wait on shutdown; the aborted wake
stays cold, skips the restore, and the loop exits before update_slots.
Also release the token right after start_loop() returns rather than in the
destructor, and only after the device buffers are actually freed: the old
order unlocked before destroy(), so the next model could start uploading into
VRAM that was still occupied. Freeing at exit uses a new
llama_context_discard_device(), which drops weights, KV, MoE cache and scratch
without capturing host shadows first.
Off production: SIGTERM mid-generation or mid-wait now exits in ~0.2 s with
exit 0 and frees the token within 5-70 ms (previously never / after the
holder's whole generation). Handoff latency and greedy output are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The doorbell was edge-triggered: a waiter wrote <arena>/doorbell/<pid> and the
holder's warden turned the inotify event into a queue flag. A ring carried no
notion of "still wanted", so it was lost whenever the holder was not already warm
and listening - during its model load (the watch does not exist yet, and the
kernel does not queue events for a watch that is not there) or inside
restore_device - and any ring that survived past the point its sender had been
satisfied caused a spurious release. Lost rings wedged the waiter permanently,
because flock(LOCK_EX) never times out and, with idle-sleep disabled, the holder
had no other reason to release. The wait blocks under mutex_tasks, so the whole
server stopped answering while /health still returned 200.
Express the request as kernel state instead. A waiter holds <arena>/want.lock
shared while it waits and drops it once it owns the token; the sleep decision
probes that lock non-blocking and releases the GPU while anyone is waiting. The
probe needs its own fd - flock treats two open file descriptions of one file
independently, so probing on the waiter's fd would convert our own lock rather
than conflict with it. Nothing can be missed, nothing goes stale, and a waiter
that dies is cleaned up by the kernel.
Probe from should_sleep() on the loop thread rather than from a warden thread.
Routing it through a flag is what made the first attempts fail: start_loop()
holds mutex_tasks from should_sleep() through the callbacks to the wait, so a
warden's request_yield() blocks on that mutex and is admitted only after the flag
has been consumed, latching a release for the next wake. Reading live state where
the decision is made has no edge to latch, and drops the warden's poll latency.
Two sleep-path bugs this exposed: wait_until_no_sleep() waited on !sleeping but
the loop clears req_stop_sleeping on the way in, so a loop that slept again
before the waiter ran stranded it forever - re-ask on every wake. And a task
queued after the waiter saw us awake could not wake us by itself, so sleep now
also breaks on a non-empty queue. Hold the GPU for 100 ms after a wake: the
request that woke us is not queued yet, and yielding at once only sends it round
again.
Measured on the RX 580 pod, two servers contending, 60 alternating handoffs:
0 stranded, median 0.309 s against the doorbell's 0.314 s.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
The vision encoder sits idle in VRAM on every text request. With
LLAMA_MMPROJ_ONDEMAND=1 the server keeps the encoder in RAM (released
after load) and brings it into VRAM just-in-time before mtmd_batch_encode,
releasing it again afterwards - so the encoder's VRAM is free for KV /
expert cache on the common text-only path.
When the encoder does not fit (that VRAM has been claimed), evict the LLM
backbone weights first: they are not needed while the encoder runs (the
decode that uses them happens afterwards), their host shadow is read-only
(cheap free, no D2H), and the KV / prompt cache is left untouched. Order
on the way back matters: free the encoder before re-uploading the weights
so the peak stays within VRAM.
Also wire the encoder release/restore into the VRAM arbiter: vram_go_cold
releases it; the just-in-time encode-path restore replaces the previous
restore in vram_ensure_warm.
Assisted-by: Claude
With LLAMA_SLEEP_EVICT_KV the KV cache device buffers are freed when a model
goes cold. update_slots() touches the KV before the decode-time vram_ensure_warm
(e.g. SWA models create a checkpoint that reads the KV via ggml_backend_tensor_get),
so the KV must already be resident by then. Move the restore to the sleep-wake
handler (handle_sleeping_state(false)), which runs before update_slots, fixing a
GGML_ASSERT(buffer) crash on iSWA models (gemma) when KV eviction is enabled.
Assisted-by: Claude
Before uploading a model's weights, acquire the shared VRAM token (ring the
doorbell so any resident model releases first). Previously load uploaded weights
to VRAM before the arbiter was active, so loading a large model while another
(e.g. the warm 4B task model) held VRAM could exceed the budget and OOM.
- add vram_arena_open() (idempotent flock/doorbell setup) and
vram_acquire_for_load(), called from load_model() before
common_init_from_params().
- a coordinated load now stays warm holding the token and serves its first
request without a re-warm (drop the init-time go_cold cold-start).
Assisted-by: Claude
The request handler only calls wait_until_no_sleep() (which wakes a server out
of its sleeping state) when sleep_idle_seconds >= 0. But the VRAM arbiter's
cross-process doorbell can put a server to sleep even when idle-sleep is
disabled, so without this a doorbell-slept server would never wake and requests
to it would hang until timeout.
Do not bypass the wake path when LLAMA_SLEEP_VRAM_ONLY is set, so the arbiter no
longer depends on --sleep-idle-seconds being configured.
Assisted-by: Claude
Extend on-demand device residency to the KV cache so that when a model's KV
plus another model would not fit in VRAM, the KV can also be evicted to a host
shadow (D2H on release, H2D on restore) instead of only the weights.
- llama_memory_i: add release_device_buffers()/restore_device_buffers()
(default no-op). Implemented in llama_kv_cache (D2H shadow of the live
ctxs_bufs, freed and reallocated like the weights); llama_memory_hybrid and
llama_kv_cache_iswa delegate to their child caches.
- llama_context::release_device(evict_kv): also evict the memory's device
buffers when requested; restore_device() rebuilds them. Public API
llama_context_release_device gains an evict_kv flag.
- server: LLAMA_SLEEP_EVICT_KV=1 enables it. Off by default (weights-only),
since the KV shadow adds a D2H/H2D copy of the live cache each cycle.
Validated on RX 580 (Vulkan), 4B @ 32k ctx: weights-only cold VRAM 1750 MB
(KV stays); weights+KV cold VRAM 726 MB (KV freed, ~1 GB reclaimed). KV
survives the round-trip: prompt cache reused after the cycle (prompt_n 4 vs
42), correct output.
Assisted-by: Claude
Add release/restore of a model's GPU weight buffers (keeping a host shadow
and the KV cache) so several always-loaded llama-server processes can
time-share a single GPU without reloading or losing the prompt cache.
- llama-model: release_device_weights()/restore_device_weights() capture a
compact host shadow (stable iteration order, view-skipping) and free then
realloc the device weight buffers; weights_resident() query.
- llama-context: release_device()/restore_device() wrappers; decode() auto-
restores; public C API llama_context_release_device/restore_device.
- server: LLAMA_SLEEP_VRAM_ONLY makes idle-sleep release only the VRAM weights
(not a full unload/reload). A cross-process flock token in LLAMA_VRAM_ARENA
enforces "resident iff holds token"; an inotify doorbell forces the holder
to release on contention. The warden thread only touches the task queue, so
releases run on the loop thread and never race a decode.
Validated on RX 580 (Vulkan): two models share 8GB, never both resident,
correct output under contention, KV cache preserved (no re-prefill).
Assisted-by: Claude
* mtmd: support hibiki realtime duplex
* fix and clean up
* fix long session quality
* fix ci
* tests : add the moshi fusion baseline for Metal
The Metal fusion check found 5 RMS_NORM+MUL fusions in the new moshi
arch and no baseline for them.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* cuda: add scatter for batched tensor writes
* update test case to mirror get_n test case
* save restore timing logs
* refactor code into batch-copy.cu file
* llama-bench : size the batch to the prompt for models without memory
Assisted-by: Claude Opus 5.5
* server : read up to 16 lfm3-d1 questions in one pass
Assisted-by: Claude Opus 5.5
* server: fix lora and recurrent
* tests: run the hybrid SWA checkpoint test with one token per ubatch
With and without the cache, the prompt was not split into the same
ubatches, and the generated tokens are decoded one at a time while the
uncached run processes them in a batch. On CUDA this moves the top
logprobs of the tiny model by up to 0.03 and swaps near ties, so the
test failed while the cache behaved as expected.
With one token per ubatch both runs go through the same kernels and
the output is compared exactly.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* jinja: impl scope-chain key lookup
* add test for call_statement lambda
* chat: add tts_dict for kokoro G2P template
* more testing
* value_float_t range check
* no json-in-meta, add llama_model_meta_arr_str/_n
* mtmd: add kokoro tts support
* detailed guide about G2P and text norm port
* refine G2P
* clean up
* migrate batch api
* nits
* mtmd: support YuE2-3B
* improve speed
* add CFG
* fix streaming
* add cfg control from api
* check ctx size
* fix n_predict
* add flow_steps config
* nits
* fix batch api
* nits
* adapt
* mtmd: yue2 samples from its own saved logits
The context logits belong to the last decode of any slot, so with
several server slots the score and the unguided codes were sampled
from another sequence. Keep the logits of each own decode in
save_logits(), which also does the guidance mix.
* mtmd: let the yue2 snake chain fuse
The beta reshape landed between sqr and mul, so the mul, sin, sqr,
mul, add chain was not consecutive and the backends never fused it.
Expand both reshapes first, like qwen3tts does.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* mtmd : allow custom video decoder
* also adapt to wllama requirements
* clean up
* add test
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
With --sleep-idle-seconds, a handler passes wait_until_no_sleep()
while the server is awake, then keeps using the server state, for
example to parse the body and tokenize. If the idle timer expires in
that window, the main loop enters sleep and the callbacks destroy the
model: the handler reads freed vocab memory, and a task posted after
that stays in the queue with nobody awake to process it.
Count the requests that passed wait_until_no_sleep() and are not done
yet, and do not enter sleep while the count is positive. The response
generator releases its count in its destructor, so every return path
of a handler gives it back.
Co-authored-by: Venkata Vineel <vyalamar@users.noreply.github.com>
* server: add oai-compat /v1/audio/speech endpoint
* add e2e pytest
* server: share the s16 rounding, type speech form fields
The WAV writer of the audio helper truncates while pcm_f32_to_s16
rounds, so wav and pcm differ by one LSB. Both now round.
pcm_f32_to_s16 takes a pointer and a count and converts the TTS
bytes without an intermediate copy.
/tts and /v1/audio/speech parse their body with parse_tts_body, so
multipart fields such as speed get their type before the speech
conversion checks them.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* mtmd, server : add temperature to /tts, default Pocket TTS to 0.3
Assisted-by: Claude Opus 5.5
* feat: add temp parameter in llama-tts
* temp: fix default hardcoded values
* fix: remove too narrow test
* fix: addressing PR comments
* server: visit parent slots before child slots
A parent slot wakes its waiting child slots once it finishes the shared
prompt prefix, while slots are visited in id order. When a child task
lands on a lower slot id than its parent, the child is visited before
the parent wakes it and waits an extra batch to process its suffix.
Non-child slots are now visited before active child slots, so every
child woken by its parent joins the same batch.
* server: check systemone test events through test tags
The systemone tests matched free-form debug messages, including the
formatted f_keep value of the prompt cache. The server now emits
__TEST_TAG_WAIT_PARENT_SLOT__ when a child slot waits for its parent,
__TEST_TAG_PROMPT_CACHE_RESTORE__ on every prompt cache restore and
__TEST_TAG_PROMPT_CACHE_FULL_RESTORE__ when the whole cached prompt is
a prefix of the new one, compared on token counts.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* checkpoints: reduce overhead
Assisted-by: Codex
* rebase
Adapt the slot save/restore checkpoint appendix from #26004 to the
shared_ptr checkpoint list and the common_state_data buffers.
Assisted-by: Claude Opus 5.5
---------
Co-authored-by: cwriter <cwriter@localhost>
* feat: support PocketTTS voices with no voice cloning
The kyutai/pocket-tts-without-voice-cloning weights ship predefined voices as the
backbone KV state after the voice prompt. The converter turns each state back into
the prompt embeddings and stores them in the mmproj (voice.<name>), the zeroed voice
encoder is left out, and the "voice" field of /tts and --tts-voice select one of them.
A speaker reference is rejected when the model has no voice encoder.
Assisted-by: Claude Fable 5.1
* convert : pocket-tts voices never fail the conversion, log the drift of the state
The voice states shipped by Kyutai are not always the ones the pack's
weights give for the prompt: italian, german and spanish match within the
bf16 rounding, english_2026-04 is 2% off and french 20% (its layer 0 state
comes from other weights), while the reference loads them as is and they
still sound like the same voice. The converter now re-prompts the backbone
with the recovered embeddings, compares the whole state and logs the
distance per voice, warning above 5%. A state that does not belong to the
weights at all is skipped, so the packs of kyutai/pocket-tts convert too.
Assisted-by: Claude Fable 5.1
* mtmd : list the predefined voices, expose them in GET /props
mtmd_gen_audio_info gains n_voices and mtmd_gen_audio_voice_name() the
names, with the voices worded as a feature of any audio generation model.
The server lists them under "voices" in /props when the model ships some.
Assisted-by: Claude Fable 5.1
* server : an unknown voice is a bad request
A model with predefined voices answers 400 naming the voice instead of 500
"failed to process TTS prompt". A model without them ignores "voice" with a
warning, as the other audio generation models do.
Assisted-by: Claude Fable 5.1
* refactor api
* fix conversion
* small renaming
* fix lint
* tts : a model with predefined voices speaks with the first one by default
Without a voice prompt such a model only produces near silence. When neither
a voice nor a speaker reference is given, llama-tts and the server now take
the first predefined voice and say so in the log, as the LFM3-Audio template
does with its default voice.
Assisted-by: Claude Fable 5.1
* mtmd : one comment per voice accessor, converter states the rule only
The clip.h comment described clip_gen_get_voice but sat above all three
functions; the converter comment quoted the drift measured per pack, which
goes stale with the next upload of the weights.
Assisted-by: Claude Fable 5.1
* clean up redundant code and comments
* mtmd: match the LFM3-Audio voice name case-insensitively
The tts chat template lowers the voice name, so Male speaks with
the male voice; the voice lookup lowers it the same way and names the
bitmap after the canonical voice.
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: Pascal <admin@serveurperso.com>
* server : add realtime conversation and transcription on /v1/realtime
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Assisted-by: Claude Opus 5.5
* examples : default the realtime client to the server port 9931
Assisted-by: Claude Opus 5.5
* server : sample realtime responses right after their own decode
Slots respond in turns on one context: sampling at the start of the next step read the logits of whichever slot decoded last, so concurrent sessions mixed their outputs.
Assisted-by: Claude Opus 5.5
* move more stuff to server-realtime, making it self-contained
* examples : fix the type check of the realtime client
Assisted-by: Claude Opus 5.5
* examples : use the model card prompts in the realtime client
Assisted-by: Claude Opus 5.5
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* model : add LiquidAI/d1-2B and LiquidAI/d1-omni-2B
Assisted-by: Claude Opus 5.5
* convert : keep the speech task templates out of the d1 GGUFs
Assisted-by: Claude Opus 5.5
* small refactoring: no_batching
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* server: expose the reasoning effort levels of the chat template
The capability probe renders a user turn for each candidate level
(minimal, low, medium, high, xhigh, max) and for an unknown level. A
level is honored when it renders without raising and differently from
the unknown level, which shows the template fallback. Levels sharing a
render are aliases and the highest one is kept.
The caps serialize to JSON, and chat_template_caps in /props reports
the list as reasoning_effort_levels, from lowest to highest.
* jinja: keep the caps free of the json dependency
The caps keep reporting their flags as a map, and the effort levels
stay a plain member. common_chat_templates_get_caps() builds the JSON
the server reports from both, so jinja depends on json only for the
probe inputs.
* model : add LiquidAI/LFM3-Audio-2B and LiquidAI/LFM3-Omni-2B
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Assisted-by: Claude Opus 5.5
* convert : read the LFM3-Audio config.json as released
A transformers that knows lfm3_audio rewrites the config into another layout (no lfm/encoder keys).
Assisted-by: Claude Opus 5.5
* mtmd : disable flash attention for the LFM3-Audio depthformer
Assisted-by: Claude Opus 5.5
* tts : add a voice option for LFM3-Audio and LFM3-Omni
The model card offers a female and a male voice; --tts-voice for llama-tts, "voice" for the server /tts.
Assisted-by: Claude Opus 5.5
* clean up template handling
* convert : detect LFM3-Audio and LFM3-Omni by model_type
Assisted-by: Claude Opus 5.5
* move banned tokens --> logits bias --> mtmd_helper
* leave mtmd-cli unchanged
* no more template_asr_system
* left over
* llama : keep the LFM3-Audio encoder convs and codec unquantized
Assisted-by: Claude Opus 5.5
* common, mtmd : shorten comments
Assisted-by: Claude Opus 5.5
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* model : add MiniCPM-OCR-Jev support
The model is a Qwen3.5-VL trunk with a value head read at the last token: it
scores an OCR text against an image rendered from it, it does not generate.
Add the openbmb-minicpm-ocr-jev decision type, served by POST /v1/systemone.
The class scores are the pooled output of the sequence, read with last pooling,
so the conversion turns score.weight into cls.output and sets
embedding_length_out. The checkpoint has no LM head, so text generation is
rejected instead of using the output weights duplicated from the token
embeddings.
The task is fixed, so its prompt is entirely in the systemone template, the
same way h2o-lightning keeps its default there. The processor of the checkpoint
allows up to 16384 merged tokens per image, more than the Qwen-VL default of
mtmd, so the mmproj keeps the image budget of the preprocessor config. The
checkpoint declares MiniCPMOCRForSequenceClassification in its config.json, as
other models do.
Signed-off-by: tc-mb <tianchi_cai@icloud.com>
* fix code
Signed-off-by: tc-mb <tianchi_cai@icloud.com>
* use gguf meta classifier_output_labels
* remove redudant code changes
---------
Signed-off-by: tc-mb <tianchi_cai@icloud.com>
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* server: realtime API: send complete after committed audio drains
* server: realtime API: complete a commit at its end row
Completion is triggered when the stream runs dry, which is a timing
proxy: audio appended while the tail drains leaks into the committed
item, a final commit before the drain merges both items, and back to
back commits collapse into one.
mtmd_audio_stream_flush() records the row count at the end of the
flushed audio (mtmd_audio_stream_get_n_rows_flushed), next to the rows
produced so far (mtmd_audio_stream_get_n_rows). The slot keeps a FIFO
of these end rows and completes each commit once every row up to it is
decoded.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* server : support typed content input for rerank endpoint
Assisted-by: Opencode/Qwen3.8 27B
* server : address rerank multimodal review findings
- reject multimodal rerank input when the model has no rerank chat template (images were silently dropped)
- translate handle_media download/base64 failures to 400 instead of 500
- reject negative top_n with 400 instead of letting resize throw
- fix comment typos, redundant comments, indentation and trailing whitespace
- README: document that TEI input accepts the same shape as Jina; drop stray blank line
Assisted-by: opencode
* Trimmed README
---------
Co-authored-by: timothywang21 <timothywang21@users.noreply.github.com>
* mtmd: add audio out stream api
* add /tts endpoint
* add docs
* wire up mtmd_helper_model_can_chat
* upload speaker_ref via form-data
* expose seed to api
* fix header for pcm
* wip
* fix penalty_last_n
* fix server_output_limits
* fix pocket-tts stream
* wip
* add server_schema for tts
* add server routing
* server: coerce seed and stream from multipart form fields
Form fields arrive as strings, so -F seed=42 and -F stream=true were
rejected by the schema. seed joins the integer fields, parsed as a long
long so a full uint32 seed fits, and stream is read as a boolean when
it comes as a string.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* server: add realtime transcription API
* drop transcription_sessions
* proxy ws
* add docs
* server: let the API key middleware guard the websocket
httplib runs the pre-routing handler before the websocket upgrade, so
the API key middleware answers 401 at handshake like any other route.
This covers the router proxy, whose children run keyless, and accepts
the same key forms as HTTP (Bearer, raw key, X-Api-Key).
* server: keep realtime slots out of the prompt loop
A realtime slot waiting for audio is GENERATING with nothing to decode,
so it fell through the prompt loop and took slot_batched. A request on
a higher slot then failed can_batch_with on the task type and starved
for the whole session. Realtime slots have no prompt: the loop sets
them up once and always leaves them.
* server: start the realtime session once its slot is live
The audio stream was created in the prompt loop, which skips a slot
that cannot batch with the one generating, and session.created went out
before the task was posted: early appends failed and the reader quit,
leaving the socket open and the slot held until the HTTP timeout.
The slot now sets up its stream at launch and signals it with the begin
result, session.created waits for it, and a reader that stops closes
the socket so the session ends with it.
* server: name the committed item in input_audio_buffer.committed
Per the Realtime spec, item_id is the item made from the committed audio
and previous_item_id the one committed before it, null for the first.
committed now comes before the transcript that completes that item, and
both go out with the item switch under one lock, so a delta cannot land
between them.
* server: use the GA transcription session shape
The session object follows the GA Realtime schema: type transcription,
with the input format, transcription model, turn detection and noise
reduction under audio.input. The rate stays the model sample rate, which
is what the client must send for now.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* return logprobs in spec path
* server: add regression test for logprobs with speculative decoding
Asserts accepted draft tokens report the target model's logprobs,
comparing a draft-enabled run against a no-draft baseline (#24271).
Fails on master (top_logprobs empty, logprob 0.0), passes with the
fix from #27196.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KARBWpPcijcBXzJEvttuwk
---------
Co-authored-by: Jeonghyun Lee <sig.jhl@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* model : add H2O-Lightning-4B decision model
Adds the h2o-lightning decision type for h2oai/h2o-lightning-4b
(Qwen3.5-4B with a merged decision LoRA).
None of the existing types match this model. It answers with " A"
rather than "A", uses up to 255 labels, and was trained on Python
json.dumps text, so floats like 12.0 must render as 12.0.
The converter writes the decision metadata and the systemone template
from the model's serve_config.json and refuses a config that doesn't
match. Tests skip unless the tiny test model is available locally.
Assisted-by: Claude Opus 5.5
* rm redundant code
* put labels to gguf
---------
Co-authored-by: Jonathan C. McKinney <jon.mckinney@h2o.ai>
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>