Commit Graph
882 Commits
Author SHA1 Message Date
Matt Thompson 711f9f4c11 server: responses api: support conversion of text.format.json_schema to completions (#26145) 2026-11-10 00:43:38 +01:00
Aldehir Rojas 26d52b0f95 server : use common_chat_input in realtime api (#31978) 2026-11-09 23:59:16 +01:00
Xuan-Son NguyenandPascal 08f945ffb5 server: add oai-compat /v1/audio/speech endpoint (#31763)
* server: add oai-compat /v1/audio/speech endpoint

* add e2e pytest

* server: share the s16 rounding, type speech form fields

The WAV writer of the audio helper truncates while pcm_f32_to_s16
rounds, so wav and pcm differ by one LSB. Both now round.

pcm_f32_to_s16 takes a pointer and a count and converts the TTS
bytes without an intermediate copy.

/tts and /v1/audio/speech parse their body with parse_tts_body, so
multipart fields such as speed get their type before the speech
conversion checks them.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-09 23:58:52 +01:00
Xuan-Son NguyenandPascal 20c2e96655 mtmd: support gemma 4 input audio stream (#31965)
* mtmd: support gemma 4 input audio stream

* use 480ms step

* server: enable realtime embeddings only for audio generation

The realtime slot forced embeddings on, which marks every row of a
batch as an output. Only a model that generates audio reserves
n_batch outputs, so a chat model with an audio stream (Gemma 4)
aborted on its first batch of audio rows. The hidden state is only
read by the audio generation pipeline.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-09 23:36:31 +01:00
Xuan-Son Nguyen 3d94ffe452 mtmd: add mtmd_helper_resampler (#31964)
* mtmd: add mtmd_helper_resampler

* update docs

* add test

* MA_MAX_FILTER_ORDER
2026-11-09 23:04:39 +01:00
Piotr Wilkin (ilintar)andXuan Son Nguyen 5f490d7bfe feat: external PLE file loading (#31185)
* model : allow loading external PLE files

Assisted-by: Codex

* convert : add separate PLE exports

Assisted-by: Codex

* tests : fix PLE export test lint

Assisted-by: Codex

* model : make --ple an alias of --shard, allow multiple shard locations

Reuse the multi-shard loading path for external PLE files instead of a
separate option.

* --ple is an alias of --shard: both take a comma separated list and may
  be given more than once, and the shards may live anywhere
* metadata comes from the main file; numbered shards are still
  auto-discovered and strictly validated
* explicit files outside the discovered set are appended as extra shards,
  exempt from split.no and split.count checks
* -isnm/--ignore-shard-num-mismatch loads only the given files, in order,
  with no auto-discovery and no split metadata validation
* ple_path, path_ple and llama_model_loader::add_ple_file are removed;
  common_model_load() uses llama_model_load_from_splits for two or more
  paths
* common/fit.{h,cpp} take the shard path vector, so --fit and the router
  memory measurement load every shard

Assisted-by: GPT-6-Astra

* rm -isnm

* make sure to dedup paths

* rm test_convert_ple

* address comments

* allow ple shard to override tensor

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-11-09 19:41:01 +01:00
nv-vankitandPascal 359dc5858e server : functionality test for new spec dec (#31027)
* server : functionality test to make sure every time when we add new spec dec method the existing ones are not broken.

* server: use compact models for speculative decoding CI

* ci: avoid GHA cache for speculative models

* ci: run speculative tests on DGX Spark

* server: address speculative test review

* server: use consolidated speculative models

* server: refine speculative test calibration

* docs: record speculative Spark validation

* ci : run the server-speculative steps with bash

* server: recalibrate speculative test floors on GB10

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-09 19:08:20 +01:00
Ethan Linandlinsen 1945793e00 server : fix response_format json_schema when schema is not nested in the json_schema wrapper (#28697)
* server : accept top-level schema in response_format with type json_schema

The OpenAI-compatible chat completions endpoint only applied the schema
when it was nested inside response_format.json_schema.schema. When the
schema was provided directly as response_format.schema (as documented in
the server README, and used by some clients), it was silently ignored
and generation ran unconstrained.

Also mention json_schema in the error message for invalid response_format
types, document the OpenAI wrapper form in the README, and add a test
case for the top-level schema form.

Fixes #10732

* server : let the json_schema wrapper take precedence by presence

Pick the schema source based on whether response_format has a
json_schema key, instead of falling back when the parsed wrapper schema
is empty. An empty schema {} is valid (any JSON value), so the old check
let a top-level schema override an explicit {} in the wrapper. Use
json_value() for the top-level schema too, like the rest of the
function, and add tests for the precedence.

---------

Co-authored-by: linsen <251731047+linsen458-spec@users.noreply.github.com>
2026-11-09 14:10:11 +01:00
Marco Colombo 2d99adb108 mtmd: Pocket TTS - add generic support for temperature set via API (#31655)
* mtmd, server : add temperature to /tts, default Pocket TTS to 0.3

Assisted-by: Claude Opus 5.5

* feat: add temp parameter in llama-tts

* temp: fix default hardcoded values

* fix: remove too narrow test

* fix: addressing PR comments
2026-11-09 11:20:45 +01:00
Georgi Gerganov f477f795cb server : downgrade per-request proxy logs to trace (#31859)
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
2026-11-08 01:29:52 +01:00
e95426f169 llama_moe_cache: run missed experts on CPU, add tuning script (#30966)
* moe-cache : compute part of the misses on the CPU

--moe-cache-cpu-share F computes a share F of the MoE cache misses on the CPU
from the host experts instead of uploading them. The graph builds the expert
FFN twice: on the device from the cache slots, where the misses left to the CPU
read zero slots, and on the CPU from the host experts, where the experts
computed on the device have id -1. The two results are added before the down
bias.

- ggml-cpu: MUL_MAT_ID skips negative ids and writes zero rows
- sched: the copy callback is called for all the inputs of a split, and before
  the host sync when there are no events
- the slot lookup callback reads the inputs of the CPU part together with the
  selected experts, so the CPU part does not wait for the device, and its
  result is copied back asynchronously
- each token uses distinct zero slots, mul_mat_id does not support an expert
  selected twice by a token
- LLAMA_MOE_CACHE_TRACE writes the selected experts of each lookup, for a
  simulator of the cache

* scripts : add moe-cache-tune.py to pick the MoE cache arguments

The script wraps a llama-server command: it starts the server at two cache
sizes and three CPU shares, fits the upload and CPU speeds of the machine, and
replays the routing of the runs through a copy of the cache logic to predict
other sizes and shares. It prints the recommended --moe-cache-mib and
--moe-cache-cpu-share.

docs/moe-cache.md describes the MoE cache arguments and the script.

* add am17an as codeowner for MoE cache

* moe-cache : upload all the misses until the cache is full

With a high --moe-cache-cpu-share, few misses are uploaded, so an empty cache
fills slowly and more misses than needed go to the CPU. The CPU share now
starts once all the slots of a cache group hold an expert. moe-cache-tune.py
replays the same rule.

* add backend sync for async transfer queue

* moe-cache : let the scheduler copy the results of the CPU part

* Fix MSVC compilation

* remove sync change

* llama-graph : extract the CPU share condition into can_cpu_share

* llama-graph : make llm_graph_params::moe_cache const

The graph construction no longer updates the MoE cache directly: the CPU
parts of the layers are accumulated in the graph result and applied to
the cache in llama_context::process_ubatch, after the graph inputs are
set and before the graph is computed.

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)

* scripts : replay the ClassLRU policy in moe-cache-tune.py

* moe-cache : pick CPU misses by CRF instead of random (#31853)

* moe-cache : remove the backend logic of the CPU part from the graph

The CPU part reads the host experts, so the scheduler places it on the
CPU without set_tensor_backend. The cache gives no CPU map for batches
that a device would offload. Graphs with a CPU part look up their own
slot map, so the graph does not register the part with the cache.

ggml-backend : the inputs of the next split that are already computed
are copied to host memory before the current split runs, so a CPU split
does not wait for the device split before it. This replaces the stash
of the cache, and the copy callback is again called for host weights
only.

* llama-graph : return the MoE cache slots and CPU ids in a struct

The scheduler prefetch no longer checks that the next split runs on
another backend: the inputs of a split are never on its own backend,
so the input loop already skips that case.

---------

Co-authored-by: Gaurav Garg <gaugarg@nvidia.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Co-authored-by: Alexey Kopytko <alexey@kopytko.com>
2026-11-08 00:20:58 +08:00
Harkanwal BediandPascal 0956684529 server: visit parent slots before child slots (#30779)
* server: visit parent slots before child slots
A parent slot wakes its waiting child slots once it finishes the shared
prompt prefix, while slots are visited in id order. When a child task
lands on a lower slot id than its parent, the child is visited before
the parent wakes it and waits an extra batch to process its suffix.
Non-child slots are now visited before active child slots, so every
child woken by its parent joins the same batch.

* server: check systemone test events through test tags

The systemone tests matched free-form debug messages, including the
formatted f_keep value of the prompt cache. The server now emits
__TEST_TAG_WAIT_PARENT_SLOT__ when a child slot waits for its parent,
__TEST_TAG_PROMPT_CACHE_RESTORE__ on every prompt cache restore and
__TEST_TAG_PROMPT_CACHE_FULL_RESTORE__ when the whole cached prompt is
a prefix of the new one, compared on token counts.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-07 13:03:38 +02:00
Adrien Gallouët 468fde90e8 server : simplify windows MCP command resolution (#31728)
* server : simplify windows MCP command resolution

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* common : remove utf8_to_wstring()

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

---------

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-11-05 15:34:54 +01:00
cwriterandcwriter e152332a15 llama-server: Use reference-counted checkpoints instead of deep-copies (#30137)
* checkpoints: reduce overhead

Assisted-by: Codex

* rebase

Adapt the slot save/restore checkpoint appendix from #26004 to the
shared_ptr checkpoint list and the common_state_data buffers.

Assisted-by: Claude Opus 5.5

---------

Co-authored-by: cwriter <cwriter@localhost>
2026-11-05 11:02:36 +01:00
Adrien Gallouët c55b094880 server : reject non-object session updates (#31714)
Prevent malformed realtime session updates from throwing an uncaught
exception and terminating the server

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-11-05 09:38:08 +01:00
Adrien Gallouët 5a081c0a70 server : drop u8path() and use generic_string() (#31682)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-11-05 08:37:13 +01:00
Tom Vaucourt 6f1e654370 server : realtime: send an error event instead of crashing on exceptions (#31632)
* server : realtime: send an error event instead of crashing on exceptions

* server : route websocket handlers through ex_wrapper, remove the realtime test

* server : realtime: keep the previous tools when the chat template rejects new ones

* server : realtime: reject tool schemas the grammar cannot convert, drop ws_ex_wrapper

* server : realtime: clarify the tool schema check comment
2026-11-04 21:34:58 +01:00
Adrien Gallouët 52e53a76df common : delete common_get_path_from_env() (#31444)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-11-04 14:21:38 +01:00
d02b4c0d66 mtmd: Pocket tts - support for voices without voice cloning (#31429)
* feat: support PocketTTS voices with no voice cloning

The kyutai/pocket-tts-without-voice-cloning weights ship predefined voices as the
backbone KV state after the voice prompt. The converter turns each state back into
the prompt embeddings and stores them in the mmproj (voice.<name>), the zeroed voice
encoder is left out, and the "voice" field of /tts and --tts-voice select one of them.
A speaker reference is rejected when the model has no voice encoder.

Assisted-by: Claude Fable 5.1

* convert : pocket-tts voices never fail the conversion, log the drift of the state

The voice states shipped by Kyutai are not always the ones the pack's
weights give for the prompt: italian, german and spanish match within the
bf16 rounding, english_2026-04 is 2% off and french 20% (its layer 0 state
comes from other weights), while the reference loads them as is and they
still sound like the same voice. The converter now re-prompts the backbone
with the recovered embeddings, compares the whole state and logs the
distance per voice, warning above 5%. A state that does not belong to the
weights at all is skipped, so the packs of kyutai/pocket-tts convert too.

Assisted-by: Claude Fable 5.1

* mtmd : list the predefined voices, expose them in GET /props

mtmd_gen_audio_info gains n_voices and mtmd_gen_audio_voice_name() the
names, with the voices worded as a feature of any audio generation model.
The server lists them under "voices" in /props when the model ships some.

Assisted-by: Claude Fable 5.1

* server : an unknown voice is a bad request

A model with predefined voices answers 400 naming the voice instead of 500
"failed to process TTS prompt". A model without them ignores "voice" with a
warning, as the other audio generation models do.

Assisted-by: Claude Fable 5.1

* refactor api

* fix conversion

* small renaming

* fix lint

* tts : a model with predefined voices speaks with the first one by default

Without a voice prompt such a model only produces near silence. When neither
a voice nor a speaker reference is given, llama-tts and the server now take
the first predefined voice and say so in the log, as the LFM3-Audio template
does with its default voice.

Assisted-by: Claude Fable 5.1

* mtmd : one comment per voice accessor, converter states the rule only

The clip.h comment described clip_gen_get_voice but sat above all three
functions; the converter comment quoted the drift measured per pack, which
goes stale with the next upload of the weights.

Assisted-by: Claude Fable 5.1

* clean up redundant code and comments

* mtmd: match the LFM3-Audio voice name case-insensitively

The tts chat template lowers the voice name, so Male speaks with
the male voice; the voice lookup lowers it the same way and names the
bitmap after the canonical voice.

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-02 18:39:51 +01:00
Sigbjørn Skjæret fcfe4b59df requirements : minor adjustments for python 3.14 wheels (#31534) 2026-11-02 11:34:45 +01:00
aiqubits f3586e66bc server: cap shared decision prefixes at the first output (#30710)
Limit the shared prefix to the first required output position of the parent and each child task.
This keeps decision output embeddings available in the final batch when questions share option tokens.

Fixes #30482

Signed-off-by: aiqubits <aiqubits@hotmail.com>
2026-10-31 19:09:48 +01:00
David Garcia 2871678a1d server : fix /v1/responses endpoint to accept extra_params (#30959) 2026-10-31 19:08:08 +01:00
Adrien Gallouët a34de8618b common : delete fs_path_to_utf8() (#31423)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-10-30 16:20:04 +01:00
Adrien Gallouët d63fbfe794 cmake : embed an app manifest in all Windows exes (#31329)
* cmake : embed an app manifest in all Windows exes

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Remove LLAMA_WIN_APP_CMAKE

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

---------

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-10-30 10:56:53 +01:00
7konigsbergandZeshawn Zahid 7df8b876b9 server : restore the spec checkpoint for drafts longer than n_rs_seq (#31350)
Assisted-by: Claude

Co-authored-by: Zeshawn Zahid <zzshawnn@gmail.com>
2026-10-29 11:28:03 +02:00
Pascal e09a78a25d tests: fix CI flakes in server cancel and sleep tests (#31330)
test_cancel_request enables context shift so the generation only ends
when the request is cancelled, then polls /slots with a deadline.
Without context shift the generation fills the context and frees the
slot on its own on fast machines, so the test passes even with
cancellation disabled.

test_server_sleep uses the existing wait_for_sleep helper instead of a
fixed sleep.
2026-10-29 01:35:01 +01:00
Xuan-Son NguyenandSigbjørn Skjæret 28cb47c7c8 gguf-py: fix np_roundf dropping the sign (#31310)
* gguf-py: fix np_roundf dropping the sign

* Update requirements/requirements-convert_legacy_llama.txt

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* requirements.txt

* Update tools/server/tests/requirements.txt

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-10-28 14:03:59 +01:00
Tarek DakhranandXuan Son Nguyen d4f1b18b3a mtmd: enable video input support for LFM3 model family (#31283)
* mtmd : add LFM3-VL and LFM3-Omni video input

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Assisted-by: Claude Opus 5.5

* mtmd : reserve the soundtrack command to avoid a GCC 11 warning

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-10-28 12:23:06 +01:00
Tarek DakhranandXuan Son Nguyen d1eb0debad server : support tool calls in /v1/realtime (#31287)
* server : add tool calls to /v1/realtime

Assisted-by: Claude Opus 5.5

* refactor

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-10-28 12:17:13 +01:00
Tarek DakhranandXuan Son Nguyen 8ca4060b7e server : add realtime conversation and transcription on /v1/realtime (#31269)
* server : add realtime conversation and transcription on /v1/realtime

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Assisted-by: Claude Opus 5.5

* examples : default the realtime client to the server port 9931

Assisted-by: Claude Opus 5.5

* server : sample realtime responses right after their own decode

Slots respond in turns on one context: sampling at the start of the next step read the logits of whichever slot decoded last, so concurrent sessions mixed their outputs.

Assisted-by: Claude Opus 5.5

* move more stuff to server-realtime, making it self-contained

* examples : fix the type check of the realtime client

Assisted-by: Claude Opus 5.5

* examples : use the model card prompts in the realtime client

Assisted-by: Claude Opus 5.5

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-10-28 01:26:12 +01:00
Tarek DakhranandXuan Son Nguyen d05706b4f3 model : add LiquidAI/d1-2B and LiquidAI/d1-omni-2B (#31275)
* model : add LiquidAI/d1-2B and LiquidAI/d1-omni-2B

Assisted-by: Claude Opus 5.5

* convert : keep the speech task templates out of the d1 GGUFs

Assisted-by: Claude Opus 5.5

* small refactoring: no_batching

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-10-27 23:36:32 +01:00
Pascal 1b4554eff1 server: expose the reasoning effort levels of the chat template (#31271)
* server: expose the reasoning effort levels of the chat template

The capability probe renders a user turn for each candidate level
(minimal, low, medium, high, xhigh, max) and for an unknown level. A
level is honored when it renders without raising and differently from
the unknown level, which shows the template fallback. Levels sharing a
render are aliases and the highest one is kept.

The caps serialize to JSON, and chat_template_caps in /props reports
the list as reasoning_effort_levels, from lowest to highest.

* jinja: keep the caps free of the json dependency

The caps keep reporting their flags as a map, and the effort levels
stay a plain member. common_chat_templates_get_caps() builds the JSON
the server reports from both, so jinja depends on json only for the
probe inputs.
2026-10-27 22:39:37 +01:00
Tarek DakhranandXuan Son Nguyen 19ff817fb2 model : add LiquidAI/LFM3-Audio-2B and LiquidAI/LFM3-Omni-2B (#31257)
* model : add LiquidAI/LFM3-Audio-2B and LiquidAI/LFM3-Omni-2B

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Assisted-by: Claude Opus 5.5

* convert : read the LFM3-Audio config.json as released

A transformers that knows lfm3_audio rewrites the config into another layout (no lfm/encoder keys).

Assisted-by: Claude Opus 5.5

* mtmd : disable flash attention for the LFM3-Audio depthformer

Assisted-by: Claude Opus 5.5

* tts : add a voice option for LFM3-Audio and LFM3-Omni

The model card offers a female and a male voice; --tts-voice for llama-tts, "voice" for the server /tts.

Assisted-by: Claude Opus 5.5

* clean up template handling

* convert : detect LFM3-Audio and LFM3-Omni by model_type

Assisted-by: Claude Opus 5.5

* move banned tokens --> logits bias --> mtmd_helper

* leave mtmd-cli unchanged

* no more template_asr_system

* left over

* llama : keep the LFM3-Audio encoder convs and codec unquantized

Assisted-by: Claude Opus 5.5

* common, mtmd : shorten comments

Assisted-by: Claude Opus 5.5

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-10-27 18:47:14 +01:00
Tarek DakhranandAldehir Rojas eb7818c0b8 common : support deferred tools and tool_reference content parts (#31256)
* common : support deferred tools and tool_reference content parts

Assisted-by: Claude Opus 5.5

* common : credit the deferred tools co-author

Co-authored-by: Aldehir Rojas <765227+aldehir@users.noreply.github.com>

Assisted-by: Claude Opus 5.5

* tests : fix the type check of the deferred tools test

Assisted-by: Claude Opus 5.5
2026-10-27 18:39:11 +01:00
85cc432e2c server: add router device memory margin parameter for dynamic unloading (#21231)
* server: add --models-memory-max parameter to allow dynamically unloading models when they exceed a memory size threshold

estimate with to-be-loaded model size included

use no_alloc to get memory requirements for model load

only set model memory_mb if not previously calculated

use memory margin instead of total size limit, apply to each device separately

add server memory debug logging

move llama_context_device_memory function to llama-ext.h

fix model count exceeded check

improve memory_per_device map naming

improve variable naming, fix style

also strip models memory margin from child processes

cont : clean-up

replace device memory map with buft memory map. Use llama_get_memory_breakdown

extract duplicated check into helper function

move model memory estimation to subprocess

precompute name->buft map, map GPU host types to CPU buft

cleanup unused variable

remove duplicated init calls

* adapt to upstream changes

* wip

* common_memory_reports internface

* SERVER_ROLE_MEASURE

* server-models-sched

* fix build

* server : fix memory accounting of measure children and claim spin

A measure child runs no_alloc, so the free memory it reports does not
include its estimate: subtracting self there erased the memory held by
other processes, down to zero with --fit on, and the next load could
overshoot the margin. Subtract self only for instance reports.

A queued load only checked --models-max before claiming, so when the
memory margin blocked it behind a busy model, load() threw and the
waiter retried in a tight loop. Refuse the claim while the limits are
exceeded and no idle model can be unloaded, so the waiter sleeps on
the condition variable until one goes idle.

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: Pascal <admin@serveurperso.com>
2026-10-27 12:14:42 +01:00
tc-mbandXuan Son Nguyen 7eed6b7685 model: Add MiniCPM-OCR-Jev (#30980)
* model : add MiniCPM-OCR-Jev support

The model is a Qwen3.5-VL trunk with a value head read at the last token: it
scores an OCR text against an image rendered from it, it does not generate.

Add the openbmb-minicpm-ocr-jev decision type, served by POST /v1/systemone.
The class scores are the pooled output of the sequence, read with last pooling,
so the conversion turns score.weight into cls.output and sets
embedding_length_out. The checkpoint has no LM head, so text generation is
rejected instead of using the output weights duplicated from the token
embeddings.

The task is fixed, so its prompt is entirely in the systemone template, the
same way h2o-lightning keeps its default there. The processor of the checkpoint
allows up to 16384 merged tokens per image, more than the Qwen-VL default of
mtmd, so the mmproj keeps the image budget of the preprocessor config. The
checkpoint declares MiniCPMOCRForSequenceClassification in its config.json, as
other models do.

Signed-off-by: tc-mb <tianchi_cai@icloud.com>

* fix code

Signed-off-by: tc-mb <tianchi_cai@icloud.com>

* use gguf meta classifier_output_labels

* remove redudant code changes

---------

Signed-off-by: tc-mb <tianchi_cai@icloud.com>
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-10-27 11:49:12 +01:00
Xuan-Son Nguyen c6fb3b5be2 mtmd: add mtmd_helper_get_capabilities (#31183)
* mtmd: add mtmd_helper_get_capabilities

* cap audio_stream

* add back old api for compat

* audio_gen_spk_ref, text_gen
2026-10-26 12:45:35 +01:00
Georgi Gerganov 224d94c02b ci : bump server timeout during startup (#31174) 2026-10-26 09:59:15 +02:00
Trunk Trick 63412d8737 llama: fix exhausting CPU and MEM caused by dry sequence breaker (#31035) 2026-10-25 12:31:09 +01:00
Yuri KhrustalevandXuan Son Nguyen b3fa69955e chat : reject JSON nested more than 128 levels before walking it (#30242)
* chat : reject JSON nested more than 128 levels before walking it

The schema builder, the jinja engine and the decision code walk request
JSON with recursion. A request nested deep enough overflows the stack of
the HTTP thread and llama-server dies (SIGBUS on macOS, SIGSEGV on Linux):
for a tool schema from a few hundred levels on macOS and about 10000 on
Linux.

Check the depth first and answer 400 past 128 levels: the chat inputs in
common_chat_templates_apply (tools, tool-call arguments,
chat_template_kwargs, json_schema), the json_schema of /completion, and the
whole /v1/systemone body. A gemma4 tool result that is too deep stays text.

* fix the root, not the call side

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-10-25 06:46:35 +01:00
Xuan-Son Nguyen f3269ef96a server: refactor server entrypoint (#31113)
* server_role

* cont
2026-10-24 20:04:28 +02:00
pzhu 184f653602 server: Substitute rerank placeholder from the original template only. (#30864) 2026-10-23 00:38:03 +02:00
Xuan-Son NguyenandPascal d814cbb88c server: realtime API: send complete after committed audio drains (#30994)
* server: realtime API: send complete after committed audio drains

* server: realtime API: complete a commit at its end row

Completion is triggered when the stream runs dry, which is a timing
proxy: audio appended while the tail drains leaks into the committed
item, a final commit before the drain merges both items, and back to
back commits collapse into one.

mtmd_audio_stream_flush() records the row count at the end of the
flushed audio (mtmd_audio_stream_get_n_rows_flushed), next to the rows
produced so far (mtmd_audio_stream_get_n_rows). The slot keeps a FIFO
of these end rows and completes each commit once every row up to it is
decoded.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-10-23 00:37:47 +02:00
Tim Wangandtimothywang21 a5d22276be server : support typed content input for RERANK endpoint (#29625)
* server : support typed content input for rerank endpoint

Assisted-by: Opencode/Qwen3.8 27B

* server : address rerank multimodal review findings

- reject multimodal rerank input when the model has no rerank chat template (images were silently dropped)
- translate handle_media download/base64 failures to 400 instead of 500
- reject negative top_n with 400 instead of letting resize throw
- fix comment typos, redundant comments, indentation and trailing whitespace
- README: document that TEI input accepts the same shape as Jina; drop stray blank line

Assisted-by: opencode

* Trimmed README

---------

Co-authored-by: timothywang21 <timothywang21@users.noreply.github.com>
2026-10-22 13:27:24 +02:00
Georgi Gerganov 1f4c2eeaa5 ci : be able to reconfigure HTTP port for server pytest workers (#30972) 2026-10-22 13:56:03 +03:00
David Garcia 261700b896 server : forward llama.cpp sampling params on /v1/messages (#30856)
* server : forward llama.cpp sampling params on /v1/messages

* server : pass the request through on /v1/messages instead of copying listed keys

* NIT: Change comment from multiline to single line

* server : accept llama.cpp params under extra_params on /v1/messages

* server : remove test case for trivial functionality

* server : show extra_params in the /v1/messages README example
2026-10-21 12:58:52 +02:00
Unbug Lee 9a3b91c680 server : fix usage.completion_tokens undercount with n > 1 (#29486)
Assisted-by: Copilot
2026-10-21 01:19:54 +02:00
Xuan-Son NguyenandPascal 5c0b49a23b server : add POST /tts endpoint for TTS models (#26603)
* mtmd: add audio out stream api

* add /tts endpoint

* add docs

* wire up mtmd_helper_model_can_chat

* upload speaker_ref via form-data

* expose seed to api

* fix header for pcm

* wip

* fix penalty_last_n

* fix server_output_limits

* fix pocket-tts stream

* wip

* add server_schema for tts

* add server routing

* server: coerce seed and stream from multipart form fields

Form fields arrive as strings, so -F seed=42 and -F stream=true were
rejected by the schema. seed joins the integer fields, parsed as a long
long so a full uint32 seed fits, and stream is read as a boolean when
it comes as a string.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-10-21 01:19:42 +02:00
Xuan-Son NguyenandPascal 1242be66f6 server: add realtime transcription API (#30636)
* server: add realtime transcription API

* drop transcription_sessions

* proxy ws

* add docs

* server: let the API key middleware guard the websocket

httplib runs the pre-routing handler before the websocket upgrade, so
the API key middleware answers 401 at handshake like any other route.
This covers the router proxy, whose children run keyless, and accepts
the same key forms as HTTP (Bearer, raw key, X-Api-Key).

* server: keep realtime slots out of the prompt loop

A realtime slot waiting for audio is GENERATING with nothing to decode,
so it fell through the prompt loop and took slot_batched. A request on
a higher slot then failed can_batch_with on the task type and starved
for the whole session. Realtime slots have no prompt: the loop sets
them up once and always leaves them.

* server: start the realtime session once its slot is live

The audio stream was created in the prompt loop, which skips a slot
that cannot batch with the one generating, and session.created went out
before the task was posted: early appends failed and the reader quit,
leaving the socket open and the slot held until the HTTP timeout.

The slot now sets up its stream at launch and signals it with the begin
result, session.created waits for it, and a reader that stops closes
the socket so the session ends with it.

* server: name the committed item in input_audio_buffer.committed

Per the Realtime spec, item_id is the item made from the committed audio
and previous_item_id the one committed before it, null for the first.
committed now comes before the transcript that completes that item, and
both go out with the item switch under one lock, so a delta cannot land
between them.

* server: use the GA transcription session shape

The session object follows the GA Realtime schema: type transcription,
with the input format, transcription model, turn detection and noise
reduction under audio.input. The rate stays the model sample rate, which
is what the client must send for now.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-10-21 00:04:15 +02:00
ead9f32058 server: Support logprobs with speculative decoding (#27196)
* return logprobs in spec path

* server: add regression test for logprobs with speculative decoding

Asserts accepted draft tokens report the target model's logprobs,
comparing a draft-enabled run against a no-draft baseline (#24271).
Fails on master (top_logprobs empty, logprob 0.0), passes with the
fix from #27196.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KARBWpPcijcBXzJEvttuwk

---------

Co-authored-by: Jeonghyun Lee <sig.jhl@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-10-20 19:36:14 +02:00