Commit Graph
906 Commits
Author SHA1 Message Date
Roshan Kumar 56fd8a375f server : ignore empty stop strings (#32093) 2026-11-15 17:23:05 +01:00
eddiechen 4e5116f476 server : fix output_config.effort being dropped in /v1/messages (#32321) 2026-11-15 11:09:46 +01:00
Xuan-Son Nguyen 429460dc0e server: refactor, add server_slot_sched, prompt running hash (#32154)
* server: refactor, add server_slot_sched, prompt running hash

* use sha256 hash
2026-11-15 09:37:17 +01:00
Aayan Karmali 58a4a6b49c server : clear slot cache when the global lora config changes (#30367) 2026-11-15 09:05:10 +01:00
Srijan Keshri cf9a9957da server : Fix connection reuse after streamed responses (#30492)
Assisted-by: Codex
2026-11-15 00:29:28 +01:00
Jhen-Jie HongandXuan Son Nguyen e3483e40b0 mtmd : allow custom video decoder (#32188)
* mtmd : allow custom video decoder

* also adapt to wllama requirements

* clean up

* add test

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-11-14 14:12:07 +01:00
Ruixiang Wang 32bdc3a89e spec + model: refactor DFlash based speculative decoding methods (#32016)
* dflash : move the DSpark markov head and backbone graphs to dspark.cpp

Move-only: build_dspark_markov_head, graph_dsv4, graph_dsv41 and
graph_mistral_large4 move verbatim. build_inp_target_features and
build_dspark_markov_head become static members of llama_model_dflash so
both files can use them.

* dflash : detect the head and the backbone once, split the loaders per file

load_arch_hparams sets llama_model_dflash::variant (DFlash, DFlash2,
DSpark, LiliCorr) and ::backbone (base, DSV4, DSV4.1, Mistral Large 4).
The head, backbone and base layer loaders become member helpers in
dflash.cpp, dflash2.cpp, lilicorr.cpp and dspark.cpp. build_arch_graph
dispatches on the backbone.

The DFlash2 and LiliCorr loaders create the DFlash2 conv tensors through
a helper in dflash2.cpp, right after their head tensors instead of in
the base layer loop; the other tensors keep their creation order. The
DFlash2 loader drops its copy of the conv group size check, which master
never reached.

* dflash : share the injection input, token embeddings and lm head lookups

build_inject_inp (target features + fc + encoder norm), get_tok_embd and
get_lm_head replace the copies in the base, DSV4, DSV4.1 and Mistral
Large 4 graphs. build_inp_target_features is file-static again.

* dflash : split the DFlash decoder into a graph per head

graph_base creates the shared inputs and provides build_inject,
build_inp_tokens, build_layers and build_logits. graph_dflash
(dflash.cpp), graph_dflash2 (dflash2.cpp), graph_dspark (dspark.cpp) and
graph_lilicorr (lilicorr.cpp) assemble them and add their head; the
selector, scorer and markov head builders move to the file of their
graph, and the DFlash2 local conv run by build_layers to dflash2.cpp.
build_arch_graph dispatches on the backbone, then on the head.

The op order and the tensor names are unchanged.

* speculative : split the DFlash driver into a class per head

common_speculative_impl_draft_dflash stays the DFlash implementation and
becomes the base of draft_dspark, draft_dflash2 and draft_lilicorr. Its
protected constructor only reads the draft metadata; init() then sets up
the contexts with the options of the draft (draft length, backend
sampling, device features, masked h_nextn). Each derived class overrides
draft() with the path of its head from the former draft(). DFlash2 also
overrides begin() to reseed its generator per request, and LiliCorr
overrides process() to store the context features of each injected
chunk, which its noise block carries as its state.

The factory keeps the current routing: draft-dspark picks DSpark,
draft-dflash picks DFlash2 or LiliCorr from the draft metadata and
DFlash otherwise. draft-dspark with a DFlash2 or LiliCorr draft now runs
the DSpark reader instead of a mix of both. sample_from_anchor is logged
by DSpark only.

* speculative : add the draft-dflash2 and draft-lilicorr types

draft-dflash2 and draft-lilicorr name the DFlash2 and LiliCorr drafts,
which draft-dflash keeps running too: it picks them from the draft
metadata. draft-dflash2 and draft-lilicorr must match the draft model,
draft-dspark rejects a DFlash2 or LiliCorr draft, a model that is not a
DFlash draft is rejected, and a run uses at most one DFlash family type;
these errors leave speculative decoding uninitialized.

The type detected from the draft GGUF is now the concrete one, and
draft-dflash2 and draft-lilicorr download the dflash sidecar.

* docs : describe the DFlash2 and LiliCorr drafts

Add the DFlash2 and LiliCorr sections and the new types, and explain
that draft-dflash also runs DFlash2 and LiliCorr drafts.

* speculative : drop the unused DFlash is_mrope

Nothing reads it since the migration to common_batch (#29385), which
picks the number of M-RoPE position rows from the rope type of the draft.

* doc: update spec dec

* address review comments
2026-11-14 09:49:27 +01:00
Pascal 4f9dfa0e36 requirements, Fix CI: support Python 3.15 (#32199)
* requirements : support Python 3.15

torch 2.11.0, numpy 2.4, matplotlib 3.10, pandas 2.3, pillow 11.3 and
cffi 1.16 have no Python 3.15 wheels, so the CI runner on 3.15 cannot
install the requirements.

Python 3.15 gets the first releases that ship 3.15 wheels through
version markers, and every version below 3.15 stays as is.
pyproject.toml now allows 3.15 and picks torch 2.15.0 there.

* requirements : apply review suggestions

numpy 2.5.4 from Python 3.14, matplotlib 3.11.2 and pandas 3.0.6 from
3.11, cffi 2.1.1 and pillow 12.3.0 for every version. cffi 1.16.0 has
no 3.14 wheel and fails to build there, so this also fixes the
tokenizer test requirements on 3.14.
2026-11-13 17:50:03 +01:00
Georgi Gerganov 28d37fe2e2 examples : clean-up old/obsolete stuff (#32184)
* examples : clean-up old/obsolete stuff

* examples : update references to removed files

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)

[no ci]
2026-11-13 10:33:17 +02:00
Pascal 0a66ac7189 server, fix CI: read a small temp file in the read_file tool test [no release] (#32168)
* server: read a small temp file in the read_file tool test

test_tools_builtin_read_file read its own file, which grows with every
added test. On Windows the CRLF checkout puts it at 16664 bytes, over
the 16384 byte read limit, and the tool refuses it.

The test now writes a small file of its own and checks the exact
content read back.

* server: use a temp cwd in the x-tool-cwd tool test

test_tools_builtin_cwd_header read test_tools_builtin.py through the
x-tool-cwd header, which hits the same 16384 byte read limit on the
Windows CRLF checkout, and wrote its shell marker into the source tree.

The test now uses a temp directory as cwd, reads a small file from it
and checks the shell marker lands there.
2026-11-12 23:47:14 +01:00
Pascal 892b40b910 server: allow private media URLs in the vision tests [no ci] (#32167)
The vision tests download their images from huggingface.co, which
resolves to a private address on the runners inside the HF network.
The media URL check refuses it with "host is not a public address"
and the server-cuda job fails.

The vision fixture enables media_allow_private_urls. The private
address check keeps its own coverage in
test_vision_image_url_private_address.
2026-11-12 23:15:28 +01:00
Pascal e970101470 server: skip LAN address tests when the host cannot reach it [no ci] (#32166)
The same-origin tests over the LAN address block forever on the Metal
runners: the request to the host's own LAN address never returns and
has no timeout, so the job runs until the 6h limit.

get_lan_ip() now probes /health on that address with a short timeout
and returns None when the host cannot reach it, which skips the test.
The test requests themselves use the default request timeout so a
hang fails the test instead of stalling the job.
2026-11-12 23:13:37 +01:00
Georgi Gerganov 8ca5a8008b llama : prune (#30565)
* llama : remove deprecated `llama_set_warmup`

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)

* llama : remove remaining deprecated API

Drop the APIs marked DEPRECATED in llama.h, the deprecated
llama_context_params::defrag_thold field and the --defrag-thold option.

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)

* fit-params : remove stale `llama-ext.h` include

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)

* llama : document `llama_model_dflash_selector_top_k`

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)

* llama : promote the NextN / speculative API from `llama-ext.h`

Move the NextN, layer input, embd row and state row APIs to `llama.h`
and rename `llama_set_embd_state_src` to `llama_set_ctx_state_src`, with
the internal `llama_context::set_state_src` following the same name.

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)

* llama : polish the NextN API promotion

Move the NextN draft model queries next to the Adapters section, rename
the `src` parameters of `llama_set_ctx_state_src` and
`llama_context::set_ctx_state_src` to `ctx_src`, and drop the now-redundant
`llama-ext.h` includes from `common/speculative.cpp` and `tools/imatrix`.

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
2026-11-12 20:59:09 +02:00
Hamish M. Blair d8c8cc4d9b server : cancel MCP tool calls on client disconnect or timeout (#32025)
* server : cancel MCP tool calls on client disconnect or timeout

* server : shorten MCP cancellation tests and fix a type-check error
2026-11-12 17:19:26 +01:00
Xuan-Son Nguyen 8954f19483 server: fix SSRF for CORS proxy and media URLs (#32115) 2026-11-12 12:33:48 +01:00
Xuan-Son Nguyen a1a68eb797 server: explicit opt-in tools via non-loopback addr, check cors host port (#32113)
* server: explicit opt-in tools via non-loopback addr, check host port

* fix ci
2026-11-12 12:16:50 +01:00
Xuan-Son Nguyen e182fdad4a server: disallow direct access child instance from browser (#32120) 2026-11-12 12:16:29 +01:00
PascalandVenkata Vineel 09452cdedd server: keep the server awake while a request uses its state (#32133)
With --sleep-idle-seconds, a handler passes wait_until_no_sleep()
while the server is awake, then keeps using the server state, for
example to parse the body and tokenize. If the idle timer expires in
that window, the main loop enters sleep and the callbacks destroy the
model: the handler reads freed vocab memory, and a task posted after
that stays in the queue with nobody awake to process it.

Count the requests that passed wait_until_no_sleep() and are not done
yet, and do not enter sleep while the count is positive. The response
generator releases its count in its destructor, so every return path
of a handler gives it back.

Co-authored-by: Venkata Vineel <vyalamar@users.noreply.github.com>
2026-11-12 12:16:16 +01:00
Xuan-Son Nguyen 60d46a5abe server: when --tools-runtime is specified, ignore x-tool-runtime (#32119) 2026-11-12 09:34:16 +01:00
Aleksander Grygier dd6b945be2 server: use the moved test model in the router CORS test (#32117)
test_router_restricted_cors_same_origin_reaches_child still named
ggml-org/tinygemma3-GGUF, which the router no longer finds since the test
models moved to ggml-org-dev, so the Server test job fails on master.

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-12 02:01:19 +01:00
Aleksander Grygier 1d81992784 ui: keep downloads in Discover during search and fix their progress with a draft (#31467)
* ui : keep downloads in Discover sidebar during search, sort base-org repos first

- Show the sticky Downloading section regardless of an active search query
- Order search results so repos published by their base model's org lead

Assisted-by: Claude (Opus 5.5 Medium)

* server : throttle download progress per file

Files of a download run in parallel threads, but progress updates were
throttled with one shared timer. A small draft sidecar reports far more
often than the main weights, so it took nearly every slot and the main
file's updates, including the first one carrying its total size, were
dropped until the draft finished. The UI then showed the draft's progress
first and restarted from 0% for the model.

Throttle each file on its own and always pass a file's first update.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: list a downloading repo once and match base orgs case-insensitively

A search hit that is downloading already shows under Downloading, so the
results leave it out. Base model orgs compare without case, as the hub
treats them.

Assisted-by: Claude (Opus 5.5 Medium)

* server : tell clients a download stopped on request

When a client paused or cancelled a download, every other client following
/models/sse got a bare download_failed and could not tell it from an error,
so the web UI showed "Download failed" for a pause made in the Llama app.
A stop asked for through unload or delete now fails with {"stopped": true}.

Deleting a paused download, which has no instance left, now sends
model_remove as deleting a running one does, so other clients drop its row.

Assisted-by: Claude (Opus 5.5 Medium)

* ui : follow downloads other clients start, pause or cancel

A download another client started (e.g. the Llama app) already showed up
from the feed; it now also picks up the sidecars the list reports for it.
A stop another client asked for reads as a pause instead of a failure, and
a removal clears the download row even when the list does not carry it yet.

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-11 23:26:39 +01:00
Xuan-Son NguyenandPascal 2cbe0ec429 server: fix --cors-origins handling + protect against DNS rebinding (#32098)
* server: fix --cors-origins localhost handling

* server: validate Host on loopback connections against DNS rebinding

A page whose own name resolves to 127.0.0.1 reaches the server with
that name in Host and a matching Origin, so the same-origin shortcut
let it through. On a loopback connection, Host must now name loopback
or the host of an allowed --cors-origins entry. The LAN tests reach
the server through a real interface instead of a forged Host.

* fix python type check

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-11 21:17:46 +01:00
Xuan-Son Nguyen 664f031f49 ci: move test models to ggml-org-dev (#32096)
* ci: move test models to ggml-org-dev

* fix test spec
2026-11-11 21:17:16 +01:00
Xuan-Son Nguyen 203b5f404c server: fix output lang for realtime api (#31982) 2026-11-10 11:35:30 +01:00
Matt Thompson 711f9f4c11 server: responses api: support conversion of text.format.json_schema to completions (#26145) 2026-11-10 00:43:38 +01:00
Aldehir Rojas 26d52b0f95 server : use common_chat_input in realtime api (#31978) 2026-11-09 23:59:16 +01:00
Xuan-Son NguyenandPascal 08f945ffb5 server: add oai-compat /v1/audio/speech endpoint (#31763)
* server: add oai-compat /v1/audio/speech endpoint

* add e2e pytest

* server: share the s16 rounding, type speech form fields

The WAV writer of the audio helper truncates while pcm_f32_to_s16
rounds, so wav and pcm differ by one LSB. Both now round.

pcm_f32_to_s16 takes a pointer and a count and converts the TTS
bytes without an intermediate copy.

/tts and /v1/audio/speech parse their body with parse_tts_body, so
multipart fields such as speed get their type before the speech
conversion checks them.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-09 23:58:52 +01:00
Xuan-Son NguyenandPascal 20c2e96655 mtmd: support gemma 4 input audio stream (#31965)
* mtmd: support gemma 4 input audio stream

* use 480ms step

* server: enable realtime embeddings only for audio generation

The realtime slot forced embeddings on, which marks every row of a
batch as an output. Only a model that generates audio reserves
n_batch outputs, so a chat model with an audio stream (Gemma 4)
aborted on its first batch of audio rows. The hidden state is only
read by the audio generation pipeline.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-09 23:36:31 +01:00
Xuan-Son Nguyen 3d94ffe452 mtmd: add mtmd_helper_resampler (#31964)
* mtmd: add mtmd_helper_resampler

* update docs

* add test

* MA_MAX_FILTER_ORDER
2026-11-09 23:04:39 +01:00
Piotr Wilkin (ilintar)andXuan Son Nguyen 5f490d7bfe feat: external PLE file loading (#31185)
* model : allow loading external PLE files

Assisted-by: Codex

* convert : add separate PLE exports

Assisted-by: Codex

* tests : fix PLE export test lint

Assisted-by: Codex

* model : make --ple an alias of --shard, allow multiple shard locations

Reuse the multi-shard loading path for external PLE files instead of a
separate option.

* --ple is an alias of --shard: both take a comma separated list and may
  be given more than once, and the shards may live anywhere
* metadata comes from the main file; numbered shards are still
  auto-discovered and strictly validated
* explicit files outside the discovered set are appended as extra shards,
  exempt from split.no and split.count checks
* -isnm/--ignore-shard-num-mismatch loads only the given files, in order,
  with no auto-discovery and no split metadata validation
* ple_path, path_ple and llama_model_loader::add_ple_file are removed;
  common_model_load() uses llama_model_load_from_splits for two or more
  paths
* common/fit.{h,cpp} take the shard path vector, so --fit and the router
  memory measurement load every shard

Assisted-by: GPT-6-Astra

* rm -isnm

* make sure to dedup paths

* rm test_convert_ple

* address comments

* allow ple shard to override tensor

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-11-09 19:41:01 +01:00
nv-vankitandPascal 359dc5858e server : functionality test for new spec dec (#31027)
* server : functionality test to make sure every time when we add new spec dec method the existing ones are not broken.

* server: use compact models for speculative decoding CI

* ci: avoid GHA cache for speculative models

* ci: run speculative tests on DGX Spark

* server: address speculative test review

* server: use consolidated speculative models

* server: refine speculative test calibration

* docs: record speculative Spark validation

* ci : run the server-speculative steps with bash

* server: recalibrate speculative test floors on GB10

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-09 19:08:20 +01:00
Ethan Linandlinsen 1945793e00 server : fix response_format json_schema when schema is not nested in the json_schema wrapper (#28697)
* server : accept top-level schema in response_format with type json_schema

The OpenAI-compatible chat completions endpoint only applied the schema
when it was nested inside response_format.json_schema.schema. When the
schema was provided directly as response_format.schema (as documented in
the server README, and used by some clients), it was silently ignored
and generation ran unconstrained.

Also mention json_schema in the error message for invalid response_format
types, document the OpenAI wrapper form in the README, and add a test
case for the top-level schema form.

Fixes #10732

* server : let the json_schema wrapper take precedence by presence

Pick the schema source based on whether response_format has a
json_schema key, instead of falling back when the parsed wrapper schema
is empty. An empty schema {} is valid (any JSON value), so the old check
let a top-level schema override an explicit {} in the wrapper. Use
json_value() for the top-level schema too, like the rest of the
function, and add tests for the precedence.

---------

Co-authored-by: linsen <251731047+linsen458-spec@users.noreply.github.com>
2026-11-09 14:10:11 +01:00
Marco Colombo 2d99adb108 mtmd: Pocket TTS - add generic support for temperature set via API (#31655)
* mtmd, server : add temperature to /tts, default Pocket TTS to 0.3

Assisted-by: Claude Opus 5.5

* feat: add temp parameter in llama-tts

* temp: fix default hardcoded values

* fix: remove too narrow test

* fix: addressing PR comments
2026-11-09 11:20:45 +01:00
Georgi Gerganov f477f795cb server : downgrade per-request proxy logs to trace (#31859)
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
2026-11-08 01:29:52 +01:00
e95426f169 llama_moe_cache: run missed experts on CPU, add tuning script (#30966)
* moe-cache : compute part of the misses on the CPU

--moe-cache-cpu-share F computes a share F of the MoE cache misses on the CPU
from the host experts instead of uploading them. The graph builds the expert
FFN twice: on the device from the cache slots, where the misses left to the CPU
read zero slots, and on the CPU from the host experts, where the experts
computed on the device have id -1. The two results are added before the down
bias.

- ggml-cpu: MUL_MAT_ID skips negative ids and writes zero rows
- sched: the copy callback is called for all the inputs of a split, and before
  the host sync when there are no events
- the slot lookup callback reads the inputs of the CPU part together with the
  selected experts, so the CPU part does not wait for the device, and its
  result is copied back asynchronously
- each token uses distinct zero slots, mul_mat_id does not support an expert
  selected twice by a token
- LLAMA_MOE_CACHE_TRACE writes the selected experts of each lookup, for a
  simulator of the cache

* scripts : add moe-cache-tune.py to pick the MoE cache arguments

The script wraps a llama-server command: it starts the server at two cache
sizes and three CPU shares, fits the upload and CPU speeds of the machine, and
replays the routing of the runs through a copy of the cache logic to predict
other sizes and shares. It prints the recommended --moe-cache-mib and
--moe-cache-cpu-share.

docs/moe-cache.md describes the MoE cache arguments and the script.

* add am17an as codeowner for MoE cache

* moe-cache : upload all the misses until the cache is full

With a high --moe-cache-cpu-share, few misses are uploaded, so an empty cache
fills slowly and more misses than needed go to the CPU. The CPU share now
starts once all the slots of a cache group hold an expert. moe-cache-tune.py
replays the same rule.

* add backend sync for async transfer queue

* moe-cache : let the scheduler copy the results of the CPU part

* Fix MSVC compilation

* remove sync change

* llama-graph : extract the CPU share condition into can_cpu_share

* llama-graph : make llm_graph_params::moe_cache const

The graph construction no longer updates the MoE cache directly: the CPU
parts of the layers are accumulated in the graph result and applied to
the cache in llama_context::process_ubatch, after the graph inputs are
set and before the graph is computed.

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)

* scripts : replay the ClassLRU policy in moe-cache-tune.py

* moe-cache : pick CPU misses by CRF instead of random (#31853)

* moe-cache : remove the backend logic of the CPU part from the graph

The CPU part reads the host experts, so the scheduler places it on the
CPU without set_tensor_backend. The cache gives no CPU map for batches
that a device would offload. Graphs with a CPU part look up their own
slot map, so the graph does not register the part with the cache.

ggml-backend : the inputs of the next split that are already computed
are copied to host memory before the current split runs, so a CPU split
does not wait for the device split before it. This replaces the stash
of the cache, and the copy callback is again called for host weights
only.

* llama-graph : return the MoE cache slots and CPU ids in a struct

The scheduler prefetch no longer checks that the next split runs on
another backend: the inputs of a split are never on its own backend,
so the input loop already skips that case.

---------

Co-authored-by: Gaurav Garg <gaugarg@nvidia.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Co-authored-by: Alexey Kopytko <alexey@kopytko.com>
2026-11-08 00:20:58 +08:00
Harkanwal BediandPascal 0956684529 server: visit parent slots before child slots (#30779)
* server: visit parent slots before child slots
A parent slot wakes its waiting child slots once it finishes the shared
prompt prefix, while slots are visited in id order. When a child task
lands on a lower slot id than its parent, the child is visited before
the parent wakes it and waits an extra batch to process its suffix.
Non-child slots are now visited before active child slots, so every
child woken by its parent joins the same batch.

* server: check systemone test events through test tags

The systemone tests matched free-form debug messages, including the
formatted f_keep value of the prompt cache. The server now emits
__TEST_TAG_WAIT_PARENT_SLOT__ when a child slot waits for its parent,
__TEST_TAG_PROMPT_CACHE_RESTORE__ on every prompt cache restore and
__TEST_TAG_PROMPT_CACHE_FULL_RESTORE__ when the whole cached prompt is
a prefix of the new one, compared on token counts.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-07 13:03:38 +02:00
Adrien Gallouët 468fde90e8 server : simplify windows MCP command resolution (#31728)
* server : simplify windows MCP command resolution

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* common : remove utf8_to_wstring()

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

---------

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-11-05 15:34:54 +01:00
cwriterandcwriter e152332a15 llama-server: Use reference-counted checkpoints instead of deep-copies (#30137)
* checkpoints: reduce overhead

Assisted-by: Codex

* rebase

Adapt the slot save/restore checkpoint appendix from #26004 to the
shared_ptr checkpoint list and the common_state_data buffers.

Assisted-by: Claude Opus 5.5

---------

Co-authored-by: cwriter <cwriter@localhost>
2026-11-05 11:02:36 +01:00
Adrien Gallouët c55b094880 server : reject non-object session updates (#31714)
Prevent malformed realtime session updates from throwing an uncaught
exception and terminating the server

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-11-05 09:38:08 +01:00
Adrien Gallouët 5a081c0a70 server : drop u8path() and use generic_string() (#31682)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-11-05 08:37:13 +01:00
Tom Vaucourt 6f1e654370 server : realtime: send an error event instead of crashing on exceptions (#31632)
* server : realtime: send an error event instead of crashing on exceptions

* server : route websocket handlers through ex_wrapper, remove the realtime test

* server : realtime: keep the previous tools when the chat template rejects new ones

* server : realtime: reject tool schemas the grammar cannot convert, drop ws_ex_wrapper

* server : realtime: clarify the tool schema check comment
2026-11-04 21:34:58 +01:00
Adrien Gallouët 52e53a76df common : delete common_get_path_from_env() (#31444)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-11-04 14:21:38 +01:00
d02b4c0d66 mtmd: Pocket tts - support for voices without voice cloning (#31429)
* feat: support PocketTTS voices with no voice cloning

The kyutai/pocket-tts-without-voice-cloning weights ship predefined voices as the
backbone KV state after the voice prompt. The converter turns each state back into
the prompt embeddings and stores them in the mmproj (voice.<name>), the zeroed voice
encoder is left out, and the "voice" field of /tts and --tts-voice select one of them.
A speaker reference is rejected when the model has no voice encoder.

Assisted-by: Claude Fable 5.1

* convert : pocket-tts voices never fail the conversion, log the drift of the state

The voice states shipped by Kyutai are not always the ones the pack's
weights give for the prompt: italian, german and spanish match within the
bf16 rounding, english_2026-04 is 2% off and french 20% (its layer 0 state
comes from other weights), while the reference loads them as is and they
still sound like the same voice. The converter now re-prompts the backbone
with the recovered embeddings, compares the whole state and logs the
distance per voice, warning above 5%. A state that does not belong to the
weights at all is skipped, so the packs of kyutai/pocket-tts convert too.

Assisted-by: Claude Fable 5.1

* mtmd : list the predefined voices, expose them in GET /props

mtmd_gen_audio_info gains n_voices and mtmd_gen_audio_voice_name() the
names, with the voices worded as a feature of any audio generation model.
The server lists them under "voices" in /props when the model ships some.

Assisted-by: Claude Fable 5.1

* server : an unknown voice is a bad request

A model with predefined voices answers 400 naming the voice instead of 500
"failed to process TTS prompt". A model without them ignores "voice" with a
warning, as the other audio generation models do.

Assisted-by: Claude Fable 5.1

* refactor api

* fix conversion

* small renaming

* fix lint

* tts : a model with predefined voices speaks with the first one by default

Without a voice prompt such a model only produces near silence. When neither
a voice nor a speaker reference is given, llama-tts and the server now take
the first predefined voice and say so in the log, as the LFM3-Audio template
does with its default voice.

Assisted-by: Claude Fable 5.1

* mtmd : one comment per voice accessor, converter states the rule only

The clip.h comment described clip_gen_get_voice but sat above all three
functions; the converter comment quoted the drift measured per pack, which
goes stale with the next upload of the weights.

Assisted-by: Claude Fable 5.1

* clean up redundant code and comments

* mtmd: match the LFM3-Audio voice name case-insensitively

The tts chat template lowers the voice name, so Male speaks with
the male voice; the voice lookup lowers it the same way and names the
bitmap after the canonical voice.

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-02 18:39:51 +01:00
Sigbjørn Skjæret fcfe4b59df requirements : minor adjustments for python 3.14 wheels (#31534) 2026-11-02 11:34:45 +01:00
aiqubits f3586e66bc server: cap shared decision prefixes at the first output (#30710)
Limit the shared prefix to the first required output position of the parent and each child task.
This keeps decision output embeddings available in the final batch when questions share option tokens.

Fixes #30482

Signed-off-by: aiqubits <aiqubits@hotmail.com>
2026-10-31 19:09:48 +01:00
David Garcia 2871678a1d server : fix /v1/responses endpoint to accept extra_params (#30959) 2026-10-31 19:08:08 +01:00
Adrien Gallouët a34de8618b common : delete fs_path_to_utf8() (#31423)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-10-30 16:20:04 +01:00
Adrien Gallouët d63fbfe794 cmake : embed an app manifest in all Windows exes (#31329)
* cmake : embed an app manifest in all Windows exes

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Remove LLAMA_WIN_APP_CMAKE

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

---------

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-10-30 10:56:53 +01:00
7konigsbergandZeshawn Zahid 7df8b876b9 server : restore the spec checkpoint for drafts longer than n_rs_seq (#31350)
Assisted-by: Claude

Co-authored-by: Zeshawn Zahid <zzshawnn@gmail.com>
2026-10-29 11:28:03 +02:00
Pascal e09a78a25d tests: fix CI flakes in server cancel and sleep tests (#31330)
test_cancel_request enables context shift so the generation only ends
when the request is cancelled, then polls /slots with a deadline.
Without context shift the generation fills the context and frees the
slot on its own on fast machines, so the test passes even with
cancellation disabled.

test_server_sleep uses the existing wait_for_sleep helper instead of a
fixed sleep.
2026-10-29 01:35:01 +01:00