Commit Graph
1632 Commits
Author SHA1 Message Date
LumpiastyandClaude Opus 5.5 fcfadfbbd2 server: fix stale VRAM-arbiter comments and the sleep log line
Comments only, plus one log string:
- the arbiter block comment said the KV cache is never evicted; with
  LLAMA_SLEEP_EVICT_KV it is (and the scheduler and scratch with it)
- vram_go_cold said vram_ensure_warm() rebuilds the encoder; only
  encode_mtmd_chunk() restores it, and only for an encode
- vram_ensure_warm and the mtmd init block named process_mtmd_chunk; the
  restore lives in encode_mtmd_chunk
- handle_sleeping_state said waking happens lazily before decode; it
  restores eagerly on the wake transition, before update_slots()
- the vram_cold ordering note now says why it still holds with the inline
  want-lock probe
- the sleep log line said "keeping KV cache" even when LLAMA_SLEEP_EVICT_KV
  evicts it; it now names what actually happens (the prefix
  "server entering sleeping state" that fork-validate.sh counts is kept)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-22 20:38:32 +01:00
LumpiastyandClaude Opus 5.5 83cc51c9da server: release the GPU token promptly on shutdown
Two ways a shutting-down server kept the VRAM arbiter's token, stalling every
other model:

- SIGTERM with a request in flight: terminate() stopped only the task queue.
  The HTTP handler kept waiting for results the loop would never send, the main
  thread blocked joining it, and the token was released only in the destructor
  after that join - i.e. never, until the client went away. Terminate the
  result queue too (readers now get "server is shutting down": a 503, or an SSE
  error event), and let wait_until_no_sleep give up once stopping.

- SIGTERM while waiting for the token: the loop thread sat in a blocking flock
  inside the wake callback, holding mutex_tasks, so terminate() waited for the
  current holder to finish and then for a pointless restore of our weights.
  Poll the lock non-blocking and abort the wait on shutdown; the aborted wake
  stays cold, skips the restore, and the loop exits before update_slots.

Also release the token right after start_loop() returns rather than in the
destructor, and only after the device buffers are actually freed: the old
order unlocked before destroy(), so the next model could start uploading into
VRAM that was still occupied. Freeing at exit uses a new
llama_context_discard_device(), which drops weights, KV, MoE cache and scratch
without capturing host shadows first.

Off production: SIGTERM mid-generation or mid-wait now exits in ~0.2 s with
exit 0 and frees the token within 5-70 ms (previously never / after the
holder's whole generation). Handoff latency and greedy output are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-22 20:38:32 +01:00
LumpiastyandClaude Opus 5.5 33f8b89f9b server: run the shutdown handler off the signal-interrupted thread
A SIGTERM that lands while the start_loop() thread is inside a sleeping
callback deadlocks the server. The callbacks run under
server_queue::mutex_tasks, and the kernel delivers the signal to that same
thread; the handler calls terminate(), which locks mutex_tasks again, and the
thread waits on itself forever.

With the VRAM arbiter that window is wide - a whole device release, or a
blocking wait for the GPU token - and llama-swap sends SIGTERM to evict
models, so it hit production: an evicted model wedged mid-release, never
released token.lock, and every other model queued behind it. SIGTERM could
not stop it either; only SIGKILL did.

Make the handler only write the signal number to a self-pipe, which is
async-signal-safe, and run shutdown_handler on a dedicated thread. terminate()
there simply waits until the loop thread drops mutex_tasks, after which the
loop exits normally. The thread is joined before the objects the handler uses
are destroyed. If pipe() fails the old direct call is kept.

Reproduced 6/6 off production (SIGTERM during release and during the token
wait); with the fix 8/8 clean exits, and handoffs, image requests and greedy
output are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-22 20:38:32 +01:00
LumpiastyandClaude Opus 5 aa1c2b4562 server: replace the VRAM doorbell with a want-lock
The doorbell was edge-triggered: a waiter wrote <arena>/doorbell/<pid> and the
holder's warden turned the inotify event into a queue flag. A ring carried no
notion of "still wanted", so it was lost whenever the holder was not already warm
and listening - during its model load (the watch does not exist yet, and the
kernel does not queue events for a watch that is not there) or inside
restore_device - and any ring that survived past the point its sender had been
satisfied caused a spurious release. Lost rings wedged the waiter permanently,
because flock(LOCK_EX) never times out and, with idle-sleep disabled, the holder
had no other reason to release. The wait blocks under mutex_tasks, so the whole
server stopped answering while /health still returned 200.

Express the request as kernel state instead. A waiter holds <arena>/want.lock
shared while it waits and drops it once it owns the token; the sleep decision
probes that lock non-blocking and releases the GPU while anyone is waiting. The
probe needs its own fd - flock treats two open file descriptions of one file
independently, so probing on the waiter's fd would convert our own lock rather
than conflict with it. Nothing can be missed, nothing goes stale, and a waiter
that dies is cleaned up by the kernel.

Probe from should_sleep() on the loop thread rather than from a warden thread.
Routing it through a flag is what made the first attempts fail: start_loop()
holds mutex_tasks from should_sleep() through the callbacks to the wait, so a
warden's request_yield() blocks on that mutex and is admitted only after the flag
has been consumed, latching a release for the next wake. Reading live state where
the decision is made has no edge to latch, and drops the warden's poll latency.

Two sleep-path bugs this exposed: wait_until_no_sleep() waited on !sleeping but
the loop clears req_stop_sleeping on the way in, so a loop that slept again
before the waiter ran stranded it forever - re-ask on every wake. And a task
queued after the waiter saw us awake could not wake us by itself, so sleep now
also breaks on a non-empty queue. Hold the GPU for 100 ms after a wake: the
request that woke us is not queued yet, and yielding at once only sends it round
again.

Measured on the RX 580 pod, two servers contending, 60 alternating handoffs:
0 stranded, median 0.309 s against the doorbell's 0.314 s.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 20:38:32 +01:00
Lumpiasty 3f61190fde server: on-demand mmproj - free encoder VRAM on the text path
The vision encoder sits idle in VRAM on every text request. With
LLAMA_MMPROJ_ONDEMAND=1 the server keeps the encoder in RAM (released
after load) and brings it into VRAM just-in-time before mtmd_batch_encode,
releasing it again afterwards - so the encoder's VRAM is free for KV /
expert cache on the common text-only path.

When the encoder does not fit (that VRAM has been claimed), evict the LLM
backbone weights first: they are not needed while the encoder runs (the
decode that uses them happens afterwards), their host shadow is read-only
(cheap free, no D2H), and the KV / prompt cache is left untouched. Order
on the way back matters: free the encoder before re-uploading the weights
so the peak stays within VRAM.

Also wire the encoder release/restore into the VRAM arbiter: vram_go_cold
releases it; the just-in-time encode-path restore replaces the previous
restore in vram_ensure_warm.

Assisted-by: Claude
2026-11-22 20:38:32 +01:00
Lumpiasty f5c6f04b4e mtmd: release/restore the encoder weights from VRAM on demand
Add clip_release_device/clip_restore_device (and mtmd_release_device/
mtmd_restore_device wrappers over the vision + audio contexts) that free
the multimodal encoder's device weight buffer to a read-only host shadow
and rebuild it on demand, using the same shadow/free/reallocate pattern
as llama_model weights. No-op for a CPU-backed encoder. This lets the
server drop the ~hundreds-of-MiB vision encoder from VRAM when it is not
encoding an image.

Assisted-by: Claude
2026-11-22 20:38:32 +01:00
Lumpiasty e457c2fd63 server: restore weights/KV on wake, not at decode, so KV eviction is safe
With LLAMA_SLEEP_EVICT_KV the KV cache device buffers are freed when a model
goes cold. update_slots() touches the KV before the decode-time vram_ensure_warm
(e.g. SWA models create a checkpoint that reads the KV via ggml_backend_tensor_get),
so the KV must already be resident by then. Move the restore to the sleep-wake
handler (handle_sleeping_state(false)), which runs before update_slots, fixing a
GGML_ASSERT(buffer) crash on iSWA models (gemma) when KV eviction is enabled.

Assisted-by: Claude
2026-11-22 20:38:31 +01:00
Lumpiasty b9c57d1c2e server: coordinate model load with the VRAM arbiter to avoid load-time OOM
Before uploading a model's weights, acquire the shared VRAM token (ring the
doorbell so any resident model releases first). Previously load uploaded weights
to VRAM before the arbiter was active, so loading a large model while another
(e.g. the warm 4B task model) held VRAM could exceed the budget and OOM.

- add vram_arena_open() (idempotent flock/doorbell setup) and
  vram_acquire_for_load(), called from load_model() before
  common_init_from_params().
- a coordinated load now stays warm holding the token and serves its first
  request without a re-warm (drop the init-time go_cold cold-start).

Assisted-by: Claude
2026-11-22 20:38:31 +01:00
Lumpiasty da03d4c1f2 server: keep the sleep-wake path active when the VRAM arbiter is enabled
The request handler only calls wait_until_no_sleep() (which wakes a server out
of its sleeping state) when sleep_idle_seconds >= 0. But the VRAM arbiter's
cross-process doorbell can put a server to sleep even when idle-sleep is
disabled, so without this a doorbell-slept server would never wake and requests
to it would hang until timeout.

Do not bypass the wake path when LLAMA_SLEEP_VRAM_ONLY is set, so the arbiter no
longer depends on --sleep-idle-seconds being configured.

Assisted-by: Claude
2026-11-22 20:38:31 +01:00
Lumpiasty 3b410b905c server: optionally evict the KV cache too (Phase 2 of VRAM sharing)
Extend on-demand device residency to the KV cache so that when a model's KV
plus another model would not fit in VRAM, the KV can also be evicted to a host
shadow (D2H on release, H2D on restore) instead of only the weights.

- llama_memory_i: add release_device_buffers()/restore_device_buffers()
  (default no-op). Implemented in llama_kv_cache (D2H shadow of the live
  ctxs_bufs, freed and reallocated like the weights); llama_memory_hybrid and
  llama_kv_cache_iswa delegate to their child caches.
- llama_context::release_device(evict_kv): also evict the memory's device
  buffers when requested; restore_device() rebuilds them. Public API
  llama_context_release_device gains an evict_kv flag.
- server: LLAMA_SLEEP_EVICT_KV=1 enables it. Off by default (weights-only),
  since the KV shadow adds a D2H/H2D copy of the live cache each cycle.

Validated on RX 580 (Vulkan), 4B @ 32k ctx: weights-only cold VRAM 1750 MB
(KV stays); weights+KV cold VRAM 726 MB (KV freed, ~1 GB reclaimed). KV
survives the round-trip: prompt cache reused after the cycle (prompt_n 4 vs
42), correct output.

Assisted-by: Claude
2026-11-22 20:38:31 +01:00
Lumpiasty cb489f3367 server: on-demand VRAM sharing to time-share one GPU between models
Add release/restore of a model's GPU weight buffers (keeping a host shadow
and the KV cache) so several always-loaded llama-server processes can
time-share a single GPU without reloading or losing the prompt cache.

- llama-model: release_device_weights()/restore_device_weights() capture a
  compact host shadow (stable iteration order, view-skipping) and free then
  realloc the device weight buffers; weights_resident() query.
- llama-context: release_device()/restore_device() wrappers; decode() auto-
  restores; public C API llama_context_release_device/restore_device.
- server: LLAMA_SLEEP_VRAM_ONLY makes idle-sleep release only the VRAM weights
  (not a full unload/reload). A cross-process flock token in LLAMA_VRAM_ARENA
  enforces "resident iff holds token"; an inotify doorbell forces the holder
  to release on contention. The warden thread only touches the task queue, so
  releases run on the loop thread and never race a decode.

Validated on RX 580 (Vulkan): two models share 8GB, never both resident,
correct output under contention, KV cache preserved (no re-prefill).

Assisted-by: Claude
2026-11-22 20:38:31 +01:00
Adrien Gallouët 56eea9e4b0 server : add offset pagination to grep_search tool (#32792)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-11-22 19:00:05 +01:00
Xuan-Son Nguyen 0b51bb2fed server: support /models/sse on single-model mode (#32398)
* server: support /models/sse on single-model mode

* nits

* nits
2026-11-22 17:52:56 +01:00
Aman Gupta 225a5345bb server : fit non-causal batches in one ubatch (#32713) 2026-11-22 11:25:13 +08:00
Aldehir Rojas afe77e18fa chat : pass reasoning_effort = none if supported (#32763) 2026-11-21 15:27:25 -06:00
saiph e6d624b9a4 mtmd : merge Parakeet rel-pos views to avoid OOB (#32722)
Assisted-by: Grok Bot
2026-11-21 19:05:02 +01:00
Xuan-Son NguyenandPascal c622f623fc mtmd: support kyutai/hibiki realtime full-duplex (#32491)
* mtmd: support hibiki realtime duplex

* fix and clean up

* fix long session quality

* fix ci

* tests : add the moshi fusion baseline for Metal

The Metal fusion check found 5 RMS_NORM+MUL fusions in the new moshi
arch and no baseline for them.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-21 14:25:19 +01:00
Xuan-Son Nguyen 40b7a546de ci: temporary disable no realloc case for mtmd (#32708) 2026-11-21 10:02:26 +02:00
Xuan-Son NguyenandPascal 9197945004 convert: qwen3-asr: put "language ..." output into thinking block (#32571)
* convert: qwen3-asr: put "language ..." output into thinking block

* convert: qwen3-asr: read the chat template from models/templates

The converter embedded its own copy of Qwen3-ASR.jinja. Read the
bundled file instead, as the Kimi-K3 and Kokoro converters do, so
the GGUF and test-chat share a single template.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-20 20:54:57 +01:00
Xuan-Son Nguyen 98709fd82b server: add more decision tests (#32578)
* server: add more decision tests

* fix ci

* also test audio
2026-11-20 18:45:31 +01:00
FeitengandXuan Son Nguyen 34ff2e047c mtmd: drama3 voice cloning (#32586)
* mtmd: drama3 voice cloning

The Fish S2 codec encoder (fish_s2_spkenc) now also serves drama3:
- conversion: drama3 writes the codec encoder and the audio encoder KV, shared with fish_s2
- fish_s2_spkenc: outputs the codes of the clip (gen-side reference) and, in a drama3_gen mmproj,
  builds the drama3 backbone rows (E_lm[semantic] + E_cb[1][code 1]) / sqrt(2)
- drama3_gen tower: teacher forcing when the caller gives the codes of the next frame
- drama3 pipeline: reference-bracket-v1 prompt with one reference (vLLM-Omni fish_omni), the
  reference frames go through the backbone as codec rows and through the tower with their codes
- mtmd_gen_state_get_ref_codes(): read-only access to the reference codes of a gen state

Claude-Session: https://claude.ai/code/session_01PpX3DmaxNSmJbxGaxnUBrX

* fix correctness

* clean up

* rm redundant code

* fish_s2: rm unused import

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-11-20 18:34:52 +01:00
yoch 8e7862411d tts : forward -t to the mtmd context (#32664)
llama-tts built the mtmd context from mtmd_context_params_default(), which hardcodes 4 threads, and never copied n_threads over, so -t never reach the mtmd side.
Copy params.cpuparams.n_threads into mtmd_params.n_threads, as mtmd-cli and the server do; with no -t this now follows the resolved default.

Verified: -t 1 now runs single-threaded, -t 4 is unchanged, and the produced WAV is identical.

Assisted-by: Claude
2026-11-20 15:34:58 +01:00
Xuan-Son Nguyen 5f837145d8 server: add more audio tests (#32579) 2026-11-20 14:21:58 +01:00
Xuan-Son Nguyen 36cd3aff40 server: add --ui-mcp-proxy-allow-private-urls (#32633) 2026-11-20 14:10:06 +01:00
JamePeng 53ecea05a0 mtmd : fix YuE2 projector loading (#32606)
- Allow cls.output.weight to be absent from the mmproj. It is an optional
Qwen3 reranking head that the YuE2 NAR decoder never uses.

- Without this fix, mtmd_init_from_file() fails with
"tensor cls.output.weight is missing from the mmproj", preventing
audio generation from starting.
2026-11-19 19:11:11 +01:00
Bhargav Krish c7468107ef cuda: Scatter kernel for set_n (#32489)
* cuda: add scatter for batched tensor writes

* update test case to mirror get_n test case

* save restore timing logs

* refactor code into batch-copy.cu file
2026-11-19 20:14:47 +05:30
Xuan-Son Nguyen e4b620af19 mtmd: support voxcpm2 (#32443)
* mtmd: support voxcpm

* speed up a bit

* various fixes
2026-11-19 02:39:05 +01:00
Xuan-Son Nguyen b044890408 server: fix sleep state for v1/realtime (#32567) 2026-11-18 23:48:02 +01:00
Tarek Dakhran bafa0420c1 server : enable packing for d1-2B and d1-Omni-2B models (#32553)
* llama-bench : size the batch to the prompt for models without memory

Assisted-by: Claude Opus 5.5

* server : read up to 16 lfm3-d1 questions in one pass

Assisted-by: Claude Opus 5.5
2026-11-18 20:15:43 +01:00
Xuan-Son Nguyen 48a32194ee server: fix transcription prompt for v1/realtime (#32507) 2026-11-18 12:09:22 +01:00
Xuan-Son NguyenandPascal 52975342ff server: slot_sched: fix lora and recurrent (#32434)
* server: fix lora and recurrent

* tests: run the hybrid SWA checkpoint test with one token per ubatch

With and without the cache, the prompt was not split into the same
ubatches, and the generated tokens are decoded one at a time while the
uncached run processes them in a batch. On CUDA this moves the top
logprobs of the tiny model by up to 0.03 and swaps near ties, so the
test failed while the cache behaved as expected.

With one token per ubatch both runs go through the same kernels and
the output is compared exactly.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-18 00:12:20 +01:00
Xuan-Son Nguyen 3574df84b7 mtmd: support tiny-voca (#32427)
* mtmd: support tiny-voca

* improve G2P
2026-11-18 00:04:15 +01:00
Xuan-Son Nguyen 307e3f1551 mtmd: add kokoro tts support (#32413)
* jinja: impl scope-chain key lookup

* add test for call_statement lambda

* chat: add tts_dict for kokoro G2P template

* more testing

* value_float_t range check

* no json-in-meta, add llama_model_meta_arr_str/_n

* mtmd: add kokoro tts support

* detailed guide about G2P and text norm port

* refine G2P

* clean up

* migrate batch api

* nits
2026-11-17 22:25:44 +01:00
James PrevettandXuan Son Nguyen 8c16babba5 mtmd: Add support for Qwen3-TTS' ICL/reference mode (#32438)
* Added support for Qwen3-TTS ICL/reference mode

Assisted-By: Claude Opus 5.5

* fix correctness

* attach transcription to bitmap

* nits

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-11-17 18:59:11 +01:00
eddiechen 955c4e8ee5 server : return 400 when grammar and ignore_eos are on (#32377)
* sampling : fix crash when both ignore_eos and grammar are on

* server : rework, throw 400 when grammar and ignore_eos are on

* sampling : move the check into the sampler (review suggestion)

* tests : remove redundant tests
2026-11-17 17:03:21 +01:00
Xuan-Son Nguyen d92ea9bb12 mtmd: support Minimax-Music3 (#32435)
* mtmd: support Minimax-Music3

* fix and clean up
2026-11-17 12:53:19 +01:00
Xuan-Son Nguyen 8d1599fca1 chat: add tts_dict for kokoro G2P template (#32397)
* chat: add tts_dict for kokoro G2P template

* more testing

* value_float_t range check

* no json-in-meta, add llama_model_meta_arr_str/_n
2026-11-17 12:52:50 +01:00
Aleksander Grygier 67cf8967eb ui: move Llama Chat to SvelteKit 3 and its built-in service worker (#32463)
* ui: replace deprecated packages and fix npm audit findings

Swap the mdast stub for @types/mdast, move the katex override to 0.19.0,
and override glob, source-map and html-encoding-sniffer under
workbox-build and http-server so installs no longer pull deprecated
packages. Apply the non-breaking npm audit fixes to the lockfile.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: migrate Llama Chat to SvelteKit 3 and its built-in service worker

Move to SvelteKit 3, Vite 8, TypeScript 6 and Storybook 10.6. The
SvelteKit options move from svelte.config.js into sveltekit() in
vite.config.ts, $lib imports become #lib subpath imports from
package.json, and $app/environment becomes $app/env. Modules named
server.* or living under a server/ folder are server-only in SvelteKit 3,
so the llama-server components, store and enums move to llama-server
names.

@vite-pwa/sveltekit and Workbox give way to SvelteKit's own service
worker in src/service-worker, registered by usePwa(), which precaches the
app files and leaves API requests to the network. A small plugin emits
manifest.webmanifest. static/sw.js removes the worker of older builds, and
llama-server serves a self-removing service-worker.js as well as sw.js
when the built-in UI is off or replaced.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: use SvelteKit 3's update detection and Svelte 5's current patterns

A page the service worker doesn't control now learns of a new build
from SvelteKit itself: `updated` from $app/state checks
_app/version.json on navigation, on focus and every hour. It replaces
the version Llama Chat kept in localStorage and compared by hand.
Reload on the update prompt pulls in the new worker and activates it
before reloading, so one click lands on the new build, whether or not
a worker controlled the page.

Following Svelte's current guidance:
- usePwa keeps its state in $state rather than a store, and no effect
  checks `browser`
- nearViewport, autoResizeTextarea and the custom CSS injection are
  attachments ({@attach}) rather than actions (use:)
- class lists use arrays rather than class: directives
- the chat contexts come from createContext, typed, in place of
  setContext/getContext with string keys
- ActionIconCopyToClipboard takes $props() rather than `export let`

Assisted-by: Claude (Opus 5.5 Medium)

* ui: move to ESLint 10

ESLint 9 is out of support. Every plugin the app uses supports 10:
typescript-eslint, eslint-plugin-svelte, perfectionist,
simple-import-sort, eslint-plugin-storybook and eslint-config-prettier.
@eslint/js and @eslint/compat follow.

ESLint 10's recommended rules ask for two things the code now does:
errors rethrown from a catch keep the caught error as their `cause`,
and no variable starts with a value that every path overwrites.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: override minimatch to clear npm audit's last finding

ESLint 10.14, @eslint/config-array 0.23.5 and typescript-eslint 8.72
depend on minimatch 10, which GHSA-g3hv-75wg-m249 (denial of service
via globparts amplification) affects below 11.0.1. Their fixed releases
move to minimatch 11 but are all younger than the repo's 7-day
min-release-age, so an override pins minimatch 11.0.5, the newest the
rule allows. The three only call new Minimatch(), match() and
minimatch(), which minimatch 11 keeps. npm audit now finds nothing.

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-17 11:43:36 +01:00
Sidhartha Reddy Potu 9581154a22 mtmd : fix Muse Glimmer vision embedding order (#32433)
Assisted-by: Claude
2026-11-17 00:29:02 +01:00
Xuan-Son NguyenandPascal 92192d31e4 mtmd: support YuE2-3B (#32369)
* mtmd: support YuE2-3B

* improve speed

* add CFG

* fix streaming

* add cfg control from api

* check ctx size

* fix n_predict

* add flow_steps config

* nits

* fix batch api

* nits

* adapt

* mtmd: yue2 samples from its own saved logits

The context logits belong to the last decode of any slot, so with
several server slots the score and the unguided codes were sampled
from another sequence. Keep the logits of each own decode in
save_logits(), which also does the guidance mix.

* mtmd: let the yue2 snake chain fuse

The beta reshape landed between sqr and mul, so the mul, sin, sqr,
mul, add chain was not consecutive and the backends never fused it.
Expand both reshapes first, like qwen3tts does.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-16 20:22:23 +01:00
Xuan-Son Nguyen a68c6136bb mtmd: support fish audio s2 (#32330)
* add fisk-audio-s2 support

* add voice cloning

* cont merge master

* nits

* nits

* clean up

* nits

* use new batch api
2026-11-16 17:32:27 +01:00
Aleksander Grygier 3a7660371d ui: move the web UI to ui/apps/chat and call it Llama Chat (#32032)
* ui: move tools/ui to ui/apps/chat

The chat app moves to its own root folder, ui/apps/chat, as the first step of splitting the UI into one library
(ui/lib) and the apps built on it (ui/apps). The code is unchanged; every path that pointed at tools/ui now points at
ui/apps/chat: CMake adds it from tools/, and the Dockerfiles, the Nix package, the workflows, the labeler, CODEOWNERS,
.editorconfig, the git hooks, the dev and demo scripts and the docs follow it.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: name Llama Chat's package @ggml-org/llama-chat

It was still called llama-ui, the name of the library that now sits beside it
in ui/lib (@ggml-org/llama-ui). The package stays private; the app's display name
and the llama-ui.zip build artifact are unchanged.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: call the built-in web UI Llama Chat in docs, help text and comments

The app in tools/ui is Llama Chat; Llama UI is the component library it is
built from. Prose that said "Web UI", "web UI" or "WebUI" for the app now
names it: the READMEs, SECURITY.md, the --ui help text, the CMake option
description, one server log line, code comments and the Storybook intro.

Names that others depend on are unchanged: the deprecated --webui flag
aliases, the LLAMA_*_WEBUI CMake and environment names, the webui_settings
field, the storage migration keys and the Nix attributes.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: name tools/ui as a former path of Llama Chat in the code-review skill

Changes under the old tools/ui path, as well as tools/server/webui, still
get the Llama Chat checklist.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: name Llama Chat's path in the security policy's exclusions

Llama Chat no longer lives under tools/server/, so the exclusion listed
under it now names ui/apps/chat. What is covered stays the same.

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-16 17:23:36 +01:00
Shubham RoyandDeveloper e8698ac163 fix:- Misc. bug: Settings/Tools is lying about tool picks inside a ch… (#32223)
* fix:- Misc. bug: Settings/Tools is lying about tool picks inside a chat #32214

* checks run lint error

---------

Co-authored-by: Developer <dev@example.com>
2026-11-16 17:15:16 +01:00
Niklas Wenzel 62a6d3a302 llama : migrate to llama_batch_ext (#32356)
* llama : migrate to llama_batch_ext

* mtmd : migrate zonos3-mini code

Added in https://github.com/ggml-org/llama.cpp/pull/32020
2026-11-16 15:16:12 +01:00
Ruixiang Wang 6b01a4767f CI: trigger spec dec testing for relevant PRs (#32281)
* ci: trigger spec dec testing after refactoring

* add options to run spec dec ci nightly manually
2026-11-16 11:14:07 +01:00
Pascal a946eb541a server: answer reasoning_effort levels the template rejects with 400 (#32332)
The reasoning effort probe already renders every candidate level, but
it dropped the ones the template raises on, so a request with such a
level (e.g. max on Qwen3.8) reached the template and came back as a
500 on /v1/chat/completions, /v1/responses and /v1/messages.

Keep the raising levels in reasoning_effort_rejected and reject them
in oaicompat_chat_params_parse with an invalid_argument, which the
server answers with a 400 listing the supported levels. Aliases and
levels a template ignores are not rejected, so high on Qwen3.8
still renders as xhigh.

Also trim the output_config.effort test to a single level.
2026-11-16 06:55:56 +01:00
Evan Huus 1719dbb0ee server : skip MCP discovery for router children (#32359) 2026-11-15 23:55:33 +01:00
Xuan-Son Nguyen fc504f91d8 mtmd: support higgs v3 STT and vibevoice-ASR (#32329)
* mtmd: support higgs v3 and vibevoice-ASR

* use proper mapping

* address review comment

* nits

* rm hacky get_audio_length, add mtmd_helper_get_media_meta
2026-11-15 21:25:34 +01:00
Xuan-Son Nguyen 7ae3c6b231 mtmd: add zonos3-mini TTS model (#32020)
* zonos3 init

* add audio gen pipeline

* fix string_format

* add default voice as gguf metadata

* better ref

* support voice clone

* add delivery params

* optimize speed

* nits

* nits 2

* add text normalization

* nits
2026-11-15 19:44:44 +01:00
Xuan-Son Nguyen 75fd7f9ec9 docs: move per-model TTS to /docs, refer it from other pages (#32304) 2026-11-15 18:35:27 +01:00