Commit Graph
1494 Commits
Author SHA1 Message Date
Aleksander Grygier d1c4e5bec1 ui: run local browser tests in installed Chrome with its sandbox on (#31596)
Playwright's bundled Chromium lags behind Chrome security releases
(1.56.1 ships 141, and even 1.64.1 ships 156.0.8078.4, before the
156.0.8078.73 fixes), and Playwright launches it with --no-sandbox by
default. Outside CI, use the installed, auto-updated Chrome with the
sandbox enabled. CI keeps the bundled Chromium, so the self-hosted
runner's preinstalled 1.56.1 browsers still work.

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-03 19:02:54 +01:00
Prabhsimran Singh 28b812d6cb mtmd : add Nemotron 3.5 Transcribe support (#31471)
* mtmd : add Nemotron 3.5 Transcribe support

Assisted-by: Claude

* convert : fix vocab of draft models for Nemotron-H targets

Assisted-by: Claude

* convert : reject non-BF16 Transcribe checkpoints

Assisted-by: Claude

* mtmd : make speaker FIFO predictions local

Assisted-by: Claude

* mtmd : normalize the mel in encode_windows

Assisted-by: Claude

* convert : map the generated speaker kernel

Assisted-by: Claude

* mtmd : validate Nemotron Transcribe mmproj shapes

Assisted-by: Claude
2026-11-03 14:27:05 +01:00
Aleksander Grygier 0943ca5472 ui: add Playwright harness for recording demo videos (#31318)
* ui: add Playwright harness for recording demo videos

Scripted walkthroughs under tools/ui/demo are recorded with Playwright and
exported as H.264 mp4 for pull requests (demo/pr) and docs (demo/docs).
Fixture MCP servers and seeded settings keep recordings deterministic, a
drawn cursor and captions make the flow readable, and DEMO_UI_DIR points
the recording at another checkout such as a PR worktree.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: record demo scripts from PR descriptions and a configurable storage

Demo scripts no longer live in the repo. `npm run demo:pr -- <number>`
records the `ts demo` block from a PR description against the PR head,
uploads the mp4, a GIF preview and the script to DEMO_STORAGE, and links
the preview in a `## Demo` section of the PR description.
`npm run demo:docs` records the docs scripts kept in DEMO_STORAGE.
DEMO_STORAGE (hf:// buckets or file://) and the public URL are read
from tools/ui/.env.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: import generated conversations into demo recordings

Demos can now start with existing chats: `test.use({ traces })` imports
llama.app JSONL traces through Settings > Import/Export before the video
starts. Built-in traces are generated deterministically, other names are
downloaded from DEMO_TRACES_DATASET, and `demo:traces` pushes the built-in
ones there. Adds `demo.scroll` for wheel scrolling.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: keep demo captions clear of the chat form and isolate concurrent runs

Captions move to the top of the frame, since at the bottom they covered
the chat form a demo is often about. Each recording run gets its own
Playwright output folder, so a run starting elsewhere no longer clears
another run's video before it is saved.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: record demos at 2x and 60 fps with human-like timing

Recording goes through Chromium's screencast instead of Playwright's
video, which encoded VP8 at about 1 Mbit/s at 1x. Frames are captured at
the device pixel ratio (DEMO_SCALE, default 2) and encoded into a 60 fps
H.264 mp4, starting once the page has settled.

The cursor now glides faster along a slight arc with Fitts-like
durations, settles briefly before each click, and typing and pauses vary
a little. The randomness is seeded per demo, so re-recording gives the
same motion.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: document the demo harness prerequisites

The demo README now lists the tools the harness needs and what each is
for, and walks through creating the Hugging Face bucket and the optional
traces dataset, with the matching .env settings.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: version linked demo URLs so re-recorded previews refresh

GitHub's image proxy caches a preview by URL, so a demo re-recorded at
the same PR head kept showing the old GIF. The Demo section now links
the files with a content hash in the query string.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: label PRs that feature a video demo

Adds a `video demo` label and a workflow that applies it when a PR
description has a demo written by the UI demo pipeline (the managed
demo:start/demo:end markers, or a preview linking to a demo video),
and removes it when that section is gone.

Assisted-by: Claude (Opus 5.5 Medium)

* ci: label video demos with gh instead of github-script

The label workflow now runs the gh CLI in a shell step, like
pr-draft-label.yml, so it needs no action outside the repository's
allowed list. The video link check only counts videos uploaded for the
PR itself, so a description linking other PRs' demos isn't labeled.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: zoom demos in on details with a two-finger pinch

demo.zoom pinches in on a control or a point, holds, and pinches back out
to 1x, so a detail too small to read in the frame can be shown up close.
It drives Chromium's own pinch gesture, so the page's fixed bars stay put,
and works at desktop and phone sizes alike.

The pinch is evened out where Chromium misses: one anchored near the top
does not zoom at all, so it starts lower, and one near the bottom
overshoots its ratio, so a small pinch from the middle corrects it. The
way out reads the real scale until the page is back at 1x, which the next
gesture needs.

The overlay also draws a fingertip per touch, so a pinch on a touch
device shows as two fingers.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: wait out a cold dev server before a demo starts

A fresh checkout's dev server optimizes its dependencies on the first
load and reloads the page, which can take well past an action's timeout,
so a demo failed before it started. The demo now waits up to 90 seconds
for the app; the recording only starts once it is ready.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: record a video per test in a PR demo, shown in a table

A large PR can split its demo into several shorter videos: each test()
in the ts demo block is recorded as its own video (<sha>-1.mp4, ...),
and the Demo section lays their previews out in a table, each column
headed by its test's title.

GIF previews now keep their longer side at DEMO_PREVIEW_WIDTH and shrink
until they are under 5 MB, since GitHub's image proxy refuses anything
larger and shows a broken image instead.

Importing traces works on a phone viewport too, where Settings sits in
the sidebar under the page until the menu key slides it aside.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: fix demo trace imports on desktop and lay landscape demos out two to a row

Importing traces failed on a desktop whose collapsed sidebar sits under
the page: it now expands the sidebar from the page's menu key on a phone
or from the sidebar's logo on a desktop, whichever makes Settings
reachable. Afterwards it forgets the expanded sidebar, so a demo starts
with it collapsed, as on a first visit.

A PR demo with several videos shows up to four portrait (phone) videos
to a row, or two landscape ones, so each preview stays readable.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: only record other people's PR demos at a reviewed head commit

demo:pr runs the PR's code and demo script on this machine, with the
gh and hf logins, so a PR by someone else now needs --trust <sha>
naming the head commit that was read, and is refused again after new
pushes. Recordings also launch Chromium with its sandbox on, which
Playwright turns off by default.

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-03 12:30:49 +01:00
Nidhi 19e8a86fe8 tools : reject mismatched shards in GGUF split merge (#31579) 2026-11-03 11:11:05 +02:00
d02b4c0d66 mtmd: Pocket tts - support for voices without voice cloning (#31429)
* feat: support PocketTTS voices with no voice cloning

The kyutai/pocket-tts-without-voice-cloning weights ship predefined voices as the
backbone KV state after the voice prompt. The converter turns each state back into
the prompt embeddings and stores them in the mmproj (voice.<name>), the zeroed voice
encoder is left out, and the "voice" field of /tts and --tts-voice select one of them.
A speaker reference is rejected when the model has no voice encoder.

Assisted-by: Claude Fable 5.1

* convert : pocket-tts voices never fail the conversion, log the drift of the state

The voice states shipped by Kyutai are not always the ones the pack's
weights give for the prompt: italian, german and spanish match within the
bf16 rounding, english_2026-04 is 2% off and french 20% (its layer 0 state
comes from other weights), while the reference loads them as is and they
still sound like the same voice. The converter now re-prompts the backbone
with the recovered embeddings, compares the whole state and logs the
distance per voice, warning above 5%. A state that does not belong to the
weights at all is skipped, so the packs of kyutai/pocket-tts convert too.

Assisted-by: Claude Fable 5.1

* mtmd : list the predefined voices, expose them in GET /props

mtmd_gen_audio_info gains n_voices and mtmd_gen_audio_voice_name() the
names, with the voices worded as a feature of any audio generation model.
The server lists them under "voices" in /props when the model ships some.

Assisted-by: Claude Fable 5.1

* server : an unknown voice is a bad request

A model with predefined voices answers 400 naming the voice instead of 500
"failed to process TTS prompt". A model without them ignores "voice" with a
warning, as the other audio generation models do.

Assisted-by: Claude Fable 5.1

* refactor api

* fix conversion

* small renaming

* fix lint

* tts : a model with predefined voices speaks with the first one by default

Without a voice prompt such a model only produces near silence. When neither
a voice nor a speaker reference is given, llama-tts and the server now take
the first predefined voice and say so in the log, as the LFM3-Audio template
does with its default voice.

Assisted-by: Claude Fable 5.1

* mtmd : one comment per voice accessor, converter states the rule only

The clip.h comment described clip_gen_get_voice but sat above all three
functions; the converter comment quoted the drift measured per pack, which
goes stale with the next upload of the weights.

Assisted-by: Claude Fable 5.1

* clean up redundant code and comments

* mtmd: match the LFM3-Audio voice name case-insensitively

The tts chat template lowers the voice name, so Male speaks with
the male voice; the voice lookup lowers it the same way and names the
bitmap after the canonical voice.

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-02 18:39:51 +01:00
Alexey Kopytko 3f8af15a76 imatrix : fix --nextn crash (#31522) 2026-11-02 14:20:02 +01:00
Aleksander Grygier 5c2ea1ab11 ui: disable model deletion the server would refuse (#31305)
The router only removes models it took from the Hugging Face cache, but
the models manager offered "Delete from disk" for every model, so a model
from --models-preset or --models-dir failed with a "not removable" toast.
Read the source and can_remove fields of /models and show the delete
action disabled for those models, with a tooltip (a hint line in the
mobile drawer) saying how to remove them instead.

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-02 13:36:00 +02:00
Aleksander Grygier 23426d797e ui: align alert dialog styling with other overlay surfaces (#31312)
Use the popover background, softer border, rounded-xl corners and
shadow-md like dropdowns, hover cards and the settings dialog, and
match the dialog title line height.

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-02 13:35:25 +02:00
Aleksander Grygier 90080ce094 ui: keep queued steering messages in their own conversation (#31304)
A message queued while a conversation was busy was re-sent through
sendMessage, which targets the active conversation, so switching chats
before the turn ended delivered it into the wrong one. Re-send only when
the owning conversation is active; otherwise keep it queued there and
flush it when that conversation is opened again.

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-02 13:34:17 +02:00
Aleksander Grygier 64be95b163 ui: show interrupted state for tool calls that never returned (#31314)
* ui: show interrupted state for tool calls that never returned

The chat-level streaming flag was applied to every assistant message, so a
tool call left without a result in an earlier message kept spinning while a
newer reply streamed. Scope streaming to the last assistant message and mark
finished calls without a result as interrupted.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: mark a shell command stopped before any output as interrupted

exec_shell_command creates its result message before running, so stopping
it mid-run leaves an empty result instead of none.

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-02 13:34:00 +02:00
Aleksander Grygier b12b405c17 ui: don't reload a loading model on select, prefer it in pre-selection (#31326)
* ui: don't reload a loading model on select, prefer it in pre-selection

Assisted-by: Claude (Opus 5.5 Medium)

* ui: skip hidden models in automatic model selection

Assisted-by: Claude (Opus 5.5 Medium)

* ui: don't count a model being unloaded as loading

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-02 13:33:28 +02:00
Sigbjørn Skjæret fcfe4b59df requirements : minor adjustments for python 3.14 wheels (#31534) 2026-11-02 11:34:45 +01:00
bosh d5ef408e20 mtmd : fix InternVL CLS token order (#30886) 2026-11-01 00:30:01 +01:00
aiqubits f3586e66bc server: cap shared decision prefixes at the first output (#30710)
Limit the shared prefix to the first required output position of the parent and each child task.
This keeps decision output embeddings available in the final batch when questions share option tokens.

Fixes #30482

Signed-off-by: aiqubits <aiqubits@hotmail.com>
2026-10-31 19:09:48 +01:00
David Garcia 2871678a1d server : fix /v1/responses endpoint to accept extra_params (#30959) 2026-10-31 19:08:08 +01:00
boshandXuan-Son Nguyen 7bd1808308 mtmd : resize Gemma 4 images without letterbox padding (#30922)
* mtmd : resize Gemma 4 images without letterbox padding

* Apply suggestion from @ngxson

---------

Co-authored-by: Xuan-Son Nguyen <thichthat@gmail.com>
2026-10-31 18:55:33 +01:00
Adrien Gallouët 5047748f11 rpc : delete fs_path_to_utf8() (#31424)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-10-30 20:36:30 +01:00
Adrien Gallouët a34de8618b common : delete fs_path_to_utf8() (#31423)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-10-30 16:20:04 +01:00
Adrien Gallouët d63fbfe794 cmake : embed an app manifest in all Windows exes (#31329)
* cmake : embed an app manifest in all Windows exes

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Remove LLAMA_WIN_APP_CMAKE

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

---------

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-10-30 10:56:53 +01:00
7konigsbergandZeshawn Zahid 7df8b876b9 server : restore the spec checkpoint for drafts longer than n_rs_seq (#31350)
Assisted-by: Claude

Co-authored-by: Zeshawn Zahid <zzshawnn@gmail.com>
2026-10-29 11:28:03 +02:00
Aleksander Grygier 8ab29faec4 ui: keep chats at the bottom until the reader scrolls up (#31311)
* ui: land at the bottom of a conversation every time it opens

Opening a long conversation often stopped short of its end:

- the press grace counted from page load (pressedAt = 0), so a chat that
  landed within 400 ms of a refresh did not follow the rows realizing
  after the jump
- the click that opened a chat (sidebar row, tab) started the same grace,
  so navigating by click had the same problem
- any scroll event away from the bottom stopped following, including the
  ones the browser fires itself while the chat settles
- the chat screen outlives its routes, so going back to the last landed
  conversation through the new-chat screen skipped the landing entirely

scrollToBottom() now clears the grace, only an upward scroll away from the
bottom stops following, and every navigation lands anew. Navigations into
a chat also disable SvelteKit's scroll reset, which moved the previous
conversation to its top for a few frames.

Where overflow-anchor is supported, a 1px pin after the messages is the
only scroll anchor while following, so the browser keeps the bottom in
place itself; WebKit keeps the resize observer.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: let nested agentic content scrollers chain to the chat

Reasoning, shell output and code blocks used overscroll-behavior:
contain, so the wheel stopped dead at their edges instead of
scrolling the conversation.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: stop following the stream as soon as the reader scrolls up

A trackpad or smooth wheel scrolls in steps smaller than the at-bottom
threshold, so each step up was re-stuck and snapped back on the next
growth while a response streamed. Any upward move that is not content
shrinking now releases at once.

The attach effect also read the hook's own state, so every flip
re-attached and re-measured, which undid such a release.

Assisted-by: Claude (Opus 5.5 Medium)
2026-10-29 10:18:13 +02:00
Aleksander Grygier f5d4d75419 ui: show submit button when the chat input has content (#31324)
ChatFormActions gated the mic and stop buttons on a `canSubmit` prop that
ChatForm never passed, so it was always false and typing text did not swap
them for the submit button. Use `canSend` instead and also hide the
skip-reasoning button while there is content to send.

Assisted-by: Claude (Opus 5.5 Medium)
2026-10-29 10:15:54 +02:00
Aleksander Grygier 3ed915e309 ui: reveal the collapsed sidebar from its logo and toggle it there (#31307)
* ui: reveal the collapsed sidebar from its logo and toggle it there

The collapsed sidebar now shows only the logo; hovering it reveals the
icon strip until the pointer leaves. The expanded sidebar drops its
top-right collapse button and the logo collapses it on hover instead.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: fade the peeked sidebar in at full width and keep the toggle slide

Hovering the logo of the collapsed sidebar now fades in the fully open
sidebar, floating over the page at its full width, instead of resizing
it. Opening or closing with the toggle still resizes and slides: the
sidebar docks to strip width while it slides, then floats out of sight,
and it stays hidden under the pointer after a toggle close. The unused
icon-only strip is removed from the sidebar actions.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: stack the sidebar above the chat tabs on desktop

The peeked sidebar floats over the page, but the chat tab bar shared its
z-index and came later in the page, so the tabs were drawn on top of it.

Assisted-by: Claude (Opus 5.5 Medium)
2026-10-29 10:15:31 +02:00
Aleksander Grygier f8fcef2546 ui: add MCP group checkboxes in tools settings and fix indeterminate checkbox colors (#31303)
MCP server groups in the settings Tools tab now get a group checkbox like the
built-in, server and browser groups, toggling the server-scoped tools key.
Indeterminate checkboxes now use the checked state colors.

Assisted-by: Claude (Opus 5.5 Medium)
2026-10-29 10:15:09 +02:00
Aleksander Grygier b7cde589e4 ui: request persistent storage on startup (#31299)
Ask the browser to keep IndexedDB and localStorage data under storage
pressure, so conversations and settings are less likely to be evicted.

Assisted-by: Claude (Opus 5.5 Medium)
2026-10-29 10:14:53 +02:00
Pascal e09a78a25d tests: fix CI flakes in server cancel and sleep tests (#31330)
test_cancel_request enables context shift so the generation only ends
when the request is cancelled, then polls /slots with a deadline.
Without context shift the generation fills the context and frees the
slot on its own on fast machines, so the test passes even with
cancellation disabled.

test_server_sleep uses the existing wait_for_sleep helper instead of a
fixed sleep.
2026-10-29 01:35:01 +01:00
Pascal cf24317bea common, mtmd: bound control vector layer indices and M-RoPE position loops (#31336)
* common: bound control vector layer indices by the model layer count

A direction tensor layer index is parsed from the tensor name with no
upper bound, and the buffer size and write offset are computed in int.
A large index overflows both, so the accumulation writes before the
start of the buffer.

Pass the model layer count to the loader, reject indices above it like
the existing zero index check, and compute the sizes in size_t.

* mtmd: fill M-RoPE positions by whole merge tiles

The position loops of the Qwen-VL, GLM, PaddleOCR and Qwen2.5-VL
encoders hardcode a 2x2 merge inside a stride of n_merge, so a patch
grid that is not a multiple of the merge writes past the positions
buffer. Iterate over whole merge tiles with n_merge inner steps, which
bounds the writes by the patch count and gives the same positions on
aligned grids.

PaddleOCR also requires a merge aligned image at graph build, as the
Qwen-VL family already does.
2026-10-28 23:15:01 +01:00
Xuan-Son NguyenandSigbjørn Skjæret 28cb47c7c8 gguf-py: fix np_roundf dropping the sign (#31310)
* gguf-py: fix np_roundf dropping the sign

* Update requirements/requirements-convert_legacy_llama.txt

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* requirements.txt

* Update tools/server/tests/requirements.txt

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-10-28 14:03:59 +01:00
Tarek DakhranandXuan Son Nguyen d4f1b18b3a mtmd: enable video input support for LFM3 model family (#31283)
* mtmd : add LFM3-VL and LFM3-Omni video input

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Assisted-by: Claude Opus 5.5

* mtmd : reserve the soundtrack command to avoid a GCC 11 warning

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-10-28 12:23:06 +01:00
Tarek DakhranandXuan Son Nguyen d1eb0debad server : support tool calls in /v1/realtime (#31287)
* server : add tool calls to /v1/realtime

Assisted-by: Claude Opus 5.5

* refactor

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-10-28 12:17:13 +01:00
Pascal e486a4b559 ui: build the reasoning menu from the template effort levels (#31273)
* ui: build the reasoning menu from the template effort levels

When the chat template reports effort levels, the menu offers Default,
Off and those levels, and a pick is sent as reasoning_effort. A
template that reports none keeps the token budget menu.

Picking a model other than the one that produced the last answer resets
the conversation effort to Default, and an effort the current menu
lacks resolves to Default.

* ui: wait for a cold router model before building the request

A router serves the props of a loaded model only, so a request sent to
a cold model was built without its template effort levels and fell
back to a token budget. The request now waits for the model to load,
joining a load already in flight, and a stop during the wait sends
nothing.

Status waiters hold a list per model so a load and the requests that
wait on it each get theirs, and a props request in flight is shared
with a second caller instead of answering null.
2026-10-28 11:35:57 +02:00
Aleksander Grygier f4d8af608a ui : show save and reset settings buttons only when relevant (#31248)
* ui : show save and reset settings buttons only when relevant

Assisted-by: Claude (Opus 5.5 Medium)

* ui : move reset settings button next to reload app, pad settings caption

Assisted-by: Claude (Opus 5.5 Medium)
2026-10-28 08:54:34 +02:00
Aleksander Grygier 02d502f314 ui: Support OGG audio attachments by converting them to WAV (#31237)
* ui: support OGG audio attachments by converting them to WAV

Fixes #24164

Assisted-by: Claude (Opus 5.5 Medium)

* ui: convert all non WAV/MP3 audio uploads to WAV

Assisted-by: Claude (Opus 5.5 Medium)
2026-10-28 08:54:15 +02:00
Aleksander Grygier b1756b23f3 ui: Fix draft for new message disappearing after using "Add System Message" action (#31214)
* ui: keep new chat draft when adding a system prompt

Fixes #24338

Assisted-by: Claude (Opus 5.5 Medium)

* ui: remove unused initialMessage from chat form

Assisted-by: Claude (Opus 5.5 Medium)
2026-10-28 08:53:56 +02:00
Aleksander Grygier 6886628cb1 ui : Search dialog (#31173)
* ui : replace the sidebar search mode with a search dialog

Search opens a spotlight dialog (Cmd+K or the Search entry under New chat in the sidebar) with keyboard navigation over recent chats and sidebar actions. Phones keep the search page.

Sidebar actions show their shortcuts, with new ones for Settings (Cmd+,) and Models (Shift+Cmd+M).

Assisted-by: Claude (Opus 5.5 Medium)

* ui : ignore search dialog keys while an IME is composing

Enter confirms an IME candidate and the arrow keys move through the candidates, so the search field leaves them to the IME instead of opening the highlighted result or moving the highlight.

Assisted-by: Claude (Opus 5.5 Medium)
2026-10-28 08:53:41 +02:00
Tarek DakhranandXuan Son Nguyen 8ca4060b7e server : add realtime conversation and transcription on /v1/realtime (#31269)
* server : add realtime conversation and transcription on /v1/realtime

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Assisted-by: Claude Opus 5.5

* examples : default the realtime client to the server port 9931

Assisted-by: Claude Opus 5.5

* server : sample realtime responses right after their own decode

Slots respond in turns on one context: sampling at the start of the next step read the logits of whichever slot decoded last, so concurrent sessions mixed their outputs.

Assisted-by: Claude Opus 5.5

* move more stuff to server-realtime, making it self-contained

* examples : fix the type check of the realtime client

Assisted-by: Claude Opus 5.5

* examples : use the model card prompts in the realtime client

Assisted-by: Claude Opus 5.5

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-10-28 01:26:12 +01:00
Tarek DakhranandXuan Son Nguyen d05706b4f3 model : add LiquidAI/d1-2B and LiquidAI/d1-omni-2B (#31275)
* model : add LiquidAI/d1-2B and LiquidAI/d1-omni-2B

Assisted-by: Claude Opus 5.5

* convert : keep the speech task templates out of the d1 GGUFs

Assisted-by: Claude Opus 5.5

* small refactoring: no_batching

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-10-27 23:36:32 +01:00
Pascal 1b4554eff1 server: expose the reasoning effort levels of the chat template (#31271)
* server: expose the reasoning effort levels of the chat template

The capability probe renders a user turn for each candidate level
(minimal, low, medium, high, xhigh, max) and for an unknown level. A
level is honored when it renders without raising and differently from
the unknown level, which shows the template fallback. Levels sharing a
render are aliases and the highest one is kept.

The caps serialize to JSON, and chat_template_caps in /props reports
the list as reasoning_effort_levels, from lowest to highest.

* jinja: keep the caps free of the json dependency

The caps keep reporting their flags as a map, and the effort levels
stay a plain member. common_chat_templates_get_caps() builds the JSON
the server reports from both, so jinja depends on json only for the
probe inputs.
2026-10-27 22:39:37 +01:00
Tarek DakhranandXuan Son Nguyen 19ff817fb2 model : add LiquidAI/LFM3-Audio-2B and LiquidAI/LFM3-Omni-2B (#31257)
* model : add LiquidAI/LFM3-Audio-2B and LiquidAI/LFM3-Omni-2B

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Assisted-by: Claude Opus 5.5

* convert : read the LFM3-Audio config.json as released

A transformers that knows lfm3_audio rewrites the config into another layout (no lfm/encoder keys).

Assisted-by: Claude Opus 5.5

* mtmd : disable flash attention for the LFM3-Audio depthformer

Assisted-by: Claude Opus 5.5

* tts : add a voice option for LFM3-Audio and LFM3-Omni

The model card offers a female and a male voice; --tts-voice for llama-tts, "voice" for the server /tts.

Assisted-by: Claude Opus 5.5

* clean up template handling

* convert : detect LFM3-Audio and LFM3-Omni by model_type

Assisted-by: Claude Opus 5.5

* move banned tokens --> logits bias --> mtmd_helper

* leave mtmd-cli unchanged

* no more template_asr_system

* left over

* llama : keep the LFM3-Audio encoder convs and codec unquantized

Assisted-by: Claude Opus 5.5

* common, mtmd : shorten comments

Assisted-by: Claude Opus 5.5

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-10-27 18:47:14 +01:00
Tarek DakhranandAldehir Rojas eb7818c0b8 common : support deferred tools and tool_reference content parts (#31256)
* common : support deferred tools and tool_reference content parts

Assisted-by: Claude Opus 5.5

* common : credit the deferred tools co-author

Co-authored-by: Aldehir Rojas <765227+aldehir@users.noreply.github.com>

Assisted-by: Claude Opus 5.5

* tests : fix the type check of the deferred tools test

Assisted-by: Claude Opus 5.5
2026-10-27 18:39:11 +01:00
19cce69199 model : add LiquidAI/LFM3-1.7B and LiquidAI/LFM3-VL-2B (#31252)
* model : add LiquidAI/LFM3-1.7B and LiquidAI/LFM3-VL-2B

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Xuan-Son Nguyen <son@huggingface.co>
Assisted-by: Claude Opus 5.5

* tests : set the SWA rope base of the LFM3 test model

Assisted-by: Claude Opus 5.5

* llama : enable recurrent state rollback for LFM3

Assisted-by: Claude Opus 5.5

* Apply batched suggestions from code review

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Tarek Dakhran <github@dakhran.com>

* gguf-py : align the lfm3vl mapping comments

Assisted-by: Claude Opus 5.5

* llama : fix CRLF line endings from the review suggestions

Assisted-by: Claude Opus 5.5

* tests : add the LFM3 Metal fusion baseline

Assisted-by: Claude Opus 5.5

* llama : keep the LFM3-VL depthwise conv kernels in F32 when quantizing

Assisted-by: Claude Opus 5.5

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Xuan-Son Nguyen <son@huggingface.co>
2026-10-27 17:06:40 +01:00
Sam Malayek 0c8c59ef3d mtmd-cli: Reduced scope of batch pointer (#31155) (#31165)
* mtmd-cli: Reduced scope of batch pointer (#31155)

* mtmd-cli: Remove CLI regression test
2026-10-27 12:56:03 +01:00
Aleksander Grygier 3ab37aedfe ui: fix system message row stretching to full prompt height in WebKit (#31246)
The expand button wrapping the system message card defaulted to
inline-block. WebKit takes an inline-block's baseline from its last
line even when that line is clipped by the card's max-height, so the
row grew to the height of the whole prompt and left a large empty gap
below the card. Make the button a block so the row matches the card.

Assisted-by: Claude (Opus 5.5 Medium)
2026-10-27 13:19:02 +02:00
85cc432e2c server: add router device memory margin parameter for dynamic unloading (#21231)
* server: add --models-memory-max parameter to allow dynamically unloading models when they exceed a memory size threshold

estimate with to-be-loaded model size included

use no_alloc to get memory requirements for model load

only set model memory_mb if not previously calculated

use memory margin instead of total size limit, apply to each device separately

add server memory debug logging

move llama_context_device_memory function to llama-ext.h

fix model count exceeded check

improve memory_per_device map naming

improve variable naming, fix style

also strip models memory margin from child processes

cont : clean-up

replace device memory map with buft memory map. Use llama_get_memory_breakdown

extract duplicated check into helper function

move model memory estimation to subprocess

precompute name->buft map, map GPU host types to CPU buft

cleanup unused variable

remove duplicated init calls

* adapt to upstream changes

* wip

* common_memory_reports internface

* SERVER_ROLE_MEASURE

* server-models-sched

* fix build

* server : fix memory accounting of measure children and claim spin

A measure child runs no_alloc, so the free memory it reports does not
include its estimate: subtracting self there erased the memory held by
other processes, down to zero with --fit on, and the next load could
overshoot the margin. Subtract self only for instance reports.

A queued load only checked --models-max before claiming, so when the
memory margin blocked it behind a busy model, load() threw and the
waiter retried in a tight loop. Refuse the claim while the limits are
exceeded and no idle model can be unloaded, so the waiter sleeps on
the condition variable until one goes idle.

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: Pascal <admin@serveurperso.com>
2026-10-27 12:14:42 +01:00
tc-mbandXuan Son Nguyen 7eed6b7685 model: Add MiniCPM-OCR-Jev (#30980)
* model : add MiniCPM-OCR-Jev support

The model is a Qwen3.5-VL trunk with a value head read at the last token: it
scores an OCR text against an image rendered from it, it does not generate.

Add the openbmb-minicpm-ocr-jev decision type, served by POST /v1/systemone.
The class scores are the pooled output of the sequence, read with last pooling,
so the conversion turns score.weight into cls.output and sets
embedding_length_out. The checkpoint has no LM head, so text generation is
rejected instead of using the output weights duplicated from the token
embeddings.

The task is fixed, so its prompt is entirely in the systemone template, the
same way h2o-lightning keeps its default there. The processor of the checkpoint
allows up to 16384 merged tokens per image, more than the Qwen-VL default of
mtmd, so the mmproj keeps the image budget of the preprocessor config. The
checkpoint declares MiniCPMOCRForSequenceClassification in its config.json, as
other models do.

Signed-off-by: tc-mb <tianchi_cai@icloud.com>

* fix code

Signed-off-by: tc-mb <tianchi_cai@icloud.com>

* use gguf meta classifier_output_labels

* remove redudant code changes

---------

Signed-off-by: tc-mb <tianchi_cai@icloud.com>
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-10-27 11:49:12 +01:00
Aleksander Grygier 0d5ca60098 ui: list org-less models first in the model selector (#31215)
The org-less group renders without a header, so when it followed a named
org its models looked like they belonged to that org. Put it ahead of the
named org groups.

Fixes #25227

Assisted-by: Claude (Opus 5.5 Medium)
2026-10-27 07:52:54 +02:00
Aleksander Grygier 823df7d0fb ui: Highlight reasoning toggle on hover and align it with model selector (#31212)
On desktop, the reasoning effort toggle now gets a muted background on
hover and while its menu is open, matches the model selector's 24px
height, uses a 13px icon and sits 5px from the selector.

Assisted-by: Claude (Opus 5.5 Medium)
2026-10-27 07:50:15 +02:00
PascalandAleksander Grygier 1a97e387d3 ui: add conversation compaction (#27401)
* ui: add conversation compaction

Add a compaction action that asks the model to summarize the
conversation so far and stores the summary in a new compaction node
appended to the branch. Later requests slice the branch at the last
compaction node, keeping the system prefix and continuing from the
summary instead of the full history, which frees up context while the
complete history stays visible and navigable in the UI.

The slice is applied once at the request funnel, covering the plain
send, the agentic flow and the KV cache pre-encode, plus the continue
path that bypasses it. The summary request is a one-shot outside the
SSE replay machinery, streamed live into the node with a full rollback
on abort, error or empty result.

The action is exposed as a /compact slash command and a Compact button
in the context gauge popup, with an overridable summarization prompt
in the settings.

* ui: pre-fill KV cache and refresh context gauge after compaction

Once the summary is stored, pre-encode the compacted branch and stamp
the returned prompt timings on the compaction node: re-encoding the
reduced prefix is part of the compaction itself, so the server KV
cache and the context gauge reflect the summary right away.

The summary stream is registered under a prefixed identity
(compact::convId), so the visibility kick and byte-offset resume
protect it through tab switches while the reload probe never
reattaches it as a chat turn. The stamped node becomes the shared
context truth for the gauge and the processing state restore, and the
stale live state that kept the gauge full is cleared.

* chore: format

* ui: allow a dedicated model for conversation compaction

An opt-in setting reveals a model selector whose options come from the
live router model list. The chosen model writes the summary and is
loaded on demand when it is not resident, while the default path keeps
resolving to the conversation's own model.

* ui: align the compaction model field with project conventions

* ui: fix the model option helpers import path

The compaction model selector pulls filterModelOptions and
groupModelOptions from the models component folder, where they no
longer live, so the build fails to resolve the module. Point the
import at the navigation folder, which now owns them.

Missed during the rebase: the helpers moved in a commit that does
not touch any compaction file, so nothing conflicted.

* ui: import the model option helpers from models/utils

* ui: follow the moved models selector in the compaction model field

The models manager moved ModelsSelectorList under ModelsSelector and
the model option helpers to /utils, where groupModelOptions takes
the options alone. Drop the stale copy the rebase kept and read the
helpers from their new home.

* ui: drop the dedicated compaction model

A summary written by another model loses the prefix the conversation
model cached, so the whole history is prefilled again, and the model
then resumes from text it did not write. The conversation model
always writes its own summary.

* ui: keep the last steps of a compacted conversation

A compaction now folds the steps the model reads but the last ones,
a step being a user message or an assistant message with its tool
results, and stores how many it kept on the compaction node. The
model then reads the system prefix, the summary, the kept steps and
what follows.

The summary request is the request the chat sends, with its options,
tools and message shaping, cut before the preserved steps: the server
reuses the prefix it cached and only the instruction is new. The
number of preserved steps is a setting, 10 by default.

* ui: run every compaction summary through one path

Three callers needed the same thing - the automatic pre-encode pass, the
action in the context gauge, and re-running a summary from the block - and
each of them had grown its own copy of the request, the streaming and the
storing.

There is one path now: a summarization owns a node, streams into it, and
reports itself through a ledger the rest of the app reads (`isCompacting`,
which node belongs to which conversation). Re-running a summary is the same
call on an existing node, so the two entry points cannot drift apart again.

* ui: give the compaction row its own summary UI

The summary was a line of text in a tool-call-shaped collapsible, which is
how a compaction reads as a side note rather than as the thing the rest of
the conversation now stands on.

The row is its own block: a divider that names the boundary, and the summary
inside it with the model that wrote it, its statistics, and the same actions
an assistant turn has - copy, edit, fork, regenerate and delete all apply to
the summary, and it collapses to the divider once a summary is in, because
the shrunk context is the headline and the prose is the detail.

The divider is a variant of the shared collapsible block, which also needed
the toggle and the title split into snippets so a caller can lay them out
without copying the markup. The action icons no longer take a role: they
render what a message offers, and a compaction offers the same as any other
stored message.

* ui: report compaction progress on the row

The row owns the run, so the row says how it is going: the prompt progress
in the header while the context is being read, the phase once the summary
starts streaming, and the token count once it is in. Before the first
summary token there is nothing to collapse, so the header stands on its own
instead of offering an empty block.

* ui: keep the summarized turns quiet while a compaction runs

The turns being summarized used to keep their own "Processing..." indicator
and live reading stats while the summary streamed, so the conversation said
the same thing twice in two places. The row reports the run, so the turns it
replaces stay silent until it is done.

* ui: hand the compact action to the context gauge

Compacting is a decision about the context, so the action belongs with the
numbers that show the context filling up rather than inside the form: it sits
at the bottom of both the hover card and the drawer, under the breakdown it
acts on, quiet enough to stay a footnote and disabled - with the row's own
spinner - while a summary is already running.

* ui: follow the bottom, otherwise leave the view alone

The screen scrolled itself with a controller of observers, intervals,
direction heuristics and landing timers, and still managed to move the view
while the reader was reading: any mutation that grew the branch was treated
as a reason to keep following.

The page is the scroller, so it follows when the reader sits at the bottom,
and stays where it is otherwise. A resize observer plus native scroll
anchoring does the rest, and `pointerdown` buys the reader a moment of grace
to be caught mid-drag.

* ui: expand collapsible bodies with a height transition

Expanding a block snapped it open: the body was mounted the moment `open`
flipped, so a tall block moved the conversation around it in one frame.

The body now goes through the same region the models table uses. The height
interpolates and the visibility follows it, so a body is unmounted once the
collapse has finished rather than the moment it starts - a closed block
still costs nothing while tokens stream past it.

* ui: open a compaction summary at the top of the screen

Toggling a summary left the reader wherever they were, so a conversation that
had just compacted showed the divider at the bottom of the screen and read
down from there.

A toggle now scrolls the page once and lands the divider a breath under the
tabs, both ways: the summary is read from its beginning, and closing it returns
to the same boundary instead of wherever the page happens to come to rest. The
scroll waits for the body to reach its height, because before that the page is
too short to scroll the divider up there and the call would clamp.

That wait is the shared collapse duration rather than a transition event: an
engine without `interpolate-size` animates the region's grid row instead of its
height, so a listener on `height` would never fire there and the page would not
move at all.

* ui: spin the compaction icon while the summary runs

The divider reported the phase in its title while the icon still showed a
finished block, so a run in progress and a summary that had landed looked the
same in the one place the eye reads first. The icon spins while the summary is
running and folds back once it is in.

* ui: let the compaction summary speak without the reasoning trimmings

The summary is the thing the conversation now stands on, not a note beside
it, and the conversation model always writes it, so the body drops the
reasoning block's rule and indent (`flushBody` keeps them for the blocks
that borrow them) and the model selector under the summary is gone - there
is nothing left to swap.

* ui: drop the model override from the compaction path

The summary row no longer offers a model selector, so nothing hands
regenerateCompaction a model anymore. Remove the parameter from the
whole path, with the on-demand load of another model it carried: the
summary always comes from the model the request funnel resolves.

* ui: let the compaction summary think as the chat does

The summary request forced thinking off, overriding the reasoning
effort the user chose. The summary is the text the conversation
resumes from, the task where reasoning helps the most, so the request
now carries the very options of the chat.

* ui: pre-encode the compacted branch as the next turn reads it

The post-compaction pre-encode sent the branch without tools, while
the tools render ahead of the messages, so it warmed a prefix the next
turn never reads and that turn prefilled the whole compacted context
again. The pre-encode now sends the tools, message shaping and
reasoning handling of the chat, so the next turn only prefills its new
message.

* ui: pre-encode the branch the next turn actually reads

The per-turn pre-encode sent the messages of the request plus one
assistant message holding the streamed text, without tools. With
tools on, it replaced the slot the next turn extends with a prefix
nobody reads again, and after an agentic turn its history did not
match the stored calls and results. Without agentic, it never ran.

Both pre-encodes now read the stored branch up to its leaf, cut at the
last compaction, and send it with the tools, message shaping and
reasoning handling of the chat. The plain completion path triggers it
too.

* ui: refuse a compaction summary the output limit cut

The stream handler read every chunk but dropped the finish reason, so
a summary cut by max_tokens folded the conversation into half a text.
The handlers now hand the finish reason to onComplete, and a summary
that ends on length rolls the compaction back with an error.

The summary was framed through a replacement string, so $& or 32560 in
it changed the text the model read; it is framed as written now.

* ui: wait for the collapse transition in the tool call tests

The collapsible body now unmounts once its collapse animation ends rather
than as soon as the block closes, so the "folds back" assertions ran before
the body was gone. Wait out COLLAPSIBLE_TRANSITION_MS first.

Assisted-by: Claude (Opus 5.5 Medium)

---------

Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>
2026-10-27 07:49:27 +02:00
Aleksander Grygier bc262f33cc ui: Open model details in a drawer without leaving the page (#31176)
* ui: open model details in a global drawer without leaving the page

The model info actions in the selector used to navigate to the models page
to open its details pane, and a phone opened a separate dialog instead. The
drawer now lives outside the models table, driven by uiStore, so the info
icon opens it over whatever page is showing.

The models dropdown and the phone picker stay open under the drawer's
backdrop, and keep focus while the drawer opens and closes, so the trigger's
tooltip no longer pops over the drawer.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: let the model details drawer reopen after a drag to dismiss

Dragging the drawer down wrote vaul's own open state, which the one-way open prop then kept closed, so the info action could not open it again until a refresh. Bind open both ways through the store instead.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: close the models selector from the details drawer chat actions

Assisted-by: Claude (Opus 5.5 Medium)
2026-10-27 07:46:25 +02:00
Aleksander Grygier f5892ca5ad ui: use shared confirmation dialog for conversation deletion (#31187)
Extract the keyboard-shortcut delete confirmation into DialogConversationDelete
and reuse it in the sidebar and mobile search page instead of the native
window.confirm prompt.

Assisted-by: Claude (Opus 5.5 Medium)
2026-10-27 07:45:53 +02:00