Commit Graph
19 Commits
Author SHA1 Message Date
Xuan-Son NguyenandPascal 92192d31e4 mtmd: support YuE2-3B (#32369)
* mtmd: support YuE2-3B

* improve speed

* add CFG

* fix streaming

* add cfg control from api

* check ctx size

* fix n_predict

* add flow_steps config

* nits

* fix batch api

* nits

* adapt

* mtmd: yue2 samples from its own saved logits

The context logits belong to the last decode of any slot, so with
several server slots the score and the unguided codes were sampled
from another sequence. Keep the logits of each own decode in
save_logits(), which also does the guidance mix.

* mtmd: let the yue2 snake chain fuse

The beta reshape landed between sqr and mul, so the mul, sin, sqr,
mul, add chain was not consecutive and the backends never fused it.
Expand both reshapes first, like qwen3tts does.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-16 20:22:23 +01:00
Xuan-Son Nguyen 7ae3c6b231 mtmd: add zonos3-mini TTS model (#32020)
* zonos3 init

* add audio gen pipeline

* fix string_format

* add default voice as gguf metadata

* better ref

* support voice clone

* add delivery params

* optimize speed

* nits

* nits 2

* add text normalization

* nits
2026-11-15 19:44:44 +01:00
Roshan Kumar 56fd8a375f server : ignore empty stop strings (#32093) 2026-11-15 17:23:05 +01:00
Marco Colombo 2d99adb108 mtmd: Pocket TTS - add generic support for temperature set via API (#31655)
* mtmd, server : add temperature to /tts, default Pocket TTS to 0.3

Assisted-by: Claude Opus 5.5

* feat: add temp parameter in llama-tts

* temp: fix default hardcoded values

* fix: remove too narrow test

* fix: addressing PR comments
2026-11-09 11:20:45 +01:00
Trunk Trick 63412d8737 llama: fix exhausting CPU and MEM caused by dry sequence breaker (#31035) 2026-10-25 12:31:09 +01:00
Xuan-Son NguyenandPascal 5c0b49a23b server : add POST /tts endpoint for TTS models (#26603)
* mtmd: add audio out stream api

* add /tts endpoint

* add docs

* wire up mtmd_helper_model_can_chat

* upload speaker_ref via form-data

* expose seed to api

* fix header for pcm

* wip

* fix penalty_last_n

* fix server_output_limits

* fix pocket-tts stream

* wip

* add server_schema for tts

* add server routing

* server: coerce seed and stream from multipart form fields

Form fields arrive as strings, so -F seed=42 and -F stream=true were
rejected by the schema. seed joins the integer fields, parsed as a long
long so a full uint32 seed fits, and stream is read as a boolean when
it comes as a string.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-10-21 01:19:42 +02:00
Aldehir Rojas 8b54361025 chat : refactor API (#30210) 2026-10-09 10:06:22 -05:00
Aldehir Rojas 18b5f8b186 server : accumulate generated text and tokens as parse input (#29876)
* server : collect text and token input

* common : add a tokenize helper that aligns tokens with bytes

* common : simplify tokenize logic
2026-10-07 11:49:17 -05:00
Aldehir Rojas acecd56032 common : implement common_schema internal representation for JSON schemas (#28736)
* common : implement common_schema types

* common : implement a json schema optimizer

* common : reduce optimizations

* common : refactor json-schema-to-grammar to use common_schema

* common : use common_trie

* common/schema : implement type/kind resolution

* cont : cleanup

* cont : remove common_chat_tool_parameters

* cont : simplify schema resolution

* cont : pass common_schema through the json-schema-to-grammar builder

* cont : cleanup

* cont : move enums under common_schema and add type enum

* cont : reduce test cases

* cont : clean up

* cont : clean up

* refactor : rename common_schema_parse to common_schema_from_json

* tests : fix gcc dangling-reference warning in test-json-schema

* tests : take the schema label as const char * to satisfy gcc dangling-reference

* refactor : rename common_schema_builder parse_* methods to build_*

* cont : fix may_be_string

* cont : properly handle empty tool parameters

* cont : add tests for empty $ref

* cont : remove dead code

* cont : update docs

* cont : make "{}" mean any object for json_object as well

* cont : restore (min|max)Length to imply string type

* cont : rename common_schema to common_chat_schema
2026-09-12 16:14:50 -05:00
Xuan-Son Nguyen d9f918d2d0 common: add json.h abstraction (#27511)
* add common/json

* migrate common

* adapt jinja

* migrate server

* big wip

* migrate tests

* wip

* revert some excessive changes

* wip

* wip 2

* revert redundant changes

* fix server crash

* various fixes

* fix ci

* harden a bit

* clean up

* rm json-shim

* add some comments

* rm redundant decl
2026-08-22 16:28:28 +02:00
Daniel Bevenius 9ba73fd1f5 server : clarify comment in eval_llama_cmpl_schema [no ci] [no release] (#26720) 2026-08-07 15:39:33 +02:00
Oliver Simons a6aa6f5450 sampler : remove "full-context windows" from history-based samplers (#26524)
* Resolve -1 to 1024 instead of ctx-len for samplers

Because of backend-sampling we initialize samplers before the complete
llama_context is there. Therefore, we cannot infer the resolved context
length yet at the time we construct the samplers.

* Shared default of 64 for history-based samplers, remove context_size
2026-08-04 21:28:55 +03:00
84075273c8 spec: add DSpark speculative decoding (#25173)
* spec: add DSpark speculative decoding

DSpark (DeepSpec, 2026) on top of the merged DFlash drafter. It reuses the
DFlash encoder/decoder graph, target feature extraction and KV-cache injection,
and the verify/accept path unchanged; the draft model is a new "dspark" arch
adding a low-rank Markov head (markov_w1/w2) and an optional (unused here)
confidence head. No new public APIs.

The proposal is the only change: the block is anchor-first (position 0 already
predicts the first draft) and the decoder graph applies a semi-autoregressive,
previous-token conditioned logit bias in-graph, chained per block position:

  logits'(i) = logits(i) + markov_w2 . markov_w1[prev(i)]
  prev(0)    = the block's anchor token, prev(i>0) = argmax(logits'(i-1))

vectorized across all blocks in the batch; the anchors are fed through a
dedicated graph input (token 0 of every block). Greedy stays lossless
(verify unchanged, same as DFlash).

- new arch "dspark" (llama_model_dspark : llama_model_dflash, reuses the graph,
  loads the markov/confidence tensors; shares the target's embed/lm_head).
- Qwen3DSparkModel converter.
- new spec type "draft-dspark" (common_speculative_impl_draft_dspark :
  common_speculative_impl_draft_dflash, overrides draft() only: submits whole
  anchor-first blocks and greedily reads back the biased logits).

* spec: read draft block size in the dflash impl

* docs: add DSpark section to speculative.md

* spec: keep dspark block size read in the dspark impl

* dspark : add TODOs for incomplete parts

- confidence head is loaded but not used yet
- confidence-scheduled prefix pruning is not implemented
- the in-graph Markov chain is greedy-only
- only Qwen3 backbones are supported for now (also noted in docs)

* spec: fold DSpark into the DFlash arch

Address review: drop LLM_ARCH_DSPARK and the dspark.block_size /
markov_rank GGUF keys. A DSpark draft now converts to a DFlash GGUF;
the Markov head tensors are detected by presence (like eagle3 d2t),
block_size is read from the existing dflash.block_size key, and the
block anchors are taken as a strided view of the decoder's token
input instead of a separate graph input.

* spec: add confidence-based draft pruning for DSpark

The DSpark confidence head predicts per-position acceptance of the
drafted block. --spec-draft-conf-min truncates the block at the first
position below the threshold (default 0 = disabled).

* fold the dspark impl into dflash, selected by spec type

* address review comments

* dspark: clean up and improve naming

* update readme

* remove trailing whitespace

* dflash: draft full n_max blocks, defer dp.n_max to the central truncation

The DSpark markov head views the draft batch as a uniform [n_seqs x block]
grid, but the per-seq dp.n_max clamp could produce blocks of different
sizes, silently corrupting the strided views and the resulting logits.

Drop the clamp and always draft the full n_max block for every sequence:
dp.n_max is already enforced by the central truncation in
common_speculative_draft(), the same way eagle3 handles it.

Co-authored-by: Zaire404 <3147879462@qq.com>

* dflash: assert the markov head block-uniformity invariant, require the conf head

With the draft batch always submitting equal-size n_max blocks, a
non-divisible token count can only mean the batch was split across
ubatches or a caller broke the layout - fail loudly instead of silently
dropping the markov bias. The block_drafts > block_size early return
stays: worst-case graph reserve passes legitimately build with
n_seq_tokens > block_size.

Also make conf_proj required when the markov head is present: the
confidence head is part of the DSpark checkpoint format, and a missing
head would otherwise leave --spec-draft-conf-min silently reading stale
embeddings instead of confidences.

Co-authored-by: Zaire404 <3147879462@qq.com>

* dspark: fold conf_min into p_min

p_min and conf_min express the same thing - the minimum predicted
survival probability for a drafted position - differing only in how the
estimate is obtained: token probability for regular drafters, the
trained confidence head for DSpark. The DSpark readback never used
p_min, so reuse it for the confidence threshold and drop the separate
--spec-draft-conf-min flag. Both defaulted to 0 (disabled), so behavior
is unchanged.

Co-authored-by: Zaire404 <3147879462@qq.com>

* dflash: note the confidence broadcast workaround

Requested in review: the ggml_repeat only adapts the [1, n_tok]
confidences to the n_embd-wide embd_nextn transport so that
llama_get_embeddings_nextn can be reused - not a placeholder.

Co-authored-by: Zaire404 <3147879462@qq.com>

* cont : clarify

[no ci]

---------

Co-authored-by: Ruixiang Wang <wangruixiang07@outlook.com>
Co-authored-by: Zaire404 <3147879462@qq.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-07-28 14:43:27 +03:00
Aldehir Rojas 910196f6b3 common : add support for multiple end sequences in the reasoning budget sampler (#25544)
* common : extract trie/ac to a separate file

* common : support multiple token sequences in the reasoning budget sampler

* common/trie : return matched word index

* common/trie : rename "word" to "pattern"

* common/reasoning-budget : expose matched end sequence

* common/sampling : replay end sequence when reasoning budget is done

* cont : update to use multiple end sequences

* cont : clean up
2026-07-25 11:58:09 +02:00
Pascal 4f37f51972 server: accept null sampling params (#25538)
* server: accept null sampling params

Extend the schema validation to treat a null value as absent, so
clients can send null on nullable params (temperature, top_p, ...)
to request the server default. This matches the OpenAI spec and the
json_value convention used elsewhere.

Add has_field() to skip null in the field eval guards.

* has_field -> has_value​
2026-07-10 22:07:29 +02:00
Pascal b5315e16e0 server + ui: ping silent SSE streams every 1s and kick only after 3s so slow prefill never drops healthy connections (#25241)
* server + ui: ping silent SSE streams every 1s and kick only after 3s so slow prefill never drops healthy connections

* server + ui: sse_ping_interval becomes a per-request body field

Address review from ngxson: the global default returns to 30 so API
clients see no behavior change, and the WebUI sends sse_ping_interval: 1
in the request body since it owns the 3s visibility-kick contract and
declares the cadence it needs. Positive values keep the existing > 0
gate, -1 keeps its disabled semantics.

* server: move sse_ping_interval into the request schema

Address review from ngxson: the field is now a typed field_num with
hard limits (-1, INT32_MAX) bound to task_params, seeded from the CLI
default alongside the other inherited parameters. The raw json_value
read and its redundant comment are gone, and schema evaluation brings
type and range validation for free.
2026-07-03 12:47:04 +02:00
Georgi Gerganov 27c8bb4f63 logs : reduce v2 (#25078)
* server : reduce logs

* cont : common

* cont : spec

* cont : CMN_ -> COM_
2026-06-28 08:52:15 +03:00
Xuan-Son Nguyen bfa3219177 server: add "verbose" field to schema (#24864) 2026-06-21 13:03:14 +02:00
Xuan-Son Nguyen e1efd0991d server: add "schema" and validation (#24150)
* wip

* working

* correct some limits

* add field name to error message
2026-06-18 15:40:58 +02:00