* model: add Muse Spark Open (text, vision and audio)
Muse Spark Open is a latent space MoE: the residual stream stays at 8192
while the 256 experts live in 4096, with a projection down and up around
them, top-8 sigmoid routing with a correction bias. Attention interleaves
three sliding layers at a 2048 window with rope for one global layer
without rope, with a sigmoid output gate and a weightless QK-norm.
Residuals are norm preserving, their gate is folded at conversion time.
Vision reuses the Muse Glimmer tower. Audio is a causal convolutional
pyramid with no attention and no spectrogram, four stages folding 16, 6,
5 and 4 samples into the channel dim, one embedding per 80 ms of 24 kHz.
The quantizer keeps the latent projections and the attention gate at
Q8_0, a little over 1% of the weights, since the whole residual stream
passes through them and the low-bit types turned the output into a
repeating loop. The model saver writes the post norm epsilon, and the
arch is registered in test-llama-archs.
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
* Muse Spark: drop the redundant forward expand
* Muse Spark: drop the redundant forward expand and tighten the comments
* Update src/llama-quant.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Update src/llama-graph.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Update conversion/muse_spark.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Update conversion/muse_spark.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Update conversion/muse_spark.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Update conversion/muse_spark.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
The first evaluated token attends to the positions above
pos - n_swa, so a SWA state starting at pos - n_swa + 1 holds the
whole window, but the reuse threshold asked for two more cells.
A saved SWA state keeps only the cells inside its window, so a
state read back from a slot file or from the RAM prompt cache was
always rejected and the slot rolled back to a checkpoint.
The SWA threshold now matches the window, while a state without SWA
keeps the exact bound. The test harness gains override_kv, used to
give the SWA test model a window smaller than the test context.
[no release]
The slot file held the target context and the context checkpoints,
but never the draft context, so a restored slot drafted with an
empty or stale draft KV until the cache refilled.
The file now ends with the live draft state, the full draft context
and the speculative state, in the same appendix style as the
checkpoints. A draft state that does not fit the draft context is
dropped and the draft starts empty, while older files and older
servers keep working.
[no release]
Slot selection takes free slots by id or by last use only, so child
tasks often land on a slot holding a parent state while empty slots
are idle, and overwrite it without saving it to the prompt cache.
Free slots are now ordered empty first, then least recently used, for
both the LRU fallback and the child slots, and a child task saves the
state it overwrites. With 4 grouped decision requests in flight and
12 states in rotation, prompt cache reads go from 5 to 36.
[no release]
The registry hard-coded 10 for AGENTIC_MAX_TURNS while DEFAULT_AGENTIC_CONFIG
said 100. The registry seeds SETTING_CONFIG_DEFAULT, so the loop's
`|| DEFAULT_AGENTIC_CONFIG.maxTurns` fallback never fires and users got 10
turns. Reference the shared constant, as the adjacent MCP timeout does.
Assisted by: llama-ui (ggml-org/Qwen3.8-Flash-Next:IQ4_NL)
* server: save only restorable slot states to the prompt cache
The idle-slot save loop stores every idle slot whenever a task starts,
while only completion tasks restore from the prompt cache. Rerank,
embedding, decision and infill states fill the cache up to its limit
without ever being read back, and slots already cached are offered
again at every task start.
Each slot now tracks with cache_pending whether its state is restorable
and missing from the cache: a completion release or an explicit slot
restore sets it, prompt_save consumes it, prompt_clear resets it. Both
save paths share this rule, and the LRU path restores only for
completion tasks.
* server: test that decision slots skip the prompt cache
Decision tasks never mark their slot as cacheable, so starting new
tasks must not save their idle slots to the RAM prompt cache, with or
without a unified KV cache.
* server: restore any reusing task from the prompt cache
Embedding and rerank tasks never reuse their state, so their slots are
cleared on release like child slots and the cache only holds states
that a later task can restore. Infill and decision tasks now restore
from the cache too, which replaces the cache_pending flag.
Decision models disable the prompt cache unless --cache-ram is given,
since their states seldom repeat.
* common : add --spec-draft-n-ubatch
Allow configuring the physical batch size of the draft context. The
drafters feed it at most the drafted tokens per sequence and chunk larger
batches by its own n_ubatch, so it does not need the ubatch of the target.
A smaller value shrinks the draft compute buffer and leaves more VRAM for
the target KV cache (the draft is fitted together with the target).
The default 0 keeps the current behavior of following --n-ubatch.
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
* cont : remove comment
[no ci]
The d1-omni and kev prompts escape "<|name|>" in the request text with
std::regex_replace. libstdc++ recurses for each char of the name, so a
state of "<|" and 30000 letters overflows the stack of the HTTP thread
and llama-server dies with SIGSEGV.
server_decision_escape_special_tokens() writes the same text with a loop.
test-server-component compares it with the regex on every string of up to
6 chars of "<|>a-".
Add an optional ggml_backend_sched_realloc_callback that user code registers with
ggml_backend_sched_set_realloc_callback to be notified when the buffers reserved for a
graph are not sufficient and have to be reserved again. The callback reports whether the
backend assignments and the graph size changed since the previous scheduling.
llama_context registers the callback when creating the scheduler (wrapped into the new
llama_context::sched_new helper) and treats the reallocations that follow a change of the
backend assignments as expected, counting the rest by graph size - unchanged or changed
from the previously scheduled graph. Both counters are reported on one line in the perf
stats.
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
* ui: fix text attachment preview overflow
Set the max height to 80dvh and the max width to full on screens
below medium and 80vw from medium onward.
* style: run formatter
* rpc : turn GGML_RPC_DEBUG into a verbosity level and add logs
GGML_RPC_DEBUG is now parsed as a number: 0/unset disables debug logs,
1-3 emit increasingly detailed output (events, per-command trace,
transport detail). Non-numeric values fall back to 1.
The duplicated env/macro blocks in ggml-rpc.cpp and transport.cpp are
replaced by a shared log.h, which becomes the single choke point for
all logging of the RPC backend: LOG_ERROR/LOG_WARN/LOG_INFO for
unconditional severity logs and LOG_DBG/LOG_DBG2/LOG_DBG3 for the
verbosity-gated ones. The transport files no longer need ggml-impl.h,
and the server banner now also goes through the ggml logger (stderr).
Missing logs are added on both the client (handshake, buffer ops,
tensor transfers, graph computes, cache decisions) and the server
(per-command dispatch, graph nodes), including the negotiated
transport via the new socket_t::transport_name().
Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-RL
* rpc : align log levels with the documented verbosity semantics
The first pass introduced GGML_RPC_DEBUG levels, but many call sites did not
follow the documented semantics and some logs bypassed the macros entirely:
- Route the remaining raw GGML_LOG_* call sites through the local LOG_* macros
so that log.h stays the single choke point for RPC logging
- Emit per-tensor transfers and per-command handler traces at level 2, and
one-shot lifecycle events (backend and buffer type creation, buffer
allocation, graph compute, graph cache eviction) at level 1
- Drop the duplicated enqueue trace in rpc_dispatcher::send, work() already
logs every dispatched command together with its round-trip timing
- Promote degraded-operation events to LOG_WARN: remote allocation failure,
RDMA falling back to TCP, unexpected peer disconnect
- Log graph cache eviction on both sides, since client and server must stay
in sync for the incremental graph update to be valid
- Distinguish Unknown command (opcode out of range) from Unhandled command
(valid opcode with no handler)
- Fix format specifiers (%zu for size_t, %u for device ids, 0x for hex
pointers) and a missing newline in an error log
- Document that llama.cpp applications drop ggml debug records unless the
verbosity threshold is raised with -lv 5
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
A request asking for a busy id_slot still ran the prompt cache update
on that slot before being deferred. When the RAM cache held a better
match, it was loaded into the slot while another request was still
generating there, and that generation continued on the wrong context.
The busy slot is now returned as is and the request waits for it.
* common : read a GGUF's trained context from its metadata
common_get_gguf_n_ctx_train opens only the file's metadata (no_alloc,
like common_get_decision_type) and reads <arch>.context_length, so a
caller can learn the trained context without loading the model. It
accepts both u32 and u64 values and returns 0 when the file is missing,
unreadable, invalid, or reports no context length.
Assisted-by: pi:zai-org/GLM-5.3-Flash
* server : report the trained context in the models listing
update_caps already resolves the model file offline to read its
modalities, so it now reads the trained context from the same GGUF
metadata, and GET /models reports it as context_length when it is
known. A router listing then carries the context without any Hub
request, which lets the UI sort and filter by it offline.
Assisted-by: pi:zai-org/GLM-5.3-Flash
* ui : take the trained context from the models listing
The router now reports context_length per model, so the option mapping
fills contextLength from it and the manager reads it before the Hub
record. The Context column, the context sort and the context filter
then work with the Hugging Face Hub API turned off. A browser suite
guards the sort and the search, the Hub-cache driven context filter and
the re-sort when details arrive after the sort was clicked.
Assisted-by: pi:zai-org/GLM-5.3-Flash
* ui : mark favorite models with a heart
A favorited model shows a rose heart in the selector even before its
row is hovered, and the crossed heart takes its place on hover, so
unfavoriting stays one hover away. The manager table marks its
favorited rows with the same heart after the badges and capabilities.
Assisted-by: pi:zai-org/GLM-5.3-Flash
* fix: UI text nit
* fix: UI nits
* fix: Favorite models grouping in models table
* feat: Remove sorting from Status column in Models Table
* server: read the GGUF metadata once per model
Read the decision type and the trained context in a single GGUF open,
accept only a UINT32 context length like the model loader, and reset
n_ctx_train with the other caps so a failed refresh drops it.
* fix: Post-review fixes
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* ui : add the drawer, sheet and grouped list primitives
Add the drawer and sheet overlay components, the toggle and toggle
group, the shared searchable input, the collapsible sections and the
grouped list, and the near viewport helper they position with.
Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash
* ui : add the shared model components and data layer
Add the model row pieces shared by the selector and the manager, the
models store with its download status feed, the huggingface and
migration services, and the model utils, enums and constants they
read through.
Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash
* ui : rework the models selector around its providers
Group the selector components under their own folder, rework the list
around the provider grouping, add the mobile trigger and the download
item, and derive the selection state in one hook.
Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash
* ui : add the models manager
Add the manager table with its repo, quant and status rows, the
filters and the toolbar, the row actions in a drawer, the downloads
section, the manage models dialog and the stories and tests.
Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash
* ui : rework the chat form actions and the mobile experience
Move the add actions into a drawer and a sheet, float the model
actions in a bar on a phone, open the context panel in a drawer, and
derive the attachment, reasoning and tools menus in hooks.
Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash
* ui : move the mcp servers to the sidebar rail and tidy the dialogs
Move the mcp servers dialog to the sidebar rail with its menu
entries, even out the dialogs on a phone, and group the mcp and
settings stores behind their own modules.
Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash
* ui : polish the shell
The secondary button gets its own look back and the chat add button its
own light surface. Pressable elements get a pointer cursor again, the
font rendering smooths on the app shell, and the agent skills stay out
of prettier's way.
The root layout props probe rework that came with this polish reads the
providers api url and stays with the providers change instead.
Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash
* ui : wrap the model id classes
Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash
* ui : load the model the new chat CTA picks
Start a new chat selected the model but left it unloaded, so the chat
opened against a server with nothing in memory. Both CTAs share a load
helper now.
Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash
* refactor: Post-review fixes
* server : preserve context checkpoints across slot save/restore
Append the checkpoints after the packed server_tokens payload added in #26640
and count them in n_written / n_read, so a restored slot can still roll back to
a checkpoint instead of re-processing the whole prompt.
* server : drop draft checkpoint data that does not match the draft context
Restoring a slot saved with a different draft KV cache type aborted in
load_dft(). Test-load one draft checkpoint on restore and drop the draft
data if it does not fit, instead of crashing. Adds a regression test.
Co-authored-by: Igor Okulist <okigan@gmail.com>
* server : harden the checkpoint appendix of slot save files
Bound each blob size by the bytes left in the file before allocating, open the
file with UTF-8 paths on Windows like the llama state payload, fall back to full
prompt re-processing when a checkpoint restored from a slot file fails to load,
and replace the 1024 count cap by keeping the last n_ctx_checkpoints while reading.
* server : report an incomplete checkpoint appendix as a failed slot save
Return an error to the client when the appendix cannot be written, like a
failed payload write, and make the oversized-blob test declare a size that
cannot be allocated, so an unbounded allocation fails the test.
* server : reject an empty target state in the checkpoint appendix
A saved checkpoint always holds a target state, an empty blob would roll back
without restoring anything. Also log with the slot id, and load the draft test
model from the HF cache instead of a second download.
* common : return bool from checkpoint load_tgt / load_dft
A checkpoint restored from a slot file falls back to full prompt re-processing
when it fails to load, a checkpoint created in memory still aborts.
---------
Co-authored-by: Igor Okulist <okigan@gmail.com>
* model : add LiquidAI/d1-omni-600M decision model
Assisted-by: Claude Opus 5.5
* mtmd : keep conformer GLU sigmoid on CUDA
Assisted-by: Claude Opus 5.5
* server : take d1omni audio through images and input_audio, scope memory-less lfm2 to non-causal
Assisted-by: Claude Opus 5.5
* common : rename decision type d1omni to lfm2-d1-omni, server : make images an alias of files
Assisted-by: Claude Opus 5.5
* model : add LiquidAI/d1-3b decision model
mtmd : read LFM2 image resize algo from GGUF
Assisted-by: Claude Opus 5.5
* common : rename decision type d1 to lfm2-d1
Assisted-by: Claude Opus 5.5
* server: support vision input for Clef
* move input_attn_causal to private
* extend old server_batch::embd
* server_batch::token::pos to multi dim
* nits
* fix abort
* fix img tokens cap
* fix yield_to_queue mutate data
* server: reject partial media truncation
* server: keep only the keep_first fix
Drop the mtmd test helper change, which no longer builds since
clip_image_f32_batch stores its entries by value, and drop the
vision test: no test fixture reaches a cut between two adjacent
media chunks with a reused cache (tinygemma3 uses SWA and wraps
images in text tokens, tinyopenjev and small-test are recurrent),
so the test passed or failed independently of the fix.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* log, server: make router child lines carry their own colors
The logger writes the color reset after the trailing newline, so the
reset opens the next line. On the shared pipe of a router child it lands
in front of the next state command, which the router then misses, and
the line break that works around it shows up as an empty log line on
every progress update.
The reset now goes before the trailing newlines, so every line is self
contained and the command goes back to its plain framing. The router
passes its effective color setting to its children, whose output ends
up in its terminal, and leaves that option out when comparing presets
on reload.
* log: enable ANSI colors on the Windows console
A Windows console renders ANSI sequences only in virtual terminal mode,
which nothing turns on for the logger, so llama-server prints raw escape
codes on the Windows 10 console while llama-cli, whose console code
enables it, shows colors. The logger now enables virtual terminal mode
on stdout and stderr when it turns colors on, and keeps colors off when
a console cannot render them. Pipes and files take the sequences as is.
* server: separate the router child commands from its logs
The child sent its state commands on the same pipe as its logs, so the
router had to pick them out of the log stream by a line prefix, and any
unterminated write in front of a command made the router miss it. This
resolves the TODO at the spawn that called for splitting stdout and
stderr.
The child now keeps stdout for the commands and points everything else
written to stdout at stderr, before anything is written. The router
reads both pipes, handles the commands from stdout and forwards stderr
as the log, and warns about any other line on the command pipe.
* server: address review from ngxson
The single server_child is now created first in the entry point and its
constructor keeps stdout for the commands, so the stream is a member of
the instance instead of a static, and init() is gone. The instance is
passed down to the server, while the CLI entry point creates its own.
* Update tools/server/server.cpp
---------
Co-authored-by: Xuan-Son Nguyen <thichthat@gmail.com>
* Make the drafter probabilistic and the target verify by rejection sampling
* Drop stale spec_draft_q before drafting
* Fallback to argmax sampling for grammar-constrained requests and adding flag for enabling probabilistic draft sampling. Default flag value is greedy.
* Support grammar-constrained requests in rejection sampling
* Fix - renormalize distribution after masking
* copy rng on sampler copy and re-accept drafted tokens on replay
* Fix draft sampler sharing the target's rng stream
* Simplify the rejection sampler's inputs and move replay to the server
* Truncate the draft candidates along with the draft
---------
Co-authored-by: praneshgo <227579474+praneshgo@users.noreply.github.com>
Co-authored-by: Pranesh Gonegandla <pgonegandla@nvidia.com>
* init conversion
* convert: ok
* model loaded
* add server code
* improve conversion script
* support shared prompt prefix
* add docs, imorove UX a bit
* add vision support
* add openjev tiny model for testing
* add dev docs
* support lev & kev
* clean up
* fix lev noul
* fix py lint
* nits docs
* clarify about not supporting date_facts