Commit Graph
381 Commits
Author SHA1 Message Date
Piotr Wilkin (ilintar)andXuan Son Nguyen 5f490d7bfe feat: external PLE file loading (#31185)
* model : allow loading external PLE files

Assisted-by: Codex

* convert : add separate PLE exports

Assisted-by: Codex

* tests : fix PLE export test lint

Assisted-by: Codex

* model : make --ple an alias of --shard, allow multiple shard locations

Reuse the multi-shard loading path for external PLE files instead of a
separate option.

* --ple is an alias of --shard: both take a comma separated list and may
  be given more than once, and the shards may live anywhere
* metadata comes from the main file; numbered shards are still
  auto-discovered and strictly validated
* explicit files outside the discovered set are appended as extra shards,
  exempt from split.no and split.count checks
* -isnm/--ignore-shard-num-mismatch loads only the given files, in order,
  with no auto-discovery and no split metadata validation
* ple_path, path_ple and llama_model_loader::add_ple_file are removed;
  common_model_load() uses llama_model_load_from_splits for two or more
  paths
* common/fit.{h,cpp} take the shard path vector, so --fit and the router
  memory measurement load every shard

Assisted-by: GPT-6-Astra

* rm -isnm

* make sure to dedup paths

* rm test_convert_ple

* address comments

* allow ple shard to override tensor

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-11-09 19:41:01 +01:00
e95426f169 llama_moe_cache: run missed experts on CPU, add tuning script (#30966)
* moe-cache : compute part of the misses on the CPU

--moe-cache-cpu-share F computes a share F of the MoE cache misses on the CPU
from the host experts instead of uploading them. The graph builds the expert
FFN twice: on the device from the cache slots, where the misses left to the CPU
read zero slots, and on the CPU from the host experts, where the experts
computed on the device have id -1. The two results are added before the down
bias.

- ggml-cpu: MUL_MAT_ID skips negative ids and writes zero rows
- sched: the copy callback is called for all the inputs of a split, and before
  the host sync when there are no events
- the slot lookup callback reads the inputs of the CPU part together with the
  selected experts, so the CPU part does not wait for the device, and its
  result is copied back asynchronously
- each token uses distinct zero slots, mul_mat_id does not support an expert
  selected twice by a token
- LLAMA_MOE_CACHE_TRACE writes the selected experts of each lookup, for a
  simulator of the cache

* scripts : add moe-cache-tune.py to pick the MoE cache arguments

The script wraps a llama-server command: it starts the server at two cache
sizes and three CPU shares, fits the upload and CPU speeds of the machine, and
replays the routing of the runs through a copy of the cache logic to predict
other sizes and shares. It prints the recommended --moe-cache-mib and
--moe-cache-cpu-share.

docs/moe-cache.md describes the MoE cache arguments and the script.

* add am17an as codeowner for MoE cache

* moe-cache : upload all the misses until the cache is full

With a high --moe-cache-cpu-share, few misses are uploaded, so an empty cache
fills slowly and more misses than needed go to the CPU. The CPU share now
starts once all the slots of a cache group hold an expert. moe-cache-tune.py
replays the same rule.

* add backend sync for async transfer queue

* moe-cache : let the scheduler copy the results of the CPU part

* Fix MSVC compilation

* remove sync change

* llama-graph : extract the CPU share condition into can_cpu_share

* llama-graph : make llm_graph_params::moe_cache const

The graph construction no longer updates the MoE cache directly: the CPU
parts of the layers are accumulated in the graph result and applied to
the cache in llama_context::process_ubatch, after the graph inputs are
set and before the graph is computed.

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)

* scripts : replay the ClassLRU policy in moe-cache-tune.py

* moe-cache : pick CPU misses by CRF instead of random (#31853)

* moe-cache : remove the backend logic of the CPU part from the graph

The CPU part reads the host experts, so the scheduler places it on the
CPU without set_tensor_backend. The cache gives no CPU map for batches
that a device would offload. Graphs with a CPU part look up their own
slot map, so the graph does not register the part with the cache.

ggml-backend : the inputs of the next split that are already computed
are copied to host memory before the current split runs, so a CPU split
does not wait for the device split before it. This replaces the stash
of the cache, and the copy callback is again called for host weights
only.

* llama-graph : return the MoE cache slots and CPU ids in a struct

The scheduler prefetch no longer checks that the next split runs on
another backend: the inputs of a split are never on its own backend,
so the input loop already skips that case.

---------

Co-authored-by: Gaurav Garg <gaugarg@nvidia.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Co-authored-by: Alexey Kopytko <alexey@kopytko.com>
2026-11-08 00:20:58 +08:00
Adrien Gallouët bbc2e4b657 common : restrict HF token to huggingface.co hosts (#31730)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-11-06 08:43:33 +01:00
Adrien Gallouët 52e53a76df common : delete common_get_path_from_env() (#31444)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-11-04 14:21:38 +01:00
Adrien Gallouët a34de8618b common : delete fs_path_to_utf8() (#31423)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-10-30 16:20:04 +01:00
Adrien Gallouët aa588a3463 common : drop the Windows argv reconstruction (#31417)
The manifest from #31329 already makes argv UTF-8.

Fixes #26416

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-10-30 14:02:00 +01:00
Tarek DakhranandXuan Son Nguyen 19ff817fb2 model : add LiquidAI/LFM3-Audio-2B and LiquidAI/LFM3-Omni-2B (#31257)
* model : add LiquidAI/LFM3-Audio-2B and LiquidAI/LFM3-Omni-2B

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Assisted-by: Claude Opus 5.5

* convert : read the LFM3-Audio config.json as released

A transformers that knows lfm3_audio rewrites the config into another layout (no lfm/encoder keys).

Assisted-by: Claude Opus 5.5

* mtmd : disable flash attention for the LFM3-Audio depthformer

Assisted-by: Claude Opus 5.5

* tts : add a voice option for LFM3-Audio and LFM3-Omni

The model card offers a female and a male voice; --tts-voice for llama-tts, "voice" for the server /tts.

Assisted-by: Claude Opus 5.5

* clean up template handling

* convert : detect LFM3-Audio and LFM3-Omni by model_type

Assisted-by: Claude Opus 5.5

* move banned tokens --> logits bias --> mtmd_helper

* leave mtmd-cli unchanged

* no more template_asr_system

* left over

* llama : keep the LFM3-Audio encoder convs and codec unquantized

Assisted-by: Claude Opus 5.5

* common, mtmd : shorten comments

Assisted-by: Claude Opus 5.5

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-10-27 18:47:14 +01:00
85cc432e2c server: add router device memory margin parameter for dynamic unloading (#21231)
* server: add --models-memory-max parameter to allow dynamically unloading models when they exceed a memory size threshold

estimate with to-be-loaded model size included

use no_alloc to get memory requirements for model load

only set model memory_mb if not previously calculated

use memory margin instead of total size limit, apply to each device separately

add server memory debug logging

move llama_context_device_memory function to llama-ext.h

fix model count exceeded check

improve memory_per_device map naming

improve variable naming, fix style

also strip models memory margin from child processes

cont : clean-up

replace device memory map with buft memory map. Use llama_get_memory_breakdown

extract duplicated check into helper function

move model memory estimation to subprocess

precompute name->buft map, map GPU host types to CPU buft

cleanup unused variable

remove duplicated init calls

* adapt to upstream changes

* wip

* common_memory_reports internface

* SERVER_ROLE_MEASURE

* server-models-sched

* fix build

* server : fix memory accounting of measure children and claim spin

A measure child runs no_alloc, so the free memory it reports does not
include its estimate: subtracting self there erased the memory held by
other processes, down to zero with --fit on, and the next load could
overshoot the margin. Subtract self only for instance reports.

A queued load only checked --models-max before claiming, so when the
memory margin blocked it behind a busy model, load() threw and the
waiter retried in a tight loop. Refuse the claim while the limits are
exceeded and no idle model can be unloaded, so the waiter sleeps on
the condition variable until one goes idle.

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: Pascal <admin@serveurperso.com>
2026-10-27 12:14:42 +01:00
Julien Chaumond 74a1fd3d03 common : --model or -hf is required (#31242)
Assisted-by: Codex
2026-10-27 11:55:48 +01:00
Adrien Gallouët 21ec307146 common : allow concurrent downloads across processes (#29312)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-10-23 23:52:56 +02:00
7adf160c41 server: add --connect, allow p2p access to server outside LAN (#28277)
* init

* wip

* wire it up

* make it clear that it's experimental

* add props.connect_code

* nits

* fix windows build

* server: do not advertise a connect code without the tunnel

--connect-code on its own had no effect other than being served in
/props, so the Web UI showed a remote access code that no tunnel was
serving. Clear it when --connect is off, and say so once at startup.

The teardown comment claimed that terminate() closes the child stdout.
It does not: the log thread leaves fgets() only once every writer of
that pipe is gone, so a descendant of llama-connect would block the
join forever. Describe the real contract instead.

Also restore the --reasoning-preserve default in the generated docs.

* Update tools/connect/CMakeLists.txt

Co-authored-by: Adrien Gallouët <angt@huggingface.co>

* server: dial llama-connect on the bound port and a reachable host

Pass ctx_http.port so --port 0 and the GCP override work, pick the
dial host from --host (wildcards map to loopback, IPv6 is bracketed)
and refuse --connect when only unix sockets are bound.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
Co-authored-by: Adrien Gallouët <angt@huggingface.co>
2026-10-20 00:23:14 +02:00
Pascal 2639a3b4e0 server: save only restorable slot states to the prompt cache (#30498)
* server: save only restorable slot states to the prompt cache

The idle-slot save loop stores every idle slot whenever a task starts,
while only completion tasks restore from the prompt cache. Rerank,
embedding, decision and infill states fill the cache up to its limit
without ever being read back, and slots already cached are offered
again at every task start.

Each slot now tracks with cache_pending whether its state is restorable
and missing from the cache: a completion release or an explicit slot
restore sets it, prompt_save consumes it, prompt_clear resets it. Both
save paths share this rule, and the LRU path restores only for
completion tasks.

* server: test that decision slots skip the prompt cache

Decision tasks never mark their slot as cacheable, so starting new
tasks must not save their idle slots to the RAM prompt cache, with or
without a unified KV cache.

* server: restore any reusing task from the prompt cache

Embedding and rerank tasks never reuse their state, so their slots are
cleared on release like child slots and the cache only holds states
that a later task can restore. Infill and decision tasks now restore
from the cache too, which replaces the cache_pending flag.

Decision models disable the prompt cache unless --cache-ram is given,
since their states seldom repeat.
2026-10-15 06:30:17 +02:00
Georgi Gerganov 3fb47677ea common : add --spec-draft-n-ubatch (#30459)
* common : add --spec-draft-n-ubatch

Allow configuring the physical batch size of the draft context. The
drafters feed it at most the drafted tokens per sequence and chunk larger
batches by its own n_ubatch, so it does not need the ubatch of the target.
A smaller value shrinks the draft compute buffer and leaves more VRAM for
the target KV cache (the draft is fitted together with the target).

The default 0 keeps the current behavior of following --n-ubatch.

Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next

* cont : remove comment

[no ci]
2026-10-14 13:58:05 +03:00
Daniel Bevenius 77757a3467 examples : fix draft sampling and rejection handling (#30095)
* examples : fix draft sampling and rejection handling

This commit fixes two issues in the speculative example which are
described in detail in the linked issue.

Resolves: https://github.com/ggml-org/llama.cpp/issues/29975

* examples : add kv-unified requirement for parallel > 1
2026-10-12 12:02:24 +02:00
Mikolaj Kucharski aa94f20861 args: add LLAMA_ARG_SLOT_SAVE_PATH env for --slot-save-path (#30272)
Allow configuring --slot-save-path via LLAMA_ARG_SLOT_SAVE_PATH so
it can be set from an EnvironmentFile in a systemd unit file.
2026-10-10 15:34:40 +02:00
Xuan-Son Nguyen b42b7e6d30 server: use port 9931 by default (#30159)
* server: use port 9931 by default

* revert unrelated changes
2026-10-09 10:40:03 +02:00
Aman Gupta c811cb8f0a llama: support MoE cache over multiple GPUs (#30112) 2026-10-08 15:50:49 +05:30
Aman Gupta d6cf9acb25 llama : add a GPU cache for MoE experts kept in host memory (#29887)
* llama : add a GPU cache for MoE experts kept in host memory

Assisted-by: Claude

* use llama_moe_cache_ptr
2026-10-07 21:07:45 +03:00
Adrien Gallouët 4d60b4d087 common, server : report model input/output modalities in GET /models (#29987)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-10-05 17:08:29 +02:00
0504396140 imatrix: calculate activation-based statistics for new format (GGUF) imatrices (#14891)
* Use activations to calculate the stats
* Determine calculation mode
* Compute entropy for activations
* Compute cosine similarity based on activations
* Compute l2 norm
* Add compute_layer_statistics() function
* Update aggregated statistic report layout
* Fix printing l2 norm when calc_mode = 1
* Refactor variable name
* Compute aggregated (per layer) l2 norm
* Update aggregated sum of squared activations per layer
* Make ZD Score two-tailed
* Update report layout
* Reverse conditional logic to match convention
* Rename report heading
* Add --activation-statistics parameter
* Add Euclidean–Cosine Score (ECS)
* Add --activation-statistics logic to avoid doubling the imatrix size by default
* Update stats output sort based on imatrix type
* Process external NextN draft files (-md / --model-draft)
* Refactor to use new llama_batch_ext

Co-authored-by: compilade <git@compilade.net>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-10-04 11:24:53 +02:00
1fb7ef3e33 spec : add probabilistic sampling for simple draft and MTP (#27694)
* Make the drafter probabilistic and the target verify by rejection sampling

* Drop stale spec_draft_q before drafting

* Fallback to argmax sampling for grammar-constrained requests and adding flag for enabling probabilistic draft sampling. Default flag value is greedy.

* Support grammar-constrained requests in rejection sampling

* Fix - renormalize distribution after masking

* copy rng on sampler copy and re-accept drafted tokens on replay

* Fix draft sampler sharing the target's rng stream

* Simplify the rejection sampler's inputs and move replay to the server

* Truncate the draft candidates along with the draft

---------

Co-authored-by: praneshgo <227579474+praneshgo@users.noreply.github.com>
Co-authored-by: Pranesh Gonegandla <pgonegandla@nvidia.com>
2026-10-02 17:52:54 +03:00
pr3pony feb9a3d6de args: fix cli download mmproj arg (#28977)
* tests: add tests for cli download arg parsing

* args: fix cli download mmproj arg
2026-09-30 20:45:39 +02:00
Adrien Gallouët 00af63567a common : use fs::path for config dir (#29649)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-09-29 17:22:49 +02:00
Adrien Gallouët 76a5bc86d1 common : use fs::path for cache dirs (#29595)
- Avoid useless string conversions on Windows.
- No need for BSD or emscripten special cases.

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-09-29 07:24:03 +02:00
Georgi Gerganov a97cce86a8 common : avoid side effects around params parsing (#29537)
- register --rpc unconditionally and call llama_supports_rpc() only from its handler
- print server "initialization ..." log after args are parsed

Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-RL
2026-09-27 20:18:56 +03:00
Emanuil RusevandXuan Son Nguyen 217f81c266 server: Add support for binding to multiple addresses (#28690)
* Add support for binding llama-server to multiple addresses

Assisted-by: Codex

* remove redundant thread handler

* make it clear about overlapping addr

* reject --port 0 with multiple tcp addr

* improve arg handler

* nits

* fix test

* nits 2

* nits

* nits 2

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-09-22 15:16:40 +02:00
Mikolaj Kucharski e0dff58475 args: add env vars for temperature, top-p, min-p and penalties (#27380)
Allow configuring --temp, --top-p, --min-p, --repeat-penalty,
--presence-penalty and --frequency-penalty via LLAMA_ARG_* so
llama-server can be fully controlled from an EnvironmentFile
(e.g. systemd on Debian).

Use `llama-gen-docs` to regenerate the readme files.
2026-09-21 12:47:38 +02:00
Xuan-Son Nguyen 8e330954ad common: add LOG_JSON macro to log structured data (#28586)
* add LOG_JSON macro

* fit: add demo LOG_JSON
2026-09-13 01:36:34 +02:00
Aldehir Rojas acecd56032 common : implement common_schema internal representation for JSON schemas (#28736)
* common : implement common_schema types

* common : implement a json schema optimizer

* common : reduce optimizations

* common : refactor json-schema-to-grammar to use common_schema

* common : use common_trie

* common/schema : implement type/kind resolution

* cont : cleanup

* cont : remove common_chat_tool_parameters

* cont : simplify schema resolution

* cont : pass common_schema through the json-schema-to-grammar builder

* cont : cleanup

* cont : move enums under common_schema and add type enum

* cont : reduce test cases

* cont : clean up

* cont : clean up

* refactor : rename common_schema_parse to common_schema_from_json

* tests : fix gcc dangling-reference warning in test-json-schema

* tests : take the schema label as const char * to satisfy gcc dangling-reference

* refactor : rename common_schema_builder parse_* methods to build_*

* cont : fix may_be_string

* cont : properly handle empty tool parameters

* cont : add tests for empty $ref

* cont : remove dead code

* cont : update docs

* cont : make "{}" mean any object for json_object as well

* cont : restore (min|max)Length to imply string type

* cont : rename common_schema to common_chat_schema
2026-09-12 16:14:50 -05:00
Aaron Teo 14a9d09f75 args: officially deprecate --mmap|mlock|dio (#28334)
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
2026-09-09 18:36:27 +08:00
Aman Gupta 415e909d84 spec: single device drafter should create meta backend wrapper (#28390) 2026-09-08 20:44:33 +08:00
Xuan-Son Nguyen 7620399f58 common: add --log-jsonl (#28437)
* common: add --log-jsonl

* rename unknown to none
2026-09-06 08:21:22 +02:00
Georgi GerganovandXuan-Son Nguyen e750b887a8 common, server : enable preserve_reasoning kwarg by default, log its effective state (#28174)
* common, server : enable preserve_reasoning kwarg by default, log its effective state

If the preserve_reasoning chat template kwarg is not specified explicitly
via --reasoning-preserve / --no-reasoning-preserve, it is enabled by
default after argument processing. The server logs the effective state of
the kwarg, warns that it is enabled by default when the template supports
it, and only warns "has no effect" when it was enabled explicitly on a
template that does not support it. Setting the kwarg via
--chat-template-kwargs is deprecated.

Assisted-by: pi:llama.cpp/Qwen3.8-27B

* cont : update comment

Co-authored-by: Xuan-Son Nguyen <son@huggingface.co>

---------

Co-authored-by: Xuan-Son Nguyen <son@huggingface.co>
2026-09-02 19:19:54 +03:00
Georgi Gerganov bebc9350ec common: rename --tensor-read-lazy to --lazy-mode, add -lzm shorthand (#27969)
Rename the --tensor-read-lazy CLI argument to --lazy-mode, to match the
internal lazy_mode parameter, and add a -lzm shorthand. Sync the READMEs.

Assisted-by: pi:llama.cpp/Qwen3.8-27B
2026-08-30 09:18:10 +03:00
Xuan-Son Nguyen 50f068ffff bench: add --tensor-read-lazy (#27881)
* bench: add --tensor-read-lazy

* rm the alias

* rename to LLAMA_LAZY_MODE_*
2026-08-28 20:51:05 +02:00
BartowskiandXuan Son Nguyen 18443257a3 server: add ctx-per-slot (--kv-unified-per-slot) (#24124)
* Add ctx-per-slot argument for unifid KV cache

* Swap out ctx fractions for ctx pool slots

* Formatting cleanup

* Remove ctx-pool-slots, make ctx-per-slot an int

* refactor it

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-08-27 22:39:14 +02:00
Xuan-Son Nguyen fac889fb38 llama: model_loader: add TENSOR_READ_LAZY (#27794)
* llama: model_loader: add TENSOR_GET_ROW_LAZY

* add --tensor-read-lazy

* rename to TENSOR_READ_LAZY

* gen docs

* address comments
2026-08-27 15:14:34 +02:00
Gaurav Garg 2bb9bddafa spec: Add benchmark-only synthetic speculative acceptance options (#27711)
* Add benchmark-only synthetic speculative acceptance to llama-server and llama-cli

* Address review comments

* Address review comments

* Add some comments in the code
2026-08-27 13:53:42 +03:00
Xuan-Son Nguyen f29551215b args: add --video-* CLI arguments (#24318)
* args: add --video-* CLI arguments

* gen docs

* nits

* add mtmd_helper_init_opt
2026-08-27 12:11:12 +02:00
Jonas J c5fc7e3488 llama : add --n-cpu-ffn option (#26622)
* common : dedupe --n-cpu-moe / --spec-draft-n-cpu-moe override loops

* common : add --n-cpu-ffn to CPU-offload dense FFN weights of first N layers

* common : generalize llm_ffn_block_regex over the FFN regex, drop TODO
2026-08-27 11:26:42 +02:00
Xuan-Son Nguyen d9f918d2d0 common: add json.h abstraction (#27511)
* add common/json

* migrate common

* adapt jinja

* migrate server

* big wip

* migrate tests

* wip

* revert some excessive changes

* wip

* wip 2

* revert redundant changes

* fix server crash

* various fixes

* fix ci

* harden a bit

* clean up

* rm json-shim

* add some comments

* rm redundant decl
2026-08-22 16:28:28 +02:00
Xuan-Son Nguyen 9fee29e943 arg: remove -no-cnv from cli [no ci] (#27542)
* arg: remove -no-cnv from cli

* clarify about not adding exccesive test cases
2026-08-22 15:53:56 +02:00
John-Henry LimandXuan Son Nguyen 681c29d36a mtmd: add --mmproj-device argument (#23255)
* feat: add --mmproj-device arg & backwards compatible MTMD_BACKEND_DEVICE env var

* feat: load mmproj device backend immediately, add -mmdev shortflag

* fix: its a pointer now get the name

* clean up

* gen docs

* nits

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-08-20 18:45:37 +02:00
Xuan-Son Nguyen ee4c505a4f server: add dedup-cache-models preset option (#27346) 2026-08-19 11:04:26 +02:00
Xuan-Son NguyenandPascal 666f8898a2 ui: move get_datetime tool to frontend (#27255)
* ui: move get_datetime tool to frontend

* clarify docs

* server: drop the now unused ctime include

strftime() and gmtime_r() were the only users, both went away with the
get_datetime tool. Also make the renderer's catch inert: the browser
executor always emits JSON, so a non-JSON result is no longer a date to
display.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-08-17 14:33:55 +02:00
Xuan-Son Nguyen 6fed9f6ff7 mtmd, common: various fixes (#27071)
* apply fixes

* cont

* revert gguf fix
2026-08-14 23:34:56 +02:00
Andy WilliamsandAlde Rojas 7e4c0a9688 chat : pass reasoning_effort to template
* chat: add reasoning_effort to common_chat_templates_inputs

Store OpenAI Chat Completions reasoning_effort and make it
available to jinja templates (with model specific translations
where required).

Assisted-by: llama.cpp:Muse-Glimmer-30B

* server : fixup reading reasoning effort from body

server_chat_convert_responses_to_chatcmpl already handles conversion of
Responses API reasoning.effort to reasoning_effort

* chat : expose reasoning effort

Assisted-by: Claude Opus 5

* chat : add reasoning_effort to generation_params

Assisted-by: Claude Opus 5

* chat : move reasoning_effort next to enable_thinking

Assisted-by: Claude Opus 5

* cont : mirror preserve_reasoning

* cont : pass context through analyze function

---------

Co-authored-by: Alde Rojas <hello@alde.dev>
2026-08-14 13:23:11 -05:00
aic0d3r f65e568fd8 common : auto-detect spec type from draft GGUF metadata (#26814)
* common : auto-detect spec type from draft GGUF metadata

When -md loads a local draft model without --spec-type, the sidecar
inference in common_models_handler_apply only checks HF repo sidecars
and misses local files. The draft model loads into VRAM but speculative
decoding never activates (types stays NONE).

Read general.architecture from the draft GGUF header and map:
  dflash + markov_w1.weight tensor -> draft-dspark
  dflash without markov head        -> draft-dflash

Assisted-by: opencode

* common : address review feedback on spec-type auto-detect PR

- Fix comment spacing to match surrounding style (/* .x = */ not /*.x =*/)
- Add LOG_INF when auto-detection fires so users can see why spec decoding enabled
- Document single-file assumption for split-GGUF edge case

Addresses bot review feedback on #26814.

* common : move spec-type GGUF auto-detect into speculative module

- add common_speculative_types_from_gguf() in speculative.cpp/.h
- use gguf_context_ptr (RAII) from ggml-cpp.h
- reduce comments to a single line per AGENTS.md style

Addresses review feedback on #26814

* common : add doc note and join SPC_INF line in spec-type auto-detect

Assisted-by: opencode
2026-08-13 12:34:27 +02:00
Johnathan Craig MaudlinandXuan Son Nguyen 8e7f22b67e common: add system-level config file (#26118)
* common: Add CLI > ENV > models-presets > INI precedence

1. CLI flags have the highest precedence
2. ENV vars have the second-highest precedence
3. System and User configs have the lowest precedence
   - Linux/BSD/Mac
     - /etc/llama.cpp/config.ini < ${XDG_CONFIG_HOME:-~/.config}/llama.cpp/config.ini
   - Windows
     - %PROGRAMDATA%\llama.cpp\config.ini < %APPDATA%\llama.cpp\config.ini

* fix UB

* use common_get_env

* ignore_unknown_keys

* nits

* add docs

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-08-13 00:02:27 +02:00
Daniel BeveniusandGeorgi Gerganov 680a9ae63d cmake : introduce semantic versioning (#26839)
* cmake : introduce semantic versioning (wip)

This commit introduces semantic versioning to llama.cpp.

* squash! cmake : introduce semantic versioning (wip)

* cmake : update test-cmake README notes [no ci]

* include libmtmd in output so show its semversioned

* ci : add make-release workflow

* ci : fix build number check in build-cmake-pkg.yml

* examples : remove trailing whitespace

* ci : abort if upstream ggml version does not exist

* ci : extract step contents into scripts

* ci : add GGML_NATIVE=OFF to ubuntu job

* examples : remove CI build information from test-cmake [no ci]

This commit removes the nightly/release information that I added
previously to keep this focused only on using building and installing
llama.cpp with cmake and being able to quickly verify changes or
troubleshoot issues.

* ci : merge scripts into single script

* remove -dev-build_number support

This commit removes the incremental build number (versioning) support
that I added. This was incorrect and we should only use the semver for
the version. Releases will be tag a nightly build and package
maintainers/managers that build from source can use the tag and it is
therefor important that the correct version is reported. So a
nightly-build will report the semver without the build number. The build
number and commit as availble via cmake and test-cmake has been updated
to include an example of using them:
```console
$ ./build.sh
[test-cmake] version: 0.1.0, build: 10360 (08c69e381)
...
```

Refs: https://github.com/ggml-org/llama.cpp/pull/26839#discussion_r3755836969

* docs: add initial release.md documentation

* cmake : clean-up and add LLAMA_BUILD_IS_DEV option

* ci : remove version input from make-release job

* ci : add LLAMA_BUILD_IS_DEV=OFF to build-cmake-pkg.yml

Refs: https://github.com/danbev/llama.cpp/actions/runs/31576801921/job/94050639145

* docs : update release notes with LLAMA_BUILD_IS_DEV info [no ci]

* ci : add TODO to winget workflow [no ci]

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-08-12 14:15:03 +02:00