Commit Graph
382 Commits
Author SHA1 Message Date
Xuan-Son Nguyen 781dbc5ac9 spec: properly handle mtmd input for mtp (#30257)
* spec: properly handle mtmd input for mtp

* nits
2026-10-10 11:22:38 +02:00
Aleksander GrygierandPascal baef3ed9a1 ui: Models Manager Follow-up Improvements (#30228)
* common : read a GGUF's trained context from its metadata

common_get_gguf_n_ctx_train opens only the file's metadata (no_alloc,
like common_get_decision_type) and reads <arch>.context_length, so a
caller can learn the trained context without loading the model. It
accepts both u32 and u64 values and returns 0 when the file is missing,
unreadable, invalid, or reports no context length.

Assisted-by: pi:zai-org/GLM-5.3-Flash

* server : report the trained context in the models listing

update_caps already resolves the model file offline to read its
modalities, so it now reads the trained context from the same GGUF
metadata, and GET /models reports it as context_length when it is
known. A router listing then carries the context without any Hub
request, which lets the UI sort and filter by it offline.

Assisted-by: pi:zai-org/GLM-5.3-Flash

* ui : take the trained context from the models listing

The router now reports context_length per model, so the option mapping
fills contextLength from it and the manager reads it before the Hub
record. The Context column, the context sort and the context filter
then work with the Hugging Face Hub API turned off. A browser suite
guards the sort and the search, the Hub-cache driven context filter and
the re-sort when details arrive after the sort was clicked.

Assisted-by: pi:zai-org/GLM-5.3-Flash

* ui : mark favorite models with a heart

A favorited model shows a rose heart in the selector even before its
row is hovered, and the crossed heart takes its place on hover, so
unfavoriting stays one hover away. The manager table marks its
favorited rows with the same heart after the badges and capabilities.

Assisted-by: pi:zai-org/GLM-5.3-Flash

* fix: UI text nit

* fix: UI nits

* fix: Favorite models grouping in models table

* feat: Remove sorting from Status column in Models Table

* server: read the GGUF metadata once per model

Read the decision type and the trained context in a single GGUF open,
accept only a UINT32 context length like the model loader, and reset
n_ctx_train with the other caps so a failed refresh drops it.

* fix: Post-review fixes

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-10-09 19:28:54 +02:00
Xuan-Son Nguyen b42b7e6d30 server: use port 9931 by default (#30159)
* server: use port 9931 by default

* revert unrelated changes
2026-10-09 10:40:03 +02:00
Aman Gupta c811cb8f0a llama: support MoE cache over multiple GPUs (#30112) 2026-10-08 15:50:49 +05:30
Leebr Data ConsultingandIgor Okulist 033df86b69 server : preserve context checkpoints across slot save/restore (#26004)
* server : preserve context checkpoints across slot save/restore

Append the checkpoints after the packed server_tokens payload added in #26640
and count them in n_written / n_read, so a restored slot can still roll back to
a checkpoint instead of re-processing the whole prompt.

* server : drop draft checkpoint data that does not match the draft context

Restoring a slot saved with a different draft KV cache type aborted in
load_dft(). Test-load one draft checkpoint on restore and drop the draft
data if it does not fit, instead of crashing. Adds a regression test.

Co-authored-by: Igor Okulist <okigan@gmail.com>

* server : harden the checkpoint appendix of slot save files

Bound each blob size by the bytes left in the file before allocating, open the
file with UTF-8 paths on Windows like the llama state payload, fall back to full
prompt re-processing when a checkpoint restored from a slot file fails to load,
and replace the 1024 count cap by keeping the last n_ctx_checkpoints while reading.

* server : report an incomplete checkpoint appendix as a failed slot save

Return an error to the client when the appendix cannot be written, like a
failed payload write, and make the oversized-blob test declare a size that
cannot be allocated, so an unbounded allocation fails the test.

* server : reject an empty target state in the checkpoint appendix

A saved checkpoint always holds a target state, an empty blob would roll back
without restoring anything. Also log with the slot id, and load the draft test
model from the HF cache instead of a second download.

* common : return bool from checkpoint load_tgt / load_dft

A checkpoint restored from a slot file falls back to full prompt re-processing
when it fails to load, a checkpoint created in memory still aborts.

---------

Co-authored-by: Igor Okulist <okigan@gmail.com>
2026-10-08 13:18:20 +03:00
Tarek Dakhran a657f7e981 model : add LiquidAI/d1-omni-600M decision model (#30114)
* model : add LiquidAI/d1-omni-600M decision model

Assisted-by: Claude Opus 5.5

* mtmd : keep conformer GLU sigmoid on CUDA

Assisted-by: Claude Opus 5.5

* server : take d1omni audio through images and input_audio, scope memory-less lfm2 to non-causal

Assisted-by: Claude Opus 5.5

* common : rename decision type d1omni to lfm2-d1-omni, server : make images an alias of files

Assisted-by: Claude Opus 5.5
2026-10-08 02:07:49 +02:00
Tarek Dakhran 88dcc460d6 model : add LiquidAI/d1-3B decision model (#30110)
* model : add LiquidAI/d1-3b decision model

mtmd : read LFM2 image resize algo from GGUF

Assisted-by: Claude Opus 5.5

* common : rename decision type d1 to lfm2-d1

Assisted-by: Claude Opus 5.5
2026-10-07 22:04:39 +02:00
Aman Gupta d6cf9acb25 llama : add a GPU cache for MoE experts kept in host memory (#29887)
* llama : add a GPU cache for MoE experts kept in host memory

Assisted-by: Claude

* use llama_moe_cache_ptr
2026-10-07 21:07:45 +03:00
Xuan-Son Nguyen da263e7275 models: support pplx-decider (#30044) 2026-10-06 16:07:46 +02:00
Adrien Gallouët 4d60b4d087 common, server : report model input/output modalities in GET /models (#29987)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-10-05 17:08:29 +02:00
PascalandXuan-Son Nguyen a7fb71fab8 log, server: self contained colors, split child commands from logs in router mode (#29895)
* log, server: make router child lines carry their own colors

The logger writes the color reset after the trailing newline, so the
reset opens the next line. On the shared pipe of a router child it lands
in front of the next state command, which the router then misses, and
the line break that works around it shows up as an empty log line on
every progress update.

The reset now goes before the trailing newlines, so every line is self
contained and the command goes back to its plain framing. The router
passes its effective color setting to its children, whose output ends
up in its terminal, and leaves that option out when comparing presets
on reload.

* log: enable ANSI colors on the Windows console

A Windows console renders ANSI sequences only in virtual terminal mode,
which nothing turns on for the logger, so llama-server prints raw escape
codes on the Windows 10 console while llama-cli, whose console code
enables it, shows colors. The logger now enables virtual terminal mode
on stdout and stderr when it turns colors on, and keeps colors off when
a console cannot render them. Pipes and files take the sequences as is.

* server: separate the router child commands from its logs

The child sent its state commands on the same pipe as its logs, so the
router had to pick them out of the log stream by a line prefix, and any
unterminated write in front of a command made the router miss it. This
resolves the TODO at the spawn that called for splitting stdout and
stderr.

The child now keeps stdout for the commands and points everything else
written to stdout at stderr, before anything is written. The router
reads both pipes, handles the commands from stdout and forwards stderr
as the log, and warns about any other line on the command pipe.

* server: address review from ngxson

The single server_child is now created first in the entry point and its
constructor keeps stdout for the commands, so the stream is a member of
the instance instead of a static, and init() is gone. The instance is
passed down to the server, while the CLI entry point creates its own.

* Update tools/server/server.cpp

---------

Co-authored-by: Xuan-Son Nguyen <thichthat@gmail.com>
2026-10-05 01:59:37 +02:00
0504396140 imatrix: calculate activation-based statistics for new format (GGUF) imatrices (#14891)
* Use activations to calculate the stats
* Determine calculation mode
* Compute entropy for activations
* Compute cosine similarity based on activations
* Compute l2 norm
* Add compute_layer_statistics() function
* Update aggregated statistic report layout
* Fix printing l2 norm when calc_mode = 1
* Refactor variable name
* Compute aggregated (per layer) l2 norm
* Update aggregated sum of squared activations per layer
* Make ZD Score two-tailed
* Update report layout
* Reverse conditional logic to match convention
* Rename report heading
* Add --activation-statistics parameter
* Add Euclidean–Cosine Score (ECS)
* Add --activation-statistics logic to avoid doubling the imatrix size by default
* Update stats output sort based on imatrix type
* Process external NextN draft files (-md / --model-draft)
* Refactor to use new llama_batch_ext

Co-authored-by: compilade <git@compilade.net>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-10-04 11:24:53 +02:00
Adrien Gallouët 6716df694b common : prepare load_from_models_dir() for path conversion (#29674)
This is part of the fs::path modernization series.
That was also the opportunity to remove fs_list().

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-10-04 11:35:33 +03:00
Adrien Gallouët edd6e2bbda common : add common_is_tty() helper and fix deprecated warnings on Windows (#29860)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-10-03 15:48:24 +02:00
Xuan-Son NguyenandSigbjørn Skjæret 99b95488ca model: add support for clef decision model (text-only) (#29831)
* init support for clef (text only)

* more static graph

* clean up

* nits

* nits 2

* Update gguf-py/gguf/constants.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-10-03 02:50:48 +02:00
1fb7ef3e33 spec : add probabilistic sampling for simple draft and MTP (#27694)
* Make the drafter probabilistic and the target verify by rejection sampling

* Drop stale spec_draft_q before drafting

* Fallback to argmax sampling for grammar-constrained requests and adding flag for enabling probabilistic draft sampling. Default flag value is greedy.

* Support grammar-constrained requests in rejection sampling

* Fix - renormalize distribution after masking

* copy rng on sampler copy and re-accept drafted tokens on replay

* Fix draft sampler sharing the target's rng stream

* Simplify the rejection sampler's inputs and move replay to the server

* Truncate the draft candidates along with the draft

---------

Co-authored-by: praneshgo <227579474+praneshgo@users.noreply.github.com>
Co-authored-by: Pranesh Gonegandla <pgonegandla@nvidia.com>
2026-10-02 17:52:54 +03:00
Xuan-Son Nguyen 46ca246de9 model: support nimble decision model (#29844) 2026-10-02 14:56:53 +02:00
Xuan-Son Nguyen a4cb4c61fd llama, server: add /v1/systemone API (models: laya, julia-1, lev, openjev, kev) (#29818)
* init conversion

* convert: ok

* model loaded

* add server code

* improve conversion script

* support shared prompt prefix

* add docs, imorove UX a bit

* add vision support

* add openjev tiny model for testing

* add dev docs

* support lev & kev

* clean up

* fix lev noul

* fix py lint

* nits docs

* clarify about not supporting date_facts
2026-10-02 11:56:04 +02:00
Adrien Gallouët 70849ee82c common : remove fs_open_ifstream() by using u8path() (#29841)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-10-02 11:43:45 +02:00
Adrien Gallouët f1cee9941b common,rpc : fix cache dir creation through symlinks on buggy libstdc++ (#29816)
See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101510

Close #29759

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-10-01 20:15:02 +02:00
Xuan-Son Nguyen 60e9cf7a7b batch: migrate the rest of examples to llama_batch_ext (#29601)
* migrate the rest

* test-thread-safety

* rm common_batch_staged
2026-09-30 18:08:43 +02:00
Adrien Gallouët 00af63567a common : use fs::path for config dir (#29649)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-09-29 17:22:49 +02:00
Adrien Gallouët 31385c9ceb common : add fs_write_atomic() (#29642)
- Check for buffered write errors when closing downloaded files.
- Use UTF-8 paths when writing ETag files on Windows.
- Write in binary mode on Windows.

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-09-29 13:46:53 +02:00
Adrien Gallouët 76a5bc86d1 common : use fs::path for cache dirs (#29595)
- Avoid useless string conversions on Windows.
- No need for BSD or emscripten special cases.

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-09-29 07:24:03 +02:00
Xuan-Son Nguyen f1ea206218 batch: migrate speculative, mtmd and server to batch_ext (#29385)
* adapt common

* add common_batch

* wip

* wip: spec

* cont

* common_speculative_process

* server_batch to use common_batch

* rm some stale calls

Assisted-by: Claude Fable 5.1

* migrate mtmd

* handle imrope, handle return val of add()/add_embd()

* add spec zeros vector

* add warning on zero fill path
2026-09-28 19:52:45 +02:00
Adrien Gallouët 6c7a87f7e5 common : fix HF cache paths on Windows (#29475)
Supersedes #29158

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-09-28 16:25:24 +02:00
Adrien Gallouët 136887b665 common : make string_split<T> throw on invalid values (#29518)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-09-27 18:04:11 +02:00
Adrien Gallouët 27b20ba8b1 common : extract shared unicode path/string helpers (#29415)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-09-25 11:40:41 +02:00
Xuan-Son Nguyen fc343a84bb llama: add llama_batch_ext (#24669)
* (wip) add llama_batch_ext

* wip

* updated design

* updated impl

* change signature

* unused var

* demo common_prompt_batch_decode

* fix pos

* tmp disable test-batch-alloc

* fix compat

* nits: add const

* no more pos_max

* add comment about llama_batch_ext_set_embd_state

* handle n_embd_out properly

* rename api --> embd_token

* llama_embd

* stub llama_batch_ext_set_embd_state

* support both token + embd + state in batch

* llama_batch_ext_add_embd

* upstream some changes

* nits

* fix test-batch-alloc

* add test for compat
2026-09-24 16:25:07 +02:00
Emanuil RusevandXuan Son Nguyen 217f81c266 server: Add support for binding to multiple addresses (#28690)
* Add support for binding llama-server to multiple addresses

Assisted-by: Codex

* remove redundant thread handler

* make it clear about overlapping addr

* reject --port 0 with multiple tcp addr

* improve arg handler

* nits

* fix test

* nits 2

* nits

* nits 2

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-09-22 15:16:40 +02:00
Georgi GerganovandXuan-Son Nguyen e750b887a8 common, server : enable preserve_reasoning kwarg by default, log its effective state (#28174)
* common, server : enable preserve_reasoning kwarg by default, log its effective state

If the preserve_reasoning chat template kwarg is not specified explicitly
via --reasoning-preserve / --no-reasoning-preserve, it is enabled by
default after argument processing. The server logs the effective state of
the kwarg, warns that it is enabled by default when the template supports
it, and only warns "has no effect" when it was enabled explicitly on a
template that does not support it. Setting the kwarg via
--chat-template-kwargs is deprecated.

Assisted-by: pi:llama.cpp/Qwen3.8-27B

* cont : update comment

Co-authored-by: Xuan-Son Nguyen <son@huggingface.co>

---------

Co-authored-by: Xuan-Son Nguyen <son@huggingface.co>
2026-09-02 19:19:54 +03:00
Xuan-Son Nguyen 50f068ffff bench: add --tensor-read-lazy (#27881)
* bench: add --tensor-read-lazy

* rm the alias

* rename to LLAMA_LAZY_MODE_*
2026-08-28 20:51:05 +02:00
BartowskiandXuan Son Nguyen 18443257a3 server: add ctx-per-slot (--kv-unified-per-slot) (#24124)
* Add ctx-per-slot argument for unifid KV cache

* Swap out ctx fractions for ctx pool slots

* Formatting cleanup

* Remove ctx-pool-slots, make ctx-per-slot an int

* refactor it

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-08-27 22:39:14 +02:00
Xuan-Son Nguyen fac889fb38 llama: model_loader: add TENSOR_READ_LAZY (#27794)
* llama: model_loader: add TENSOR_GET_ROW_LAZY

* add --tensor-read-lazy

* rename to TENSOR_READ_LAZY

* gen docs

* address comments
2026-08-27 15:14:34 +02:00
Gaurav Garg 2bb9bddafa spec: Add benchmark-only synthetic speculative acceptance options (#27711)
* Add benchmark-only synthetic speculative acceptance to llama-server and llama-cli

* Address review comments

* Address review comments

* Add some comments in the code
2026-08-27 13:53:42 +03:00
Xuan-Son Nguyen f29551215b args: add --video-* CLI arguments (#24318)
* args: add --video-* CLI arguments

* gen docs

* nits

* add mtmd_helper_init_opt
2026-08-27 12:11:12 +02:00
Jonas J c5fc7e3488 llama : add --n-cpu-ffn option (#26622)
* common : dedupe --n-cpu-moe / --spec-draft-n-cpu-moe override loops

* common : add --n-cpu-ffn to CPU-offload dense FFN weights of first N layers

* common : generalize llm_ffn_block_regex over the FFN regex, drop TODO
2026-08-27 11:26:42 +02:00
John-Henry LimandXuan Son Nguyen 681c29d36a mtmd: add --mmproj-device argument (#23255)
* feat: add --mmproj-device arg & backwards compatible MTMD_BACKEND_DEVICE env var

* feat: load mmproj device backend immediately, add -mmdev shortflag

* fix: its a pointer now get the name

* clean up

* gen docs

* nits

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-08-20 18:45:37 +02:00
Niklas Wenzel a94d563ed8 common: apply CPU parameters across tools (#27026) 2026-08-13 20:40:59 +02:00
Johnathan Craig MaudlinandXuan Son Nguyen 8e7f22b67e common: add system-level config file (#26118)
* common: Add CLI > ENV > models-presets > INI precedence

1. CLI flags have the highest precedence
2. ENV vars have the second-highest precedence
3. System and User configs have the lowest precedence
   - Linux/BSD/Mac
     - /etc/llama.cpp/config.ini < ${XDG_CONFIG_HOME:-~/.config}/llama.cpp/config.ini
   - Windows
     - %PROGRAMDATA%\llama.cpp\config.ini < %APPDATA%\llama.cpp\config.ini

* fix UB

* use common_get_env

* ignore_unknown_keys

* nits

* add docs

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-08-13 00:02:27 +02:00
153d324bcf llama: add default load-mode auto, which avoids mmap on iGPUs (#26081)
* llama: add new default load-mode auto which picks mmap unless a non-Metal iGPU is used

* Update ggml/src/ggml-hexagon/ggml-hexagon.cpp

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>

* set mmap_support to false on OpenCL backend

* fix order of load modes

* use -1 for auto

* resolve load mode auto earlier to correctly pick gpu host or cpu memory

* add load mode auto to llama-bench

* bump virtgpu api version, regenerate docs

---------

Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com>
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-08-11 09:20:46 +03:00
Gaurav GargandGeorgi Gerganov dd1ea52433 llama : support multi-output backend sampling (#25532)
* Enable backend sampling with token speculation

* Clamp the mask sum before converting it into the sampled index

* Add a numeric context parameter declaring the maximum outputs one sequence

* More fixes

* Don't reuse memory for output views.

* Match dist between CPU and GPU

* Fix CPU and backend sampling mismatches

* Simpify some of the changes

* Fix tests on Vulkan

* More test fixes

* Rebase changes

* Rebase and address review comments

* Address review comments

* Address review comments

* Update src/llama-sampler.cpp

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-08-10 16:58:56 +03:00
Xuan-Son NguyenandPascal dd2c7c4471 server: add initial tool isolation support (via docker) (#26507)
* server: add initial tool isolation support (via docker)

* add docs

* adapt get_info

* py: fix type check

* cont

* separate tools_io_sandbox / tools_io_docker

* rename sandbox --> isolate

* x-tool-docker --> x-tool-runtime

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-08-08 16:35:53 +02:00
Oliver Simons a6aa6f5450 sampler : remove "full-context windows" from history-based samplers (#26524)
* Resolve -1 to 1024 instead of ctx-len for samplers

Because of backend-sampling we initialize samplers before the complete
llama_context is there. Therefore, we cannot infer the resolved context
length yet at the time we construct the samplers.

* Shared default of 64 for history-based samplers, remove context_size
2026-08-04 21:28:55 +03:00
Xuan-Son NguyenandPascal 0713275082 mtmd: support Qwen3-TTS (note: breaking change to llama-tts binary) (#26254)
* convert text model

* main model load ok

* convert encoder ok

* speaker encoder loading ok

* speaker enc graph

* adapt vocab for backbone (with some tricks)

* add suppress_tokens

* poc new mtmd gen api

* convert code_predictor to gguf

* load gen_code model ok

* add clip_encode

* wire up

* code gen cgraph init version

Co-authored-by: Pascal <admin@serveurperso.com>

* code2wav convert to gguf

* code2wav graph ok

* wire up in/out

* (wip) subgraph

* wire up

* wip, correct code2wav

* demo (to be removed)

* code2wav preserve kv between calls

* demo voice clone

* llama: add llama_model_get_tok_embd

* mtmd_helper_gen_audio API

* fix clamp cold prefix

Co-authored-by: Pascal <admin@serveurperso.com>

* fuse snake op

Co-authored-by: Pascal <admin@serveurperso.com>

* demo: use proper sampling

* update dev docs

* polymorphism helper

* revamp llama-tts binary

* update docs

* fix compile

* fix lint

* nits

* add guide + docs

* more timings info

* clean up code comments

* security fixes

* update docs

* use ggml_build_forward_select, clean up comments

* fix ci

* use ISO 639-1 language code

* rename CODE2WAV --> GEN_WAV, update docs

* clean up

* clean up tts.cpp

* add seq_id

* add step_prompt()

* mtmd_helper_model_can_chat

* clean up comments

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-08-04 17:26:15 +02:00
PascalandXuan Son Nguyen ee0445c99c tests: add model resolution test on synthetic repo listings (#26172)
* tests: add model resolution test on synthetic repo listings

Include download.cpp and arg.cpp inside a namespace with hf_cache
monkey patched to serve hardcoded listings, so the resolution and the
model handler assembly are tested end-to-end through the real CLI
parsing, without modifying the tested code and without network access.

Covers the primary, shard, mmproj, sidecar and preset resolution on
layouts mimicking real vendor conventions, replays every case on
permutations of the listing to assert determinism, and asserts the
final wired paths for the spec type auto-selection, the -md precedence
and the fallback suppression.

* tests: keep model resolution checks active and let the handler resolve

Replace assert with a REQUIRE macro alive in Release builds, key the
fake hf_cache by repo id so the real handler init resolves every plan
itself, check the exact shard sets, restrict the permutation exception
to the order dependent picks, and cover dflash and eagle3.

* tests: fix model resolution build on fatal warnings CI and Windows

The namespaced copy of the sources leaves many static functions unused
in this TU, exempt it from the unused warnings. Pre-include the
windows headers so arg.cpp does not pull them inside the namespace.
Declare the renamed copies of the download.h functions, verbatim from
the header and renamed in sync by the macros, so missing declaration
and missing prototype warnings are satisfied on every toolchain.

* tests: fix winsock inclusion order for the model resolution test

WIN32_LEAN_AND_MEAN and winsock2.h before windows.h, so http.h does
not redefine the socket types afterwards.

* tests: link cpp-httplib to the model resolution test

The test compiles its own copy of download.cpp, which calls httplib
directly, and the private link of llama-common does not propagate the
symbols under lld-link.

* common_http_client

* common: finish the http client wrapper

Add the virtual Head, Get and Post methods and the passthrough
setters to the common_http_client skeleton, move follow_location
into the constructor, expose the underlying client for the ranged
pull path, and rename the missed common_http_client_init call sites.

* tests: rewrite model resolution on the http client stub

Replace the namespace inclusion of the sources by a plain TU: the
common_http_client factory returns a stub serving hardcoded HF API
responses, so the real hf_cache parsing, resolution and CLI handler
run against synthetic listings in an isolated cache directory.

Failures print the named case, the reordering and the actual versus
expected values, the assembly cases use the full command line as
context, and the empty result cases are checked once to keep the
logs short.

* tests: fix the model resolution on Windows and the builds without TLS

Assert the exact expected paths composed like the cache does instead of
suffix matching on forward slashes, set the environment portably, and
serve the stub through an http endpoint so the builds rejecting the
https scheme still reach it. Pause the log so the negative cases can be
replayed on every reordering.

* tests: make the model resolution failures self explanatory

Resume the paused log before the failure report so the CI shows why
the tested code bailed, and format the stub oids portably.

* common: hold the http client factory behind exported functions

The factory was an inline variable, and the Windows shared builds
export functions but not data, so the executable and the DLL each had
their own instance: the stub installed by the test was invisible to
the library, which reached for the real endpoint and resolved nothing.
Route the creation through functions compiled into the library and
format the stub oids portably.

* common: add the http client factory source missed in the previous commit

* common: typedef the http client factory callback

Address review from @ngxson

* tests: serve the model resolution repos over the loopback

Replace the client stub by a real httplib server bound to the
loopback, so no C++ object crosses the module boundary anymore and
the library exercises its own client and transport end to end. The
debug shared build on Windows crashed inside the stubbed path.

* common: add portable common_get_env and common_set_env helpers

Address review from @ngxson

* common: drop the http client factory left without a caller

The loopback server made the stub substitution unnecessary, the client
init builds the real client directly again.

* common: read the model endpoint through the env helpers

* nit: drop the stub leftovers from the model resolution test

* common: align common_set_env and isolate the test cache per run

The POSIX branch now behaves like _putenv_s, so the helper has a single
contract on every platform, and common_get_env already reads an unset and
an empty variable alike.

The model resolution test keys its cache directory on the loopback port,
where two concurrent runs on the same machine used to share one directory
and the initial cleanup of either wiped the other.

* tests: move the model resolution server into main

* tests: support the DSpark sidecar resolution

* common: revert the http client to the plain httplib client

address review from @ngxson

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-08-03 18:58:15 +02:00
Xuan-Son NguyenandPascal 99111b19ce server: add get_info tool (#26522)
* server: add get_info tool

* fix --rpc in docs

* server: harden get_info probe result handling

Report the OS as unknown when the probe process fails to spawn or times
out, so the diagnostic text from run() is never returned as an OS name.
Strip the probe output on both ends, which also drops the blank line
that ver prints before the version on Windows. Name the output and
timeout limits, and report an unreadable working directory as unknown
instead of an empty string.

* server: simplify get_info result handling

Drop the named limits and the working directory error branch, keeping
the probe result handling to a single expression.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-08-03 18:51:02 +02:00
Georgi Gerganov 47f686f53f tests : avoid building get-model.cpp many times (#26317)
* tests : remove get-model.cpp

* tests : fix quant type selection
2026-07-30 19:34:04 +03:00
Xuan-Son Nguyen afeebe103b llama: move suppress_tokens handling to common/sampling (#26276)
* llama: move suppress_tokens handling to common/sampling

* address security issues

* rm has_logit_bias
2026-07-29 18:02:30 +02:00
Xuan-Son Nguyen ee3d1b54c1 server: abstract llama_memory calls to common_memory (#26221) 2026-07-28 16:35:20 +02:00