* model : allow loading external PLE files
Assisted-by: Codex
* convert : add separate PLE exports
Assisted-by: Codex
* tests : fix PLE export test lint
Assisted-by: Codex
* model : make --ple an alias of --shard, allow multiple shard locations
Reuse the multi-shard loading path for external PLE files instead of a
separate option.
* --ple is an alias of --shard: both take a comma separated list and may
be given more than once, and the shards may live anywhere
* metadata comes from the main file; numbered shards are still
auto-discovered and strictly validated
* explicit files outside the discovered set are appended as extra shards,
exempt from split.no and split.count checks
* -isnm/--ignore-shard-num-mismatch loads only the given files, in order,
with no auto-discovery and no split metadata validation
* ple_path, path_ple and llama_model_loader::add_ple_file are removed;
common_model_load() uses llama_model_load_from_splits for two or more
paths
* common/fit.{h,cpp} take the shard path vector, so --fit and the router
memory measurement load every shard
Assisted-by: GPT-6-Astra
* rm -isnm
* make sure to dedup paths
* rm test_convert_ple
* address comments
* allow ple shard to override tensor
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* fit : account for skipped NextN layers when setting n_gpu_layers
Without MTP, fit counts n_layer + 1 layers while the model loader counts
n_layer_all + 1 for n_gpu_layers, including the NextN layers which hold no
weights. A fit result that offloads all layers then leaves layer 0 on the CPU
("offloaded 65/66 layers" for Qwen3.8-27B on two GPUs).
Add the skipped NextN layers to n_gpu_layers and to the tensor split of the
device that holds the output layer so the loader assigns the layers that
fit measured.
Fixes#31531
* fit : pass hp_ngl_skip by reference like the other hparams
* fit : add hp_ngl_skip to common_get_device_memory_data
* server: add --models-memory-max parameter to allow dynamically unloading models when they exceed a memory size threshold
estimate with to-be-loaded model size included
use no_alloc to get memory requirements for model load
only set model memory_mb if not previously calculated
use memory margin instead of total size limit, apply to each device separately
add server memory debug logging
move llama_context_device_memory function to llama-ext.h
fix model count exceeded check
improve memory_per_device map naming
improve variable naming, fix style
also strip models memory margin from child processes
cont : clean-up
replace device memory map with buft memory map. Use llama_get_memory_breakdown
extract duplicated check into helper function
move model memory estimation to subprocess
precompute name->buft map, map GPU host types to CPU buft
cleanup unused variable
remove duplicated init calls
* adapt to upstream changes
* wip
* common_memory_reports internface
* SERVER_ROLE_MEASURE
* server-models-sched
* fix build
* server : fix memory accounting of measure children and claim spin
A measure child runs no_alloc, so the free memory it reports does not
include its estimate: subtracting self there erased the memory held by
other processes, down to zero with --fit on, and the next load could
overshoot the margin. Subtract self only for instance reports.
A queued load only checked --models-max before claiming, so when the
memory margin blocked it behind a busy model, load() threw and the
waiter retried in a tight loop. Refuse the claim while the limits are
exceeded and no idle model can be unloaded, so the waiter sleeps on
the condition variable until one goes idle.
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: Pascal <admin@serveurperso.com>
* fit: also take into account n_streams
* server: make the draft context follow the target context
With a non-unified KV cache the target context now holds n_ctx_train
tokens per sequence, while the draft context was still created with
n_ctx = 0 and fell back to n_ctx_train / n_streams per sequence. A slot
filled beyond that point makes the draft batch fail to decode, and the
server answers 500 on the request.
The draft context now takes its size from the target context, so both
hold the same number of tokens per sequence. Contexts that share their
cells with the target no longer need the kv_size override.
The memory reserved for the draft model before fitting is measured at
the largest context the target can take, since the draft context grows
with the target and a fixed byte margin cannot express that.
* fit: take an optional second model into account
Illustrates the alternative discussed on the draft context fix. The
memory of a draft or MTP context is currently handed to the fit as a
fixed byte margin, which cannot express a memory that grows with the
context the fit is still deciding on.
common_fit_params now takes an optional second model that shares the
devices of the main one. Its context follows the main context and its
memory is measured again whenever that context changes, so the reduce
path stays exact instead of conservative. A model that cannot be
measured on its own, such as a shared cell MTP context, is skipped with
a warning and the main model is fitted alone.
This drops the reservation block in the server, which no longer has to
probe the trained context size of the target to guess an upper bound.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* common: do not fit to unknown device memory
Signed-off-by: Florian Reinle <f.reinle@otec.de>
* common: preserve host fallback for non-GPU fit devices
Signed-off-by: Florian Reinle <f.reinle@otec.de>
* common: keep unknown GPU fit memory at zero
Signed-off-by: Florian Reinle <f.reinle@otec.de>
---------
Signed-off-by: Florian Reinle <f.reinle@otec.de>