Files
llama.cpp/docs/models.md
T
Piotr Wilkin (ilintar)andXuan Son Nguyen 5f490d7bfe feat: external PLE file loading (#31185)
* model : allow loading external PLE files

Assisted-by: Codex

* convert : add separate PLE exports

Assisted-by: Codex

* tests : fix PLE export test lint

Assisted-by: Codex

* model : make --ple an alias of --shard, allow multiple shard locations

Reuse the multi-shard loading path for external PLE files instead of a
separate option.

* --ple is an alias of --shard: both take a comma separated list and may
  be given more than once, and the shards may live anywhere
* metadata comes from the main file; numbered shards are still
  auto-discovered and strictly validated
* explicit files outside the discovered set are appended as extra shards,
  exempt from split.no and split.count checks
* -isnm/--ignore-shard-num-mismatch loads only the given files, in order,
  with no auto-discovery and no split metadata validation
* ple_path, path_ple and llama_model_loader::add_ple_file are removed;
  common_model_load() uses llama_model_load_from_splits for two or more
  paths
* common/fit.{h,cpp} take the shard path vector, so --fit and the router
  memory measurement load every shard

Assisted-by: GPT-6-Astra

* rm -isnm

* make sure to dedup paths

* rm test_convert_ple

* address comments

* allow ple shard to override tensor

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-11-09 19:41:01 +01:00

4.0 KiB

Obtaining and quantizing models

The Hugging Face platform hosts thousands of models compatible with llama.cpp:

You can use any llama.cpp-compatible model from Hugging Face using this CLI argument: -hf <user>/<model>[:quant]. For example:

llama cli -hf ggml-org/gemma-3-1b-it-GGUF

You can use the same CLI invocation to download from other sites, by pointing the MODEL_ENDPOINT environment variable to an endpoint compatible with the Hugging Face API. llama.cpp can also run models you have downloaded locally to your filesystem.

After downloading a model, use the CLI tools to run it locally - see below.

llama.cpp requires the model to be stored in the GGUF file format. Models in other data formats can be converted to GGUF using the convert_*.py Python scripts in this repo. To learn more about model quantization, read this documentation

The Hugging Face platform provides a variety of online tools for converting, quantizing and hosting models with llama.cpp:

Exporting per-layer embeddings separately

For models with a per-layer embedding (PLE) table, export the backbone and table from the same checkpoint:

python3 convert_hf_to_gguf.py /path/to/model --no-ple --outfile model-no-ple.gguf
python3 convert_hf_to_gguf.py /path/to/model --ple-only --outfile model-ple.gguf
llama-cli -m model-no-ple.gguf --shard model-ple.gguf

--no-ple excludes only per_layer_token_embd.weight and keeps the PLE metadata and projection weights. --ple-only exports only that table, without a tokenizer. Each export supports --outtype, so the table and backbone can use different precisions. Default output names start with no-ple- and ple- respectively.

These modes are mutually exclusive and require a model with a PLE table, such as Qwen3.8 Flash Next or Gemma 4. They cannot be combined with vocabulary, multimodal projector, MTP-only, or DSpark-only export. --no-ple can be combined with --no-mtp.

--ple is an alias for --shard. Extra shards can be comma separated or supplied with repeated options, for example -m backbone.gguf --shard /fast/ple.gguf,/other/token-embd.gguf. Paths may be on different drives. Tensor names must be unique across all files, except for the PLE case below.

A numbered backbone is auto-discovered and checked as usual; explicitly supplied paths outside that numbered set are appended as extra shards. Their split numbering is ignored, and the loader uses the actual combined tensor count. Paths already in the discovered set are loaded once. If the main filename does not follow the numbered naming scheme, an explicit backbone list must still match its declared split count and order.

If an extra shard holds only per_layer_token_embd.weight, it replaces the PLE table of the model. If the model already has the table, the extra shard overrides it. A backbone split that holds only that tensor is skipped, and it can be absent from disk. This lets one PLE file be shared across quants and releases, for example -m model-Q4_K_M-00001-of-00003.gguf --shard /fast/ple-Q8_0.gguf.