* dflash : move the DSpark markov head and backbone graphs to dspark.cpp Move-only: build_dspark_markov_head, graph_dsv4, graph_dsv41 and graph_mistral_large4 move verbatim. build_inp_target_features and build_dspark_markov_head become static members of llama_model_dflash so both files can use them. * dflash : detect the head and the backbone once, split the loaders per file load_arch_hparams sets llama_model_dflash::variant (DFlash, DFlash2, DSpark, LiliCorr) and ::backbone (base, DSV4, DSV4.1, Mistral Large 4). The head, backbone and base layer loaders become member helpers in dflash.cpp, dflash2.cpp, lilicorr.cpp and dspark.cpp. build_arch_graph dispatches on the backbone. The DFlash2 and LiliCorr loaders create the DFlash2 conv tensors through a helper in dflash2.cpp, right after their head tensors instead of in the base layer loop; the other tensors keep their creation order. The DFlash2 loader drops its copy of the conv group size check, which master never reached. * dflash : share the injection input, token embeddings and lm head lookups build_inject_inp (target features + fc + encoder norm), get_tok_embd and get_lm_head replace the copies in the base, DSV4, DSV4.1 and Mistral Large 4 graphs. build_inp_target_features is file-static again. * dflash : split the DFlash decoder into a graph per head graph_base creates the shared inputs and provides build_inject, build_inp_tokens, build_layers and build_logits. graph_dflash (dflash.cpp), graph_dflash2 (dflash2.cpp), graph_dspark (dspark.cpp) and graph_lilicorr (lilicorr.cpp) assemble them and add their head; the selector, scorer and markov head builders move to the file of their graph, and the DFlash2 local conv run by build_layers to dflash2.cpp. build_arch_graph dispatches on the backbone, then on the head. The op order and the tensor names are unchanged. * speculative : split the DFlash driver into a class per head common_speculative_impl_draft_dflash stays the DFlash implementation and becomes the base of draft_dspark, draft_dflash2 and draft_lilicorr. Its protected constructor only reads the draft metadata; init() then sets up the contexts with the options of the draft (draft length, backend sampling, device features, masked h_nextn). Each derived class overrides draft() with the path of its head from the former draft(). DFlash2 also overrides begin() to reseed its generator per request, and LiliCorr overrides process() to store the context features of each injected chunk, which its noise block carries as its state. The factory keeps the current routing: draft-dspark picks DSpark, draft-dflash picks DFlash2 or LiliCorr from the draft metadata and DFlash otherwise. draft-dspark with a DFlash2 or LiliCorr draft now runs the DSpark reader instead of a mix of both. sample_from_anchor is logged by DSpark only. * speculative : add the draft-dflash2 and draft-lilicorr types draft-dflash2 and draft-lilicorr name the DFlash2 and LiliCorr drafts, which draft-dflash keeps running too: it picks them from the draft metadata. draft-dflash2 and draft-lilicorr must match the draft model, draft-dspark rejects a DFlash2 or LiliCorr draft, a model that is not a DFlash draft is rejected, and a run uses at most one DFlash family type; these errors leave speculative decoding uninitialized. The type detected from the draft GGUF is now the concrete one, and draft-dflash2 and draft-lilicorr download the dflash sidecar. * docs : describe the DFlash2 and LiliCorr drafts Add the DFlash2 and LiliCorr sections and the new types, and explain that draft-dflash also runs DFlash2 and LiliCorr drafts. * speculative : drop the unused DFlash is_mrope Nothing reads it since the migration to common_batch (#29385), which picks the number of M-RoPE position rows from the rope type of the draft. * doc: update spec dec * address review comments
31 KiB
31 KiB
llama.cpp/tools/cli
Usage
Common params
| Argument | Explanation |
|---|---|
-h, --help, --usage |
print usage and exit |
--version |
show version and build info |
-cl, --cache-list |
show list of models in cache |
--completion-bash |
print source-able bash completion script for llama.cpp |
-t, --threads N |
number of CPU threads to use during generation (default: -1) (env: LLAMA_ARG_THREADS) |
-tb, --threads-batch N |
number of threads to use during batch and prompt processing (default: same as --threads) |
-C, --cpu-mask M |
CPU affinity mask: arbitrarily long hex. Complements cpu-range (default: "") |
-Cr, --cpu-range lo-hi |
range of CPUs for affinity. Complements --cpu-mask |
--cpu-strict <0|1> |
use strict CPU placement (default: 0) |
--prio N |
set process/thread priority : low(-1), normal(0), medium(1), high(2), realtime(3) (default: 0) |
--poll <0...100> |
use polling level to wait for work (0 - no polling, default: 50) |
-Cb, --cpu-mask-batch M |
CPU affinity mask: arbitrarily long hex. Complements cpu-range-batch (default: same as --cpu-mask) |
-Crb, --cpu-range-batch lo-hi |
ranges of CPUs for affinity. Complements --cpu-mask-batch |
--cpu-strict-batch <0|1> |
use strict CPU placement (default: same as --cpu-strict) |
--prio-batch N |
set process/thread priority : 0-normal, 1-medium, 2-high, 3-realtime (default: 0) |
--poll-batch <0|1> |
use polling to wait for work (default: same as --poll) |
-c, --ctx-size N |
size of the prompt context (default: 0, 0 = loaded from model) (env: LLAMA_ARG_CTX_SIZE) |
-n, --predict, --n-predict N |
number of tokens to predict (default: -1, -1 = infinity) (env: LLAMA_ARG_N_PREDICT) |
-b, --batch-size N |
logical maximum batch size (default: 2048) (env: LLAMA_ARG_BATCH) |
-ub, --ubatch-size N |
physical maximum batch size (default: 512) (env: LLAMA_ARG_UBATCH) |
--keep N |
number of tokens to keep from the initial prompt (default: 0, -1 = all) |
--swa-full |
use full-size SWA cache (default: false) (more info) (env: LLAMA_ARG_SWA_FULL) |
-fa, --flash-attn [on|off|auto] |
set Flash Attention use ('on', 'off', or 'auto', default: 'auto') (env: LLAMA_ARG_FLASH_ATTN) |
-p, --prompt PROMPT |
prompt to start generation with; for system message, use -sys |
--perf, --no-perf |
whether to enable internal libllama performance timings (default: false) (env: LLAMA_ARG_PERF) |
-f, --file FNAME |
a file containing the prompt (default: none) |
-bf, --binary-file FNAME |
binary file containing the prompt (default: none) |
-e, --escape, --no-escape |
whether to process escapes sequences (\n, \r, \t, ', ", \) (default: true) |
--rope-scaling {none,linear,yarn} |
RoPE frequency scaling method, defaults to linear unless specified by the model (env: LLAMA_ARG_ROPE_SCALING_TYPE) |
--rope-scale N |
RoPE context scaling factor, expands context by a factor of N (env: LLAMA_ARG_ROPE_SCALE) |
--rope-freq-base N |
RoPE base frequency, used by NTK-aware scaling (default: loaded from model) (env: LLAMA_ARG_ROPE_FREQ_BASE) |
--rope-freq-scale N |
RoPE frequency scaling factor, expands context by a factor of 1/N (env: LLAMA_ARG_ROPE_FREQ_SCALE) |
--yarn-orig-ctx N |
YaRN: original context size of model (default: 0 = model training context size) (env: LLAMA_ARG_YARN_ORIG_CTX) |
--yarn-ext-factor N |
YaRN: extrapolation mix factor (default: -1.00, 0.0 = full interpolation) (env: LLAMA_ARG_YARN_EXT_FACTOR) |
--yarn-attn-factor N |
YaRN: scale sqrt(t) or attention magnitude (default: -1.00) (env: LLAMA_ARG_YARN_ATTN_FACTOR) |
--yarn-beta-slow N |
YaRN: high correction dim or alpha (default: -1.00) (env: LLAMA_ARG_YARN_BETA_SLOW) |
--yarn-beta-fast N |
YaRN: low correction dim or beta (default: -1.00) (env: LLAMA_ARG_YARN_BETA_FAST) |
-kvo, --kv-offload, -nkvo, --no-kv-offload |
whether to enable KV cache offloading (default: enabled) (env: LLAMA_ARG_KV_OFFLOAD) |
--repack, -nr, --no-repack |
whether to enable weight repacking (default: enabled) (env: LLAMA_ARG_REPACK) |
--no-host |
bypass host buffer allowing extra buffers to be used (env: LLAMA_ARG_NO_HOST) |
-ctk, --cache-type-k TYPE |
KV cache data type for K allowed values: f32, f16, bf16, f8_e4m3, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1 (default: f16) (env: LLAMA_ARG_CACHE_TYPE_K) |
-ctv, --cache-type-v TYPE |
KV cache data type for V allowed values: f32, f16, bf16, f8_e4m3, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1 (default: f16) (env: LLAMA_ARG_CACHE_TYPE_V) |
-np, --parallel N |
number of parallel sequences to decode (default: 1) (env: LLAMA_ARG_N_PARALLEL) |
--rpc SERVERS |
comma-separated list of RPC servers (host:port) (env: LLAMA_ARG_RPC) |
-lm, --load-mode MODE |
model loading mode (default: auto) - auto: mmap, unless a device does not support it - none: no special loading mode - mmap: memory-map model (if mmap disabled, slower load but may reduce pageouts if not using mlock) - mlock: force system to keep model in RAM rather than swapping or compressing - mmap+mlock: mmap + force system to keep model in RAM rather than swapping or compressing - dio: use DirectIO if available (env: LLAMA_ARG_LOAD_MODE) |
-lzm, --lazy-mode MODE |
on-demand reading of certain tensors, for example per-layer embeddings (default: auto) - on: read the rows of such tensors from disk on demand instead of keeping them resident (requires mmap) - auto: on, but only for tensors larger than 4 GiB - off: always keep them resident (env: LLAMA_ARG_LAZY_MODE) |
--numa TYPE |
attempt optimizations that help on some NUMA systems - distribute: spread execution evenly over all nodes - isolate: only spawn threads on CPUs on the node that execution started on - numactl: use the CPU map provided by numactl if run without this previously, it is recommended to drop the system page cache before using this see https://github.com/ggml-org/llama.cpp/issues/1437 (env: LLAMA_ARG_NUMA) |
-dev, --device <dev1,dev2,..> |
comma-separated list of devices to use for offloading (none = don't offload) use --list-devices to see a list of available devices (env: LLAMA_ARG_DEVICE) |
--list-devices |
print list of available devices and exit |
-ot, --override-tensor <tensor name pattern>=<buffer type>,... |
override tensor buffer type (env: LLAMA_ARG_OVERRIDE_TENSOR) |
-cmoe, --cpu-moe |
keep all Mixture of Experts (MoE) weights in the CPU (env: LLAMA_ARG_CPU_MOE) |
-ncmoe, --n-cpu-moe N |
keep the Mixture of Experts (MoE) weights of the first N layers in the CPU (env: LLAMA_ARG_N_CPU_MOE) |
--moe-cache-mib N |
GPU cache size in MiB for the MoE experts kept in the CPU. with multiple GPUs, it is split among them like the layers (--tensor-split) (default: 0, disabled) (env: LLAMA_ARG_MOE_CACHE_MIB) |
--moe-cache-cpu-share F |
share of the MoE cache misses that the CPU computes from the host experts instead of uploading them to the GPU once the cache is full, from 0 to 1 (default: 0.00) (env: LLAMA_ARG_MOE_CACHE_CPU_SHARE) |
-ncffn, --n-cpu-ffn N |
keep the dense FFN weights of the first N layers in the CPU (dense models; for MoE expert weights use --n-cpu-moe) (env: LLAMA_ARG_N_CPU_FFN) |
-ngl, --gpu-layers, --n-gpu-layers N |
max. number of layers to store in VRAM, either an exact number, 'auto', or 'all' (default: auto) (env: LLAMA_ARG_N_GPU_LAYERS) |
-sm, --split-mode {none,layer,row,tensor} |
how to split the model across multiple GPUs, one of: - none: use one GPU only - layer (default): split layers and KV across GPUs (pipelined) - row: split weight across GPUs by rows (parallelized) - tensor: split weights and KV across GPUs (parallelized, EXPERIMENTAL) (env: LLAMA_ARG_SPLIT_MODE) |
-ts, --tensor-split N0,N1,N2,... |
fraction of the model to offload to each GPU, comma-separated list of proportions, e.g. 3,1 (env: LLAMA_ARG_TENSOR_SPLIT) |
-mg, --main-gpu INDEX |
the GPU to use for the model (with split-mode = none), or for intermediate results and KV (with split-mode = row) (default: 0) (env: LLAMA_ARG_MAIN_GPU) |
-fit, --fit [on|off] |
whether to adjust unset arguments to fit in device memory ('on' or 'off', default: 'on') (env: LLAMA_ARG_FIT) |
-fitt, --fit-target MiB0,MiB1,MiB2,... |
target margin per device for --fit, comma-separated list of values, single value is broadcast across all devices, default: 1024 (env: LLAMA_ARG_FIT_TARGET) |
-fitc, --fit-ctx N |
minimum ctx size that can be set by --fit option, default: 4096 (env: LLAMA_ARG_FIT_CTX) |
--check-tensors |
check model tensor data for invalid values (default: false) |
--override-kv KEY=TYPE:VALUE,... |
advanced option to override model metadata by key. to specify multiple overrides, either use comma-separated values. types: int, float, bool, str. example: --override-kv tokenizer.ggml.add_bos_token=bool:false,tokenizer.ggml.add_eos_token=bool:false |
--op-offload, --no-op-offload |
whether to offload host tensor operations to device (default: true) |
--lora FNAME |
path to LoRA adapter (use comma-separated values to load multiple adapters) |
--lora-scaled FNAME:SCALE,... |
path to LoRA adapter with user defined scaling (format: FNAME:SCALE,...) note: use comma-separated values |
--control-vector FNAME |
add a control vector note: use comma-separated values to add multiple control vectors |
--control-vector-scaled FNAME:SCALE,... |
add a control vector with user defined scaling SCALE note: use comma-separated values (format: FNAME:SCALE,...) |
--control-vector-layer-range START END |
layer range to apply the control vector(s) to, start and end inclusive |
-m, --model FNAME |
model path to load (env: LLAMA_ARG_MODEL) |
--ple, --shard FNAME,... |
extra GGUF shards for the model, comma separated; may be given more than once paths may live anywhere, including on different drives; --ple is an alias for --shard (env: LLAMA_ARG_SHARD) |
-mu, --model-url MODEL_URL |
model download url (default: unused) (env: LLAMA_ARG_MODEL_URL) |
-dr, --docker-repo [<repo>/]<model>[:quant] |
Docker Hub model repository. repo is optional, default to ai/. quant is optional, default to :latest. example: gemma3 (default: unused) (env: LLAMA_ARG_DOCKER_REPO) |
-hf, -hfr, --hf-repo <user>/<model>[:quant] |
Hugging Face model repository; quant is optional, case-insensitive, default to Q4_K_M, or falls back to the first file in the repo if Q4_K_M doesn't exist. mmproj is also downloaded automatically if available. to disable, add --no-mmproj example: ggml-org/GLM-4.7-Flash-GGUF:Q4_K_M (default: unused) (env: LLAMA_ARG_HF_REPO) |
-hff, --hf-file FILE |
Hugging Face model file. If specified, it will override the quant in --hf-repo (default: unused) (env: LLAMA_ARG_HF_FILE) |
-hft, --hf-token TOKEN |
Hugging Face access token (default: value from HF_TOKEN environment variable) (env: HF_TOKEN) |
--log-disable |
Log disable |
--log-file FNAME |
Log to file (env: LLAMA_ARG_LOG_FILE) |
--log-jsonl, --no-log-jsonl |
Log as JSONL (one JSON object per line) to stdout, this also disables colored logging (default: disabled) (env: LLAMA_ARG_LOG_JSONL) |
--log-colors [on|off|auto] |
Set colored logging ('on', 'off', or 'auto', default: 'auto') 'auto' enables colors when output is to a terminal (env: LLAMA_ARG_LOG_COLORS) |
-v, --verbose, --log-verbose |
Set verbosity level to infinity (i.e. log all messages, useful for debugging) |
--offline |
Offline mode: forces use of cache, prevents network access (env: LLAMA_ARG_OFFLINE) |
-lv, --verbosity, --log-verbosity N |
Set the verbosity threshold. Messages with a higher verbosity will be ignored. Values: - 0: generic output - 1: error - 2: warning - 3: info - 4: trace (more info) - 5: debug (default: 3) (env: LLAMA_ARG_LOG_VERBOSITY) |
--log-prefix, --no-log-prefix |
Enable prefix in log messages (env: LLAMA_ARG_LOG_PREFIX) |
--log-timestamps, --no-log-timestamps |
Enable timestamps in log messages (env: LLAMA_ARG_LOG_TIMESTAMPS) |
--spec-draft-type-k, -ctkd, --cache-type-k-draft TYPE |
KV cache data type for K for the draft model allowed values: f32, f16, bf16, f8_e4m3, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1 (default: f16) (env: LLAMA_ARG_SPEC_DRAFT_CACHE_TYPE_K) |
--spec-draft-type-v, -ctvd, --cache-type-v-draft TYPE |
KV cache data type for V for the draft model allowed values: f32, f16, bf16, f8_e4m3, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1 (default: f16) (env: LLAMA_ARG_SPEC_DRAFT_CACHE_TYPE_V) |
Sampling params
| Argument | Explanation |
|---|---|
--samplers SAMPLERS |
samplers that will be used for generation in the order, separated by ';' (default: penalties;dry;top_n_sigma;top_k;typ_p;top_p;min_p;xtc;temperature) |
-s, --seed SEED |
RNG seed (default: -1, use random seed for -1) |
--sampler-seq, --sampling-seq SEQUENCE |
simplified sequence for samplers that will be used (default: edskypmxt) |
--ignore-eos |
ignore end of stream token and continue generating (implies --logit-bias EOS-inf) |
--temp, --temperature N |
temperature (default: 0.80) (env: LLAMA_ARG_TEMPERATURE) |
--top-k N |
top-k sampling (default: 40, 0 = disabled) (env: LLAMA_ARG_TOP_K) |
--top-p N |
top-p sampling (default: 0.95, 1.0 = disabled) (env: LLAMA_ARG_TOP_P) |
--min-p N |
min-p sampling (default: 0.05, 0.0 = disabled) (env: LLAMA_ARG_MIN_P) |
--top-nsigma, --top-n-sigma N |
top-n-sigma sampling (default: -1.00, -1.0 = disabled) |
--xtc-probability N |
xtc probability (default: 0.00, 0.0 = disabled) |
--xtc-threshold N |
xtc threshold (default: 0.10, 1.0 = disabled) |
--typical, --typical-p N |
locally typical sampling, parameter p (default: 1.00, 1.0 = disabled) |
--repeat-last-n N |
last n tokens to consider for penalize (default: 64, 0 = disabled) |
--repeat-penalty N |
penalize repeat sequence of tokens (default: 1.00, 1.0 = disabled) (env: LLAMA_ARG_REPEAT_PENALTY) |
--presence-penalty N |
repeat alpha presence penalty (default: 0.00, 0.0 = disabled) (env: LLAMA_ARG_PRESENCE_PENALTY) |
--frequency-penalty N |
repeat alpha frequency penalty (default: 0.00, 0.0 = disabled) (env: LLAMA_ARG_FREQUENCY_PENALTY) |
--dry-multiplier N |
set DRY sampling multiplier (default: 0.00, 0.0 = disabled) |
--dry-base N |
set DRY sampling base value (default: 1.75) |
--dry-allowed-length N |
set allowed length for DRY sampling (default: 2) |
--dry-penalty-last-n N |
set DRY penalty for the last n tokens (default: 64, 0 = disable) |
--dry-sequence-breaker STRING |
add sequence breaker for DRY sampling, clearing out default breakers ('\n', ':', '"', '*') in the process; use "none" to not use any sequence breakers |
--adaptive-target N |
adaptive-p: select tokens near this probability (valid range 0.0 to 1.0; negative = disabled) (default: -1.00) (more info) |
--adaptive-decay N |
adaptive-p: decay rate for target adaptation over time. lower values are more reactive, higher values are more stable. (valid range 0.0 to 0.99) (default: 0.90) |
--dynatemp-range N |
dynamic temperature range (default: 0.00, 0.0 = disabled) |
--dynatemp-exp N |
dynamic temperature exponent (default: 1.00) |
--mirostat N |
use Mirostat sampling. Top K, Nucleus and Locally Typical samplers are ignored if used. (default: 0, 0 = disabled, 1 = Mirostat, 2 = Mirostat 2.0) |
--mirostat-lr N |
Mirostat learning rate, parameter eta (default: 0.10) |
--mirostat-ent N |
Mirostat target entropy, parameter tau (default: 5.00) |
-l, --logit-bias TOKEN_ID(+/-)BIAS |
modifies the likelihood of token appearing in the completion, i.e. --logit-bias 15043+1 to increase likelihood of token ' Hello',or --logit-bias 15043-1 to decrease likelihood of token ' Hello' |
--grammar GRAMMAR |
BNF-like grammar to constrain generations (see samples in grammars/ dir) |
--grammar-file FNAME |
file to read grammar from |
-j, --json-schema SCHEMA |
JSON schema to constrain generations (https://json-schema.org/), e.g. {"type": "object"} for any JSON object |
-jf, --json-schema-file FILE |
File containing a JSON schema to constrain generations (https://json-schema.org/), e.g. {"type": "object"} for any JSON object |
-bs, --backend-sampling |
enable backend sampling (experimental) (default: disabled) (env: LLAMA_ARG_BACKEND_SAMPLING) |
CLI-specific params
| Argument | Explanation |
|---|---|
--server-base URL |
connect to this server instead of starting a new one, example: 'http://localhost:9931' (default: none) |
--verbose-prompt |
print a verbose prompt before generation (default: false) |
--display-prompt, --no-display-prompt |
whether to print prompt at generation (default: true) |
-co, --color [on|off|auto] |
Colorize output to distinguish prompt and user input from generations ('on', 'off', or 'auto', default: 'auto') 'auto' enables colors when output is to a terminal |
-ctxcp, --ctx-checkpoints, --swa-checkpoints N |
max number of context checkpoints to create per slot (default: 32)(more info) (env: LLAMA_ARG_CTX_CHECKPOINTS) |
-cram, --cache-ram N |
set the maximum cache size in MiB (default: 8192, 0 for decision models, -1 - no limit, 0 - disable)(more info) (env: LLAMA_ARG_CACHE_RAM) |
--context-shift, --no-context-shift |
whether to use context shift on infinite text generation (default: disabled) (env: LLAMA_ARG_CONTEXT_SHIFT) |
-sys, --system-prompt PROMPT |
system prompt to use with model (if applicable, depending on chat template) |
--show-timings, --no-show-timings |
whether to show timing information after each response (default: true) (env: LLAMA_ARG_SHOW_TIMINGS) |
-sysf, --system-prompt-file FNAME |
a file containing the system prompt (default: none) |
-r, --reverse-prompt PROMPT |
halt generation at PROMPT, return control in interactive mode |
-sp, --special |
special tokens output enabled (default: false) |
-st, --single-turn |
run conversation for a single turn only, then exit when done will not be interactive if first turn is predefined with --prompt (default: false) |
-mli, --multiline-input |
allows you to write or paste multiple lines without ending each in '' |
--warmup, --no-warmup |
whether to perform warmup with an empty run (default: enabled) |
-mm, --mmproj FILE |
path to a multimodal projector file. see tools/mtmd/README.md note: if -hf is used, this argument can be omitted (env: LLAMA_ARG_MMPROJ) |
-mmu, --mmproj-url URL |
URL to a multimodal projector file. see tools/mtmd/README.md (env: LLAMA_ARG_MMPROJ_URL) |
--mmproj-auto, --no-mmproj, --no-mmproj-auto |
whether to use multimodal projector file (if available), useful when using -hf (default: enabled) (env: LLAMA_ARG_MMPROJ_AUTO) |
--mmproj-offload, --no-mmproj-offload |
whether to enable GPU offloading for multimodal projector (default: enabled) (env: LLAMA_ARG_MMPROJ_OFFLOAD) |
-mmdev, --mmproj-device DEVICE |
device to use for multimodal projector (none = don't offload, default: follows --device) use --list-devices to see a list of available devices (env: MTMD_BACKEND_DEVICE) |
--image, --audio, --video FILE |
path to an image, audio, or video file. use with multimodal models, use comma-separated values for multiple files |
--image-min-tokens N |
minimum number of tokens each image can take, only used by vision models with dynamic resolution (default: read from model) (env: LLAMA_ARG_IMAGE_MIN_TOKENS) |
--image-max-tokens N |
maximum number of tokens each image can take, only used by vision models with dynamic resolution (default: read from model) (env: LLAMA_ARG_IMAGE_MAX_TOKENS) |
--video-fps N |
target video frame rate (default: 4.0) (env: LLAMA_ARG_VIDEO_FPS) |
--video-timestamp-interval N |
interval in milliseconds between text timestamps (default: 5000) (env: LLAMA_ARG_VIDEO_TIMESTAMP_INTERVAL) |
--video-ffmpeg-dir DIR |
path to the directory containing ffmpeg and ffprobe (default: search in PATH) (env: LLAMA_ARG_VIDEO_FFMPEG_DIR) |
-o, --output, --output-file FNAME |
output file (default: '') |
--chat-template-kwargs STRING |
sets additional params for the json template parser, must be a valid json object string, e.g. '{"key1":"value1","key2":"value2"}' (env: LLAMA_ARG_CHAT_TEMPLATE_KWARGS) |
--jinja, --no-jinja |
whether to use jinja template engine for chat (default: enabled) (env: LLAMA_ARG_JINJA) |
--reasoning-format FORMAT |
controls whether thought tags are allowed and/or extracted from the response, and in which format they're returned; one of: - none: leaves thoughts unparsed in message.content- deepseek: puts thoughts in message.reasoning_content- deepseek-legacy: keeps <think> tags in message.content while also populating message.reasoning_content(default: auto) (env: LLAMA_ARG_THINK) |
-rea, --reasoning [on|off|auto] |
Use reasoning/thinking in the chat ('on', 'off', or 'auto', default: 'auto' (detect from template)) (env: LLAMA_ARG_REASONING) |
--reasoning-effort LEVEL |
reasoning effort level given to the chat template: 'default' to keep the template default, or a level such as 'minimal', 'low', 'medium', 'high', 'xhigh' or 'max' (default: default) (env: LLAMA_ARG_REASONING_EFFORT) |
--reasoning-budget N |
token budget for thinking: -1 for unrestricted, 0 for immediate end, N>0 for token budget (default: -1) (env: LLAMA_ARG_THINK_BUDGET) |
--reasoning-budget-message MESSAGE |
message injected before the end-of-thinking tag when reasoning budget is exhausted (default: none) (env: LLAMA_ARG_THINK_BUDGET_MESSAGE) |
--reasoning-preserve, --no-reasoning-preserve |
preserve reasoning trace in the full history, not just the last assistant message (default: enabled) compatible with certain templates having 'supports_preserve_reasoning' capability example: https://docs.z.ai/guides/capabilities/thinking-mode#preserved-thinking (env: LLAMA_ARG_REASONING_PRESERVE) |
--chat-template JINJA_TEMPLATE |
set custom jinja chat template (default: template taken from model's metadata) if suffix/prefix are specified, template will be disabled only commonly used templates are accepted (unless --jinja is set before this flag): list of built-in templates: bailing, bailing-think, bailing2, chatglm3, chatglm4, chatml, command-r, deepseek, deepseek-ocr, deepseek2, deepseek3, exaone-moe, exaone3, exaone4, falcon3, gemma, gigachat, glmedge, gpt-oss, granite, granite-4.0, granite-4.1, grok-2, hunyuan-dense, hunyuan-moe, hunyuan-vl, kimi-k2, llama2, llama2-sys, llama2-sys-bos, llama2-sys-strip, llama3, llama4, megrez, minicpm, mistral-v1, mistral-v3, mistral-v3-tekken, mistral-v7, mistral-v7-tekken, monarch, openchat, orion, pangu-embedded, phi3, phi4, rwkv-world, seed_oss, smolvlm, solar-open, vicuna, vicuna-orca, yandex, zephyr (env: LLAMA_ARG_CHAT_TEMPLATE) |
--chat-template-file JINJA_TEMPLATE_FILE |
set custom jinja chat template file (default: template taken from model's metadata) if suffix/prefix are specified, template will be disabled only commonly used templates are accepted (unless --jinja is set before this flag): list of built-in templates: bailing, bailing-think, bailing2, chatglm3, chatglm4, chatml, command-r, deepseek, deepseek-ocr, deepseek2, deepseek3, exaone-moe, exaone3, exaone4, falcon3, gemma, gigachat, glmedge, gpt-oss, granite, granite-4.0, granite-4.1, grok-2, hunyuan-dense, hunyuan-moe, hunyuan-vl, kimi-k2, llama2, llama2-sys, llama2-sys-bos, llama2-sys-strip, llama3, llama4, megrez, minicpm, mistral-v1, mistral-v3, mistral-v3-tekken, mistral-v7, mistral-v7-tekken, monarch, openchat, orion, pangu-embedded, phi3, phi4, rwkv-world, seed_oss, smolvlm, solar-open, vicuna, vicuna-orca, yandex, zephyr (env: LLAMA_ARG_CHAT_TEMPLATE_FILE) |
--skip-chat-parsing, --no-skip-chat-parsing |
force a pure content parser, even if a Jinja template is specified; model will output everything in the content section, including any reasoning and/or tool calls (default: disabled) (env: LLAMA_ARG_SKIP_CHAT_PARSING) |
--simple-io |
use basic IO for better compatibility in subprocesses and limited consoles |
--log-prompts-dir PATH |
Log prompts to directory (auto-created if not present; only used for debugging, default: disabled) |
--spec-draft-hf, -hfd, -hfrd, --hf-repo-draft <user>/<model>[:quant] |
Same as --hf-repo, but for the draft model (default: unused) (env: LLAMA_ARG_SPEC_DRAFT_HF_REPO) |
--spec-draft-threads, -td, --threads-draft N |
number of threads to use during generation (default: same as --threads) |
--spec-draft-threads-batch, -tbd, --threads-batch-draft N |
number of threads to use during batch and prompt processing (default: same as --threads-draft) |
--spec-draft-cpu-mask, -Cd, --cpu-mask-draft M |
Draft model CPU affinity mask. Complements cpu-range-draft (default: same as --cpu-mask) |
--spec-draft-cpu-range, -Crd, --cpu-range-draft lo-hi |
Ranges of CPUs for affinity. Complements --cpu-mask-draft |
--spec-draft-cpu-strict, --cpu-strict-draft <0|1> |
Use strict CPU placement for draft model (default: same as --cpu-strict) |
--spec-draft-prio, --prio-draft N |
set draft process/thread priority : 0-normal, 1-medium, 2-high, 3-realtime (default: 0) |
--spec-draft-poll, --poll-draft <0|1> |
Use polling to wait for draft model work (default: same as --poll) |
--spec-draft-cpu-mask-batch, -Cbd, --cpu-mask-batch-draft M |
Draft model CPU affinity mask. Complements cpu-range-draft (default: same as --cpu-mask) |
--spec-draft-cpu-strict-batch, --cpu-strict-batch-draft <0|1> |
Use strict CPU placement for draft model (default: --cpu-strict-draft) |
--spec-draft-prio-batch, --prio-batch-draft N |
set draft process/thread priority : 0-normal, 1-medium, 2-high, 3-realtime (default: 0) |
--spec-draft-poll-batch, --poll-batch-draft <0|1> |
Use polling to wait for draft model work (default: --poll-draft) |
--spec-draft-override-tensor, -otd, --override-tensor-draft <tensor name pattern>=<buffer type>,... |
override tensor buffer type for draft model |
--spec-draft-cpu-moe, -cmoed, --cpu-moe-draft |
keep all Mixture of Experts (MoE) weights in the CPU for the draft model (env: LLAMA_ARG_SPEC_DRAFT_CPU_MOE) |
--spec-draft-n-cpu-moe, --spec-draft-ncmoe, -ncmoed, --n-cpu-moe-draft N |
keep the Mixture of Experts (MoE) weights of the first N layers in the CPU for the draft model (env: LLAMA_ARG_SPEC_DRAFT_N_CPU_MOE) |
--spec-draft-n-max N |
number of tokens to draft for speculative decoding (default: 3) (env: LLAMA_ARG_SPEC_DRAFT_N_MAX) |
--spec-draft-n-min N |
minimum number of draft tokens to use for speculative decoding (default: 0) (env: LLAMA_ARG_SPEC_DRAFT_N_MIN) |
--spec-draft-n-ubatch N |
physical maximum batch size of the draft context, use a smaller value to spare VRAM (default: 0 - same as --n-ubatch) (env: LLAMA_ARG_SPEC_DRAFT_N_UBATCH) |
--spec-synth-len L |
target mean synthetic acceptance length, including the target token (benchmarking only) (env: LLAMA_ARG_SPEC_SYNTH_LEN) |
--spec-synth-rates P0,P1,... |
comma-separated unconditional per-position synthetic acceptance probabilities (benchmarking only) (env: LLAMA_ARG_SPEC_SYNTH_RATES) |
--spec-draft-p-split, --draft-p-split P |
speculative decoding split probability (default: 0.10) (env: LLAMA_ARG_SPEC_DRAFT_P_SPLIT) |
--spec-draft-p-min, --draft-p-min P |
minimum speculative decoding probability (greedy) (default: 0.00) (env: LLAMA_ARG_SPEC_DRAFT_P_MIN) |
--spec-draft-backend-sampling, --no-spec-draft-backend-sampling |
offload draft sampling to the backend (default: enabled) (env: LLAMA_ARG_SPEC_DRAFT_BACKEND_SAMPLING) |
--spec-draft-sampling {greedy,probabilistic} |
how the draft is sampled: greedy takes its argmax, probabilistic samples it and has the target verify by rejection sampling (default: greedy) (env: LLAMA_ARG_SPEC_DRAFT_SAMPLING) |
--spec-draft-device, -devd, --device-draft <dev1,dev2,..> |
comma-separated list of devices to use for offloading the draft model (none = don't offload, default: follows --device) use --list-devices to see a list of available devices |
--spec-draft-ngl, -ngld, --gpu-layers-draft, --n-gpu-layers-draft N |
max. number of draft model layers to store in VRAM, either an exact number, 'auto', or 'all' (default: auto) (env: LLAMA_ARG_N_GPU_LAYERS_DRAFT) |
--spec-draft-model, -md, --model-draft FNAME |
draft model for speculative decoding (default: unused) (env: LLAMA_ARG_SPEC_DRAFT_MODEL) |
--spec-type none,draft-simple,draft-eagle3,draft-mtp,draft-dflash,draft-dspark,draft-dflash2,draft-lilicorr,ngram-simple,ngram-map-k,ngram-map-k4v,ngram-mod,ngram-cache |
comma-separated list of types of speculative decoding to use (default: none) (env: LLAMA_ARG_SPEC_TYPE) |
--spec-ngram-mod-n-min N |
minimum number of ngram tokens to use for ngram-based speculative decoding (default: 48) |
--spec-ngram-mod-n-max N |
maximum number of ngram tokens to use for ngram-based speculative decoding (default: 64) |
--spec-ngram-mod-n-match N |
ngram-mod lookup length (default: 24) |
--spec-ngram-simple-size-n N |
ngram size N for ngram-simple speculative decoding, length of lookup n-gram (default: 12) |
--spec-ngram-simple-size-m N |
ngram size M for ngram-simple speculative decoding, length of draft m-gram (default: 48) |
--spec-ngram-simple-min-hits N |
minimum hits for ngram-simple speculative decoding (default: 1) |
--spec-ngram-map-k-size-n N |
ngram size N for ngram-map-k speculative decoding, length of lookup n-gram (default: 12) |
--spec-ngram-map-k-size-m N |
ngram size M for ngram-map-k speculative decoding, length of draft m-gram (default: 48) |
--spec-ngram-map-k-min-hits N |
minimum hits for ngram-map-k speculative decoding (default: 1) |
--spec-ngram-map-k4v-size-n N |
ngram size N for ngram-map-k4v speculative decoding, length of lookup n-gram (default: 12) |
--spec-ngram-map-k4v-size-m N |
ngram size M for ngram-map-k4v speculative decoding, length of draft m-gram (default: 48) |
--spec-ngram-map-k4v-min-hits N |
minimum hits for ngram-map-k4v speculative decoding (default: 1) |
--draft, --draft-n, --draft-max N |
the argument has been removed. use --spec-draft-n-max or --spec-ngram-mod-n-max (env: LLAMA_ARG_DRAFT_MAX) |
--draft-min, --draft-n-min N |
the argument has been removed. use --spec-draft-n-min or --spec-ngram-mod-n-min (env: LLAMA_ARG_DRAFT_MIN) |
--gpt-oss-20b-default |
use gpt-oss-20b (note: can download weights from the internet) |
--gpt-oss-120b-default |
use gpt-oss-120b (note: can download weights from the internet) |
--vision-gemma-4b-default |
use Gemma 3 4B QAT (note: can download weights from the internet) |
--vision-gemma-12b-default |
use Gemma 3 12B QAT (note: can download weights from the internet) |
--spec-default |
enable default speculative decoding config |