docs: move per-model TTS to /docs, refer it from other pages (#32304)
This commit is contained in:
1 parent
a68799c609
commit
75fd7f9ec9
7 files changed
+192
-172
No files matched your search
@@ -102,6 +102,7 @@ The `llama.cpp` project is build on top of the [ggml](https://github.com/ggml-or
|
||||
- [cli](tools/cli/README.md)
|
||||
- [completion](tools/completion/README.md)
|
||||
- [server](tools/server/README.md)
|
||||
- [Text-to-speech (TTS) models](docs/multimodal/tts.md)
|
||||
- [GBNF grammars](grammars/README.md)
|
||||
|
||||
#### Development
|
||||
|
||||
+2
-2
@@ -4502,7 +4502,7 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
|
||||
add_opt(common_arg(
|
||||
{"--tts-lang"}, "FNAME",
|
||||
"language (ISO 639-1) for audio generation\n"
|
||||
"see tts/README.md for per-model usage notes",
|
||||
"see docs/multimodal/tts.md for per-model usage notes",
|
||||
[](common_params & params, const std::string & value) {
|
||||
params.tts_lang = value;
|
||||
}
|
||||
@@ -4510,7 +4510,7 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
|
||||
add_opt(common_arg(
|
||||
{"--tts-voice"}, "VOICE",
|
||||
"voice for audio generation (model-dependent)\n"
|
||||
"see tts/README.md for per-model usage notes",
|
||||
"see docs/multimodal/tts.md for per-model usage notes",
|
||||
[](common_params & params, const std::string & value) {
|
||||
params.tts_voice = value;
|
||||
}
|
||||
|
||||
@@ -0,0 +1,182 @@
|
||||
# Text-to-speech (TTS)
|
||||
|
||||
llama.cpp supports audio generation via `libmtmd`. The following tools can be used:
|
||||
- [llama-tts](../../tools/tts/README.md), see its README for common usage and params
|
||||
- [llama-server](../../tools/server/README.md) via the `/tts` endpoint and the OpenAI-compatible `/v1/audio/speech` API
|
||||
|
||||
Supported models:
|
||||
- [Qwen3-TTS](#qwen3-tts)
|
||||
- [Pocket TTS](#pocket-tts)
|
||||
- [Soprano](#soprano)
|
||||
- [LFM3-Audio](#lfm3-audio)
|
||||
- [Chatterbox](#chatterbox)
|
||||
- [Drama 3](#drama-3)
|
||||
|
||||
The sections below list the params supported by each model, with `llama-tts` examples. In `llama-server`, the same options are set with `--tts-lang` and `--tts-voice`, or per request (see the [`/tts` endpoint](../../tools/server/README.md#post-tts-generate-speech-audio-from-text)).
|
||||
|
||||
## Qwen3-TTS
|
||||
|
||||
Available params:
|
||||
- `--tts-lang` can be `zh`, `en`, `de`, `it`, `pt`, `es`, `ja`, `ko`, `fr`, `ru` (default: `en`)
|
||||
- `--tts-speaker-file` should point to a speaker reference audio file (wav, mp3)
|
||||
|
||||
Example usage:
|
||||
|
||||
```sh
|
||||
llama-tts -hf ggml-org/Qwen3-TTS-12Hz-1.7B-Base-GGUF \
|
||||
-p "Hello world" \
|
||||
--tts-lang english \
|
||||
--tts-speaker-file speaker.mp3 \
|
||||
--output out.wav
|
||||
```
|
||||
|
||||
## Pocket TTS
|
||||
|
||||
Available params:
|
||||
- `--tts-speaker-file` should point to a speaker reference audio file (wav, mp3). It is required, the model produces almost no audio without it
|
||||
- `--tts-voice` selects one of the predefined voices shipped with the weights (e.g. `giovanni`), when the GGUF holds them. The converter reads them from `embeddings/*.safetensors` next to the weights, of both repositories, and logs how closely each one is reproduced (some packs ship voice states computed with other weights, the converter warns above 5% and skips a voice only when it does not belong to the model at all)
|
||||
- Note: `lang` is not used, the language is a property of the weights
|
||||
|
||||
Example usage:
|
||||
|
||||
```sh
|
||||
llama-tts -m pocket-tts.gguf \
|
||||
-mm mmproj-pocket-tts.gguf \
|
||||
-p "Hello world" \
|
||||
--tts-speaker-file speaker.mp3 \
|
||||
--output out.wav
|
||||
```
|
||||
|
||||
The weights [without voice cloning](https://huggingface.co/kyutai/pocket-tts-without-voice-cloning) ship a set of predefined voices (`embeddings/*.safetensors`) next to each language; the converter stores them in the mmproj, as the voice prompt embeddings, and leaves out the zeroed voice encoder. With such a model `--tts-speaker-file` is rejected and a voice is selected with `--tts-voice` (the first one when it is not given):
|
||||
|
||||
```sh
|
||||
llama-tts -m pocket-tts.gguf \
|
||||
-mm mmproj-pocket-tts.gguf \
|
||||
-p "Ciao a tutti" \
|
||||
--tts-voice giovanni \
|
||||
--output out.wav
|
||||
```
|
||||
|
||||
**Note for GGUF conversion:**
|
||||
|
||||
The [upstream repository](https://huggingface.co/kyutai/pocket-tts) holds one complete model per language under `languages/`, next to a set of shared files at the root. Convert one of the `languages/<name>` directories, **not** the root directory:
|
||||
|
||||
```sh
|
||||
python convert_hf_to_gguf.py path/to/pocket-tts/languages/english --outfile pocket-tts.gguf
|
||||
python convert_hf_to_gguf.py path/to/pocket-tts/languages/english --mmproj --outfile mmproj-pocket-tts.gguf
|
||||
```
|
||||
|
||||
## Soprano
|
||||
|
||||
Available params:
|
||||
- `--tts-speaker-file` is not supported; the model uses a fixed voice
|
||||
- Note: `lang` is not used. Supply normalized English text
|
||||
|
||||
Example usage:
|
||||
|
||||
```sh
|
||||
llama-tts -m soprano.gguf \
|
||||
-mm mmproj-soprano.gguf \
|
||||
-p "Hello world" \
|
||||
--temp 0 \
|
||||
--output out.wav
|
||||
```
|
||||
|
||||
**Note for GGUF conversion:**
|
||||
|
||||
Both [Soprano-80M](https://huggingface.co/ekwek/Soprano-80M) and [Soprano-1.1-80M](https://huggingface.co/ekwek/Soprano-1.1-80M) are supported. Convert both files from the same model directory, including `decoder.pth`:
|
||||
|
||||
```sh
|
||||
python convert_hf_to_gguf.py path/to/Soprano-1.1-80M --outtype f32 --outfile soprano.gguf
|
||||
python convert_hf_to_gguf.py path/to/Soprano-1.1-80M --mmproj --outtype f16 --outfile mmproj-soprano.gguf
|
||||
```
|
||||
|
||||
## LFM3-Audio
|
||||
|
||||
Available params:
|
||||
- `--tts-lang` can be `ar`, `da`, `de`, `el`, `en`, `en-gb`, `es`, `fi`, `fr`, `he`, `it`, `ja`, `ko`, `nl`, `pl`, `pt`, `sv`, `tr`, `zh` (default: `en`), other values are used as the language name
|
||||
- `--tts-voice` can be `female` (default) or `male`
|
||||
- Note: `--tts-speaker-file` is not used, the model has no voice cloning
|
||||
- Note: `--top-k` and `--top-p` are not used, audio frames are sampled with the settings of the model card (LFM3-Audio: temperature 1.0, top-k 128, repetition penalty 1.1 over 4096 frames; LFM3-Omni: temperature 0.4, repetition penalty 1.2 over 64 frames), the `audio_*` keys of `generation_config.json` override them at conversion
|
||||
- Each frame is 80 ms, generation stops at the end-of-audio frame
|
||||
|
||||
Example usage:
|
||||
|
||||
```sh
|
||||
llama-tts -m lfm3-audio.gguf \
|
||||
-mm mmproj-lfm3-audio.gguf \
|
||||
-p "Hello world" \
|
||||
--tts-voice male \
|
||||
--output out.wav
|
||||
```
|
||||
|
||||
The mmproj holds the audio encoder (for ASR), the depthformer and the codec decoder, the latter two are exported when the checkpoint has an `audio_tokenizer` directory:
|
||||
|
||||
```sh
|
||||
python convert_hf_to_gguf.py path/to/lfm3-audio --outfile lfm3-audio.gguf
|
||||
python convert_hf_to_gguf.py path/to/lfm3-audio --mmproj --outfile mmproj-lfm3-audio.gguf
|
||||
```
|
||||
|
||||
## Chatterbox
|
||||
|
||||
Two variants are supported: [Chatterbox Turbo](https://huggingface.co/ResembleAI/chatterbox-turbo) (English) and [Chatterbox Multilingual V3](https://huggingface.co/ResembleAI/chatterbox) (23 languages).
|
||||
|
||||
Available params:
|
||||
- `--tts-speaker-file` should point to a speaker reference audio file (wav, mp3). Optional, the predefined voice is used without it
|
||||
- `--tts-voice` can be `default`, the voice shipped with the model (`conds.pt`)
|
||||
- `--tts-lang` sets the language of the multilingual variant, for example `fr` (default: none). It is not used by Turbo
|
||||
|
||||
Example usage:
|
||||
|
||||
```sh
|
||||
llama-tts -m chatterbox-mtl.gguf \
|
||||
-mm mmproj-chatterbox-mtl.gguf \
|
||||
-p "Bonjour tout le monde" \
|
||||
--tts-lang fr \
|
||||
--tts-speaker-file speaker.mp3 \
|
||||
--output out.wav
|
||||
```
|
||||
|
||||
Note: the multilingual variant runs classifier-free guidance with a second sequence. `llama-tts` sets this up by itself, but `llama-server` does not support it yet.
|
||||
|
||||
**Note for GGUF conversion:**
|
||||
|
||||
The upstream repositories ship raw checkpoints without `config.json`. Download the files below into a directory, then convert it as usual (the variant is detected from the talker file):
|
||||
- Turbo: `t3_turbo_v1.safetensors`, `s3gen_meanflow.safetensors`, `ve.safetensors`, `conds.pt` and the GPT-2 tokenizer files (`vocab.json`, `merges.txt`, `added_tokens.json`, `special_tokens_map.json`, `tokenizer_config.json`)
|
||||
- Multilingual V3: `t3_mtl23ls_v3.safetensors`, `s3gen_v3.safetensors`, `ve.safetensors`, `conds.pt`, `mtl_tokenizer.json`
|
||||
|
||||
```sh
|
||||
python convert_hf_to_gguf.py path/to/chatterbox --outfile chatterbox.gguf
|
||||
python convert_hf_to_gguf.py path/to/chatterbox --mmproj --outfile mmproj-chatterbox.gguf
|
||||
```
|
||||
|
||||
## Drama 3
|
||||
|
||||
[Fish Audio Drama 3](https://huggingface.co/fishaudio/drama-3): a Qwen3.5-MoE backbone (35B, 3B active) samples each frame's main token, a codebook tower samples the 9 other codes, and the Fish DAC decodes them to 44.1 kHz audio.
|
||||
|
||||
Available params:
|
||||
- Speaker tags `<|speaker:0|>` to `<|speaker:15|>` in the text give each character its own voice, a text without a tag starts with `<|speaker:0|>`
|
||||
- The language is the one of the text, `--tts-lang` is not used
|
||||
- Note: `--tts-speaker-file` is not supported yet (voice cloning)
|
||||
- Note: `--top-k` and `--top-p` are not used, frames are sampled with the settings of the reference (main token: temperature 1.0, top-p 0.9, repetition-aware redraw over the last 16 frames; other codes: temperature 1.0, top-k 30, top-p 0.9)
|
||||
- Each frame is 2048 samples (about 46 ms), generation stops at the end-of-speech token
|
||||
- The codebook tower keeps the whole sequence in its KV cache, the prompt and the generated frames are limited to 4096 positions
|
||||
|
||||
Example usage:
|
||||
|
||||
```sh
|
||||
llama-tts -m drama3.gguf \
|
||||
-mm mmproj-drama3.gguf \
|
||||
-p "<|speaker:0|> Did you remember to buy the bread? <|speaker:1|> Of course I did, it is on the kitchen table." \
|
||||
-n 1000 \
|
||||
--output out.wav
|
||||
```
|
||||
|
||||
**Note for GGUF conversion:**
|
||||
|
||||
The routed experts of the checkpoint are FP8, they are dequantized at conversion. The mmproj holds the codebook tower and the decoder half of `codec.pth`, it is best kept in BF16:
|
||||
|
||||
```sh
|
||||
python convert_hf_to_gguf.py path/to/drama-3 --outtype q8_0 --outfile drama3.gguf
|
||||
python convert_hf_to_gguf.py path/to/drama-3 --mmproj --outtype bf16 --outfile mmproj-drama3.gguf
|
||||
```
|
||||
@@ -108,7 +108,7 @@ Due to wide variety of audio generation pipelines, the `mtmd_gen_audio` system i
|
||||
- 60% changes inside `mtmd-helper-gen.cpp`
|
||||
- 10% changes inside `libmtmd` core (`mtmd-model`, `mtmd-subcontext`, `models/`)
|
||||
- The rest downstream code (CLI, server) should have no changes at all
|
||||
5. Update usage documentation in `tools/tts/README.md`
|
||||
5. Update usage documentation in `docs/multimodal/tts.md`
|
||||
|
||||
IMPORTANT: If your model needs changes that don't fit the existing infrastructure, **open an issue first for discussion**.
|
||||
|
||||
|
||||
@@ -295,8 +295,8 @@ For the full list of features, please refer to [server's changelog](https://gith
|
||||
| `--spec-ngram-size-n N` | the argument has been removed. use the respective --spec-ngram-*-size-n or --spec-ngram-mod-n-match |
|
||||
| `--spec-ngram-size-m N` | the argument has been removed. use the respective --spec-ngram-*-size-m |
|
||||
| `--spec-ngram-min-hits N` | the argument has been removed. use the respective --spec-ngram-*-min-hits |
|
||||
| `--tts-lang FNAME` | language (ISO 639-1) for audio generation<br/>see tts/README.md for per-model usage notes |
|
||||
| `--tts-voice VOICE` | voice for audio generation (model-dependent)<br/>see tts/README.md for per-model usage notes |
|
||||
| `--tts-lang FNAME` | language (ISO 639-1) for audio generation<br/>see docs/multimodal/tts.md for per-model usage notes |
|
||||
| `--tts-voice VOICE` | voice for audio generation (model-dependent)<br/>see docs/multimodal/tts.md for per-model usage notes |
|
||||
| `--embd-gemma-default` | use default EmbeddingGemma model (note: can download weights from the internet) |
|
||||
| `--fim-qwen-1.5b-default` | use default Qwen 2.5 Coder 1.5B (note: can download weights from the internet) |
|
||||
| `--fim-qwen-3b-default` | use default Qwen 2.5 Coder 3B (note: can download weights from the internet) |
|
||||
@@ -854,7 +854,7 @@ Causal decoder rerankers (e.g. `Qwen3-VL-Reranker`) have no such limitation.
|
||||
|
||||
### POST `/tts`: Generate speech audio from text
|
||||
|
||||
Returns raw audio bytes (`audio/wav` by default) rather than JSON. For more info, see [tts/README.md](../tts/README.md)
|
||||
Returns raw audio bytes (`audio/wav` by default) rather than JSON. For the list of supported models and their params, see [docs/multimodal/tts.md](../../docs/multimodal/tts.md)
|
||||
|
||||
*Options:*
|
||||
|
||||
|
||||
+2
-165
@@ -19,169 +19,6 @@ Common params:
|
||||
- `-n <number_of_frames>` limits the output length, e.g. `-n 500`. Note that how many milliseconds each frame represents varies by model
|
||||
- Core inference params such as `-ngl`, `-b`, `-ub`, etc.
|
||||
|
||||
## Qwen3-TTS
|
||||
## Supported models
|
||||
|
||||
Available params:
|
||||
- `--tts-lang` can be `zh`, `en`, `de`, `it`, `pt`, `es`, `ja`, `ko`, `fr`, `ru` (default: `en`)
|
||||
- `--tts-speaker-file` should point to a speaker reference audio file (wav, mp3)
|
||||
|
||||
Example usage:
|
||||
|
||||
```sh
|
||||
llama-tts -hf ggml-org/Qwen3-TTS-12Hz-1.7B-Base-GGUF \
|
||||
-p "Hello world" \
|
||||
--tts-lang english \
|
||||
--tts-speaker-file speaker.mp3 \
|
||||
--output out.wav
|
||||
```
|
||||
|
||||
## Pocket TTS
|
||||
|
||||
Available params:
|
||||
- `--tts-speaker-file` should point to a speaker reference audio file (wav, mp3). It is required, the model produces almost no audio without it
|
||||
- `--tts-voice` selects one of the predefined voices shipped with the weights (e.g. `giovanni`), when the GGUF holds them. The converter reads them from `embeddings/*.safetensors` next to the weights, of both repositories, and logs how closely each one is reproduced (some packs ship voice states computed with other weights, the converter warns above 5% and skips a voice only when it does not belong to the model at all)
|
||||
- Note: `lang` is not used, the language is a property of the weights
|
||||
|
||||
Example usage:
|
||||
|
||||
```sh
|
||||
llama-tts -m pocket-tts.gguf \
|
||||
-mm mmproj-pocket-tts.gguf \
|
||||
-p "Hello world" \
|
||||
--tts-speaker-file speaker.mp3 \
|
||||
--output out.wav
|
||||
```
|
||||
|
||||
The weights [without voice cloning](https://huggingface.co/kyutai/pocket-tts-without-voice-cloning) ship a set of predefined voices (`embeddings/*.safetensors`) next to each language; the converter stores them in the mmproj, as the voice prompt embeddings, and leaves out the zeroed voice encoder. With such a model `--tts-speaker-file` is rejected and a voice is selected with `--tts-voice` (the first one when it is not given):
|
||||
|
||||
```sh
|
||||
llama-tts -m pocket-tts.gguf \
|
||||
-mm mmproj-pocket-tts.gguf \
|
||||
-p "Ciao a tutti" \
|
||||
--tts-voice giovanni \
|
||||
--output out.wav
|
||||
```
|
||||
|
||||
**Note for GGUF conversion:**
|
||||
|
||||
The [upstream repository](https://huggingface.co/kyutai/pocket-tts) holds one complete model per language under `languages/`, next to a set of shared files at the root. Convert one of the `languages/<name>` directories, **not** the root directory:
|
||||
|
||||
```sh
|
||||
python convert_hf_to_gguf.py path/to/pocket-tts/languages/english --outfile pocket-tts.gguf
|
||||
python convert_hf_to_gguf.py path/to/pocket-tts/languages/english --mmproj --outfile mmproj-pocket-tts.gguf
|
||||
```
|
||||
|
||||
## Soprano
|
||||
|
||||
Available params:
|
||||
- `--tts-speaker-file` is not supported; the model uses a fixed voice
|
||||
- Note: `lang` is not used. Supply normalized English text
|
||||
|
||||
Example usage:
|
||||
|
||||
```sh
|
||||
llama-tts -m soprano.gguf \
|
||||
-mm mmproj-soprano.gguf \
|
||||
-p "Hello world" \
|
||||
--temp 0 \
|
||||
--output out.wav
|
||||
```
|
||||
|
||||
**Note for GGUF conversion:**
|
||||
|
||||
Both [Soprano-80M](https://huggingface.co/ekwek/Soprano-80M) and [Soprano-1.1-80M](https://huggingface.co/ekwek/Soprano-1.1-80M) are supported. Convert both files from the same model directory, including `decoder.pth`:
|
||||
|
||||
```sh
|
||||
python convert_hf_to_gguf.py path/to/Soprano-1.1-80M --outtype f32 --outfile soprano.gguf
|
||||
python convert_hf_to_gguf.py path/to/Soprano-1.1-80M --mmproj --outtype f16 --outfile mmproj-soprano.gguf
|
||||
```
|
||||
|
||||
## LFM3-Audio
|
||||
|
||||
Available params:
|
||||
- `--tts-lang` can be `ar`, `da`, `de`, `el`, `en`, `en-gb`, `es`, `fi`, `fr`, `he`, `it`, `ja`, `ko`, `nl`, `pl`, `pt`, `sv`, `tr`, `zh` (default: `en`), other values are used as the language name
|
||||
- `--tts-voice` can be `female` (default) or `male`
|
||||
- Note: `--tts-speaker-file` is not used, the model has no voice cloning
|
||||
- Note: `--top-k` and `--top-p` are not used, audio frames are sampled with the settings of the model card (LFM3-Audio: temperature 1.0, top-k 128, repetition penalty 1.1 over 4096 frames; LFM3-Omni: temperature 0.4, repetition penalty 1.2 over 64 frames), the `audio_*` keys of `generation_config.json` override them at conversion
|
||||
- Each frame is 80 ms, generation stops at the end-of-audio frame
|
||||
|
||||
Example usage:
|
||||
|
||||
```sh
|
||||
llama-tts -m lfm3-audio.gguf \
|
||||
-mm mmproj-lfm3-audio.gguf \
|
||||
-p "Hello world" \
|
||||
--tts-voice male \
|
||||
--output out.wav
|
||||
```
|
||||
|
||||
The mmproj holds the audio encoder (for ASR), the depthformer and the codec decoder, the latter two are exported when the checkpoint has an `audio_tokenizer` directory:
|
||||
|
||||
```sh
|
||||
python convert_hf_to_gguf.py path/to/lfm3-audio --outfile lfm3-audio.gguf
|
||||
python convert_hf_to_gguf.py path/to/lfm3-audio --mmproj --outfile mmproj-lfm3-audio.gguf
|
||||
```
|
||||
|
||||
## Chatterbox
|
||||
|
||||
Two variants are supported: [Chatterbox Turbo](https://huggingface.co/ResembleAI/chatterbox-turbo) (English) and [Chatterbox Multilingual V3](https://huggingface.co/ResembleAI/chatterbox) (23 languages).
|
||||
|
||||
Available params:
|
||||
- `--tts-speaker-file` should point to a speaker reference audio file (wav, mp3). Optional, the predefined voice is used without it
|
||||
- `--tts-voice` can be `default`, the voice shipped with the model (`conds.pt`)
|
||||
- `--tts-lang` sets the language of the multilingual variant, for example `fr` (default: none). It is not used by Turbo
|
||||
|
||||
Example usage:
|
||||
|
||||
```sh
|
||||
llama-tts -m chatterbox-mtl.gguf \
|
||||
-mm mmproj-chatterbox-mtl.gguf \
|
||||
-p "Bonjour tout le monde" \
|
||||
--tts-lang fr \
|
||||
--tts-speaker-file speaker.mp3 \
|
||||
--output out.wav
|
||||
```
|
||||
|
||||
Note: the multilingual variant runs classifier-free guidance with a second sequence. `llama-tts` sets this up by itself, but `llama-server` does not support it yet.
|
||||
|
||||
**Note for GGUF conversion:**
|
||||
|
||||
The upstream repositories ship raw checkpoints without `config.json`. Download the files below into a directory, then convert it as usual (the variant is detected from the talker file):
|
||||
- Turbo: `t3_turbo_v1.safetensors`, `s3gen_meanflow.safetensors`, `ve.safetensors`, `conds.pt` and the GPT-2 tokenizer files (`vocab.json`, `merges.txt`, `added_tokens.json`, `special_tokens_map.json`, `tokenizer_config.json`)
|
||||
- Multilingual V3: `t3_mtl23ls_v3.safetensors`, `s3gen_v3.safetensors`, `ve.safetensors`, `conds.pt`, `mtl_tokenizer.json`
|
||||
|
||||
```sh
|
||||
python convert_hf_to_gguf.py path/to/chatterbox --outfile chatterbox.gguf
|
||||
python convert_hf_to_gguf.py path/to/chatterbox --mmproj --outfile mmproj-chatterbox.gguf
|
||||
```
|
||||
|
||||
## Drama 3
|
||||
|
||||
[Fish Audio Drama 3](https://huggingface.co/fishaudio/drama-3): a Qwen3.5-MoE backbone (35B, 3B active) samples each frame's main token, a codebook tower samples the 9 other codes, and the Fish DAC decodes them to 44.1 kHz audio.
|
||||
|
||||
Available params:
|
||||
- Speaker tags `<|speaker:0|>` to `<|speaker:15|>` in the text give each character its own voice, a text without a tag starts with `<|speaker:0|>`
|
||||
- The language is the one of the text, `--tts-lang` is not used
|
||||
- Note: `--tts-speaker-file` is not supported yet (voice cloning)
|
||||
- Note: `--top-k` and `--top-p` are not used, frames are sampled with the settings of the reference (main token: temperature 1.0, top-p 0.9, repetition-aware redraw over the last 16 frames; other codes: temperature 1.0, top-k 30, top-p 0.9)
|
||||
- Each frame is 2048 samples (about 46 ms), generation stops at the end-of-speech token
|
||||
- The codebook tower keeps the whole sequence in its KV cache, the prompt and the generated frames are limited to 4096 positions
|
||||
|
||||
Example usage:
|
||||
|
||||
```sh
|
||||
llama-tts -m drama3.gguf \
|
||||
-mm mmproj-drama3.gguf \
|
||||
-p "<|speaker:0|> Did you remember to buy the bread? <|speaker:1|> Of course I did, it is on the kitchen table." \
|
||||
-n 1000 \
|
||||
--output out.wav
|
||||
```
|
||||
|
||||
**Note for GGUF conversion:**
|
||||
|
||||
The routed experts of the checkpoint are FP8, they are dequantized at conversion. The mmproj holds the codebook tower and the decoder half of `codec.pth`, it is best kept in BF16:
|
||||
|
||||
```sh
|
||||
python convert_hf_to_gguf.py path/to/drama-3 --outtype q8_0 --outfile drama3.gguf
|
||||
python convert_hf_to_gguf.py path/to/drama-3 --mmproj --outtype bf16 --outfile mmproj-drama3.gguf
|
||||
```
|
||||
See [docs/multimodal/tts.md](../../docs/multimodal/tts.md) for the list of supported models, their params and GGUF conversion notes.
|
||||
+1
-1
@@ -39,7 +39,7 @@ static void print_usage(int, char ** argv) {
|
||||
LOG("\n %s -hf user/model -p \"text to speak\" -o output.wav\n", argv[0]);
|
||||
LOG("\nnote: --tts-lang, --tts-voice and --tts-speaker-file may not be supported in all models");
|
||||
LOG("\n use -n to limit the output length");
|
||||
LOG("\n see tts/README.md for per-model usage notes");
|
||||
LOG("\n see docs/multimodal/tts.md for per-model usage notes");
|
||||
LOG("\n\n");
|
||||
}
|
||||
|
||||
|
||||
Reference in new issue
Block a user