diff --git a/README.md b/README.md index 80b73cc52..3cacc8d53 100644 --- a/README.md +++ b/README.md @@ -102,6 +102,7 @@ The `llama.cpp` project is build on top of the [ggml](https://github.com/ggml-or - [cli](tools/cli/README.md) - [completion](tools/completion/README.md) - [server](tools/server/README.md) +- [Text-to-speech (TTS) models](docs/multimodal/tts.md) - [GBNF grammars](grammars/README.md) #### Development diff --git a/common/arg.cpp b/common/arg.cpp index 00f9659cb..15cc9abd0 100644 --- a/common/arg.cpp +++ b/common/arg.cpp @@ -4502,7 +4502,7 @@ common_params_context common_params_parser_init(common_params & params, llama_ex add_opt(common_arg( {"--tts-lang"}, "FNAME", "language (ISO 639-1) for audio generation\n" - "see tts/README.md for per-model usage notes", + "see docs/multimodal/tts.md for per-model usage notes", [](common_params & params, const std::string & value) { params.tts_lang = value; } @@ -4510,7 +4510,7 @@ common_params_context common_params_parser_init(common_params & params, llama_ex add_opt(common_arg( {"--tts-voice"}, "VOICE", "voice for audio generation (model-dependent)\n" - "see tts/README.md for per-model usage notes", + "see docs/multimodal/tts.md for per-model usage notes", [](common_params & params, const std::string & value) { params.tts_voice = value; } diff --git a/docs/multimodal/tts.md b/docs/multimodal/tts.md new file mode 100644 index 000000000..3ffc6d45e --- /dev/null +++ b/docs/multimodal/tts.md @@ -0,0 +1,182 @@ +# Text-to-speech (TTS) + +llama.cpp supports audio generation via `libmtmd`. The following tools can be used: +- [llama-tts](../../tools/tts/README.md), see its README for common usage and params +- [llama-server](../../tools/server/README.md) via the `/tts` endpoint and the OpenAI-compatible `/v1/audio/speech` API + +Supported models: +- [Qwen3-TTS](#qwen3-tts) +- [Pocket TTS](#pocket-tts) +- [Soprano](#soprano) +- [LFM3-Audio](#lfm3-audio) +- [Chatterbox](#chatterbox) +- [Drama 3](#drama-3) + +The sections below list the params supported by each model, with `llama-tts` examples. In `llama-server`, the same options are set with `--tts-lang` and `--tts-voice`, or per request (see the [`/tts` endpoint](../../tools/server/README.md#post-tts-generate-speech-audio-from-text)). + +## Qwen3-TTS + +Available params: +- `--tts-lang` can be `zh`, `en`, `de`, `it`, `pt`, `es`, `ja`, `ko`, `fr`, `ru` (default: `en`) +- `--tts-speaker-file` should point to a speaker reference audio file (wav, mp3) + +Example usage: + +```sh +llama-tts -hf ggml-org/Qwen3-TTS-12Hz-1.7B-Base-GGUF \ + -p "Hello world" \ + --tts-lang english \ + --tts-speaker-file speaker.mp3 \ + --output out.wav +``` + +## Pocket TTS + +Available params: +- `--tts-speaker-file` should point to a speaker reference audio file (wav, mp3). It is required, the model produces almost no audio without it +- `--tts-voice` selects one of the predefined voices shipped with the weights (e.g. `giovanni`), when the GGUF holds them. The converter reads them from `embeddings/*.safetensors` next to the weights, of both repositories, and logs how closely each one is reproduced (some packs ship voice states computed with other weights, the converter warns above 5% and skips a voice only when it does not belong to the model at all) +- Note: `lang` is not used, the language is a property of the weights + +Example usage: + +```sh +llama-tts -m pocket-tts.gguf \ + -mm mmproj-pocket-tts.gguf \ + -p "Hello world" \ + --tts-speaker-file speaker.mp3 \ + --output out.wav +``` + +The weights [without voice cloning](https://huggingface.co/kyutai/pocket-tts-without-voice-cloning) ship a set of predefined voices (`embeddings/*.safetensors`) next to each language; the converter stores them in the mmproj, as the voice prompt embeddings, and leaves out the zeroed voice encoder. With such a model `--tts-speaker-file` is rejected and a voice is selected with `--tts-voice` (the first one when it is not given): + +```sh +llama-tts -m pocket-tts.gguf \ + -mm mmproj-pocket-tts.gguf \ + -p "Ciao a tutti" \ + --tts-voice giovanni \ + --output out.wav +``` + +**Note for GGUF conversion:** + +The [upstream repository](https://huggingface.co/kyutai/pocket-tts) holds one complete model per language under `languages/`, next to a set of shared files at the root. Convert one of the `languages/` directories, **not** the root directory: + +```sh +python convert_hf_to_gguf.py path/to/pocket-tts/languages/english --outfile pocket-tts.gguf +python convert_hf_to_gguf.py path/to/pocket-tts/languages/english --mmproj --outfile mmproj-pocket-tts.gguf +``` + +## Soprano + +Available params: +- `--tts-speaker-file` is not supported; the model uses a fixed voice +- Note: `lang` is not used. Supply normalized English text + +Example usage: + +```sh +llama-tts -m soprano.gguf \ + -mm mmproj-soprano.gguf \ + -p "Hello world" \ + --temp 0 \ + --output out.wav +``` + +**Note for GGUF conversion:** + +Both [Soprano-80M](https://huggingface.co/ekwek/Soprano-80M) and [Soprano-1.1-80M](https://huggingface.co/ekwek/Soprano-1.1-80M) are supported. Convert both files from the same model directory, including `decoder.pth`: + +```sh +python convert_hf_to_gguf.py path/to/Soprano-1.1-80M --outtype f32 --outfile soprano.gguf +python convert_hf_to_gguf.py path/to/Soprano-1.1-80M --mmproj --outtype f16 --outfile mmproj-soprano.gguf +``` + +## LFM3-Audio + +Available params: +- `--tts-lang` can be `ar`, `da`, `de`, `el`, `en`, `en-gb`, `es`, `fi`, `fr`, `he`, `it`, `ja`, `ko`, `nl`, `pl`, `pt`, `sv`, `tr`, `zh` (default: `en`), other values are used as the language name +- `--tts-voice` can be `female` (default) or `male` +- Note: `--tts-speaker-file` is not used, the model has no voice cloning +- Note: `--top-k` and `--top-p` are not used, audio frames are sampled with the settings of the model card (LFM3-Audio: temperature 1.0, top-k 128, repetition penalty 1.1 over 4096 frames; LFM3-Omni: temperature 0.4, repetition penalty 1.2 over 64 frames), the `audio_*` keys of `generation_config.json` override them at conversion +- Each frame is 80 ms, generation stops at the end-of-audio frame + +Example usage: + +```sh +llama-tts -m lfm3-audio.gguf \ + -mm mmproj-lfm3-audio.gguf \ + -p "Hello world" \ + --tts-voice male \ + --output out.wav +``` + +The mmproj holds the audio encoder (for ASR), the depthformer and the codec decoder, the latter two are exported when the checkpoint has an `audio_tokenizer` directory: + +```sh +python convert_hf_to_gguf.py path/to/lfm3-audio --outfile lfm3-audio.gguf +python convert_hf_to_gguf.py path/to/lfm3-audio --mmproj --outfile mmproj-lfm3-audio.gguf +``` + +## Chatterbox + +Two variants are supported: [Chatterbox Turbo](https://huggingface.co/ResembleAI/chatterbox-turbo) (English) and [Chatterbox Multilingual V3](https://huggingface.co/ResembleAI/chatterbox) (23 languages). + +Available params: +- `--tts-speaker-file` should point to a speaker reference audio file (wav, mp3). Optional, the predefined voice is used without it +- `--tts-voice` can be `default`, the voice shipped with the model (`conds.pt`) +- `--tts-lang` sets the language of the multilingual variant, for example `fr` (default: none). It is not used by Turbo + +Example usage: + +```sh +llama-tts -m chatterbox-mtl.gguf \ + -mm mmproj-chatterbox-mtl.gguf \ + -p "Bonjour tout le monde" \ + --tts-lang fr \ + --tts-speaker-file speaker.mp3 \ + --output out.wav +``` + +Note: the multilingual variant runs classifier-free guidance with a second sequence. `llama-tts` sets this up by itself, but `llama-server` does not support it yet. + +**Note for GGUF conversion:** + +The upstream repositories ship raw checkpoints without `config.json`. Download the files below into a directory, then convert it as usual (the variant is detected from the talker file): +- Turbo: `t3_turbo_v1.safetensors`, `s3gen_meanflow.safetensors`, `ve.safetensors`, `conds.pt` and the GPT-2 tokenizer files (`vocab.json`, `merges.txt`, `added_tokens.json`, `special_tokens_map.json`, `tokenizer_config.json`) +- Multilingual V3: `t3_mtl23ls_v3.safetensors`, `s3gen_v3.safetensors`, `ve.safetensors`, `conds.pt`, `mtl_tokenizer.json` + +```sh +python convert_hf_to_gguf.py path/to/chatterbox --outfile chatterbox.gguf +python convert_hf_to_gguf.py path/to/chatterbox --mmproj --outfile mmproj-chatterbox.gguf +``` + +## Drama 3 + +[Fish Audio Drama 3](https://huggingface.co/fishaudio/drama-3): a Qwen3.5-MoE backbone (35B, 3B active) samples each frame's main token, a codebook tower samples the 9 other codes, and the Fish DAC decodes them to 44.1 kHz audio. + +Available params: +- Speaker tags `<|speaker:0|>` to `<|speaker:15|>` in the text give each character its own voice, a text without a tag starts with `<|speaker:0|>` +- The language is the one of the text, `--tts-lang` is not used +- Note: `--tts-speaker-file` is not supported yet (voice cloning) +- Note: `--top-k` and `--top-p` are not used, frames are sampled with the settings of the reference (main token: temperature 1.0, top-p 0.9, repetition-aware redraw over the last 16 frames; other codes: temperature 1.0, top-k 30, top-p 0.9) +- Each frame is 2048 samples (about 46 ms), generation stops at the end-of-speech token +- The codebook tower keeps the whole sequence in its KV cache, the prompt and the generated frames are limited to 4096 positions + +Example usage: + +```sh +llama-tts -m drama3.gguf \ + -mm mmproj-drama3.gguf \ + -p "<|speaker:0|> Did you remember to buy the bread? <|speaker:1|> Of course I did, it is on the kitchen table." \ + -n 1000 \ + --output out.wav +``` + +**Note for GGUF conversion:** + +The routed experts of the checkpoint are FP8, they are dequantized at conversion. The mmproj holds the codebook tower and the decoder half of `codec.pth`, it is best kept in BF16: + +```sh +python convert_hf_to_gguf.py path/to/drama-3 --outtype q8_0 --outfile drama3.gguf +python convert_hf_to_gguf.py path/to/drama-3 --mmproj --outtype bf16 --outfile mmproj-drama3.gguf +``` diff --git a/tools/mtmd/README-dev.md b/tools/mtmd/README-dev.md index 736bc9575..8feb7a647 100644 --- a/tools/mtmd/README-dev.md +++ b/tools/mtmd/README-dev.md @@ -108,7 +108,7 @@ Due to wide variety of audio generation pipelines, the `mtmd_gen_audio` system i - 60% changes inside `mtmd-helper-gen.cpp` - 10% changes inside `libmtmd` core (`mtmd-model`, `mtmd-subcontext`, `models/`) - The rest downstream code (CLI, server) should have no changes at all -5. Update usage documentation in `tools/tts/README.md` +5. Update usage documentation in `docs/multimodal/tts.md` IMPORTANT: If your model needs changes that don't fit the existing infrastructure, **open an issue first for discussion**. diff --git a/tools/server/README.md b/tools/server/README.md index 9f5cd62cc..753530a86 100644 --- a/tools/server/README.md +++ b/tools/server/README.md @@ -295,8 +295,8 @@ For the full list of features, please refer to [server's changelog](https://gith | `--spec-ngram-size-n N` | the argument has been removed. use the respective --spec-ngram-*-size-n or --spec-ngram-mod-n-match | | `--spec-ngram-size-m N` | the argument has been removed. use the respective --spec-ngram-*-size-m | | `--spec-ngram-min-hits N` | the argument has been removed. use the respective --spec-ngram-*-min-hits | -| `--tts-lang FNAME` | language (ISO 639-1) for audio generation
see tts/README.md for per-model usage notes | -| `--tts-voice VOICE` | voice for audio generation (model-dependent)
see tts/README.md for per-model usage notes | +| `--tts-lang FNAME` | language (ISO 639-1) for audio generation
see docs/multimodal/tts.md for per-model usage notes | +| `--tts-voice VOICE` | voice for audio generation (model-dependent)
see docs/multimodal/tts.md for per-model usage notes | | `--embd-gemma-default` | use default EmbeddingGemma model (note: can download weights from the internet) | | `--fim-qwen-1.5b-default` | use default Qwen 2.5 Coder 1.5B (note: can download weights from the internet) | | `--fim-qwen-3b-default` | use default Qwen 2.5 Coder 3B (note: can download weights from the internet) | @@ -854,7 +854,7 @@ Causal decoder rerankers (e.g. `Qwen3-VL-Reranker`) have no such limitation. ### POST `/tts`: Generate speech audio from text -Returns raw audio bytes (`audio/wav` by default) rather than JSON. For more info, see [tts/README.md](../tts/README.md) +Returns raw audio bytes (`audio/wav` by default) rather than JSON. For the list of supported models and their params, see [docs/multimodal/tts.md](../../docs/multimodal/tts.md) *Options:* diff --git a/tools/tts/README.md b/tools/tts/README.md index 39d773d85..9608db569 100644 --- a/tools/tts/README.md +++ b/tools/tts/README.md @@ -19,169 +19,6 @@ Common params: - `-n ` limits the output length, e.g. `-n 500`. Note that how many milliseconds each frame represents varies by model - Core inference params such as `-ngl`, `-b`, `-ub`, etc. -## Qwen3-TTS +## Supported models -Available params: -- `--tts-lang` can be `zh`, `en`, `de`, `it`, `pt`, `es`, `ja`, `ko`, `fr`, `ru` (default: `en`) -- `--tts-speaker-file` should point to a speaker reference audio file (wav, mp3) - -Example usage: - -```sh -llama-tts -hf ggml-org/Qwen3-TTS-12Hz-1.7B-Base-GGUF \ - -p "Hello world" \ - --tts-lang english \ - --tts-speaker-file speaker.mp3 \ - --output out.wav -``` - -## Pocket TTS - -Available params: -- `--tts-speaker-file` should point to a speaker reference audio file (wav, mp3). It is required, the model produces almost no audio without it -- `--tts-voice` selects one of the predefined voices shipped with the weights (e.g. `giovanni`), when the GGUF holds them. The converter reads them from `embeddings/*.safetensors` next to the weights, of both repositories, and logs how closely each one is reproduced (some packs ship voice states computed with other weights, the converter warns above 5% and skips a voice only when it does not belong to the model at all) -- Note: `lang` is not used, the language is a property of the weights - -Example usage: - -```sh -llama-tts -m pocket-tts.gguf \ - -mm mmproj-pocket-tts.gguf \ - -p "Hello world" \ - --tts-speaker-file speaker.mp3 \ - --output out.wav -``` - -The weights [without voice cloning](https://huggingface.co/kyutai/pocket-tts-without-voice-cloning) ship a set of predefined voices (`embeddings/*.safetensors`) next to each language; the converter stores them in the mmproj, as the voice prompt embeddings, and leaves out the zeroed voice encoder. With such a model `--tts-speaker-file` is rejected and a voice is selected with `--tts-voice` (the first one when it is not given): - -```sh -llama-tts -m pocket-tts.gguf \ - -mm mmproj-pocket-tts.gguf \ - -p "Ciao a tutti" \ - --tts-voice giovanni \ - --output out.wav -``` - -**Note for GGUF conversion:** - -The [upstream repository](https://huggingface.co/kyutai/pocket-tts) holds one complete model per language under `languages/`, next to a set of shared files at the root. Convert one of the `languages/` directories, **not** the root directory: - -```sh -python convert_hf_to_gguf.py path/to/pocket-tts/languages/english --outfile pocket-tts.gguf -python convert_hf_to_gguf.py path/to/pocket-tts/languages/english --mmproj --outfile mmproj-pocket-tts.gguf -``` - -## Soprano - -Available params: -- `--tts-speaker-file` is not supported; the model uses a fixed voice -- Note: `lang` is not used. Supply normalized English text - -Example usage: - -```sh -llama-tts -m soprano.gguf \ - -mm mmproj-soprano.gguf \ - -p "Hello world" \ - --temp 0 \ - --output out.wav -``` - -**Note for GGUF conversion:** - -Both [Soprano-80M](https://huggingface.co/ekwek/Soprano-80M) and [Soprano-1.1-80M](https://huggingface.co/ekwek/Soprano-1.1-80M) are supported. Convert both files from the same model directory, including `decoder.pth`: - -```sh -python convert_hf_to_gguf.py path/to/Soprano-1.1-80M --outtype f32 --outfile soprano.gguf -python convert_hf_to_gguf.py path/to/Soprano-1.1-80M --mmproj --outtype f16 --outfile mmproj-soprano.gguf -``` - -## LFM3-Audio - -Available params: -- `--tts-lang` can be `ar`, `da`, `de`, `el`, `en`, `en-gb`, `es`, `fi`, `fr`, `he`, `it`, `ja`, `ko`, `nl`, `pl`, `pt`, `sv`, `tr`, `zh` (default: `en`), other values are used as the language name -- `--tts-voice` can be `female` (default) or `male` -- Note: `--tts-speaker-file` is not used, the model has no voice cloning -- Note: `--top-k` and `--top-p` are not used, audio frames are sampled with the settings of the model card (LFM3-Audio: temperature 1.0, top-k 128, repetition penalty 1.1 over 4096 frames; LFM3-Omni: temperature 0.4, repetition penalty 1.2 over 64 frames), the `audio_*` keys of `generation_config.json` override them at conversion -- Each frame is 80 ms, generation stops at the end-of-audio frame - -Example usage: - -```sh -llama-tts -m lfm3-audio.gguf \ - -mm mmproj-lfm3-audio.gguf \ - -p "Hello world" \ - --tts-voice male \ - --output out.wav -``` - -The mmproj holds the audio encoder (for ASR), the depthformer and the codec decoder, the latter two are exported when the checkpoint has an `audio_tokenizer` directory: - -```sh -python convert_hf_to_gguf.py path/to/lfm3-audio --outfile lfm3-audio.gguf -python convert_hf_to_gguf.py path/to/lfm3-audio --mmproj --outfile mmproj-lfm3-audio.gguf -``` - -## Chatterbox - -Two variants are supported: [Chatterbox Turbo](https://huggingface.co/ResembleAI/chatterbox-turbo) (English) and [Chatterbox Multilingual V3](https://huggingface.co/ResembleAI/chatterbox) (23 languages). - -Available params: -- `--tts-speaker-file` should point to a speaker reference audio file (wav, mp3). Optional, the predefined voice is used without it -- `--tts-voice` can be `default`, the voice shipped with the model (`conds.pt`) -- `--tts-lang` sets the language of the multilingual variant, for example `fr` (default: none). It is not used by Turbo - -Example usage: - -```sh -llama-tts -m chatterbox-mtl.gguf \ - -mm mmproj-chatterbox-mtl.gguf \ - -p "Bonjour tout le monde" \ - --tts-lang fr \ - --tts-speaker-file speaker.mp3 \ - --output out.wav -``` - -Note: the multilingual variant runs classifier-free guidance with a second sequence. `llama-tts` sets this up by itself, but `llama-server` does not support it yet. - -**Note for GGUF conversion:** - -The upstream repositories ship raw checkpoints without `config.json`. Download the files below into a directory, then convert it as usual (the variant is detected from the talker file): -- Turbo: `t3_turbo_v1.safetensors`, `s3gen_meanflow.safetensors`, `ve.safetensors`, `conds.pt` and the GPT-2 tokenizer files (`vocab.json`, `merges.txt`, `added_tokens.json`, `special_tokens_map.json`, `tokenizer_config.json`) -- Multilingual V3: `t3_mtl23ls_v3.safetensors`, `s3gen_v3.safetensors`, `ve.safetensors`, `conds.pt`, `mtl_tokenizer.json` - -```sh -python convert_hf_to_gguf.py path/to/chatterbox --outfile chatterbox.gguf -python convert_hf_to_gguf.py path/to/chatterbox --mmproj --outfile mmproj-chatterbox.gguf -``` - -## Drama 3 - -[Fish Audio Drama 3](https://huggingface.co/fishaudio/drama-3): a Qwen3.5-MoE backbone (35B, 3B active) samples each frame's main token, a codebook tower samples the 9 other codes, and the Fish DAC decodes them to 44.1 kHz audio. - -Available params: -- Speaker tags `<|speaker:0|>` to `<|speaker:15|>` in the text give each character its own voice, a text without a tag starts with `<|speaker:0|>` -- The language is the one of the text, `--tts-lang` is not used -- Note: `--tts-speaker-file` is not supported yet (voice cloning) -- Note: `--top-k` and `--top-p` are not used, frames are sampled with the settings of the reference (main token: temperature 1.0, top-p 0.9, repetition-aware redraw over the last 16 frames; other codes: temperature 1.0, top-k 30, top-p 0.9) -- Each frame is 2048 samples (about 46 ms), generation stops at the end-of-speech token -- The codebook tower keeps the whole sequence in its KV cache, the prompt and the generated frames are limited to 4096 positions - -Example usage: - -```sh -llama-tts -m drama3.gguf \ - -mm mmproj-drama3.gguf \ - -p "<|speaker:0|> Did you remember to buy the bread? <|speaker:1|> Of course I did, it is on the kitchen table." \ - -n 1000 \ - --output out.wav -``` - -**Note for GGUF conversion:** - -The routed experts of the checkpoint are FP8, they are dequantized at conversion. The mmproj holds the codebook tower and the decoder half of `codec.pth`, it is best kept in BF16: - -```sh -python convert_hf_to_gguf.py path/to/drama-3 --outtype q8_0 --outfile drama3.gguf -python convert_hf_to_gguf.py path/to/drama-3 --mmproj --outtype bf16 --outfile mmproj-drama3.gguf -``` +See [docs/multimodal/tts.md](../../docs/multimodal/tts.md) for the list of supported models, their params and GGUF conversion notes. diff --git a/tools/tts/tts.cpp b/tools/tts/tts.cpp index 5f5c2b973..7edf605b8 100644 --- a/tools/tts/tts.cpp +++ b/tools/tts/tts.cpp @@ -39,7 +39,7 @@ static void print_usage(int, char ** argv) { LOG("\n %s -hf user/model -p \"text to speak\" -o output.wav\n", argv[0]); LOG("\nnote: --tts-lang, --tts-voice and --tts-speaker-file may not be supported in all models"); LOG("\n use -n to limit the output length"); - LOG("\n see tts/README.md for per-model usage notes"); + LOG("\n see docs/multimodal/tts.md for per-model usage notes"); LOG("\n\n"); }