mtmd: support Qwen3-TTS (note: breaking change to llama-tts binary) (#26254)
* convert text model * main model load ok * convert encoder ok * speaker encoder loading ok * speaker enc graph * adapt vocab for backbone (with some tricks) * add suppress_tokens * poc new mtmd gen api * convert code_predictor to gguf * load gen_code model ok * add clip_encode * wire up * code gen cgraph init version Co-authored-by: Pascal <admin@serveurperso.com> * code2wav convert to gguf * code2wav graph ok * wire up in/out * (wip) subgraph * wire up * wip, correct code2wav * demo (to be removed) * code2wav preserve kv between calls * demo voice clone * llama: add llama_model_get_tok_embd * mtmd_helper_gen_audio API * fix clamp cold prefix Co-authored-by: Pascal <admin@serveurperso.com> * fuse snake op Co-authored-by: Pascal <admin@serveurperso.com> * demo: use proper sampling * update dev docs * polymorphism helper * revamp llama-tts binary * update docs * fix compile * fix lint * nits * add guide + docs * more timings info * clean up code comments * security fixes * update docs * use ggml_build_forward_select, clean up comments * fix ci * use ISO 639-1 language code * rename CODE2WAV --> GEN_WAV, update docs * clean up * clean up tts.cpp * add seq_id * add step_prompt() * mtmd_helper_model_can_chat * clean up comments --------- Co-authored-by: Pascal <admin@serveurperso.com>
This commit is contained in:
Xuan-Son Nguyen
co-authored by
Pascal
Pascal
parent
1c3c9674de
commit
0713275082
@@ -18,6 +18,8 @@ add_library(mtmd
|
||||
mtmd-image.cpp
|
||||
mtmd.h
|
||||
mtmd-helper.cpp
|
||||
mtmd-helper-gen.cpp
|
||||
mtmd-helper-common.h
|
||||
mtmd-helper.h
|
||||
clip.cpp
|
||||
clip.h
|
||||
@@ -52,6 +54,8 @@ add_library(mtmd
|
||||
models/mimovl.cpp
|
||||
models/qwen3a.cpp
|
||||
models/mimo-audio.cpp
|
||||
models/qwen3tts-spkenc.cpp
|
||||
models/qwen3tts-gen.cpp
|
||||
models/step3vl.cpp
|
||||
models/siglip.cpp
|
||||
models/whisper-enc.cpp
|
||||
|
||||
@@ -33,3 +33,52 @@ A typical pipeline of the core libmtmd is as follows:
|
||||
We provide a set of helper functions via `mtmd_helper` to make using libmtmd easier. The helper provides:
|
||||
- Image, audio and video file decoding (for example, decode raw JPEG into RGB bitmap)
|
||||
- Manage `llama_batch` and calls to `llama_decode`
|
||||
|
||||
## Audio generation support
|
||||
|
||||
Audio generation is added to mtmd in PR [#26254](https://github.com/ggml-org/llama.cpp/pull/26254)
|
||||
|
||||
Currently, we support the 3-stage pipeline below which should cover most TTS models:
|
||||
- Stage 1: Backbone / Semantic Stage: Backbone model accepts text prompt and reference voice as input
|
||||
- Stage 2: Acoustic Detail Generator: A model takes the hidden state from backbone and generate audio details (usually as audio codes or mel-spectrogram)
|
||||
- Stage 3: Waveform Reconstruction: Convert the semantic and acoustic data from previous stages to the final waveform
|
||||
|
||||
For example, Qwen3-TTS:
|
||||
- Reference voice is encoded using ECAPA-TDNN speaker encoder (`speaker_encoder`)
|
||||
- Text prompt and reference voice are processed via a backbone (`talker.model`)
|
||||
- A model converts sampled semantic token and hidden state from stage 2 into a list of 15 acoustic codes (`talker.code_predictor`)
|
||||
- 16 generated codes are converted into waveform (`code2wav`)
|
||||
|
||||
### API design constraints
|
||||
|
||||
Due to wide variety of audio generation pipelines, the `mtmd_gen_audio` system is designed to be flexible and reusable by new models.
|
||||
|
||||
`mtmd_gen_audio` is split into 2 main API:
|
||||
- Core API `mtmd.h`: handles main inference. Important: the API surface must be stateless; caller must handle state management and audio frame accumulation.
|
||||
- Helper API `mtmd-helper.h`: provides a model-agnostic stateful API. Usage example can be found in the `tools/tts` directory.
|
||||
|
||||
### Checklist for porting new audio generation models to mtmd
|
||||
|
||||
1. Establish a list of reusable and missing components from the current mtmd implementation.
|
||||
2. For GGUF conversion:
|
||||
- Backbone model should be converted to a normal text model (loadable via `libllama`)
|
||||
- If model used hard-coded embedding row ID, append them to token embeddings and assign token name for them (see `qwen3tts.py`)
|
||||
- If model have a specific output logits head for audio codes (usually semantic code), keep the head as-is and pad the logits at inference time (see `src/models/qwen3vl.cpp`)
|
||||
- Sidecar models (code2wav, bigvgan, etc) must live inside the mmproj GGUF (but can be in different `clip_context` if necessary)
|
||||
- Note: it should use `ggml_build_forward_select` to select graphs if multiple graphs living in the same context
|
||||
- Reuse existing GGUF metadata key name and tensor name whenever possible; think twice before adding extensive changes to GGUF writer. For example, Qwen3-TTS hard-code part of the hparams to `clip.cpp` as they won't likely to change.
|
||||
- For tensor naming:
|
||||
- Prefixed with `a.*` for tensors used by speaker encoder pipeline
|
||||
- Prefixed with `a.gen.*` for generation stages (code / mel-spectrogram / PCM generation)
|
||||
3. Make sure most of the changes happen inside `mtmd-helper-gen.cpp`. A good PR looks like this:
|
||||
- 10-20% changes is to add new backbone (text) model and conversion
|
||||
- 60% changes inside `mtmd-helper-gen.cpp`
|
||||
- 10% changes inside `libmtmd` and `clip.cpp` systems
|
||||
- The rest downstream code (CLI, server) should have no changes at all
|
||||
4. Update usage documentation in `tools/tts/README.md`
|
||||
|
||||
IMPORTANT: If your model needs changes that don't fit the existing infrastructure, **open an issue first for discussion**.
|
||||
|
||||
No-go checklist (these will get the PR rejected and require discussion before proceeding):
|
||||
- Violating the API design constraints stated above
|
||||
- Adding a new model-specific binary: the API and binary surface must stay model-agnostic
|
||||
|
||||
@@ -54,6 +54,9 @@ struct clip_graph {
|
||||
|
||||
clip_graph(clip_ctx * ctx, const clip_image_f32 & img);
|
||||
|
||||
// build sub-graph, reuse buf from parent
|
||||
clip_graph(const clip_graph & parent);
|
||||
|
||||
virtual ~clip_graph() = default;
|
||||
virtual ggml_cgraph * build() = 0;
|
||||
|
||||
|
||||
@@ -32,6 +32,7 @@
|
||||
#define KEY_PROJ_TYPE "clip.projector_type"
|
||||
#define KEY_HAS_AUDIO_ENC "clip.has_audio_encoder"
|
||||
#define KEY_HAS_VISION_ENC "clip.has_vision_encoder"
|
||||
#define KEY_HAS_GEN_AUDIO_ENC "clip.has_gen_audio_encoder"
|
||||
#define KEY_USE_GELU "clip.use_gelu"
|
||||
#define KEY_USE_SILU "clip.use_silu"
|
||||
|
||||
@@ -89,6 +90,8 @@
|
||||
#define KEY_A_ATTN_WINDOW_SIZE "clip.audio.window_size" // mimo-audio-tokenizer: sliding-window radius
|
||||
#define KEY_A_LOCAL_BLOCK_COUNT "clip.audio.local_block_count" // mimo-v2.5: input_local_transformer layer count
|
||||
#define KEY_A_LOCAL_GROUP_SIZE "clip.audio.local_group_size" // mimo-v2.5: input_local_transformer grouping size
|
||||
// audio generation (gen-audio)-specific
|
||||
#define KEY_GEN_AUDIO_PROJ_TYPE "clip.gen.audio.projector_type" // for models with mixed modalities
|
||||
#define KEY_AUDIO_SUBSAMPLING_FACTOR "clip.audio.subsampling_factor"
|
||||
|
||||
//
|
||||
@@ -201,6 +204,48 @@
|
||||
#define TN_MM_A_LOCAL_LN2 "mm.a.local_blk.%d.ln2.%s"
|
||||
#define TN_MM_A_LOCAL_NORM "mm.a.local_norm.%s"
|
||||
|
||||
// qwen3tts speaker encoder (ECAPA-TDNN)
|
||||
#define TN_A_SE_CONV1 "a.blk.%d.se_conv1.%s"
|
||||
#define TN_A_SE_CONV2 "a.blk.%d.se_conv2.%s"
|
||||
#define TN_A_CONV_RES2 "a.blk.%d.res2.%d.%s"
|
||||
#define TN_A_ASP_ATTN "a.asp_attn.%s"
|
||||
#define TN_A_ASP_TDNN "a.asp_tdnn.%s"
|
||||
|
||||
// qwen3tts code_predictor
|
||||
#define TN_A_GEN_CODE_PROJ_IN "a.gen.code.proj_in.%s"
|
||||
#define TN_A_GEN_CODE_EMBD "a.gen.code.embd.%s"
|
||||
#define TN_A_GEN_CODE_HEAD "a.gen.code.head.%s"
|
||||
#define TN_A_GEN_CODE_OUT_EMBD "a.gen.code.out_embd.%s"
|
||||
#define TN_A_GEN_CODE_NORM "a.gen.code.output_norm.%s"
|
||||
|
||||
// qwen3tts code2wav (RVQ codes -> raw PCM)
|
||||
// pre_transformer layers use the generic TN_ATTN_*/TN_FFN_*/TN_LN_*/TN_LS_* macros, prefix "a.gen.wav.tfm"
|
||||
#define TN_A_GEN_WAV_QUANT_FIRST_IN "a.gen.wav.quant.first.in_proj.%s"
|
||||
#define TN_A_GEN_WAV_QUANT_FIRST_OUT "a.gen.wav.quant.first.out_proj.%s"
|
||||
#define TN_A_GEN_WAV_QUANT_FIRST_CB "a.gen.wav.quant.first.codebook.%s"
|
||||
#define TN_A_GEN_WAV_QUANT_REST_IN "a.gen.wav.quant.rest.in_proj.%s"
|
||||
#define TN_A_GEN_WAV_QUANT_REST_OUT "a.gen.wav.quant.rest.out_proj.%s"
|
||||
#define TN_A_GEN_WAV_QUANT_REST_CB "a.gen.wav.quant.rest.codebook.%s"
|
||||
#define TN_A_GEN_WAV_PRE_CONV "a.gen.wav.pre_conv.%s"
|
||||
#define TN_A_GEN_WAV_TFM_IN_PROJ "a.gen.wav.tfm.in_proj.%s"
|
||||
#define TN_A_GEN_WAV_TFM_OUT_PROJ "a.gen.wav.tfm.out_proj.%s"
|
||||
#define TN_A_GEN_WAV_TFM_OUT_NORM "a.gen.wav.tfm.output_norm.%s"
|
||||
#define TN_A_GEN_WAV_UP_CONV "a.gen.wav.up.blk.%d.conv.%s"
|
||||
#define TN_A_GEN_WAV_UP_DWCONV "a.gen.wav.up.blk.%d.dwconv.%s"
|
||||
#define TN_A_GEN_WAV_UP_NORM "a.gen.wav.up.blk.%d.norm.%s"
|
||||
#define TN_A_GEN_WAV_UP_PW1 "a.gen.wav.up.blk.%d.pw1.%s"
|
||||
#define TN_A_GEN_WAV_UP_PW2 "a.gen.wav.up.blk.%d.pw2.%s"
|
||||
#define TN_A_GEN_WAV_UP_GAMMA "a.gen.wav.up.blk.%d.gamma"
|
||||
#define TN_A_GEN_WAV_DAC_ENTRY "a.gen.wav.dac.entry.%s"
|
||||
#define TN_A_GEN_WAV_DAC_SNAKE "a.gen.wav.dac.blk.%d.snake.%s"
|
||||
#define TN_A_GEN_WAV_DAC_CONV "a.gen.wav.dac.blk.%d.conv.%s"
|
||||
#define TN_A_GEN_WAV_DAC_RES_ACT1 "a.gen.wav.dac.blk.%d.res.%d.act1.%s"
|
||||
#define TN_A_GEN_WAV_DAC_RES_CONV1 "a.gen.wav.dac.blk.%d.res.%d.conv1.%s"
|
||||
#define TN_A_GEN_WAV_DAC_RES_ACT2 "a.gen.wav.dac.blk.%d.res.%d.act2.%s"
|
||||
#define TN_A_GEN_WAV_DAC_RES_CONV2 "a.gen.wav.dac.blk.%d.res.%d.conv2.%s"
|
||||
#define TN_A_GEN_WAV_DAC_POST_SNAKE "a.gen.wav.dac.post_snake.%s"
|
||||
#define TN_A_GEN_WAV_DAC_POST_CONV "a.gen.wav.dac.post_conv.%s"
|
||||
|
||||
// cogvlm
|
||||
#define TN_MM_POST_FC_NORM "mm.post_fc_norm.%s"
|
||||
#define TN_MM_H_TO_4H "mm.up.%s"
|
||||
@@ -408,6 +453,8 @@ enum projector_type {
|
||||
PROJECTOR_TYPE_MINIMAX_M3,
|
||||
PROJECTOR_TYPE_GRANITE4_VISION,
|
||||
PROJECTOR_TYPE_MIMO_AUDIO,
|
||||
PROJECTOR_TYPE_QWEN3TTS_SPKENC,
|
||||
PROJECTOR_TYPE_QWEN3TTS_GEN,
|
||||
PROJECTOR_TYPE_UNKNOWN,
|
||||
};
|
||||
|
||||
@@ -465,6 +512,8 @@ static std::map<projector_type, std::string> PROJECTOR_TYPE_NAMES = {
|
||||
{ PROJECTOR_TYPE_GRANITE4_VISION, "granite4_vision"},
|
||||
{ PROJECTOR_TYPE_MIMO_AUDIO, "mimo_audio"},
|
||||
{ PROJECTOR_TYPE_PARAKEET, "parakeet"},
|
||||
{ PROJECTOR_TYPE_QWEN3TTS_SPKENC, "qwen3tts_spkenc"},
|
||||
{ PROJECTOR_TYPE_QWEN3TTS_GEN, "qwen3tts_gen"},
|
||||
};
|
||||
|
||||
static projector_type clip_projector_type_from_string(const std::string & str) {
|
||||
|
||||
@@ -136,6 +136,19 @@ struct clip_hparams {
|
||||
int32_t rvq_num_quantizers = 0;
|
||||
std::vector<int32_t> rvq_codebook_size; // per-quantizer bin count (ragged, e.g. 1024/1024/256/128x17)
|
||||
|
||||
// qwen3tts code2wav
|
||||
int32_t wav_tfm_n_layer = 0;
|
||||
int32_t wav_tfm_n_embd = 0;
|
||||
int32_t wav_tfm_n_ff = 0;
|
||||
int32_t wav_tfm_n_head = 0;
|
||||
int32_t wav_tfm_n_head_kv = 0;
|
||||
float wav_tfm_eps = 1e-5f;
|
||||
float wav_tfm_rope_theta = 10000.0f;
|
||||
int32_t wav_upsample_n_block = 0;
|
||||
int32_t wav_dac_n_block = 0;
|
||||
int32_t wav_dac_n_res = 0;
|
||||
int32_t wav_tfm_swa = 0; // pre_transformer's KV cache size, in frames
|
||||
|
||||
// mimo-v2.5: LLM-side connector (input_local_transformer)
|
||||
int32_t audio_local_n_layer = 0;
|
||||
int32_t audio_local_group_size = 0;
|
||||
@@ -286,6 +299,14 @@ struct clip_layer {
|
||||
ggml_tensor * cross_attn_norm_w = nullptr;
|
||||
ggml_tensor * cross_attn_norm_b = nullptr;
|
||||
|
||||
// qwen3tts speaker encoder: SE-Res2Net block, tdnn1/tdnn2 reuse conv_pw1_w/b and conv_pw2_w/b above
|
||||
ggml_tensor * se_conv1_w = nullptr;
|
||||
ggml_tensor * se_conv1_b = nullptr;
|
||||
ggml_tensor * se_conv2_w = nullptr;
|
||||
ggml_tensor * se_conv2_b = nullptr;
|
||||
std::vector<ggml_tensor *> res2_conv_w; // Res2Net hierarchical branches
|
||||
std::vector<ggml_tensor *> res2_conv_b;
|
||||
|
||||
bool has_deepstack() const {
|
||||
return deepstack_fc1_w != nullptr;
|
||||
}
|
||||
@@ -365,6 +386,73 @@ struct qf_block {
|
||||
std::vector<clip_layer> qf_proj_layers;
|
||||
};
|
||||
|
||||
// qwen3tts code2wav: RVQ codes -> raw PCM
|
||||
struct clip_code2wav {
|
||||
// "upsample" stage: one ConvNeXt block plus the causal ConvTranspose1d before it
|
||||
struct upsample_block {
|
||||
ggml_tensor * conv_w = nullptr; // causal ConvTranspose1d, 2x
|
||||
ggml_tensor * conv_b = nullptr;
|
||||
ggml_tensor * dwconv_w = nullptr; // depthwise causal conv, k=7
|
||||
ggml_tensor * dwconv_b = nullptr;
|
||||
ggml_tensor * norm_w = nullptr; // LayerNorm
|
||||
ggml_tensor * norm_b = nullptr;
|
||||
ggml_tensor * pw1_w = nullptr; // pointwise expand
|
||||
ggml_tensor * pw1_b = nullptr;
|
||||
ggml_tensor * pw2_w = nullptr; // pointwise project
|
||||
ggml_tensor * pw2_b = nullptr;
|
||||
ggml_tensor * gamma = nullptr; // layer scale
|
||||
};
|
||||
|
||||
// one DAC residual unit: SnakeBeta -> dilated causal conv -> SnakeBeta -> pointwise causal conv
|
||||
struct dac_res {
|
||||
ggml_tensor * act1_alpha = nullptr;
|
||||
ggml_tensor * act1_beta = nullptr;
|
||||
ggml_tensor * conv1_w = nullptr;
|
||||
ggml_tensor * conv1_b = nullptr;
|
||||
ggml_tensor * act2_alpha = nullptr;
|
||||
ggml_tensor * act2_beta = nullptr;
|
||||
ggml_tensor * conv2_w = nullptr;
|
||||
ggml_tensor * conv2_b = nullptr;
|
||||
};
|
||||
|
||||
// one DAC upsample block (SnakeBeta -> causal ConvTranspose1d -> 3 residual units)
|
||||
struct dac_block {
|
||||
ggml_tensor * snake_alpha = nullptr;
|
||||
ggml_tensor * snake_beta = nullptr;
|
||||
ggml_tensor * conv_w = nullptr; // causal ConvTranspose1d
|
||||
ggml_tensor * conv_b = nullptr;
|
||||
std::vector<dac_res> res;
|
||||
};
|
||||
|
||||
// quantizer: RVQ codebook decode
|
||||
ggml_tensor * quant_first_in_w = nullptr; // semantic RVQ, in_proj (1x1 conv, loaded as 2D)
|
||||
ggml_tensor * quant_first_out_w = nullptr;
|
||||
ggml_tensor * quant_first_cb_w = nullptr; // codebook (1 layer)
|
||||
ggml_tensor * quant_rest_in_w = nullptr; // acoustic RVQ
|
||||
ggml_tensor * quant_rest_out_w = nullptr;
|
||||
ggml_tensor * quant_rest_cb_w = nullptr; // codebooks, merged 3D [15, vocab, dim]
|
||||
|
||||
ggml_tensor * pre_conv_w = nullptr;
|
||||
ggml_tensor * pre_conv_b = nullptr;
|
||||
|
||||
ggml_tensor * tfm_in_proj_w = nullptr;
|
||||
ggml_tensor * tfm_in_proj_b = nullptr;
|
||||
ggml_tensor * tfm_out_proj_w = nullptr;
|
||||
ggml_tensor * tfm_out_proj_b = nullptr;
|
||||
ggml_tensor * tfm_output_norm_w = nullptr;
|
||||
std::vector<clip_layer> tfm_layers; // reuses the generic block fields (ln_1/attn/ln_2/ffn/ls_1/ls_2)
|
||||
|
||||
std::vector<upsample_block> upsample;
|
||||
|
||||
ggml_tensor * dac_entry_w = nullptr;
|
||||
ggml_tensor * dac_entry_b = nullptr;
|
||||
std::vector<dac_block> dac;
|
||||
ggml_tensor * dac_post_snake_alpha = nullptr;
|
||||
ggml_tensor * dac_post_snake_beta = nullptr;
|
||||
ggml_tensor * dac_post_conv_w = nullptr;
|
||||
ggml_tensor * dac_post_conv_b = nullptr;
|
||||
};
|
||||
|
||||
struct clip_model {
|
||||
clip_modality modality = CLIP_MODALITY_VISION;
|
||||
projector_type proj_type = PROJECTOR_TYPE_MLP;
|
||||
@@ -577,6 +665,24 @@ struct clip_model {
|
||||
ggml_tensor * conv2d_3_w = nullptr;
|
||||
ggml_tensor * conv2d_3_b = nullptr;
|
||||
|
||||
// qwen3tts speaker encoder (ECAPA-TDNN)
|
||||
// reused tensors: stem conv is conv1d_1_w/b, feature aggregation is conv_out_w/b, output proj is mm_fc_w/b
|
||||
ggml_tensor * spk_asp_attn_w = nullptr;
|
||||
ggml_tensor * spk_asp_attn_b = nullptr;
|
||||
ggml_tensor * spk_asp_tdnn_w = nullptr;
|
||||
ggml_tensor * spk_asp_tdnn_b = nullptr;
|
||||
|
||||
// qwen3tts code_predictor
|
||||
ggml_tensor * gen_code_proj_in_w = nullptr; // small_to_mtp_projection
|
||||
ggml_tensor * gen_code_proj_in_b = nullptr;
|
||||
ggml_tensor * gen_code_embd_w = nullptr; // per-codebook embedding, merged 3D
|
||||
ggml_tensor * gen_code_head_w = nullptr; // per-codebook output head, merged 3D
|
||||
ggml_tensor * gen_code_out_embd_w = nullptr; // codebook-0 embedding, fed back into the talker
|
||||
ggml_tensor * gen_code_norm_w = nullptr; // final norm
|
||||
|
||||
// qwen3tts code2wav: RVQ codes -> raw PCM
|
||||
clip_code2wav c2w;
|
||||
|
||||
// cogvlm
|
||||
ggml_tensor * mm_post_fc_norm_w = nullptr;
|
||||
ggml_tensor * mm_post_fc_norm_b = nullptr;
|
||||
|
||||
+416
-39
@@ -17,6 +17,7 @@
|
||||
#include <cstring>
|
||||
#include <fstream>
|
||||
#include <map>
|
||||
#include <random>
|
||||
#include <stdexcept>
|
||||
#include <unordered_set>
|
||||
#include <vector>
|
||||
@@ -269,6 +270,29 @@ clip_graph::clip_graph(clip_ctx * ctx, const clip_image_f32 & img) :
|
||||
gf = ggml_new_graph_custom(ctx0, ctx->max_nodes, false);
|
||||
}
|
||||
|
||||
clip_graph::clip_graph(const clip_graph & parent) :
|
||||
model(parent.model),
|
||||
hparams(parent.hparams),
|
||||
proj_type(parent.proj_type),
|
||||
img(parent.img),
|
||||
patch_size(parent.patch_size),
|
||||
n_patches_x(parent.n_patches_x),
|
||||
n_patches_y(parent.n_patches_y),
|
||||
n_patches(parent.n_patches),
|
||||
n_embd(parent.n_embd),
|
||||
n_head(parent.n_head),
|
||||
n_head_kv(parent.n_head_kv),
|
||||
d_head(parent.d_head),
|
||||
n_layer(parent.n_layer),
|
||||
n_mmproj_embd(parent.n_mmproj_embd),
|
||||
eps(parent.eps),
|
||||
kq_scale(parent.kq_scale),
|
||||
flash_attn_type(parent.flash_attn_type) {
|
||||
// reuse from parent
|
||||
ctx0 = parent.ctx0;
|
||||
gf = parent.gf;
|
||||
}
|
||||
|
||||
ggml_tensor * clip_graph::build_mm(ggml_tensor * w, ggml_tensor * x) const {
|
||||
return ggml_mul_mat(ctx0, w, x);
|
||||
}
|
||||
@@ -873,7 +897,8 @@ ggml_tensor * clip_graph::build_patch_merge_permute(ggml_tensor * cur, int scale
|
||||
return cur;
|
||||
}
|
||||
|
||||
static std::unique_ptr<clip_graph> clip_get_graph_builder(clip_ctx * ctx, const clip_image_f32_batch & imgs) {
|
||||
static std::unique_ptr<clip_graph> clip_get_graph_builder(clip_ctx * ctx, const clip_image_f32_batch & imgs,
|
||||
const clip_encode_params * params = nullptr) {
|
||||
const clip_image_f32 & img = imgs.entries[0];
|
||||
std::unique_ptr<clip_graph> builder;
|
||||
|
||||
@@ -1025,6 +1050,17 @@ static std::unique_ptr<clip_graph> clip_get_graph_builder(clip_ctx * ctx, const
|
||||
{
|
||||
builder = std::make_unique<clip_graph_mimo_audio>(ctx, img);
|
||||
} break;
|
||||
case PROJECTOR_TYPE_QWEN3TTS_SPKENC:
|
||||
{
|
||||
builder = std::make_unique<clip_graph_qwen3tts_spkenc>(ctx, img);
|
||||
} break;
|
||||
case PROJECTOR_TYPE_QWEN3TTS_GEN:
|
||||
{
|
||||
const auto gen_process = params ? params->gen_process : CLIP_GEN_PROCESS_GEN_CODE;
|
||||
const int top_k = params ? params->top_k : 50;
|
||||
const float top_p = params ? params->top_p : 1.0f;
|
||||
builder = std::make_unique<clip_graph_qwen3tts_gen>(ctx, img, gen_process, top_k, top_p);
|
||||
} break;
|
||||
case PROJECTOR_TYPE_YOUTUVL:
|
||||
{
|
||||
builder = std::make_unique<clip_graph_youtuvl>(ctx, img);
|
||||
@@ -1065,8 +1101,9 @@ struct clip_model_loader {
|
||||
|
||||
size_t model_size = 0; // in bytes
|
||||
|
||||
bool has_vision = false;
|
||||
bool has_audio = false;
|
||||
bool has_vision = false;
|
||||
bool has_audio = false;
|
||||
bool has_gen_audio = false;
|
||||
|
||||
mtmd_progress_callback progress_callback = nullptr;
|
||||
void * progress_callback_user_data = nullptr;
|
||||
@@ -1112,8 +1149,9 @@ struct clip_model_loader {
|
||||
|
||||
// modalities
|
||||
{
|
||||
get_bool(KEY_HAS_VISION_ENC, has_vision, false);
|
||||
get_bool(KEY_HAS_AUDIO_ENC, has_audio, false);
|
||||
get_bool(KEY_HAS_VISION_ENC, has_vision, false);
|
||||
get_bool(KEY_HAS_AUDIO_ENC, has_audio, false);
|
||||
get_bool(KEY_HAS_GEN_AUDIO_ENC, has_gen_audio, false);
|
||||
|
||||
if (has_vision) {
|
||||
LOG_INF("%s: has vision encoder\n", __func__);
|
||||
@@ -1121,6 +1159,9 @@ struct clip_model_loader {
|
||||
if (has_audio) {
|
||||
LOG_INF("%s: has audio encoder\n", __func__);
|
||||
}
|
||||
if (has_gen_audio) {
|
||||
LOG_INF("%s: has audio generation (gen) encoder\n", __func__);
|
||||
}
|
||||
}
|
||||
|
||||
// tensors
|
||||
@@ -1147,6 +1188,8 @@ struct clip_model_loader {
|
||||
GGML_ASSERT(has_vision);
|
||||
} else if (modality == CLIP_MODALITY_AUDIO) {
|
||||
GGML_ASSERT(has_audio);
|
||||
} else if (modality == CLIP_MODALITY_GEN_AUDIO) {
|
||||
GGML_ASSERT(has_gen_audio);
|
||||
}
|
||||
model.modality = modality;
|
||||
|
||||
@@ -1163,6 +1206,8 @@ struct clip_model_loader {
|
||||
get_string(KEY_VISION_PROJ_TYPE, proj_type, false);
|
||||
} else if (modality == CLIP_MODALITY_AUDIO) {
|
||||
get_string(KEY_AUDIO_PROJ_TYPE, proj_type, false);
|
||||
} else if (modality == CLIP_MODALITY_GEN_AUDIO) {
|
||||
get_string(KEY_GEN_AUDIO_PROJ_TYPE, proj_type, false);
|
||||
} else {
|
||||
GGML_ABORT("unknown modality");
|
||||
}
|
||||
@@ -1182,12 +1227,13 @@ struct clip_model_loader {
|
||||
}
|
||||
}
|
||||
|
||||
const bool is_vision = model.modality == CLIP_MODALITY_VISION;
|
||||
const bool is_audio = model.modality == CLIP_MODALITY_AUDIO;
|
||||
const bool is_vision = model.modality == CLIP_MODALITY_VISION;
|
||||
const bool is_audio = model.modality == CLIP_MODALITY_AUDIO;
|
||||
const bool is_gen_audio = model.modality == CLIP_MODALITY_GEN_AUDIO;
|
||||
|
||||
// other hparams
|
||||
{
|
||||
const char * prefix = is_vision ? "vision" : "audio";
|
||||
const char * prefix = is_vision ? "vision" : (is_audio ? "audio" : "gen.audio");
|
||||
get_u32(string_format(KEY_N_EMBD, prefix), hparams.n_embd);
|
||||
get_u32(string_format(KEY_N_HEAD, prefix), hparams.n_head);
|
||||
get_u32(string_format(KEY_N_EMBD_HEAD, prefix), hparams.n_embd_head, false);
|
||||
@@ -1198,6 +1244,7 @@ struct clip_model_loader {
|
||||
|
||||
// n_head_kv is optional (for GQA), default to n_head
|
||||
hparams.n_head_kv = hparams.n_head;
|
||||
get_u32(string_format(KEY_N_HEAD_KV, prefix), hparams.n_head_kv, false);
|
||||
|
||||
if (is_vision) {
|
||||
get_u32(KEY_IMAGE_SIZE, hparams.image_size);
|
||||
@@ -1226,6 +1273,11 @@ struct clip_model_loader {
|
||||
hparams.image_size = 0;
|
||||
hparams.patch_size = 1;
|
||||
|
||||
} else if (is_gen_audio) {
|
||||
// these are unused, but still need to be set to avoid issues
|
||||
hparams.image_size = 0;
|
||||
hparams.patch_size = 1;
|
||||
|
||||
} else {
|
||||
GGML_ASSERT(false && "unknown modality");
|
||||
}
|
||||
@@ -1647,6 +1699,33 @@ struct clip_model_loader {
|
||||
"%s: mimo_audio: %s must be > 0\n", __func__, KEY_A_LOCAL_GROUP_SIZE));
|
||||
}
|
||||
} break;
|
||||
case PROJECTOR_TYPE_QWEN3TTS_SPKENC:
|
||||
{
|
||||
// ECAPA-TDNN speaker encoder, mel front-end uses the Slaney default (fmin=0, fmax=sr/2)
|
||||
hparams.audio_sample_rate = 24000;
|
||||
hparams.audio_n_fft = 1024;
|
||||
hparams.audio_window_len = 1024;
|
||||
hparams.audio_hop_len = 256;
|
||||
} break;
|
||||
case PROJECTOR_TYPE_QWEN3TTS_GEN:
|
||||
{
|
||||
// TODO: hardcoded for now, read from code_predictor_config instead
|
||||
hparams.rope_theta = 1000000.0f;
|
||||
|
||||
// code2wav params
|
||||
hparams.wav_tfm_n_layer = 8;
|
||||
hparams.wav_tfm_n_embd = 512;
|
||||
hparams.wav_tfm_n_ff = 1024;
|
||||
hparams.wav_tfm_n_head = 16;
|
||||
hparams.wav_tfm_n_head_kv = 16;
|
||||
hparams.wav_tfm_eps = 1e-5f;
|
||||
hparams.wav_tfm_rope_theta = 10000.0f;
|
||||
hparams.wav_upsample_n_block = 2;
|
||||
hparams.wav_dac_n_block = 4;
|
||||
hparams.wav_dac_n_res = 3;
|
||||
// matches the reference decoder's sliding_window (speech_tokenizer/config.json)
|
||||
hparams.wav_tfm_swa = 72;
|
||||
} break;
|
||||
case PROJECTOR_TYPE_PADDLEOCR:
|
||||
{
|
||||
hparams.n_merge = 2;
|
||||
@@ -1871,7 +1950,9 @@ struct clip_model_loader {
|
||||
}
|
||||
|
||||
// TODO @ngxson : support both audio and video in the future
|
||||
const char * prefix = model.modality == CLIP_MODALITY_AUDIO ? "a" : "v";
|
||||
const char * prefix = model.modality == CLIP_MODALITY_AUDIO ? "a"
|
||||
: model.modality == CLIP_MODALITY_GEN_AUDIO ? "a.gen.code"
|
||||
: "v";
|
||||
|
||||
// get offsets
|
||||
for (int64_t i = 0; i < gguf_get_n_tensors(ctx_gguf.get()); ++i) {
|
||||
@@ -1973,7 +2054,8 @@ struct clip_model_loader {
|
||||
model.position_embeddings = get_tensor(string_format(TN_POS_EMBD, prefix), false);
|
||||
|
||||
const bool has_standard_layers = (
|
||||
model.proj_type != PROJECTOR_TYPE_GEMMA3NV);
|
||||
model.proj_type != PROJECTOR_TYPE_GEMMA3NV &&
|
||||
model.proj_type != PROJECTOR_TYPE_QWEN3TTS_SPKENC);
|
||||
|
||||
// layers
|
||||
const int n_layers_to_load = has_standard_layers ? hparams.n_layer : 0;
|
||||
@@ -2599,6 +2681,144 @@ struct clip_model_loader {
|
||||
model.mm_1_w = get_tensor(string_format(TN_MM_AUDIO_MLP, 1, "weight"));
|
||||
model.mm_2_w = get_tensor(string_format(TN_MM_AUDIO_MLP, 2, "weight"));
|
||||
} break;
|
||||
case PROJECTOR_TYPE_QWEN3TTS_SPKENC:
|
||||
{
|
||||
// stem TDNN (block 0)
|
||||
model.conv1d_1_w = get_tensor(string_format(TN_CONV1D, 0, "weight"));
|
||||
model.conv1d_1_b = get_tensor(string_format(TN_CONV1D, 0, "bias"));
|
||||
|
||||
// SE-Res2Net blocks (GGUF bid 1..3, one per hparams.n_layer)
|
||||
model.layers.resize(hparams.n_layer);
|
||||
for (int il = 0; il < hparams.n_layer; il++) {
|
||||
auto & layer = model.layers[il];
|
||||
int bid = il + 1;
|
||||
layer.conv_pw1_w = get_tensor(string_format(TN_CONV_PW1, prefix, bid, "weight"));
|
||||
layer.conv_pw1_b = get_tensor(string_format(TN_CONV_PW1, prefix, bid, "bias"));
|
||||
layer.conv_pw2_w = get_tensor(string_format(TN_CONV_PW2, prefix, bid, "weight"));
|
||||
layer.conv_pw2_b = get_tensor(string_format(TN_CONV_PW2, prefix, bid, "bias"));
|
||||
layer.se_conv1_w = get_tensor(string_format(TN_A_SE_CONV1, bid, "weight"));
|
||||
layer.se_conv1_b = get_tensor(string_format(TN_A_SE_CONV1, bid, "bias"));
|
||||
layer.se_conv2_w = get_tensor(string_format(TN_A_SE_CONV2, bid, "weight"));
|
||||
layer.se_conv2_b = get_tensor(string_format(TN_A_SE_CONV2, bid, "bias"));
|
||||
layer.res2_conv_w.resize(7);
|
||||
layer.res2_conv_b.resize(7);
|
||||
for (int xid = 0; xid < 7; xid++) {
|
||||
layer.res2_conv_w[xid] = get_tensor(string_format(TN_A_CONV_RES2, bid, xid, "weight"));
|
||||
layer.res2_conv_b[xid] = get_tensor(string_format(TN_A_CONV_RES2, bid, xid, "bias"));
|
||||
}
|
||||
}
|
||||
|
||||
// multi-layer feature aggregation
|
||||
model.conv_out_w = get_tensor(string_format(TN_CONV_OUT, "weight"));
|
||||
model.conv_out_b = get_tensor(string_format(TN_CONV_OUT, "bias"));
|
||||
|
||||
// attentive statistics pooling
|
||||
model.spk_asp_attn_w = get_tensor(string_format(TN_A_ASP_ATTN, "weight"));
|
||||
model.spk_asp_attn_b = get_tensor(string_format(TN_A_ASP_ATTN, "bias"));
|
||||
model.spk_asp_tdnn_w = get_tensor(string_format(TN_A_ASP_TDNN, "weight"));
|
||||
model.spk_asp_tdnn_b = get_tensor(string_format(TN_A_ASP_TDNN, "bias"));
|
||||
|
||||
// final speaker embedding projection
|
||||
model.mm_fc_w = get_tensor(string_format(TN_MM_AUDIO_FC, "weight"));
|
||||
model.mm_fc_b = get_tensor(string_format(TN_MM_AUDIO_FC, "bias"));
|
||||
} break;
|
||||
case PROJECTOR_TYPE_QWEN3TTS_GEN:
|
||||
{
|
||||
// code_predictor
|
||||
model.gen_code_proj_in_w = get_tensor(string_format(TN_A_GEN_CODE_PROJ_IN, "weight"));
|
||||
model.gen_code_proj_in_b = get_tensor(string_format(TN_A_GEN_CODE_PROJ_IN, "bias"));
|
||||
model.gen_code_embd_w = get_tensor(string_format(TN_A_GEN_CODE_EMBD, "weight"));
|
||||
model.gen_code_head_w = get_tensor(string_format(TN_A_GEN_CODE_HEAD, "weight"));
|
||||
model.gen_code_out_embd_w = get_tensor(string_format(TN_A_GEN_CODE_OUT_EMBD, "weight"));
|
||||
model.gen_code_norm_w = get_tensor(string_format(TN_A_GEN_CODE_NORM, "weight"));
|
||||
|
||||
// code2wav: RVQ codes -> raw PCM, lives in the same ctx as code_predictor
|
||||
{
|
||||
auto & c2w = model.c2w;
|
||||
|
||||
c2w.quant_first_in_w = get_tensor(string_format(TN_A_GEN_WAV_QUANT_FIRST_IN, "weight"));
|
||||
c2w.quant_first_out_w = get_tensor(string_format(TN_A_GEN_WAV_QUANT_FIRST_OUT, "weight"));
|
||||
c2w.quant_first_cb_w = get_tensor(string_format(TN_A_GEN_WAV_QUANT_FIRST_CB, "weight"));
|
||||
c2w.quant_rest_in_w = get_tensor(string_format(TN_A_GEN_WAV_QUANT_REST_IN, "weight"));
|
||||
c2w.quant_rest_out_w = get_tensor(string_format(TN_A_GEN_WAV_QUANT_REST_OUT, "weight"));
|
||||
c2w.quant_rest_cb_w = get_tensor(string_format(TN_A_GEN_WAV_QUANT_REST_CB, "weight"));
|
||||
|
||||
c2w.pre_conv_w = get_tensor(string_format(TN_A_GEN_WAV_PRE_CONV, "weight"));
|
||||
c2w.pre_conv_b = get_tensor(string_format(TN_A_GEN_WAV_PRE_CONV, "bias"));
|
||||
|
||||
c2w.tfm_in_proj_w = get_tensor(string_format(TN_A_GEN_WAV_TFM_IN_PROJ, "weight"));
|
||||
c2w.tfm_in_proj_b = get_tensor(string_format(TN_A_GEN_WAV_TFM_IN_PROJ, "bias"));
|
||||
c2w.tfm_out_proj_w = get_tensor(string_format(TN_A_GEN_WAV_TFM_OUT_PROJ, "weight"));
|
||||
c2w.tfm_out_proj_b = get_tensor(string_format(TN_A_GEN_WAV_TFM_OUT_PROJ, "bias"));
|
||||
c2w.tfm_output_norm_w = get_tensor(string_format(TN_A_GEN_WAV_TFM_OUT_NORM, "weight"));
|
||||
|
||||
// loaded manually, the generic model.layers loop is taken by code_predictor
|
||||
c2w.tfm_layers.resize(hparams.wav_tfm_n_layer);
|
||||
for (int il = 0; il < hparams.wav_tfm_n_layer; il++) {
|
||||
auto & layer = c2w.tfm_layers[il];
|
||||
const char * p = "a.gen.wav.tfm";
|
||||
layer.q_w = get_tensor(string_format(TN_ATTN_Q, p, il, "weight"));
|
||||
layer.k_w = get_tensor(string_format(TN_ATTN_K, p, il, "weight"));
|
||||
layer.v_w = get_tensor(string_format(TN_ATTN_V, p, il, "weight"));
|
||||
layer.o_w = get_tensor(string_format(TN_ATTN_OUTPUT, p, il, "weight"));
|
||||
layer.ln_1_w = get_tensor(string_format(TN_LN_1, p, il, "weight"));
|
||||
layer.ln_2_w = get_tensor(string_format(TN_LN_2, p, il, "weight"));
|
||||
layer.ls_1_w = get_tensor(string_format(TN_LS_1, p, il, "weight"));
|
||||
layer.ls_2_w = get_tensor(string_format(TN_LS_2, p, il, "weight"));
|
||||
layer.ff_gate_w = get_tensor(string_format(TN_FFN_GATE, p, il, "weight"));
|
||||
layer.ff_up_w = get_tensor(string_format(TN_FFN_UP, p, il, "weight"));
|
||||
layer.ff_down_w = get_tensor(string_format(TN_FFN_DOWN, p, il, "weight"));
|
||||
}
|
||||
|
||||
// upsample: 2x (causal ConvTranspose1d + ConvNeXt block)
|
||||
c2w.upsample.resize(hparams.wav_upsample_n_block);
|
||||
for (int il = 0; il < hparams.wav_upsample_n_block; il++) {
|
||||
auto & up = c2w.upsample[il];
|
||||
up.conv_w = get_tensor(string_format(TN_A_GEN_WAV_UP_CONV, il, "weight"));
|
||||
up.conv_b = get_tensor(string_format(TN_A_GEN_WAV_UP_CONV, il, "bias"));
|
||||
up.dwconv_w = get_tensor(string_format(TN_A_GEN_WAV_UP_DWCONV, il, "weight"));
|
||||
up.dwconv_b = get_tensor(string_format(TN_A_GEN_WAV_UP_DWCONV, il, "bias"));
|
||||
up.norm_w = get_tensor(string_format(TN_A_GEN_WAV_UP_NORM, il, "weight"));
|
||||
up.norm_b = get_tensor(string_format(TN_A_GEN_WAV_UP_NORM, il, "bias"));
|
||||
up.pw1_w = get_tensor(string_format(TN_A_GEN_WAV_UP_PW1, il, "weight"));
|
||||
up.pw1_b = get_tensor(string_format(TN_A_GEN_WAV_UP_PW1, il, "bias"));
|
||||
up.pw2_w = get_tensor(string_format(TN_A_GEN_WAV_UP_PW2, il, "weight"));
|
||||
up.pw2_b = get_tensor(string_format(TN_A_GEN_WAV_UP_PW2, il, "bias"));
|
||||
up.gamma = get_tensor(string_format(TN_A_GEN_WAV_UP_GAMMA, il));
|
||||
}
|
||||
|
||||
// DAC decoder: conv_pre + n upsample blocks (each with n_res residual units) + conv_post
|
||||
c2w.dac_entry_w = get_tensor(string_format(TN_A_GEN_WAV_DAC_ENTRY, "weight"));
|
||||
c2w.dac_entry_b = get_tensor(string_format(TN_A_GEN_WAV_DAC_ENTRY, "bias"));
|
||||
|
||||
c2w.dac.resize(hparams.wav_dac_n_block);
|
||||
for (int il = 0; il < hparams.wav_dac_n_block; il++) {
|
||||
auto & blk = c2w.dac[il];
|
||||
blk.snake_alpha = get_tensor(string_format(TN_A_GEN_WAV_DAC_SNAKE, il, "alpha"));
|
||||
blk.snake_beta = get_tensor(string_format(TN_A_GEN_WAV_DAC_SNAKE, il, "beta"));
|
||||
blk.conv_w = get_tensor(string_format(TN_A_GEN_WAV_DAC_CONV, il, "weight"));
|
||||
blk.conv_b = get_tensor(string_format(TN_A_GEN_WAV_DAC_CONV, il, "bias"));
|
||||
|
||||
blk.res.resize(hparams.wav_dac_n_res);
|
||||
for (int ir = 0; ir < hparams.wav_dac_n_res; ir++) {
|
||||
auto & res = blk.res[ir];
|
||||
res.act1_alpha = get_tensor(string_format(TN_A_GEN_WAV_DAC_RES_ACT1, il, ir, "alpha"));
|
||||
res.act1_beta = get_tensor(string_format(TN_A_GEN_WAV_DAC_RES_ACT1, il, ir, "beta"));
|
||||
res.conv1_w = get_tensor(string_format(TN_A_GEN_WAV_DAC_RES_CONV1, il, ir, "weight"));
|
||||
res.conv1_b = get_tensor(string_format(TN_A_GEN_WAV_DAC_RES_CONV1, il, ir, "bias"));
|
||||
res.act2_alpha = get_tensor(string_format(TN_A_GEN_WAV_DAC_RES_ACT2, il, ir, "alpha"));
|
||||
res.act2_beta = get_tensor(string_format(TN_A_GEN_WAV_DAC_RES_ACT2, il, ir, "beta"));
|
||||
res.conv2_w = get_tensor(string_format(TN_A_GEN_WAV_DAC_RES_CONV2, il, ir, "weight"));
|
||||
res.conv2_b = get_tensor(string_format(TN_A_GEN_WAV_DAC_RES_CONV2, il, ir, "bias"));
|
||||
}
|
||||
}
|
||||
|
||||
c2w.dac_post_snake_alpha = get_tensor(string_format(TN_A_GEN_WAV_DAC_POST_SNAKE, "alpha"));
|
||||
c2w.dac_post_snake_beta = get_tensor(string_format(TN_A_GEN_WAV_DAC_POST_SNAKE, "beta"));
|
||||
c2w.dac_post_conv_w = get_tensor(string_format(TN_A_GEN_WAV_DAC_POST_CONV, "weight"));
|
||||
c2w.dac_post_conv_b = get_tensor(string_format(TN_A_GEN_WAV_DAC_POST_CONV, "bias"));
|
||||
}
|
||||
} break;
|
||||
case PROJECTOR_TYPE_VOXTRAL:
|
||||
{
|
||||
model.conv1d_1_w = get_tensor(string_format(TN_CONV1D, 1, "weight"));
|
||||
@@ -3427,6 +3647,7 @@ struct clip_model_loader {
|
||||
struct clip_init_result clip_init(const char * fname, struct clip_context_params ctx_params) {
|
||||
clip_ctx * ctx_vision = nullptr;
|
||||
clip_ctx * ctx_audio = nullptr;
|
||||
clip_ctx * ctx_gen_audio = nullptr;
|
||||
|
||||
try {
|
||||
clip_model_loader loader(fname,
|
||||
@@ -3459,16 +3680,25 @@ struct clip_init_result clip_init(const char * fname, struct clip_context_params
|
||||
}
|
||||
}
|
||||
|
||||
if (loader.has_gen_audio) {
|
||||
ctx_gen_audio = new clip_ctx(ctx_params);
|
||||
loader.load_hparams(ctx_gen_audio->model, CLIP_MODALITY_GEN_AUDIO);
|
||||
loader.load_tensors(*ctx_gen_audio);
|
||||
// TODO: fix warmup
|
||||
ctx_gen_audio->buf_compute_meta.resize(ctx_gen_audio->max_nodes * ggml_tensor_overhead() + ggml_graph_overhead());
|
||||
}
|
||||
|
||||
} catch (const std::exception & e) {
|
||||
LOG_ERR("%s: failed to load model '%s': %s\n", __func__, fname, e.what());
|
||||
|
||||
delete ctx_vision;
|
||||
delete ctx_audio;
|
||||
delete ctx_gen_audio;
|
||||
|
||||
return {nullptr, nullptr};
|
||||
return {nullptr, nullptr, nullptr};
|
||||
}
|
||||
|
||||
return {ctx_vision, ctx_audio};
|
||||
return {ctx_vision, ctx_audio, ctx_gen_audio};
|
||||
}
|
||||
|
||||
struct clip_cap clip_get_cap(const char * fname) {
|
||||
@@ -3784,6 +4014,16 @@ int clip_n_output_tokens(const clip_ctx * ctx, const clip_image_f32 * img) {
|
||||
const int ds = ctx->model.hparams.audio_proj_downsample_rate;
|
||||
n_patches = ((img->nx() + ws - 1) / ws) * (ws / ds);
|
||||
} break;
|
||||
case PROJECTOR_TYPE_QWEN3TTS_SPKENC:
|
||||
{
|
||||
// pooling gives one speaker embedding, whatever the clip length is
|
||||
n_patches = 1;
|
||||
} break;
|
||||
case PROJECTOR_TYPE_QWEN3TTS_GEN:
|
||||
{
|
||||
// one hidden-state vector fed back to the talker per call
|
||||
n_patches = 1;
|
||||
} break;
|
||||
case PROJECTOR_TYPE_GRANITE4_VISION:
|
||||
{
|
||||
// Per-tile output token count: each projector block outputs
|
||||
@@ -3817,7 +4057,16 @@ bool clip_image_encode(struct clip_ctx * ctx, int n_threads, const clip_image_f3
|
||||
}
|
||||
|
||||
bool clip_image_batch_encode(clip_ctx * ctx, int n_threads, const clip_image_f32_batch * imgs_c_ptr, std::vector<float> & out_batch_embd) {
|
||||
const clip_image_f32_batch & imgs = *imgs_c_ptr;
|
||||
clip_encode_params params;
|
||||
params.imgs = imgs_c_ptr;
|
||||
params.n_threads = n_threads;
|
||||
params.out_embd = &out_batch_embd;
|
||||
|
||||
return clip_encode(ctx, ¶ms);
|
||||
}
|
||||
|
||||
bool clip_encode(struct clip_ctx * ctx, struct clip_encode_params * params) {
|
||||
const clip_image_f32_batch & imgs = *params->imgs;
|
||||
int n_batch_cur = imgs.entries.size();
|
||||
|
||||
// [QWEN_VIDEO] for video models, the batch dimension is used as temporal dimension for merged frames
|
||||
@@ -3828,12 +4077,12 @@ bool clip_image_batch_encode(clip_ctx * ctx, int n_threads, const clip_image_f32
|
||||
|
||||
// if buffers are not allocated, we need to do a warmup run to allocate them
|
||||
if (!ctx->is_allocated) {
|
||||
clip_model_loader::warmup(*ctx, *imgs_c_ptr);
|
||||
clip_model_loader::warmup(*ctx, *params->imgs);
|
||||
}
|
||||
|
||||
// build the inference graph
|
||||
ggml_backend_sched_reset(ctx->sched.get());
|
||||
ggml_cgraph * gf = clip_get_graph_builder(ctx, imgs)->build();
|
||||
ggml_cgraph * gf = clip_get_graph_builder(ctx, imgs, params)->build();
|
||||
ggml_backend_sched_alloc_graph(ctx->sched.get(), gf);
|
||||
|
||||
// set inputs
|
||||
@@ -3918,8 +4167,8 @@ bool clip_image_batch_encode(clip_ctx * ctx, int n_threads, const clip_image_f32
|
||||
}
|
||||
set_input_f32("inp_raw", inp_raw);
|
||||
|
||||
} else {
|
||||
// audio input
|
||||
} else if (!(ctx->proj_type() == PROJECTOR_TYPE_QWEN3TTS_GEN && params->gen_process == CLIP_GEN_PROCESS_GEN_WAV)) {
|
||||
// audio input, code2wav is not here: its only input is "inp_codes", set in the switch below
|
||||
GGML_ASSERT(imgs.entries.size() == 1);
|
||||
|
||||
const auto & mel_inp = imgs.entries[0];
|
||||
@@ -4475,9 +4724,77 @@ bool clip_image_batch_encode(clip_ctx * ctx, int n_threads, const clip_image_f32
|
||||
case PROJECTOR_TYPE_COGVLM:
|
||||
case PROJECTOR_TYPE_YASA2:
|
||||
case PROJECTOR_TYPE_GEMMA4UA:
|
||||
case PROJECTOR_TYPE_QWEN3TTS_SPKENC:
|
||||
{
|
||||
// do nothing
|
||||
} break;
|
||||
case PROJECTOR_TYPE_QWEN3TTS_GEN:
|
||||
{
|
||||
if (params->gen_process == CLIP_GEN_PROCESS_GEN_WAV) {
|
||||
GGML_ASSERT(params->codes != nullptr);
|
||||
|
||||
// frame-major input to group-major, rear-padded with code 0 up to one window
|
||||
const int64_t n_codes = model.gen_code_head_w->ne[2] + 1;
|
||||
const int64_t n_frames_w = hparams.wav_tfm_swa;
|
||||
const int64_t n_frames = (int64_t) params->codes->size() / n_codes;
|
||||
GGML_ASSERT(n_frames > 0 && n_frames <= n_frames_w);
|
||||
|
||||
// codes are used as ggml_get_rows indices, so check them against the codebook vocab
|
||||
const int64_t vocab_first = model.c2w.quant_first_cb_w->ne[1];
|
||||
const int64_t vocab_rest = model.c2w.quant_rest_cb_w->ne[1];
|
||||
for (int64_t f = 0; f < n_frames; f++) {
|
||||
for (int64_t g = 0; g < n_codes; g++) {
|
||||
const int32_t c = (*params->codes)[f * n_codes + g];
|
||||
const int64_t vocab = (g == 0) ? vocab_first : vocab_rest;
|
||||
if (c < 0 || (int64_t) c >= vocab) {
|
||||
LOG_ERR("%s: code out of range (frame %lld, group %lld, code %d, vocab %lld)\n",
|
||||
__func__, (long long) f, (long long) g, c, (long long) vocab);
|
||||
return false;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
std::vector<int32_t> codes(n_frames_w * n_codes, 0);
|
||||
for (int64_t f = 0; f < n_frames; f++) {
|
||||
for (int64_t g = 0; g < n_codes; g++) {
|
||||
codes[g * n_frames_w + f] = (*params->codes)[f * n_codes + g];
|
||||
}
|
||||
}
|
||||
set_input_i32("inp_codes", codes);
|
||||
|
||||
// upload the state from the previous call, or zero-fill on a cold start
|
||||
size_t offset = 0;
|
||||
for (const auto & slot : list_c2w_state_slots(hparams, model)) {
|
||||
ggml_tensor * t = get_inp_tensor(("state_in_" + slot.name).c_str());
|
||||
const size_t nb = ggml_nbytes(t);
|
||||
if (params->state_in && params->state_in->size() >= offset + nb) {
|
||||
ggml_backend_tensor_set(t, params->state_in->data() + offset, 0, nb);
|
||||
} else {
|
||||
std::vector<uint8_t> zeros(nb, 0);
|
||||
ggml_backend_tensor_set(t, zeros.data(), 0, nb);
|
||||
}
|
||||
offset += nb;
|
||||
}
|
||||
} else {
|
||||
// code0 indexes gen_code_out_embd_w via ggml_get_rows; bound it
|
||||
const int64_t vocab0 = model.gen_code_out_embd_w->ne[1];
|
||||
if (params->code0 < 0 || (int64_t) params->code0 >= vocab0) {
|
||||
LOG_ERR("%s: code0 out of range (%d, vocab %lld)\n", __func__, params->code0, (long long) vocab0);
|
||||
return false;
|
||||
}
|
||||
std::vector<int32_t> code0 = { params->code0 };
|
||||
set_input_i32("inp_code0", code0);
|
||||
|
||||
// one uniform(0,1) draw per codebook, used by do_sampling()
|
||||
static std::mt19937 rng{ std::random_device{}() };
|
||||
std::uniform_real_distribution<float> dist(0.0f, 1.0f);
|
||||
const int64_t n_acoustic = model.gen_code_head_w->ne[2];
|
||||
for (int64_t g = 0; g < n_acoustic; g++) {
|
||||
std::vector<float> r = { dist(rng) };
|
||||
set_input_f32(("inp_rand_" + std::to_string(g)).c_str(), r);
|
||||
}
|
||||
}
|
||||
} break;
|
||||
case PROJECTOR_TYPE_HUNYUANVL:
|
||||
{
|
||||
// Compute the HunyuanVL 2D position embedding on CPU (with the
|
||||
@@ -4883,7 +5200,7 @@ bool clip_image_batch_encode(clip_ctx * ctx, int n_threads, const clip_image_f32
|
||||
if (reg) {
|
||||
auto ggml_backend_set_n_threads_fn = (ggml_backend_set_n_threads_t) ggml_backend_reg_get_proc_address(reg, "ggml_backend_set_n_threads");
|
||||
if (ggml_backend_set_n_threads_fn) {
|
||||
ggml_backend_set_n_threads_fn(ctx->backend_cpu, n_threads);
|
||||
ggml_backend_set_n_threads_fn(ctx->backend_cpu, params->n_threads);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -4893,34 +5210,90 @@ bool clip_image_batch_encode(clip_ctx * ctx, int n_threads, const clip_image_f32
|
||||
return false;
|
||||
}
|
||||
|
||||
// the last node is the embedding tensor
|
||||
ggml_tensor * embeddings = ggml_graph_node(gf, -1);
|
||||
// the last node is the embedding tensor, code2wav has no out_embd
|
||||
ggml_tensor * embeddings = params->out_embd ? ggml_graph_node(gf, -1) : nullptr;
|
||||
|
||||
// sanity check (assuming that all images in batch have the same number of tokens, so we only check the first one)
|
||||
const int n_tokens_out = embeddings->ne[1];
|
||||
const int expected_n_tokens_out = clip_n_output_tokens(ctx, &imgs.entries[0]);
|
||||
if (n_tokens_out != expected_n_tokens_out) {
|
||||
LOG_ERR("%s: expected output %d tokens, got %d\n", __func__, expected_n_tokens_out, n_tokens_out);
|
||||
GGML_ABORT("Invalid number of output tokens");
|
||||
}
|
||||
if (embeddings != nullptr) {
|
||||
// sanity check (assuming that all images in batch have the same number of tokens, so we only check the first one)
|
||||
const int n_tokens_out = embeddings->ne[1];
|
||||
const int expected_n_tokens_out = clip_n_output_tokens(ctx, &imgs.entries[0]);
|
||||
if (n_tokens_out != expected_n_tokens_out) {
|
||||
LOG_ERR("%s: expected output %d tokens, got %d\n", __func__, expected_n_tokens_out, n_tokens_out);
|
||||
GGML_ABORT("Invalid number of output tokens");
|
||||
}
|
||||
|
||||
LOG_DBG("%s: output embedding shape [%d, %d, %d]\n", __func__,
|
||||
(int)embeddings->ne[0], (int)embeddings->ne[1], (int)embeddings->ne[2]);
|
||||
LOG_DBG("%s: output embedding shape [%d, %d, %d]\n", __func__,
|
||||
(int)embeddings->ne[0], (int)embeddings->ne[1], (int)embeddings->ne[2]);
|
||||
|
||||
// copy output to user buffer if provided
|
||||
// if output is empty, skip the copy
|
||||
if (!out_batch_embd.empty()) {
|
||||
if (out_batch_embd.size() != (size_t)ggml_nelements(embeddings)) {
|
||||
LOG_ERR("%s: output buffer has %zu elements but expected %zu\n", __func__, out_batch_embd.size(), (size_t)ggml_nelements(embeddings));
|
||||
GGML_ABORT("Output buffer size mismatch");
|
||||
// copy output to user buffer if provided
|
||||
// if output is empty, skip the copy
|
||||
auto & out_batch_embd = *params->out_embd;
|
||||
if (!out_batch_embd.empty()) {
|
||||
if (out_batch_embd.size() != (size_t)ggml_nelements(embeddings)) {
|
||||
LOG_ERR("%s: output buffer has %zu elements but expected %zu\n", __func__, out_batch_embd.size(), (size_t)ggml_nelements(embeddings));
|
||||
GGML_ABORT("Output buffer size mismatch");
|
||||
}
|
||||
ggml_backend_tensor_get(embeddings, out_batch_embd.data(), 0, ggml_nbytes(embeddings));
|
||||
} else {
|
||||
LOG_WRN("%s: output buffer is empty, skipping copy\n", __func__);
|
||||
}
|
||||
ggml_backend_tensor_get(embeddings, out_batch_embd.data(), 0, ggml_nbytes(embeddings));
|
||||
} else {
|
||||
LOG_WRN("%s: output buffer is empty, skipping copy\n", __func__);
|
||||
}
|
||||
|
||||
//
|
||||
// for audio gen models
|
||||
//
|
||||
|
||||
if (params->out_codes != nullptr) {
|
||||
ggml_tensor * codes = ggml_graph_get_tensor(gf, "out_codes");
|
||||
if (codes == nullptr) {
|
||||
GGML_ABORT("out_codes requested but graph has no \"out_codes\" tensor");
|
||||
}
|
||||
auto & out_codes = *params->out_codes;
|
||||
out_codes.resize(ggml_nelements(codes));
|
||||
ggml_backend_tensor_get(codes, out_codes.data(), 0, ggml_nbytes(codes));
|
||||
}
|
||||
if (params->out_audio != nullptr) {
|
||||
ggml_tensor * audio = ggml_graph_get_tensor(gf, "out_audio");
|
||||
if (audio == nullptr) {
|
||||
GGML_ABORT("out_audio requested but graph has no \"out_audio\" tensor");
|
||||
}
|
||||
auto & out_audio = *params->out_audio;
|
||||
out_audio.resize(ggml_nelements(audio));
|
||||
ggml_backend_tensor_get(audio, out_audio.data(), 0, ggml_nbytes(audio));
|
||||
|
||||
// drop the tail audio that comes from the code-0 rear padding
|
||||
const int64_t n_codes = model.gen_code_head_w->ne[2] + 1;
|
||||
const int64_t n_frames_w = hparams.wav_tfm_swa;
|
||||
const int64_t n_frames = (int64_t) params->codes->size() / n_codes;
|
||||
if (n_frames < n_frames_w) {
|
||||
const size_t hop = out_audio.size() / n_frames_w;
|
||||
out_audio.resize((size_t) n_frames * hop);
|
||||
}
|
||||
}
|
||||
if (params->state_out != nullptr) {
|
||||
auto & state_out = *params->state_out;
|
||||
size_t total = 0;
|
||||
for (const auto & slot : list_c2w_state_slots(hparams, model)) {
|
||||
total += (size_t) (slot.ne0 * slot.ne1) * sizeof(float);
|
||||
}
|
||||
state_out.resize(total);
|
||||
size_t offset = 0;
|
||||
for (const auto & slot : list_c2w_state_slots(hparams, model)) {
|
||||
ggml_tensor * t = ggml_graph_get_tensor(gf, ("state_out_" + slot.name).c_str());
|
||||
if (t == nullptr) {
|
||||
GGML_ABORT("state_out requested but graph has no \"state_out_%s\" tensor", slot.name.c_str());
|
||||
}
|
||||
const size_t nb = ggml_nbytes(t);
|
||||
ggml_backend_tensor_get(t, state_out.data() + offset, 0, nb);
|
||||
offset += nb;
|
||||
}
|
||||
}
|
||||
|
||||
//
|
||||
// Debug: dump final embeddings if MTMD_DEBUG_EMBEDDINGS is set
|
||||
if (ctx->debug_output_embeddings) {
|
||||
//
|
||||
|
||||
if (ctx->debug_output_embeddings && embeddings != nullptr) {
|
||||
const int64_t n_embd = embeddings->ne[0];
|
||||
const int64_t n_tokens = embeddings->ne[1];
|
||||
std::vector<float> emb_data(ggml_nelements(embeddings));
|
||||
@@ -5047,6 +5420,10 @@ int clip_n_mmproj_embd(const struct clip_ctx * ctx) {
|
||||
return ctx->model.mm_ffn_down_w->ne[1];
|
||||
case PROJECTOR_TYPE_MIMO_AUDIO:
|
||||
return ctx->model.mm_2_w->ne[1];
|
||||
case PROJECTOR_TYPE_QWEN3TTS_SPKENC:
|
||||
return ctx->model.mm_fc_w->ne[2];
|
||||
case PROJECTOR_TYPE_QWEN3TTS_GEN:
|
||||
return ctx->model.gen_code_out_embd_w->ne[0];
|
||||
case PROJECTOR_TYPE_PARAKEET:
|
||||
return ctx->model.mm_1_w->ne[1];
|
||||
default:
|
||||
|
||||
@@ -37,6 +37,7 @@ struct clip_image_f32_batch;
|
||||
enum clip_modality {
|
||||
CLIP_MODALITY_VISION,
|
||||
CLIP_MODALITY_AUDIO,
|
||||
CLIP_MODALITY_GEN_AUDIO,
|
||||
};
|
||||
|
||||
enum clip_flash_attn_type {
|
||||
@@ -61,6 +62,7 @@ struct clip_context_params {
|
||||
struct clip_init_result {
|
||||
struct clip_ctx * ctx_v; // vision context
|
||||
struct clip_ctx * ctx_a; // audio context
|
||||
struct clip_ctx * ctx_gen_a; // audio generation context
|
||||
};
|
||||
|
||||
struct clip_init_result clip_init(const char * fname, struct clip_context_params ctx_params);
|
||||
@@ -84,6 +86,33 @@ int clip_n_mmproj_embd(const struct clip_ctx * ctx);
|
||||
bool clip_image_encode (struct clip_ctx * ctx, int n_threads, const clip_image_f32 * img, std::vector<float> & out_vec);
|
||||
bool clip_image_batch_encode(struct clip_ctx * ctx, int n_threads, const struct clip_image_f32_batch * imgs, std::vector<float> & out_batch_embd);
|
||||
|
||||
enum clip_gen_process_type {
|
||||
CLIP_GEN_PROCESS_GEN_UNKNOWN,
|
||||
CLIP_GEN_PROCESS_GEN_CODE, // h_state to codes
|
||||
CLIP_GEN_PROCESS_GEN_WAV, // codes to raw PCM audio
|
||||
};
|
||||
struct clip_encode_params {
|
||||
int n_threads = 1;
|
||||
const clip_image_f32_batch * imgs = nullptr;
|
||||
std::vector<float> * out_embd = nullptr;
|
||||
|
||||
// for audio gen, imgs has exactly one entry: hidden state from backbone (GEN_CODE) or unused (GEN_WAV)
|
||||
clip_gen_process_type gen_process = CLIP_GEN_PROCESS_GEN_UNKNOWN;
|
||||
|
||||
// GEN_CODE: out_embd receives the embd to feed back to the backbone
|
||||
int32_t code0 = 0; // semantic code sampled by the backbone
|
||||
int32_t top_k = 50;
|
||||
float top_p = 1.0f;
|
||||
std::vector<int32_t> * out_codes = nullptr; // this frame's 16 sampled codes
|
||||
|
||||
// GEN_WAV
|
||||
const std::vector<int32_t> * codes = nullptr; // this frame's 16 RVQ codes
|
||||
std::vector<float> * out_audio = nullptr; // decoded PCM samples, F32
|
||||
const std::vector<uint8_t> * state_in = nullptr; // state from previous call, null or wrong size means cold start
|
||||
std::vector<uint8_t> * state_out = nullptr; // state for the next call
|
||||
};
|
||||
bool clip_encode(struct clip_ctx * ctx, struct clip_encode_params * params);
|
||||
|
||||
bool clip_is_llava(const struct clip_ctx * ctx);
|
||||
// note for contributor: this clip_is_(model) pattern is deprecated
|
||||
// do NOT add new functions like this
|
||||
|
||||
@@ -2,6 +2,11 @@
|
||||
|
||||
#include "../clip-graph.h"
|
||||
|
||||
#include <map>
|
||||
#include <string>
|
||||
#include <utility>
|
||||
#include <vector>
|
||||
|
||||
/*
|
||||
* IMPORTANT: The mtmd module does NOT accept pull requests that are fully or predominantly AI-generated.
|
||||
* We encourage human contributors to ensure the quality and reliability of the codebase.
|
||||
@@ -215,6 +220,111 @@ struct clip_graph_mimo_audio : clip_graph {
|
||||
ggml_cgraph * build() override;
|
||||
};
|
||||
|
||||
struct clip_graph_qwen3tts_spkenc : clip_graph {
|
||||
clip_graph_qwen3tts_spkenc(clip_ctx * ctx, const clip_image_f32 & img) : clip_graph(ctx, img) {}
|
||||
ggml_cgraph * build() override;
|
||||
|
||||
ggml_tensor * conv1d_same(ggml_tensor * x, ggml_tensor * w, ggml_tensor * b, int dilation) const;
|
||||
ggml_tensor * res2net(ggml_tensor * x, const clip_layer & layer, int dilation, int scale) const;
|
||||
ggml_tensor * se_block(ggml_tensor * x, const clip_layer & layer) const;
|
||||
ggml_tensor * se_res2net_block(ggml_tensor * x, const clip_layer & layer, int dilation, int scale) const;
|
||||
ggml_tensor * attentive_stats_pool(ggml_tensor * x) const;
|
||||
};
|
||||
|
||||
struct clip_graph_qwen3tts_gen : clip_graph {
|
||||
clip_graph_qwen3tts_gen(clip_ctx * ctx, const clip_image_f32 & img, clip_gen_process_type gen_process, int top_k, float top_p)
|
||||
: clip_graph(ctx, img), gen_process(gen_process), top_k(top_k), top_p(top_p) {}
|
||||
ggml_cgraph * build() override;
|
||||
|
||||
// which sub-graph build() constructs, fixed at graph-build time
|
||||
clip_gen_process_type gen_process;
|
||||
|
||||
// sampling params, fixed at graph-build time (GEN_CODE only)
|
||||
int top_k;
|
||||
float top_p;
|
||||
|
||||
//
|
||||
// code_gen: backbone hidden state + sampled code0 -> 16 RVQ codes
|
||||
// MTP-style code predictor, one token per codebook
|
||||
//
|
||||
struct code_gen : clip_graph {
|
||||
code_gen(const clip_graph & parent, int top_k, float top_p)
|
||||
: clip_graph(parent), top_k(top_k), top_p(top_p) {}
|
||||
ggml_cgraph * build() override { GGML_ABORT("call prefill()/step() instead"); }
|
||||
|
||||
int top_k;
|
||||
float top_p;
|
||||
|
||||
ggml_tensor * cache_set(ggml_tensor * cache, int row_idx, ggml_tensor * value) const;
|
||||
ggml_tensor * do_sampling(ggml_tensor * logits, ggml_tensor * inp_rand) const;
|
||||
|
||||
ggml_tensor * const_i32(ggml_tensor * anchor, float value) const;
|
||||
ggml_tensor * causal_mask_row(int64_t n_kv_pad, int pos) const;
|
||||
ggml_tensor * project_in(ggml_tensor * cur) const;
|
||||
|
||||
ggml_tensor * layer_forward(
|
||||
ggml_tensor * cur,
|
||||
const clip_layer & layer,
|
||||
ggml_tensor * inp_pos,
|
||||
ggml_tensor * kq_mask,
|
||||
ggml_tensor *& k_cache_layer,
|
||||
ggml_tensor *& v_cache_layer,
|
||||
int64_t n_kv_pad,
|
||||
int pos,
|
||||
int il) const;
|
||||
|
||||
void prefill(
|
||||
std::vector<ggml_tensor *> & k_cache,
|
||||
std::vector<ggml_tensor *> & v_cache,
|
||||
ggml_tensor *& out_code_cache,
|
||||
ggml_tensor * h_state,
|
||||
ggml_tensor * code0_embd,
|
||||
ggml_tensor * inp_rand) const;
|
||||
|
||||
ggml_tensor * step(
|
||||
std::vector<ggml_tensor *> & k_cache,
|
||||
std::vector<ggml_tensor *> & v_cache,
|
||||
ggml_tensor * out_code_cache,
|
||||
ggml_tensor * inp_rand,
|
||||
int step_idx) const;
|
||||
};
|
||||
|
||||
//
|
||||
// code2wav: RVQ codes -> raw PCM (quantizer + pre_conv + pre_transformer + upsample + DAC).
|
||||
//
|
||||
struct code2wav : clip_graph {
|
||||
code2wav(const clip_graph & parent) : clip_graph(parent) {}
|
||||
ggml_cgraph * build() override { GGML_ABORT("call decode() instead"); }
|
||||
|
||||
// state_in: previous call's persisted state, by slot name (see list_c2w_state_slots())
|
||||
std::map<std::string, ggml_tensor *> state_in;
|
||||
// state_out: this call's state to persist, added to the graph outputs by build()
|
||||
mutable std::vector<std::pair<std::string, ggml_tensor *>> state_out;
|
||||
|
||||
// stateful conv ops: read/update their state via state_in/state_out[state_name]
|
||||
ggml_tensor * causal_conv1d(ggml_tensor * x, ggml_tensor * w, ggml_tensor * b, int dilation, const std::string & state_name) const;
|
||||
ggml_tensor * causal_conv1d_dw(ggml_tensor * x, ggml_tensor * w, ggml_tensor * b, const std::string & state_name) const;
|
||||
ggml_tensor * causal_conv_transpose1d(ggml_tensor * x, ggml_tensor * w, ggml_tensor * b, int stride, const std::string & state_name) const;
|
||||
ggml_tensor * snake(ggml_tensor * x, ggml_tensor * alpha, ggml_tensor * beta) const;
|
||||
|
||||
ggml_tensor * quant_decode(ggml_tensor * inp_codes) const;
|
||||
ggml_tensor * tfm_layer_forward(ggml_tensor * cur, const clip_layer & layer, int il) const;
|
||||
ggml_tensor * convnext_block(ggml_tensor * x, const clip_code2wav::upsample_block & blk, const std::string & state_prefix) const;
|
||||
ggml_tensor * dac_res_unit(ggml_tensor * x, const clip_code2wav::dac_res & res, int dilation, const std::string & state_name) const;
|
||||
|
||||
// inp_codes [1, n_codes] I32 -> this frame's audio samples [n_samples] F32, clamped to [-1, 1]
|
||||
ggml_tensor * decode(ggml_tensor * inp_codes) const;
|
||||
};
|
||||
};
|
||||
|
||||
// one persisted state buffer used by code2wav, see qwen3tts-gen.cpp
|
||||
struct c2w_state_slot {
|
||||
std::string name;
|
||||
int64_t ne0;
|
||||
int64_t ne1;
|
||||
};
|
||||
std::vector<c2w_state_slot> list_c2w_state_slots(const clip_hparams & hparams, const clip_model & model);
|
||||
|
||||
struct clip_graph_kimik25 : clip_graph {
|
||||
clip_graph_kimik25(clip_ctx * ctx, const clip_image_f32 & img) : clip_graph(ctx, img) {}
|
||||
ggml_cgraph * build() override;
|
||||
|
||||
@@ -0,0 +1,766 @@
|
||||
#include "models.h"
|
||||
|
||||
#include <string>
|
||||
|
||||
// on-device sampling: top-k, top-p, then a random draw
|
||||
ggml_tensor * clip_graph_qwen3tts_gen::code_gen::do_sampling(ggml_tensor * logits, ggml_tensor * inp_rand) const {
|
||||
logits = ggml_reshape_1d(ctx0, logits, ggml_nelements(logits));
|
||||
const int64_t n_vocab = logits->ne[0];
|
||||
|
||||
// sort a's rows by idx
|
||||
auto sort_by = [this](ggml_tensor * a, ggml_tensor * idx) {
|
||||
ggml_tensor * a2d = ggml_reshape_2d(ctx0, a, 1, a->ne[0]);
|
||||
return ggml_reshape_1d(ctx0, ggml_get_rows(ctx0, a2d, idx), idx->ne[0]);
|
||||
};
|
||||
|
||||
ggml_tensor * cur = logits;
|
||||
ggml_tensor * candidates = nullptr; // maps row index back to vocab id
|
||||
|
||||
if (top_k > 0 && top_k < n_vocab) {
|
||||
ggml_tensor * idx = ggml_top_k(ctx0, cur, top_k);
|
||||
candidates = idx;
|
||||
cur = sort_by(cur, idx);
|
||||
cb(cur, "sample_top_k_logits", -1);
|
||||
}
|
||||
|
||||
if (top_p < 1.0f) {
|
||||
ggml_tensor * sorted_idx = ggml_argsort(ctx0, cur, GGML_SORT_ORDER_DESC);
|
||||
ggml_tensor * sorted_logits = sort_by(cur, sorted_idx);
|
||||
candidates = candidates ? sort_by(candidates, sorted_idx) : sorted_idx;
|
||||
|
||||
ggml_tensor * probs = ggml_soft_max(ctx0, sorted_logits);
|
||||
ggml_tensor * cdf = ggml_cumsum(ctx0, probs);
|
||||
|
||||
// keep_mask[i] = 1 once cdf[i] crosses top_p
|
||||
ggml_tensor * cdf_scaled = ggml_scale_bias(ctx0, cdf, -1.0f, top_p);
|
||||
ggml_tensor * keep_mask = ggml_step(ctx0, cdf_scaled);
|
||||
ggml_tensor * idxf = ggml_sum(ctx0, keep_mask);
|
||||
idxf = ggml_clamp(ctx0, idxf, 0.0f, (float) keep_mask->ne[0] - 1);
|
||||
ggml_tensor * ones = ggml_scale_bias(ctx0, idxf, 0.0f, 1.0f);
|
||||
|
||||
// top-p must include the crossing element, so force it to 1
|
||||
ggml_tensor * keep_mask_2d = ggml_reshape_2d(ctx0, keep_mask, 1, keep_mask->ne[0]);
|
||||
keep_mask_2d = ggml_set_rows(ctx0, keep_mask_2d, ones, ggml_cast(ctx0, idxf, GGML_TYPE_I32));
|
||||
keep_mask = ggml_reshape_1d(ctx0, keep_mask_2d, keep_mask->ne[0]);
|
||||
|
||||
// log(1) = 0 (keep), log(0) = -inf (drop)
|
||||
ggml_tensor * bias = ggml_log(ctx0, keep_mask);
|
||||
cur = ggml_add(ctx0, sorted_logits, bias);
|
||||
cb(cur, "sample_top_p_logits", -1);
|
||||
}
|
||||
|
||||
// draw one token: find where the cdf crosses inp_rand
|
||||
ggml_tensor * probs = ggml_soft_max(ctx0, cur);
|
||||
ggml_tensor * cumsum = ggml_cumsum(ctx0, probs);
|
||||
|
||||
ggml_tensor * diff = ggml_sub(ctx0, cumsum, inp_rand);
|
||||
ggml_tensor * cross_mask = ggml_step(ctx0, diff);
|
||||
ggml_tensor * idxf = ggml_sum(ctx0, cross_mask);
|
||||
ggml_tensor * idx = ggml_cast(ctx0, ggml_scale_bias(ctx0, idxf, -1.0f, (float) cross_mask->ne[0]), GGML_TYPE_I32);
|
||||
|
||||
if (candidates) {
|
||||
ggml_tensor * cand_2d = ggml_reshape_2d(ctx0, candidates, 1, candidates->ne[0]);
|
||||
idx = ggml_get_rows(ctx0, cand_2d, idx);
|
||||
}
|
||||
cb(idx, "sample_token_id", -1);
|
||||
|
||||
return idx;
|
||||
}
|
||||
|
||||
// returns a new cache with row row_idx set to value
|
||||
ggml_tensor * clip_graph_qwen3tts_gen::code_gen::cache_set(ggml_tensor * cache, int row_idx, ggml_tensor * value) const {
|
||||
const int64_t n_embd = cache->ne[0];
|
||||
const int64_t n_cache = cache->ne[1];
|
||||
GGML_ASSERT(row_idx >= 0 && row_idx < n_cache);
|
||||
|
||||
// append value as the last row, then gather it back into place
|
||||
ggml_tensor * value_2d = ggml_reshape_2d(ctx0, value, n_embd, 1);
|
||||
ggml_tensor * cache_ext = ggml_concat(ctx0, cache, value_2d, 1); // [n_embd, n_cache + 1]
|
||||
|
||||
// gather indices [0..row_idx-1, n_cache, row_idx+1..n_cache-1]
|
||||
// built via concat, since ggml_set_rows needs F32/F16 values, not an I32 index array
|
||||
ggml_tensor * idx = const_i32(cache, (float) n_cache);
|
||||
if (row_idx > 0) {
|
||||
ggml_tensor * prefix = ggml_cast(ctx0, ggml_arange(ctx0, 0.0f, (float) row_idx, 1.0f), GGML_TYPE_I32);
|
||||
idx = ggml_concat(ctx0, prefix, idx, 0);
|
||||
}
|
||||
if (row_idx < n_cache - 1) {
|
||||
ggml_tensor * suffix = ggml_cast(ctx0, ggml_arange(ctx0, (float) (row_idx + 1), (float) n_cache, 1.0f), GGML_TYPE_I32);
|
||||
idx = ggml_concat(ctx0, idx, suffix, 0);
|
||||
}
|
||||
|
||||
ggml_tensor * result = ggml_get_rows(ctx0, cache_ext, idx);
|
||||
cb(result, "cache_set_out", -1);
|
||||
return result;
|
||||
}
|
||||
|
||||
// builds a const i32 with no host upload: view a tensor, zero it via scale, add value, cast to i32
|
||||
ggml_tensor * clip_graph_qwen3tts_gen::code_gen::const_i32(ggml_tensor * anchor, float value) const {
|
||||
ggml_tensor * v = ggml_view_1d(ctx0, anchor, 1, 0);
|
||||
if (v->type != GGML_TYPE_F32) {
|
||||
v = ggml_cast(ctx0, v, GGML_TYPE_F32);
|
||||
}
|
||||
return ggml_cast(ctx0, ggml_scale_bias(ctx0, v, 0.0f, value), GGML_TYPE_I32);
|
||||
}
|
||||
|
||||
// causal keep-mask row for a query at position pos, window size n_kv_pad
|
||||
ggml_tensor * clip_graph_qwen3tts_gen::code_gen::causal_mask_row(int64_t n_kv_pad, int pos) const {
|
||||
ggml_tensor * ones = ggml_fill(ctx0, ggml_new_tensor_2d(ctx0, GGML_TYPE_F32, n_kv_pad, n_kv_pad), 1.0f);
|
||||
ggml_tensor * keep = ggml_tri(ctx0, ones, GGML_TRI_TYPE_LOWER_DIAG);
|
||||
ggml_tensor * row = ggml_view_1d(ctx0, keep, n_kv_pad, (size_t) pos * keep->nb[1]);
|
||||
ggml_tensor * mask = ggml_log(ctx0, row); // 0 = keep, -inf = masked
|
||||
return ggml_reshape_4d(ctx0, mask, n_kv_pad, 1, 1, 1);
|
||||
}
|
||||
|
||||
// talker hidden size -> predictor hidden size (small_to_mtp_projection)
|
||||
ggml_tensor * clip_graph_qwen3tts_gen::code_gen::project_in(ggml_tensor * cur) const {
|
||||
if (!model.gen_code_proj_in_w) {
|
||||
return cur;
|
||||
}
|
||||
cur = ggml_mul_mat(ctx0, model.gen_code_proj_in_w, cur);
|
||||
if (model.gen_code_proj_in_b) {
|
||||
cur = ggml_add(ctx0, cur, model.gen_code_proj_in_b);
|
||||
}
|
||||
return cur;
|
||||
}
|
||||
|
||||
// one transformer layer at position pos; writes k/v into k_cache_layer/v_cache_layer at row pos
|
||||
ggml_tensor * clip_graph_qwen3tts_gen::code_gen::layer_forward(
|
||||
ggml_tensor * cur,
|
||||
const clip_layer & layer,
|
||||
ggml_tensor * inp_pos,
|
||||
ggml_tensor * kq_mask,
|
||||
ggml_tensor *& k_cache_layer,
|
||||
ggml_tensor *& v_cache_layer,
|
||||
int64_t n_kv_pad,
|
||||
int pos,
|
||||
int il) const {
|
||||
const int n_head = hparams.n_head;
|
||||
const int n_head_kv = hparams.n_head_kv;
|
||||
const int64_t d_head = layer.q_w->ne[1] / n_head; // real head_dim, not n_embd / n_head
|
||||
const float kq_scale = 1.0f / sqrtf((float) d_head);
|
||||
|
||||
ggml_tensor * residual = cur;
|
||||
|
||||
ggml_tensor * h = ggml_rms_norm(ctx0, cur, hparams.eps);
|
||||
h = ggml_mul(ctx0, h, layer.ln_1_w);
|
||||
|
||||
ggml_tensor * q = ggml_mul_mat(ctx0, layer.q_w, h);
|
||||
ggml_tensor * k = ggml_mul_mat(ctx0, layer.k_w, h);
|
||||
ggml_tensor * v = ggml_mul_mat(ctx0, layer.v_w, h);
|
||||
|
||||
q = ggml_reshape_3d(ctx0, q, d_head, n_head, 1);
|
||||
k = ggml_reshape_3d(ctx0, k, d_head, n_head_kv, 1);
|
||||
|
||||
q = ggml_rms_norm(ctx0, q, hparams.eps);
|
||||
q = ggml_mul(ctx0, q, layer.q_norm);
|
||||
k = ggml_rms_norm(ctx0, k, hparams.eps);
|
||||
k = ggml_mul(ctx0, k, layer.k_norm);
|
||||
|
||||
q = ggml_rope_ext(ctx0, q, inp_pos, nullptr, (int) d_head, GGML_ROPE_TYPE_NEOX, 0,
|
||||
hparams.rope_theta, 1.0f, 0.0f, 1.0f, 0.0f, 0.0f);
|
||||
k = ggml_rope_ext(ctx0, k, inp_pos, nullptr, (int) d_head, GGML_ROPE_TYPE_NEOX, 0,
|
||||
hparams.rope_theta, 1.0f, 0.0f, 1.0f, 0.0f, 0.0f);
|
||||
|
||||
// write k/v into the cache at row pos, flat layout
|
||||
ggml_tensor * k_flat = ggml_reshape_1d(ctx0, k, d_head * n_head_kv);
|
||||
k_cache_layer = cache_set(k_cache_layer, pos, k_flat);
|
||||
v_cache_layer = cache_set(v_cache_layer, pos, v);
|
||||
|
||||
ggml_tensor * q_cur = ggml_reshape_4d(ctx0, q, d_head, n_head, 1, 1);
|
||||
ggml_tensor * k_cur = ggml_reshape_4d(ctx0, k_cache_layer, d_head, n_head_kv, n_kv_pad, 1);
|
||||
ggml_tensor * v_cur = ggml_reshape_4d(ctx0, v_cache_layer, d_head, n_head_kv, n_kv_pad, 1);
|
||||
|
||||
ggml_tensor * attn_out = build_attn(layer.o_w, layer.o_b, q_cur, k_cur, v_cur, kq_mask, kq_scale, il);
|
||||
|
||||
cur = ggml_add(ctx0, residual, attn_out);
|
||||
|
||||
ggml_tensor * h2 = ggml_rms_norm(ctx0, cur, hparams.eps);
|
||||
h2 = ggml_mul(ctx0, h2, layer.ln_2_w);
|
||||
|
||||
ggml_tensor * gate = ggml_mul_mat(ctx0, layer.ff_gate_w, h2);
|
||||
ggml_tensor * up = ggml_mul_mat(ctx0, layer.ff_up_w, h2);
|
||||
ggml_tensor * gu = ggml_swiglu_split(ctx0, gate, up);
|
||||
ggml_tensor * down = ggml_mul_mat(ctx0, layer.ff_down_w, gu);
|
||||
|
||||
return ggml_add(ctx0, cur, down);
|
||||
}
|
||||
|
||||
// position 0: hidden bridge, seeds the k/v cache, no sampling
|
||||
// position 1: embed(code0), sample with lm_head[0], write out_code_cache[1]
|
||||
void clip_graph_qwen3tts_gen::code_gen::prefill(
|
||||
std::vector<ggml_tensor *> & k_cache,
|
||||
std::vector<ggml_tensor *> & v_cache,
|
||||
ggml_tensor *& out_code_cache,
|
||||
ggml_tensor * h_state,
|
||||
ggml_tensor * code0_embd,
|
||||
ggml_tensor * inp_rand) const {
|
||||
const int64_t n_kv_pad = k_cache[0]->ne[1];
|
||||
|
||||
{
|
||||
ggml_tensor * cur = project_in(h_state);
|
||||
ggml_tensor * kq_mask = causal_mask_row(n_kv_pad, 0);
|
||||
ggml_tensor * inp_pos = const_i32(k_cache[0], 0.0f);
|
||||
for (size_t il = 0; il < model.layers.size(); il++) {
|
||||
cur = layer_forward(cur, model.layers[il], inp_pos, kq_mask, k_cache[il], v_cache[il], n_kv_pad, 0, (int) il);
|
||||
}
|
||||
// position 0's output is unused, it only seeded the cache
|
||||
}
|
||||
|
||||
{
|
||||
ggml_tensor * cur = project_in(code0_embd);
|
||||
ggml_tensor * kq_mask = causal_mask_row(n_kv_pad, 1);
|
||||
ggml_tensor * inp_pos = const_i32(k_cache[0], 1.0f);
|
||||
for (size_t il = 0; il < model.layers.size(); il++) {
|
||||
cur = layer_forward(cur, model.layers[il], inp_pos, kq_mask, k_cache[il], v_cache[il], n_kv_pad, 1, (int) il);
|
||||
}
|
||||
|
||||
cur = ggml_rms_norm(ctx0, cur, hparams.eps);
|
||||
cur = ggml_mul(ctx0, cur, model.gen_code_norm_w);
|
||||
|
||||
ggml_tensor * head_w = model.gen_code_head_w;
|
||||
ggml_tensor * head_g = ggml_view_2d(ctx0, head_w, head_w->ne[0], head_w->ne[1], head_w->nb[1], 0); // lm_head[0]
|
||||
ggml_tensor * logits = ggml_mul_mat(ctx0, head_g, cur);
|
||||
|
||||
ggml_tensor * sampled = do_sampling(logits, inp_rand);
|
||||
out_code_cache = cache_set(out_code_cache, 1, sampled);
|
||||
}
|
||||
}
|
||||
|
||||
// one decode step of code_predictor
|
||||
// at step_idx g:
|
||||
// - read code from out_code_cache[g], then embed it with codebook table g-1
|
||||
// - write new kv at cache row g+1, sample with lm_head[g]
|
||||
// - write result to out_code_cache[g+1]
|
||||
// step_idx must be in [1, n_acoustic - 1]
|
||||
ggml_tensor * clip_graph_qwen3tts_gen::code_gen::step(
|
||||
std::vector<ggml_tensor *> & k_cache,
|
||||
std::vector<ggml_tensor *> & v_cache,
|
||||
ggml_tensor * out_code_cache,
|
||||
ggml_tensor * inp_rand,
|
||||
int step_idx) const {
|
||||
const int64_t n_acoustic = model.gen_code_head_w->ne[2];
|
||||
GGML_ASSERT(step_idx >= 1 && step_idx < n_acoustic);
|
||||
GGML_ASSERT(k_cache.size() == model.layers.size());
|
||||
GGML_ASSERT(v_cache.size() == model.layers.size());
|
||||
|
||||
const int64_t n_kv_pad = k_cache[0]->ne[1];
|
||||
const int pos = step_idx + 1; // new cache row and RoPE position
|
||||
|
||||
// embed the previous code via this step's codebook table (rows are already scalars)
|
||||
ggml_tensor * code_in = ggml_view_1d(ctx0, out_code_cache, 1, (size_t) step_idx * out_code_cache->nb[1]);
|
||||
|
||||
ggml_tensor * embd_w = model.gen_code_embd_w; // [n_embd_talker, vocab, n_acoustic]
|
||||
ggml_tensor * embd_g = ggml_view_2d(ctx0, embd_w, embd_w->ne[0], embd_w->ne[1], embd_w->nb[1],
|
||||
(size_t) (step_idx - 1) * embd_w->nb[2]);
|
||||
ggml_tensor * cur = ggml_get_rows(ctx0, embd_g, code_in);
|
||||
cur = ggml_reshape_1d(ctx0, cur, cur->ne[0]);
|
||||
cb(cur, "step_embd_in", step_idx);
|
||||
|
||||
cur = project_in(cur);
|
||||
cb(cur, "step_proj_in", step_idx);
|
||||
|
||||
ggml_tensor * kq_mask = causal_mask_row(n_kv_pad, pos);
|
||||
ggml_tensor * inp_pos = const_i32(k_cache[0], (float) pos);
|
||||
|
||||
for (size_t il = 0; il < model.layers.size(); il++) {
|
||||
cur = layer_forward(cur, model.layers[il], inp_pos, kq_mask, k_cache[il], v_cache[il], n_kv_pad, pos, (int) il);
|
||||
cb(cur, "step_layer_out", (int) il);
|
||||
}
|
||||
|
||||
// final norm, this step's lm_head, sample, write the result
|
||||
cur = ggml_rms_norm(ctx0, cur, hparams.eps);
|
||||
cur = ggml_mul(ctx0, cur, model.gen_code_norm_w);
|
||||
|
||||
ggml_tensor * head_w = model.gen_code_head_w; // [n_embd_pred, vocab, n_acoustic]
|
||||
ggml_tensor * head_g = ggml_view_2d(ctx0, head_w, head_w->ne[0], head_w->ne[1], head_w->nb[1],
|
||||
(size_t) step_idx * head_w->nb[2]);
|
||||
ggml_tensor * logits = ggml_mul_mat(ctx0, head_g, cur);
|
||||
cb(logits, "step_logits", step_idx);
|
||||
|
||||
ggml_tensor * sampled = do_sampling(logits, inp_rand);
|
||||
cb(sampled, "step_sampled", step_idx);
|
||||
|
||||
return cache_set(out_code_cache, pos, sampled);
|
||||
}
|
||||
|
||||
// causal conv1d, stride 1: prepend persisted left-context instead of zero-padding, then a plain conv
|
||||
// x: [T, IC] (T-first). w: [K, IC, OC]. state_name empty means K == 1 (no left-context). returns [T, OC]
|
||||
ggml_tensor * clip_graph_qwen3tts_gen::code2wav::causal_conv1d(ggml_tensor * x, ggml_tensor * w, ggml_tensor * b, int dilation, const std::string & state_name) const {
|
||||
const int K = (int) w->ne[0];
|
||||
const int pad = (K - 1) * dilation;
|
||||
|
||||
ggml_tensor * x_full = x;
|
||||
if (pad > 0) {
|
||||
ggml_tensor * left = state_in.at(state_name); // [pad, IC]
|
||||
x_full = ggml_concat(ctx0, left, x, 0);
|
||||
}
|
||||
ggml_tensor * y = ggml_conv_1d(ctx0, w, x_full, 1, 0, dilation); // [T, OC, 1]
|
||||
y = ggml_reshape_2d(ctx0, y, y->ne[0], y->ne[1]);
|
||||
if (b) {
|
||||
y = ggml_add(ctx0, y, ggml_reshape_2d(ctx0, b, 1, b->ne[0]));
|
||||
}
|
||||
if (pad > 0) {
|
||||
ggml_tensor * new_left = ggml_cont(ctx0, ggml_view_2d(ctx0, x_full, pad, x_full->ne[1], x_full->nb[1],
|
||||
(size_t) (x_full->ne[0] - pad) * x_full->nb[0]));
|
||||
state_out.push_back({state_name, new_left});
|
||||
}
|
||||
return y;
|
||||
}
|
||||
|
||||
// causal depthwise conv1d, stride 1, dilation 1, kernel from w's shape.
|
||||
// x: [T, C]. w: [K, 1, C]. returns [T, C]. see causal_conv1d for the state contract.
|
||||
ggml_tensor * clip_graph_qwen3tts_gen::code2wav::causal_conv1d_dw(ggml_tensor * x, ggml_tensor * w, ggml_tensor * b, const std::string & state_name) const {
|
||||
const int K = (int) w->ne[0];
|
||||
const int pad = K - 1;
|
||||
|
||||
ggml_tensor * x_full = x;
|
||||
if (pad > 0) {
|
||||
ggml_tensor * left = state_in.at(state_name); // [pad, C]
|
||||
x_full = ggml_concat(ctx0, left, x, 0);
|
||||
}
|
||||
ggml_tensor * y = ggml_conv_1d_dw(ctx0, w, x_full, 1, 0, 1); // [T, C, 1]
|
||||
y = ggml_reshape_2d(ctx0, y, y->ne[0], y->ne[1]);
|
||||
if (b) {
|
||||
y = ggml_add(ctx0, y, ggml_reshape_2d(ctx0, b, 1, b->ne[0]));
|
||||
}
|
||||
if (pad > 0) {
|
||||
ggml_tensor * new_left = ggml_cont(ctx0, ggml_view_2d(ctx0, x_full, pad, x_full->ne[1], x_full->nb[1],
|
||||
(size_t) (x_full->ne[0] - pad) * x_full->nb[0]));
|
||||
state_out.push_back({state_name, new_left});
|
||||
}
|
||||
return y;
|
||||
}
|
||||
|
||||
// causal ConvTranspose1d, the (kernel - stride) overlap tail is kept as state for the next call
|
||||
// x: [T, IC], w: [K, OC, IC]. state_name empty means K == stride (no overlap). returns [T * stride, OC]
|
||||
ggml_tensor * clip_graph_qwen3tts_gen::code2wav::causal_conv_transpose1d(ggml_tensor * x, ggml_tensor * w, ggml_tensor * b, int stride, const std::string & state_name) const {
|
||||
const int K = (int) w->ne[0];
|
||||
const int OC = (int) w->ne[1];
|
||||
const int trim = K - stride;
|
||||
const int64_t emit_len = x->ne[0] * stride;
|
||||
|
||||
// transposed conv as GEMM + col2im scatter-add, y: [emit_len + trim, OC]
|
||||
ggml_tensor * w2 = ggml_reshape_2d(ctx0, w, (int64_t) K * OC, w->ne[2]);
|
||||
w2 = ggml_cont(ctx0, ggml_transpose(ctx0, w2));
|
||||
ggml_tensor * xt = ggml_cont(ctx0, ggml_transpose(ctx0, x));
|
||||
ggml_tensor * col = ggml_mul_mat(ctx0, w2, xt);
|
||||
ggml_tensor * y = ggml_col2im_1d(ctx0, col, stride, OC, 0);
|
||||
|
||||
ggml_tensor * out = y;
|
||||
if (trim > 0) {
|
||||
ggml_tensor * tail = state_in.at(state_name); // [trim, OC]
|
||||
ggml_tensor * head = ggml_add(ctx0, ggml_view_2d(ctx0, y, trim, y->ne[1], y->nb[1], 0), tail);
|
||||
if (emit_len > trim) {
|
||||
ggml_tensor * middle = ggml_view_2d(ctx0, y, emit_len - trim, y->ne[1], y->nb[1], (size_t) trim * y->nb[0]);
|
||||
out = ggml_concat(ctx0, head, middle, 0);
|
||||
} else {
|
||||
out = head;
|
||||
}
|
||||
ggml_tensor * new_tail = ggml_cont(ctx0, ggml_view_2d(ctx0, y, trim, y->ne[1], y->nb[1], (size_t) emit_len * y->nb[0]));
|
||||
state_out.push_back({state_name, new_tail});
|
||||
}
|
||||
if (b) {
|
||||
out = ggml_add(ctx0, out, ggml_reshape_2d(ctx0, b, 1, b->ne[0]));
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
// SnakeBeta activation: y = x + sin(alpha*x)^2 * inv_beta (alpha/inv_beta folded via exp/reciprocal at conversion time)
|
||||
// x: [T, C]. alpha/beta: [C], broadcasts over T
|
||||
ggml_tensor * clip_graph_qwen3tts_gen::code2wav::snake(ggml_tensor * x, ggml_tensor * alpha, ggml_tensor * beta) const {
|
||||
ggml_tensor * a = ggml_reshape_2d(ctx0, alpha, 1, alpha->ne[0]);
|
||||
ggml_tensor * b = ggml_reshape_2d(ctx0, beta, 1, beta->ne[0]);
|
||||
|
||||
// expand reshapes first so mul/sin/sqr/mul/add lands as consecutive nodes, letting backends fuse them
|
||||
ggml_build_forward_expand(gf, a);
|
||||
ggml_build_forward_expand(gf, b);
|
||||
|
||||
ggml_tensor * s = ggml_sin(ctx0, ggml_mul(ctx0, x, a));
|
||||
s = ggml_sqr(ctx0, s);
|
||||
s = ggml_mul(ctx0, s, b);
|
||||
return ggml_add(ctx0, x, s);
|
||||
}
|
||||
|
||||
// RVQ codebook decode: T frames of 16 codes -> 512-dim hidden (C-first, [512, T])
|
||||
// codebook 0 (semantic) and 1..15 (acoustic) sum within their group, project separately, then add
|
||||
ggml_tensor * clip_graph_qwen3tts_gen::code2wav::quant_decode(ggml_tensor * inp_codes) const {
|
||||
const auto & c2w = model.c2w;
|
||||
const int64_t T = inp_codes->ne[0];
|
||||
|
||||
// ids for codebook group g over all T frames, [T] I32
|
||||
auto group_ids = [&](int g) {
|
||||
return ggml_view_1d(ctx0, inp_codes, T, (size_t) g * inp_codes->nb[1]);
|
||||
};
|
||||
|
||||
ggml_tensor * sem = ggml_get_rows(ctx0, c2w.quant_first_cb_w, group_ids(0)); // [256, T]
|
||||
ggml_tensor * sem_out = ggml_mul_mat(ctx0, c2w.quant_first_out_w, sem); // [512, T]
|
||||
|
||||
ggml_tensor * acc = nullptr;
|
||||
const int64_t n_acoustic = c2w.quant_rest_cb_w->ne[2];
|
||||
for (int g = 1; g <= n_acoustic; g++) {
|
||||
ggml_tensor * cb_g = ggml_view_2d(ctx0, c2w.quant_rest_cb_w, c2w.quant_rest_cb_w->ne[0], c2w.quant_rest_cb_w->ne[1],
|
||||
c2w.quant_rest_cb_w->nb[1], (size_t) (g - 1) * c2w.quant_rest_cb_w->nb[2]);
|
||||
ggml_tensor * embd = ggml_get_rows(ctx0, cb_g, group_ids(g)); // [256, T]
|
||||
acc = acc ? ggml_add(ctx0, acc, embd) : embd;
|
||||
}
|
||||
ggml_tensor * ac_out = ggml_mul_mat(ctx0, c2w.quant_rest_out_w, acc); // [512, T]
|
||||
|
||||
ggml_tensor * hidden = ggml_add(ctx0, sem_out, ac_out);
|
||||
cb(hidden, "wav_quant_hidden", -1);
|
||||
return hidden;
|
||||
}
|
||||
|
||||
// one pre_transformer layer over a batch of N = sliding_window new frames
|
||||
// attention runs over [(W-1)-frame prefix from the last batch] + [N new frames]
|
||||
// RoPE positions come from a persisted counter, so phases line up across batches
|
||||
ggml_tensor * clip_graph_qwen3tts_gen::code2wav::tfm_layer_forward(ggml_tensor * cur, const clip_layer & layer, int il) const {
|
||||
const int n_head = hparams.wav_tfm_n_head;
|
||||
const int n_head_kv = hparams.wav_tfm_n_head_kv;
|
||||
const int64_t d_head = layer.q_w->ne[1] / n_head;
|
||||
const float kq_scale = 1.0f / sqrtf((float) d_head);
|
||||
const int64_t W = hparams.wav_tfm_swa; // == N, frames per batch
|
||||
const int64_t N = cur->ne[1];
|
||||
const int64_t prefix = W - 1;
|
||||
const int64_t total_kv = prefix + N;
|
||||
|
||||
ggml_tensor * residual = cur;
|
||||
ggml_tensor * h = ggml_rms_norm(ctx0, cur, hparams.wav_tfm_eps);
|
||||
h = ggml_mul(ctx0, h, layer.ln_1_w);
|
||||
|
||||
ggml_tensor * q = ggml_mul_mat(ctx0, layer.q_w, h); // [n_head*d_head, N]
|
||||
ggml_tensor * k = ggml_mul_mat(ctx0, layer.k_w, h); // [n_head_kv*d_head, N]
|
||||
ggml_tensor * v = ggml_mul_mat(ctx0, layer.v_w, h); // [n_head_kv*d_head, N]
|
||||
|
||||
q = ggml_reshape_3d(ctx0, q, d_head, n_head, N);
|
||||
k = ggml_reshape_3d(ctx0, k, d_head, n_head_kv, N);
|
||||
|
||||
// real, ever-increasing positions: base (persisted) .. base+N-1
|
||||
ggml_tensor * base = ggml_reshape_1d(ctx0, state_in.at("tfm_pos"), 1);
|
||||
ggml_tensor * offset = ggml_arange(ctx0, 0.0f, (float) N, 1.0f);
|
||||
ggml_tensor * pos = ggml_cast(ctx0, ggml_add(ctx0, offset, base), GGML_TYPE_I32);
|
||||
|
||||
q = ggml_rope_ext(ctx0, q, pos, nullptr, (int) d_head, GGML_ROPE_TYPE_NEOX, 0,
|
||||
hparams.wav_tfm_rope_theta, 1.0f, 0.0f, 1.0f, 0.0f, 0.0f);
|
||||
k = ggml_rope_ext(ctx0, k, pos, nullptr, (int) d_head, GGML_ROPE_TYPE_NEOX, 0,
|
||||
hparams.wav_tfm_rope_theta, 1.0f, 0.0f, 1.0f, 0.0f, 0.0f);
|
||||
|
||||
// the position counter is the same for all layers, push it once from layer 0
|
||||
if (il == 0) {
|
||||
state_out.push_back({"tfm_pos", ggml_scale_bias(ctx0, state_in.at("tfm_pos"), 1.0f, (float) N)});
|
||||
}
|
||||
|
||||
ggml_tensor * k_new = ggml_reshape_2d(ctx0, k, d_head * n_head_kv, N);
|
||||
ggml_tensor * v_new = ggml_reshape_2d(ctx0, v, d_head * n_head_kv, N);
|
||||
|
||||
ggml_tensor * old_k = state_in.at("tfm_k_" + std::to_string(il)); // [d_head*n_head_kv, W-1]
|
||||
ggml_tensor * old_v = state_in.at("tfm_v_" + std::to_string(il));
|
||||
|
||||
ggml_tensor * k_full = ggml_concat(ctx0, old_k, k_new, 1); // [.., prefix+N]
|
||||
ggml_tensor * v_full = ggml_concat(ctx0, old_v, v_new, 1);
|
||||
|
||||
// next batch's prefix: the last (W-1) frames of this batch
|
||||
state_out.push_back({"tfm_k_" + std::to_string(il),
|
||||
ggml_cont(ctx0, ggml_view_2d(ctx0, k_full, k_full->ne[0], prefix, k_full->nb[1], (size_t) N * k_full->nb[1]))});
|
||||
state_out.push_back({"tfm_v_" + std::to_string(il),
|
||||
ggml_cont(ctx0, ggml_view_2d(ctx0, v_full, v_full->ne[0], prefix, v_full->nb[1], (size_t) N * v_full->nb[1]))});
|
||||
|
||||
// banded causal mask: key j is visible to query i iff 0 <= (prefix+i) - j < W
|
||||
ggml_tensor * pos_k = ggml_reshape_2d(ctx0, ggml_arange(ctx0, 0.0f, (float) total_kv, 1.0f), total_kv, 1);
|
||||
ggml_tensor * pos_q = ggml_reshape_2d(ctx0, ggml_arange(ctx0, (float) prefix, (float) (prefix + N), 1.0f), 1, N);
|
||||
ggml_tensor * pos_q_grid = ggml_repeat_4d(ctx0, pos_q, total_kv, N, 1, 1);
|
||||
ggml_tensor * diff = ggml_sub(ctx0, pos_q_grid, pos_k); // [total_kv, N]
|
||||
|
||||
ggml_tensor * causal_keep = ggml_step(ctx0, ggml_scale_bias(ctx0, diff, 1.0f, 0.5f)); // diff >= 0
|
||||
ggml_tensor * in_window = ggml_step(ctx0, ggml_scale_bias(ctx0, diff, -1.0f, (float) W - 0.5f)); // diff < W
|
||||
ggml_tensor * keep = ggml_mul(ctx0, causal_keep, in_window);
|
||||
|
||||
// on a cold start, key j is real state only when j >= prefix - tfm_pos, mask out the rest
|
||||
ggml_tensor * warm = ggml_step(ctx0, ggml_scale_bias(ctx0, ggml_add(ctx0, pos_k, base),
|
||||
1.0f, 0.5f - (float) prefix)); // j + pos > prefix - 0.5
|
||||
keep = ggml_mul(ctx0, keep, warm);
|
||||
|
||||
ggml_tensor * mask = ggml_reshape_4d(ctx0, ggml_log(ctx0, keep), total_kv, N, 1, 1); // 0 = keep, -inf = masked
|
||||
|
||||
ggml_tensor * q_cur = ggml_reshape_4d(ctx0, q, d_head, n_head, N, 1);
|
||||
ggml_tensor * k_cur = ggml_reshape_4d(ctx0, k_full, d_head, n_head_kv, total_kv, 1);
|
||||
ggml_tensor * v_cur = ggml_reshape_4d(ctx0, v_full, d_head, n_head_kv, total_kv, 1);
|
||||
|
||||
ggml_tensor * attn_out = build_attn(layer.o_w, layer.o_b, q_cur, k_cur, v_cur, mask, kq_scale, il);
|
||||
if (layer.ls_1_w) {
|
||||
attn_out = ggml_mul(ctx0, attn_out, layer.ls_1_w);
|
||||
}
|
||||
cur = ggml_add(ctx0, residual, attn_out);
|
||||
|
||||
ggml_tensor * residual2 = cur;
|
||||
ggml_tensor * h2 = ggml_rms_norm(ctx0, cur, hparams.wav_tfm_eps);
|
||||
h2 = ggml_mul(ctx0, h2, layer.ln_2_w);
|
||||
|
||||
ggml_tensor * gate = ggml_mul_mat(ctx0, layer.ff_gate_w, h2);
|
||||
ggml_tensor * up = ggml_mul_mat(ctx0, layer.ff_up_w, h2);
|
||||
ggml_tensor * gu = ggml_swiglu_split(ctx0, gate, up);
|
||||
ggml_tensor * down = ggml_mul_mat(ctx0, layer.ff_down_w, gu);
|
||||
if (layer.ls_2_w) {
|
||||
down = ggml_mul(ctx0, down, layer.ls_2_w);
|
||||
}
|
||||
return ggml_add(ctx0, residual2, down);
|
||||
}
|
||||
|
||||
// dwconv -> LayerNorm -> pwconv1 -> GELU -> pwconv2 -> layer scale -> residual
|
||||
// x: [T, C] T-first; LayerNorm/pwconv need C on ne0, so this transposes in and back out
|
||||
ggml_tensor * clip_graph_qwen3tts_gen::code2wav::convnext_block(ggml_tensor * x, const clip_code2wav::upsample_block & blk, const std::string & state_prefix) const {
|
||||
ggml_tensor * residual = x;
|
||||
|
||||
ggml_tensor * h = causal_conv1d_dw(x, blk.dwconv_w, blk.dwconv_b, state_prefix + "_dwconv"); // [T, C]
|
||||
ggml_tensor * hc = ggml_cont(ctx0, ggml_transpose(ctx0, h)); // [C, T]
|
||||
|
||||
hc = ggml_norm(ctx0, hc, 1e-6f);
|
||||
hc = ggml_mul(ctx0, hc, blk.norm_w);
|
||||
hc = ggml_add(ctx0, hc, blk.norm_b);
|
||||
|
||||
ggml_tensor * g = ggml_mul_mat(ctx0, blk.pw1_w, hc);
|
||||
g = ggml_add(ctx0, g, blk.pw1_b);
|
||||
g = ggml_gelu(ctx0, g);
|
||||
g = ggml_mul_mat(ctx0, blk.pw2_w, g);
|
||||
g = ggml_add(ctx0, g, blk.pw2_b);
|
||||
g = ggml_mul(ctx0, g, blk.gamma);
|
||||
|
||||
ggml_tensor * g_t = ggml_cont(ctx0, ggml_transpose(ctx0, g)); // back to [T, C]
|
||||
return ggml_add(ctx0, residual, g_t);
|
||||
}
|
||||
|
||||
// SnakeBeta -> dilated causal conv (k=7) -> SnakeBeta -> pointwise causal conv (k=1) -> residual.
|
||||
// x: [T, C]. returns [T, C].
|
||||
ggml_tensor * clip_graph_qwen3tts_gen::code2wav::dac_res_unit(ggml_tensor * x, const clip_code2wav::dac_res & res, int dilation, const std::string & state_name) const {
|
||||
ggml_tensor * residual = x;
|
||||
ggml_tensor * h = snake(x, res.act1_alpha, res.act1_beta);
|
||||
h = causal_conv1d(h, res.conv1_w, res.conv1_b, dilation, state_name);
|
||||
h = snake(h, res.act2_alpha, res.act2_beta);
|
||||
h = causal_conv1d(h, res.conv2_w, res.conv2_b, 1, ""); // k=1, no left-context needed
|
||||
return ggml_add(ctx0, residual, h);
|
||||
}
|
||||
|
||||
// RVQ codes -> raw PCM for a batch of N = sliding_window frames
|
||||
ggml_tensor * clip_graph_qwen3tts_gen::code2wav::decode(ggml_tensor * inp_codes) const {
|
||||
const auto & c2w = model.c2w;
|
||||
|
||||
// 1. quantizer decode: N frames of 16 codes -> [512, N] (C-first)
|
||||
ggml_tensor * hidden = quant_decode(inp_codes);
|
||||
|
||||
// 2. pre_conv: [512, N] -> T-first [N, 512] -> causal conv k=3 -> [N, 1024]
|
||||
ggml_tensor * x = ggml_cont(ctx0, ggml_transpose(ctx0, hidden)); // [N, 512]
|
||||
x = causal_conv1d(x, c2w.pre_conv_w, c2w.pre_conv_b, 1, "pre_conv"); // [N, 1024]
|
||||
cb(x, "wav_pre_conv_out", -1);
|
||||
|
||||
// 3. pre_transformer: back to C-first [1024, N], project down, run the layers, project back up
|
||||
ggml_tensor * cur = ggml_cont(ctx0, ggml_transpose(ctx0, x)); // [1024, N]
|
||||
cur = ggml_mul_mat(ctx0, c2w.tfm_in_proj_w, cur);
|
||||
cur = ggml_add(ctx0, cur, c2w.tfm_in_proj_b); // [512 (tfm hidden), N]
|
||||
|
||||
for (int il = 0; il < hparams.wav_tfm_n_layer; il++) {
|
||||
cur = tfm_layer_forward(cur, c2w.tfm_layers[il], il);
|
||||
}
|
||||
|
||||
cur = ggml_rms_norm(ctx0, cur, hparams.wav_tfm_eps);
|
||||
cur = ggml_mul(ctx0, cur, c2w.tfm_output_norm_w);
|
||||
cur = ggml_mul_mat(ctx0, c2w.tfm_out_proj_w, cur);
|
||||
cur = ggml_add(ctx0, cur, c2w.tfm_out_proj_b); // [1024, N]
|
||||
cb(cur, "wav_tfm_out", -1);
|
||||
|
||||
// 4. upsample: 2x (causal ConvTranspose1d, stride 2 + ConvNeXt block), back to T-first
|
||||
// kernel == stride here, so there is no overlap tail to persist
|
||||
x = ggml_cont(ctx0, ggml_transpose(ctx0, cur)); // [N, 1024]
|
||||
for (size_t il = 0; il < c2w.upsample.size(); il++) {
|
||||
const auto & up = c2w.upsample[il];
|
||||
x = causal_conv_transpose1d(x, up.conv_w, up.conv_b, 2, "");
|
||||
x = convnext_block(x, up, "up" + std::to_string(il));
|
||||
cb(x, "wav_upsample_out", (int) il);
|
||||
}
|
||||
|
||||
// 5. DAC decoder: conv_pre -> n blocks (SnakeBeta -> ConvTranspose1d -> 3 res units) -> conv_post
|
||||
static constexpr int DAC_DILATIONS[3] = { 1, 3, 9 };
|
||||
|
||||
x = causal_conv1d(x, c2w.dac_entry_w, c2w.dac_entry_b, 1, "dac_entry");
|
||||
cb(x, "wav_dac_entry_out", -1);
|
||||
|
||||
for (size_t il = 0; il < c2w.dac.size(); il++) {
|
||||
const auto & blk = c2w.dac[il];
|
||||
const int stride = (int) (blk.conv_w->ne[0] / 2); // kernel == 2*stride for all 4 blocks
|
||||
const std::string blk_name = "dac" + std::to_string(il);
|
||||
x = snake(x, blk.snake_alpha, blk.snake_beta);
|
||||
x = causal_conv_transpose1d(x, blk.conv_w, blk.conv_b, stride, blk_name + "_tail");
|
||||
for (size_t ir = 0; ir < blk.res.size(); ir++) {
|
||||
x = dac_res_unit(x, blk.res[ir], DAC_DILATIONS[ir], blk_name + "_res" + std::to_string(ir));
|
||||
}
|
||||
cb(x, "wav_dac_block_out", (int) il);
|
||||
}
|
||||
|
||||
x = snake(x, c2w.dac_post_snake_alpha, c2w.dac_post_snake_beta);
|
||||
x = causal_conv1d(x, c2w.dac_post_conv_w, c2w.dac_post_conv_b, 1, "dac_post_conv"); // [n_samples, 1]
|
||||
|
||||
x = ggml_clamp(ctx0, x, -1.0f, 1.0f);
|
||||
x = ggml_reshape_1d(ctx0, x, x->ne[0]);
|
||||
cb(x, "wav_audio_out", -1);
|
||||
return x;
|
||||
}
|
||||
|
||||
// code2wav's persisted state buffers: RoPE position counter, K/V per pre_transformer layer,
|
||||
// left-context/tail per stateful conv. shape lookup only, no graph needed
|
||||
std::vector<c2w_state_slot> list_c2w_state_slots(const clip_hparams & hparams, const clip_model & model) {
|
||||
const auto & c2w = model.c2w;
|
||||
std::vector<c2w_state_slot> slots;
|
||||
|
||||
slots.push_back({"tfm_pos", 1, 1});
|
||||
|
||||
// prefix is (W-1) frames, the batch itself gives the other N=W frames (see tfm_layer_forward)
|
||||
const int64_t d_head = c2w.tfm_layers[0].q_w->ne[1] / hparams.wav_tfm_n_head;
|
||||
const int64_t kv_ch = d_head * hparams.wav_tfm_n_head_kv;
|
||||
const int64_t prefix = hparams.wav_tfm_swa - 1;
|
||||
for (int il = 0; il < hparams.wav_tfm_n_layer; il++) {
|
||||
slots.push_back({"tfm_k_" + std::to_string(il), kv_ch, prefix});
|
||||
slots.push_back({"tfm_v_" + std::to_string(il), kv_ch, prefix});
|
||||
}
|
||||
|
||||
slots.push_back({"pre_conv", c2w.pre_conv_w->ne[0] - 1, c2w.pre_conv_w->ne[1]});
|
||||
|
||||
for (size_t il = 0; il < c2w.upsample.size(); il++) {
|
||||
const auto & up = c2w.upsample[il];
|
||||
slots.push_back({"up" + std::to_string(il) + "_dwconv", up.dwconv_w->ne[0] - 1, up.dwconv_w->ne[2]});
|
||||
}
|
||||
|
||||
slots.push_back({"dac_entry", c2w.dac_entry_w->ne[0] - 1, c2w.dac_entry_w->ne[1]});
|
||||
|
||||
static constexpr int DAC_DILATIONS[3] = { 1, 3, 9 };
|
||||
for (size_t il = 0; il < c2w.dac.size(); il++) {
|
||||
const auto & blk = c2w.dac[il];
|
||||
const int64_t stride = blk.conv_w->ne[0] / 2; // kernel == 2*stride for all 4 blocks
|
||||
const std::string blk_name = "dac" + std::to_string(il);
|
||||
slots.push_back({blk_name + "_tail", stride, blk.conv_w->ne[1]});
|
||||
for (size_t ir = 0; ir < blk.res.size(); ir++) {
|
||||
const auto & res = blk.res[ir];
|
||||
slots.push_back({blk_name + "_res" + std::to_string(ir),
|
||||
(res.conv1_w->ne[0] - 1) * DAC_DILATIONS[ir], res.conv1_w->ne[1]});
|
||||
}
|
||||
}
|
||||
|
||||
slots.push_back({"dac_post_conv", c2w.dac_post_conv_w->ne[0] - 1, c2w.dac_post_conv_w->ne[1]});
|
||||
|
||||
return slots;
|
||||
}
|
||||
|
||||
// both sub-graphs are always built, so the topology stays constant
|
||||
// ggml_build_forward_select() then picks the one that actually runs
|
||||
ggml_cgraph * clip_graph_qwen3tts_gen::build() {
|
||||
GGML_ASSERT(n_batch == 1); // this module only ever processes one frame at a time
|
||||
|
||||
int idx;
|
||||
switch (gen_process) {
|
||||
case CLIP_GEN_PROCESS_GEN_CODE: idx = 0; break;
|
||||
case CLIP_GEN_PROCESS_GEN_WAV: idx = 1; break;
|
||||
default: GGML_ABORT("unknown gen_process");
|
||||
}
|
||||
|
||||
// ---- CLIP_GEN_PROCESS_GEN_CODE: backbone hidden state -> 16 RVQ codes + next-step embd ----
|
||||
// not build_inp_raw(), a GEN_WAV call's `img` has no hidden-state data
|
||||
ggml_tensor * h_state = ggml_new_tensor_1d(ctx0, GGML_TYPE_F32, n_mmproj_embd);
|
||||
ggml_set_name(h_state, "inp_raw"); // must keep this exact name, clip_encode() sets it by name
|
||||
ggml_set_input(h_state);
|
||||
|
||||
ggml_tensor * code0 = ggml_new_tensor_1d(ctx0, GGML_TYPE_I32, 1);
|
||||
ggml_set_name(code0, "inp_code0");
|
||||
ggml_set_input(code0);
|
||||
|
||||
ggml_tensor * code0_embd = ggml_get_rows(ctx0, model.gen_code_out_embd_w, code0);
|
||||
code0_embd = ggml_reshape_1d(ctx0, code0_embd, code0_embd->ne[0]);
|
||||
cb(code0_embd, "code0_embd", -1);
|
||||
|
||||
const int64_t n_acoustic = model.gen_code_head_w->ne[2]; // 15
|
||||
const int n_codes = (int) n_acoustic + 1; // 16
|
||||
const int64_t n_kv_pad = n_codes;
|
||||
const int n_layer = (int) model.layers.size();
|
||||
const int n_head = hparams.n_head;
|
||||
const int n_head_kv = hparams.n_head_kv;
|
||||
const int64_t d_head = model.layers[0].q_w->ne[1] / n_head;
|
||||
|
||||
// zero-filled per layer k/v caches, so masked-out rows can't hold garbage
|
||||
std::vector<ggml_tensor *> k_cache(n_layer), v_cache(n_layer);
|
||||
for (int il = 0; il < n_layer; il++) {
|
||||
k_cache[il] = ggml_fill(ctx0, ggml_new_tensor_2d(ctx0, GGML_TYPE_F32, d_head * n_head_kv, n_kv_pad), 0.0f);
|
||||
v_cache[il] = ggml_fill(ctx0, ggml_new_tensor_2d(ctx0, GGML_TYPE_F32, d_head * n_head_kv, n_kv_pad), 0.0f);
|
||||
}
|
||||
|
||||
code_gen cg(*this, top_k, top_p);
|
||||
|
||||
ggml_tensor * out_code_cache = ggml_new_tensor_2d(ctx0, GGML_TYPE_I32, 1, n_codes);
|
||||
out_code_cache = cg.cache_set(out_code_cache, 0, code0);
|
||||
|
||||
ggml_tensor * inp_rand0 = ggml_new_tensor_1d(ctx0, GGML_TYPE_F32, 1);
|
||||
ggml_set_name(inp_rand0, "inp_rand_0");
|
||||
ggml_set_input(inp_rand0);
|
||||
|
||||
cg.prefill(k_cache, v_cache, out_code_cache, h_state, code0_embd, inp_rand0);
|
||||
|
||||
for (int g = 1; g < n_acoustic; g++) {
|
||||
ggml_tensor * inp_rand = ggml_new_tensor_1d(ctx0, GGML_TYPE_F32, 1);
|
||||
ggml_set_name(inp_rand, ("inp_rand_" + std::to_string(g)).c_str());
|
||||
ggml_set_input(inp_rand);
|
||||
out_code_cache = cg.step(k_cache, v_cache, out_code_cache, inp_rand, g);
|
||||
}
|
||||
|
||||
// output 1: this frame's 16 sampled codes, for the caller's code2wav window
|
||||
ggml_tensor * out_codes = ggml_cont(ctx0, out_code_cache);
|
||||
ggml_set_name(out_codes, "out_codes");
|
||||
ggml_set_output(out_codes);
|
||||
|
||||
// output 2: sum of all 16 codebook embeddings, fed back to the talker for the next frame
|
||||
ggml_tensor * out_embd = code0_embd;
|
||||
for (int g = 1; g <= n_acoustic; g++) {
|
||||
ggml_tensor * code_g = ggml_view_1d(ctx0, out_code_cache, 1, (size_t) g * out_code_cache->nb[1]);
|
||||
|
||||
ggml_tensor * embd_g = ggml_view_2d(ctx0, model.gen_code_embd_w, model.gen_code_embd_w->ne[0], model.gen_code_embd_w->ne[1],
|
||||
model.gen_code_embd_w->nb[1], (size_t) (g - 1) * model.gen_code_embd_w->nb[2]);
|
||||
ggml_tensor * e = ggml_get_rows(ctx0, embd_g, code_g);
|
||||
e = ggml_reshape_1d(ctx0, e, e->ne[0]);
|
||||
|
||||
out_embd = ggml_add(ctx0, out_embd, e);
|
||||
}
|
||||
out_embd = ggml_reshape_2d(ctx0, out_embd, out_embd->ne[0], 1);
|
||||
cb(out_embd, "gen_audio_out", -1);
|
||||
|
||||
// ---- CLIP_GEN_PROCESS_GEN_WAV: 16 RVQ codes -> raw PCM ----
|
||||
const int n_frames = hparams.wav_tfm_swa; // frames per batch, == the attention window
|
||||
|
||||
ggml_tensor * inp_codes = ggml_new_tensor_2d(ctx0, GGML_TYPE_I32, n_frames, n_codes);
|
||||
ggml_set_name(inp_codes, "inp_codes");
|
||||
ggml_set_input(inp_codes);
|
||||
|
||||
code2wav c2w(*this);
|
||||
for (const auto & slot : list_c2w_state_slots(hparams, model)) {
|
||||
ggml_tensor * t = ggml_new_tensor_2d(ctx0, GGML_TYPE_F32, slot.ne0, slot.ne1);
|
||||
ggml_set_name(t, ("state_in_" + slot.name).c_str());
|
||||
ggml_set_input(t);
|
||||
c2w.state_in[slot.name] = t;
|
||||
}
|
||||
|
||||
ggml_tensor * out_audio = c2w.decode(inp_codes);
|
||||
ggml_set_name(out_audio, "out_audio");
|
||||
ggml_set_output(out_audio);
|
||||
|
||||
for (auto & slot : c2w.state_out) {
|
||||
ggml_set_name(slot.second, ("state_out_" + slot.first).c_str());
|
||||
ggml_set_output(slot.second);
|
||||
}
|
||||
|
||||
// out_embd goes last, clip_encode() reads it back via ggml_graph_node(gf, -1)
|
||||
ggml_tensor * outs[2];
|
||||
outs[0] = out_codes; outs[1] = out_audio;
|
||||
ggml_build_forward_select(gf, outs, 2, idx);
|
||||
for (auto & slot : c2w.state_out) {
|
||||
outs[0] = out_codes; outs[1] = slot.second;
|
||||
ggml_build_forward_select(gf, outs, 2, idx);
|
||||
}
|
||||
outs[0] = out_embd; outs[1] = out_audio;
|
||||
ggml_build_forward_select(gf, outs, 2, idx);
|
||||
|
||||
return gf;
|
||||
}
|
||||
@@ -0,0 +1,197 @@
|
||||
#include "models.h"
|
||||
|
||||
static constexpr int SPK_RES2NET_SCALE = 8; // enc_res2net_scale
|
||||
static constexpr int SPK_DILATIONS[3] = { 2, 3, 4 }; // enc_dilations[1..3]
|
||||
|
||||
// conv1d, kernel K, padding "same" (reflect), dilation d
|
||||
// x: [C, T] (ne[0]=C, ne[1]=T) -> [out_c, T]
|
||||
ggml_tensor * clip_graph_qwen3tts_spkenc::conv1d_same(ggml_tensor * x, ggml_tensor * w, ggml_tensor * b, int dilation) const {
|
||||
const int K = (int) w->ne[0];
|
||||
const int IC = (int) w->ne[1];
|
||||
const int OC = (int) w->ne[2];
|
||||
const int pad = ((K - 1) * dilation) / 2;
|
||||
|
||||
// ggml_pad_reflect_1d pads ne[0], so bring T onto ne[0] first, same layout as im2col wants
|
||||
ggml_tensor * x_t = ggml_cont(ctx0, ggml_transpose(ctx0, x)); // [T, IC]
|
||||
if (pad > 0) {
|
||||
x_t = ggml_pad_reflect_1d(ctx0, x_t, pad, pad); // [T + 2*pad, IC]
|
||||
}
|
||||
ggml_tensor * x4d = ggml_reshape_4d(ctx0, x_t, x_t->ne[0], IC, 1, 1);
|
||||
|
||||
// dummy F32 kernel, im2col only reads its shape, so a quantized w does not assert
|
||||
ggml_tensor * dummy = ggml_new_tensor_4d(ctx0, GGML_TYPE_F32, K, IC, 1, 1);
|
||||
|
||||
ggml_tensor * col = ggml_im2col(ctx0, dummy, x4d, 1, 1, 0, 0, dilation, 1, false, GGML_TYPE_F32);
|
||||
const int64_t T_out = col->ne[1];
|
||||
col = ggml_reshape_2d(ctx0, col, (int64_t) K * IC, T_out);
|
||||
|
||||
ggml_tensor * w2d = ggml_reshape_2d(ctx0, w, (int64_t) K * IC, OC);
|
||||
ggml_tensor * y = ggml_mul_mat(ctx0, w2d, col); // [OC, T_out]
|
||||
ggml_mul_mat_set_prec(y, GGML_PREC_F32);
|
||||
|
||||
ggml_tensor * b2d = ggml_reshape_2d(ctx0, b, OC, 1);
|
||||
y = ggml_add(ctx0, y, b2d);
|
||||
return y;
|
||||
}
|
||||
|
||||
// Res2Net: split channel axis into `scale` chunks, chain dilated conv1d branches
|
||||
// x: [C, T] -> [C, T]
|
||||
ggml_tensor * clip_graph_qwen3tts_spkenc::res2net(ggml_tensor * x, const clip_layer & layer, int dilation, int scale) const {
|
||||
const int64_t C = x->ne[0];
|
||||
const int64_t T = x->ne[1];
|
||||
const int64_t Cs = C / scale;
|
||||
|
||||
std::vector<ggml_tensor *> outs;
|
||||
outs.reserve(scale);
|
||||
|
||||
auto chunk = [&](int i) -> ggml_tensor * {
|
||||
return ggml_view_2d(ctx0, x, Cs, T, x->nb[1], (size_t) i * Cs * x->nb[0]);
|
||||
};
|
||||
|
||||
ggml_tensor * prev = nullptr;
|
||||
for (int i = 0; i < scale; i++) {
|
||||
ggml_tensor * c = ggml_cont(ctx0, chunk(i));
|
||||
if (i == 0) {
|
||||
outs.push_back(c);
|
||||
continue;
|
||||
}
|
||||
ggml_tensor * inp = (i >= 2) ? ggml_add(ctx0, c, prev) : c;
|
||||
ggml_tensor * y = conv1d_same(inp, layer.res2_conv_w[i - 1], layer.res2_conv_b[i - 1], dilation);
|
||||
y = ggml_relu(ctx0, y);
|
||||
outs.push_back(y);
|
||||
prev = y;
|
||||
}
|
||||
|
||||
ggml_tensor * acc = outs[0];
|
||||
for (int i = 1; i < scale; i++) {
|
||||
acc = ggml_concat(ctx0, acc, outs[i], 0);
|
||||
}
|
||||
return acc;
|
||||
}
|
||||
|
||||
// squeeze-and-excitation gate. x: [C, T] -> [C, T]
|
||||
ggml_tensor * clip_graph_qwen3tts_spkenc::se_block(ggml_tensor * x, const clip_layer & layer) const {
|
||||
// temporal mean, keepdim: transpose so T is on ne[0], reduce, transpose back
|
||||
ggml_tensor * x_t = ggml_cont(ctx0, ggml_transpose(ctx0, x)); // [T, C]
|
||||
ggml_tensor * mean = ggml_mean(ctx0, x_t); // [1, C]
|
||||
mean = ggml_cont(ctx0, ggml_transpose(ctx0, mean)); // [C, 1]
|
||||
|
||||
ggml_tensor * h = conv1d_same(mean, layer.se_conv1_w, layer.se_conv1_b, 1);
|
||||
h = ggml_relu(ctx0, h);
|
||||
h = conv1d_same(h, layer.se_conv2_w, layer.se_conv2_b, 1);
|
||||
h = ggml_sigmoid(ctx0, h); // [C, 1]
|
||||
|
||||
return ggml_mul(ctx0, x, h); // broadcast gate over T
|
||||
}
|
||||
|
||||
// tdnn1 -> res2net -> tdnn2 -> se, plus residual. x: [C, T] -> [C, T]
|
||||
ggml_tensor * clip_graph_qwen3tts_spkenc::se_res2net_block(ggml_tensor * x, const clip_layer & layer, int dilation, int scale) const {
|
||||
ggml_tensor * residual = x;
|
||||
ggml_tensor * h = conv1d_same(x, layer.conv_pw1_w, layer.conv_pw1_b, 1); // tdnn1
|
||||
h = ggml_relu(ctx0, h);
|
||||
h = res2net(h, layer, dilation, scale);
|
||||
h = conv1d_same(h, layer.conv_pw2_w, layer.conv_pw2_b, 1); // tdnn2
|
||||
h = ggml_relu(ctx0, h);
|
||||
h = se_block(h, layer);
|
||||
return ggml_add(ctx0, h, residual);
|
||||
}
|
||||
|
||||
// attentive statistics pooling. x: [C, T] -> [2*C, 1]
|
||||
ggml_tensor * clip_graph_qwen3tts_spkenc::attentive_stats_pool(ggml_tensor * x) const {
|
||||
const int64_t T = x->ne[1];
|
||||
|
||||
// mean over T: [C, 1]
|
||||
ggml_tensor * x_t = ggml_cont(ctx0, ggml_transpose(ctx0, x));
|
||||
ggml_tensor * mean = ggml_mean(ctx0, x_t);
|
||||
mean = ggml_cont(ctx0, ggml_transpose(ctx0, mean));
|
||||
|
||||
// std over T: sqrt(clamp(mean((x - mean)^2), eps))
|
||||
ggml_tensor * mean_rep = ggml_repeat(ctx0, mean, x);
|
||||
ggml_tensor * centered = ggml_sub(ctx0, x, mean_rep);
|
||||
ggml_tensor * var_t = ggml_cont(ctx0, ggml_transpose(ctx0, ggml_sqr(ctx0, centered)));
|
||||
ggml_tensor * var = ggml_mean(ctx0, var_t);
|
||||
var = ggml_cont(ctx0, ggml_transpose(ctx0, var));
|
||||
var = ggml_scale_bias(ctx0, var, 1.0f, 1e-12f);
|
||||
ggml_tensor * std = ggml_sqrt(ctx0, var);
|
||||
|
||||
// attention input: cat([x, mean, std]) along channel axis -> [3C, T]
|
||||
ggml_tensor * std_rep = ggml_repeat(ctx0, std, x);
|
||||
ggml_tensor * cat = ggml_concat(ctx0, x, mean_rep, 0);
|
||||
cat = ggml_concat(ctx0, cat, std_rep, 0);
|
||||
|
||||
// attention TDNN (3C -> attn_c) + ReLU, tanh, then 1x1 conv (attn_c -> C)
|
||||
ggml_tensor * a = conv1d_same(cat, model.spk_asp_tdnn_w, model.spk_asp_tdnn_b, 1);
|
||||
a = ggml_relu(ctx0, a);
|
||||
a = ggml_tanh(ctx0, a);
|
||||
a = conv1d_same(a, model.spk_asp_attn_w, model.spk_asp_attn_b, 1);
|
||||
|
||||
// softmax over T
|
||||
ggml_tensor * a_t = ggml_cont(ctx0, ggml_transpose(ctx0, a)); // [T, C]
|
||||
ggml_tensor * w_t = ggml_soft_max(ctx0, a_t);
|
||||
ggml_tensor * w = ggml_cont(ctx0, ggml_transpose(ctx0, w_t)); // [C, T]
|
||||
|
||||
// weighted mean: sum(w * x) over T, multiply by T to undo ggml_mean's 1/T scaling
|
||||
ggml_tensor * wx = ggml_mul(ctx0, w, x);
|
||||
ggml_tensor * wx_t = ggml_cont(ctx0, ggml_transpose(ctx0, wx));
|
||||
ggml_tensor * w_mean = ggml_mean(ctx0, wx_t);
|
||||
w_mean = ggml_scale(ctx0, w_mean, (float) T);
|
||||
w_mean = ggml_cont(ctx0, ggml_transpose(ctx0, w_mean)); // [C, 1]
|
||||
|
||||
// weighted std: sum(w * (x - w_mean)^2) over T
|
||||
ggml_tensor * w_mean_rep = ggml_repeat(ctx0, w_mean, x);
|
||||
ggml_tensor * dev = ggml_sub(ctx0, x, w_mean_rep);
|
||||
ggml_tensor * w_var_in = ggml_mul(ctx0, w, ggml_sqr(ctx0, dev));
|
||||
ggml_tensor * w_var_t = ggml_cont(ctx0, ggml_transpose(ctx0, w_var_in));
|
||||
ggml_tensor * w_var = ggml_mean(ctx0, w_var_t);
|
||||
w_var = ggml_scale(ctx0, w_var, (float) T);
|
||||
w_var = ggml_cont(ctx0, ggml_transpose(ctx0, w_var));
|
||||
w_var = ggml_scale_bias(ctx0, w_var, 1.0f, 1e-12f);
|
||||
ggml_tensor * w_std = ggml_sqrt(ctx0, w_var);
|
||||
|
||||
return ggml_concat(ctx0, w_mean, w_std, 0); // [2C, 1]
|
||||
}
|
||||
|
||||
ggml_cgraph * clip_graph_qwen3tts_spkenc::build() {
|
||||
// inp_raw: [T, n_mel, 1, 1], from mtmd_audio_preprocessor_qwen3tts_spk
|
||||
ggml_tensor * inp = build_inp_raw(1);
|
||||
inp = ggml_reshape_2d(ctx0, inp, inp->ne[0], inp->ne[1]);
|
||||
|
||||
// this file's convention is [C, T]; the preprocessor delivers [T, C]
|
||||
ggml_tensor * mel = ggml_cont(ctx0, ggml_transpose(ctx0, inp)); // [n_mel, T]
|
||||
cb(mel, "mel", -1);
|
||||
|
||||
// frontend conv0 TDNN k=5, dilation=1: 128 -> 512
|
||||
ggml_tensor * cur = conv1d_same(mel, model.conv1d_1_w, model.conv1d_1_b, 1);
|
||||
cur = ggml_relu(ctx0, cur);
|
||||
cb(cur, "frontend", -1);
|
||||
|
||||
// 3 SE-Res2Net blocks at dilations 2, 3, 4
|
||||
GGML_ASSERT((int) model.layers.size() == 3);
|
||||
std::vector<ggml_tensor *> blk_out(3);
|
||||
for (int il = 0; il < 3; il++) {
|
||||
cur = se_res2net_block(cur, model.layers[il], SPK_DILATIONS[il], SPK_RES2NET_SCALE);
|
||||
blk_out[il] = cur;
|
||||
cb(cur, "block_out", il);
|
||||
}
|
||||
|
||||
// multi-layer feature aggregation: cat blk[0..2] then TDNN k=1 + ReLU
|
||||
ggml_tensor * cat = ggml_concat(ctx0, blk_out[0], blk_out[1], 0);
|
||||
cat = ggml_concat(ctx0, cat, blk_out[2], 0); // [1536, T]
|
||||
ggml_tensor * mfa = conv1d_same(cat, model.conv_out_w, model.conv_out_b, 1);
|
||||
mfa = ggml_relu(ctx0, mfa);
|
||||
cb(mfa, "mfa", -1);
|
||||
|
||||
// attentive statistics pooling: [1536, T] -> [3072, 1]
|
||||
ggml_tensor * stats = attentive_stats_pool(mfa);
|
||||
cb(stats, "asp", -1);
|
||||
|
||||
// final FC k=1: [3072, 1] -> [enc_dim, 1]
|
||||
ggml_tensor * emb = conv1d_same(stats, model.mm_fc_w, model.mm_fc_b, 1);
|
||||
|
||||
emb = ggml_reshape_1d(ctx0, emb, emb->ne[0]);
|
||||
emb = ggml_cont(ctx0, emb);
|
||||
cb(emb, "spk_embedding", -1);
|
||||
|
||||
ggml_build_forward_expand(gf, emb);
|
||||
return gf;
|
||||
}
|
||||
@@ -791,6 +791,66 @@ bool mtmd_audio_preprocessor_mimo_audio::preprocess(const float *
|
||||
return true;
|
||||
}
|
||||
|
||||
//
|
||||
// mtmd_audio_preprocessor_qwen3tts_spk
|
||||
//
|
||||
// same as mel_spectrogram() in modeling_qwen3_tts.py
|
||||
// ECAPA-TDNN takes the whole clip in one pass, so no Whisper-style chunking or normalization
|
||||
//
|
||||
|
||||
void mtmd_audio_preprocessor_qwen3tts_spk::initialize() {
|
||||
cache.fill_sin_cos_table(hparams.audio_n_fft);
|
||||
cache.fill_hann_window(hparams.audio_window_len, true);
|
||||
cache.fill_mel_filterbank_matrix(hparams.n_mel_bins, hparams.audio_n_fft, hparams.audio_sample_rate);
|
||||
}
|
||||
|
||||
bool mtmd_audio_preprocessor_qwen3tts_spk::preprocess(const float * samples,
|
||||
size_t n_samples,
|
||||
std::vector<mtmd_audio_mel> & output) {
|
||||
if (n_samples == 0) {
|
||||
return false;
|
||||
}
|
||||
|
||||
GGML_ASSERT(!cache.sin_vals.empty());
|
||||
GGML_ASSERT(!cache.cos_vals.empty());
|
||||
GGML_ASSERT(!cache.filters.data.empty());
|
||||
|
||||
// reflect pad by (n_fft - hop) / 2 = 384, matching center=False STFT framing
|
||||
const int pad = (hparams.audio_n_fft - hparams.audio_hop_len) / 2;
|
||||
if (n_samples < (size_t) pad + 1) {
|
||||
return false;
|
||||
}
|
||||
|
||||
std::vector<float> padded(n_samples + 2 * pad, 0.0f);
|
||||
for (int i = 0; i < pad; i++) {
|
||||
padded[i] = samples[pad - i];
|
||||
}
|
||||
std::copy(samples, samples + n_samples, padded.begin() + pad);
|
||||
for (int i = 0; i < pad; i++) {
|
||||
padded[n_samples + pad + i] = samples[n_samples - 2 - i];
|
||||
}
|
||||
|
||||
filter_params params;
|
||||
params.n_mel = hparams.n_mel_bins;
|
||||
params.n_fft_bins = 1 + (hparams.audio_n_fft / 2);
|
||||
params.hann_window_size = hparams.audio_window_len;
|
||||
params.hop_length = hparams.audio_hop_len;
|
||||
params.sample_rate = hparams.audio_sample_rate;
|
||||
params.no_padding = true; // reflect padding already applied above
|
||||
params.use_natural_log = true;
|
||||
params.use_magnitude = true;
|
||||
params.mel_floor = 1e-5f;
|
||||
|
||||
mtmd_audio_mel out;
|
||||
bool ok = log_mel_spectrogram(padded.data(), (int) padded.size(), 4, params, cache, out);
|
||||
if (!ok) {
|
||||
return false;
|
||||
}
|
||||
|
||||
output.push_back(std::move(out));
|
||||
return true;
|
||||
}
|
||||
|
||||
//
|
||||
// mtmd_audio_preprocessor_conformer
|
||||
//
|
||||
|
||||
@@ -120,6 +120,15 @@ struct mtmd_audio_preprocessor_mimo_audio : mtmd_audio_preprocessor {
|
||||
mtmd_audio_cache cache;
|
||||
};
|
||||
|
||||
struct mtmd_audio_preprocessor_qwen3tts_spk : mtmd_audio_preprocessor {
|
||||
mtmd_audio_preprocessor_qwen3tts_spk(const clip_ctx * ctx) : mtmd_audio_preprocessor(ctx) {}
|
||||
void initialize() override;
|
||||
bool preprocess(const float * samples, size_t n_samples, std::vector<mtmd_audio_mel> & output) override;
|
||||
|
||||
private:
|
||||
mtmd_audio_cache cache;
|
||||
};
|
||||
|
||||
struct mtmd_audio_preprocessor_parakeet : mtmd_audio_preprocessor {
|
||||
mtmd_audio_preprocessor_parakeet(clip_ctx * ctx) : mtmd_audio_preprocessor(ctx) { }
|
||||
void initialize() override;
|
||||
|
||||
@@ -116,6 +116,14 @@ struct mtmd_cli_context {
|
||||
exit(1);
|
||||
}
|
||||
|
||||
init_vision_context(params);
|
||||
|
||||
if (!mtmd_helper_model_can_chat(lctx, ctx_vision.get())) {
|
||||
LOG_ERR("Model does not support chat mode\n");
|
||||
LOG_ERR("Hint: for TTS models, please use llama-tts\n");
|
||||
exit(1);
|
||||
}
|
||||
|
||||
if (!llama_model_chat_template(model, nullptr) && params.chat_template.empty()) {
|
||||
LOG_ERR("Model does not have chat template.\n");
|
||||
LOG_ERR(" For old llava models, you may need to use '--chat-template vicuna'\n");
|
||||
@@ -129,8 +137,6 @@ struct mtmd_cli_context {
|
||||
chat_history.clear();
|
||||
LOG_INF("%s: chat template example:\n%s\n", __func__, common_chat_format_example(tmpls.get(), params.use_jinja, params.default_template_kwargs).c_str());
|
||||
|
||||
init_vision_context(params);
|
||||
|
||||
// load antiprompt tokens for legacy templates
|
||||
if (params.chat_template == "vicuna") {
|
||||
antiprompt_tokens = common_tokenize(lctx, "ASSISTANT:", false, true);
|
||||
|
||||
@@ -0,0 +1,180 @@
|
||||
#pragma once
|
||||
|
||||
// shared internal utilities for the mtmd-helper-*.cpp translation units
|
||||
// (mtmd-helper.cpp, mtmd-helper-gen.cpp)
|
||||
// NOT part of the public mtmd-helper.h API
|
||||
|
||||
#include "ggml.h"
|
||||
#include "llama.h"
|
||||
#include "mtmd.h"
|
||||
|
||||
#include <cstdarg>
|
||||
#include <cstdio>
|
||||
#include <cstdlib>
|
||||
#include <vector>
|
||||
|
||||
//
|
||||
// logging
|
||||
//
|
||||
|
||||
struct mtmd_helper_logger {
|
||||
ggml_log_callback default_callback = [](ggml_log_level level, const char * text, void * user_data) {
|
||||
(void) level;
|
||||
(void) user_data;
|
||||
fputs(text, stderr);
|
||||
fflush(stderr);
|
||||
};
|
||||
|
||||
ggml_log_callback log_callback = default_callback;
|
||||
void * log_callback_user_data;
|
||||
|
||||
void log_v(enum ggml_log_level level, const char * format, va_list args) {
|
||||
if (format == NULL) {
|
||||
return;
|
||||
}
|
||||
va_list args_copy;
|
||||
va_copy(args_copy, args);
|
||||
char buffer[128];
|
||||
int len = vsnprintf(buffer, 128, format, args);
|
||||
if (len < 128) {
|
||||
log_callback(level, buffer, log_callback_user_data);
|
||||
} else {
|
||||
char * buffer2 = (char *) calloc(len + 1, sizeof(char));
|
||||
vsnprintf(buffer2, len + 1, format, args_copy);
|
||||
buffer2[len] = 0;
|
||||
log_callback(level, buffer2, log_callback_user_data);
|
||||
free(buffer2);
|
||||
}
|
||||
va_end(args_copy);
|
||||
}
|
||||
|
||||
void log(enum ggml_log_level level, const char * format, ...) {
|
||||
va_list args;
|
||||
va_start(args, format);
|
||||
log_v(level, format, args);
|
||||
va_end(args);
|
||||
}
|
||||
};
|
||||
|
||||
// inline, so all TUs including this header share one instance
|
||||
inline mtmd_helper_logger g_logger;
|
||||
|
||||
#define LOG_DBG(...) g_logger.log(GGML_LOG_LEVEL_DEBUG, __VA_ARGS__)
|
||||
#define LOG_INF(...) g_logger.log(GGML_LOG_LEVEL_INFO, __VA_ARGS__)
|
||||
#define LOG_WRN(...) g_logger.log(GGML_LOG_LEVEL_WARN, __VA_ARGS__)
|
||||
#define LOG_ERR(...) g_logger.log(GGML_LOG_LEVEL_ERROR, __VA_ARGS__)
|
||||
|
||||
//
|
||||
// embd batch
|
||||
//
|
||||
|
||||
// helper struct to make working with embd batch easier
|
||||
// note: this will be removed after llama_batch_ext refactoring
|
||||
struct decode_embd_batch {
|
||||
int n_pos_per_embd;
|
||||
int n_mmproj_embd;
|
||||
std::vector<llama_pos> pos;
|
||||
std::vector<llama_pos> pos_view; // used by mrope
|
||||
std::vector<int32_t> n_seq_id;
|
||||
std::vector<llama_seq_id> seq_id_0;
|
||||
std::vector<llama_seq_id *> seq_ids;
|
||||
std::vector<int8_t> logits;
|
||||
llama_batch batch;
|
||||
decode_embd_batch(float * embd, int32_t n_tokens, int n_pos_per_embd, int n_mmproj_embd) : n_pos_per_embd(n_pos_per_embd), n_mmproj_embd(n_mmproj_embd) {
|
||||
GGML_ASSERT(n_tokens > 0 && n_pos_per_embd > 0 && n_mmproj_embd > 0);
|
||||
pos .resize(n_tokens * n_pos_per_embd);
|
||||
n_seq_id.resize(n_tokens);
|
||||
seq_ids .resize(n_tokens + 1);
|
||||
logits .resize(n_tokens);
|
||||
seq_id_0.resize(1);
|
||||
seq_ids [n_tokens] = nullptr;
|
||||
batch = {
|
||||
/*n_tokens =*/ n_tokens,
|
||||
/*tokens =*/ nullptr,
|
||||
/*embd =*/ embd,
|
||||
/*pos =*/ pos.data(),
|
||||
/*n_seq_id =*/ n_seq_id.data(),
|
||||
/*seq_id =*/ seq_ids.data(),
|
||||
/*logits =*/ logits.data(),
|
||||
};
|
||||
}
|
||||
|
||||
void set_position_normal(llama_pos pos_0, llama_seq_id seq_id) {
|
||||
seq_id_0[0] = seq_id;
|
||||
for (int i = 0; i < batch.n_tokens; i++) {
|
||||
batch.pos [i] = pos_0 + i;
|
||||
batch.n_seq_id[i] = 1;
|
||||
batch.seq_id [i] = seq_id_0.data();
|
||||
batch.logits [i] = false;
|
||||
}
|
||||
}
|
||||
|
||||
// M-RoPE for image
|
||||
void set_position_mrope_2d(const std::vector<mtmd_decoder_pos> & rel_pos, llama_seq_id seq_id) {
|
||||
GGML_ASSERT(n_pos_per_embd == 4);
|
||||
GGML_ASSERT(!rel_pos.empty() && (int32_t)rel_pos.size() == batch.n_tokens);
|
||||
seq_id_0[0] = seq_id;
|
||||
for (int32_t i = 0; i < batch.n_tokens; i++) {
|
||||
pos[i ] = rel_pos[i].t;
|
||||
pos[i + batch.n_tokens ] = rel_pos[i].y;
|
||||
pos[i + batch.n_tokens * 2] = rel_pos[i].x;
|
||||
pos[i + batch.n_tokens * 3] = rel_pos[i].z;
|
||||
}
|
||||
for (int i = 0; i < batch.n_tokens; i++) {
|
||||
batch.n_seq_id[i] = 1;
|
||||
batch.seq_id [i] = seq_id_0.data();
|
||||
batch.logits [i] = false;
|
||||
}
|
||||
}
|
||||
|
||||
// M-RoPE for audio
|
||||
void set_position_mrope_1d(llama_pos pos_0, llama_seq_id seq_id) {
|
||||
GGML_ASSERT(n_pos_per_embd == 4);
|
||||
seq_id_0[0] = seq_id;
|
||||
for (int i = 0; i < batch.n_tokens; i++) {
|
||||
pos[i ] = pos_0 + i;
|
||||
pos[i + batch.n_tokens ] = pos_0 + i;
|
||||
pos[i + batch.n_tokens * 2] = pos_0 + i;
|
||||
pos[i + batch.n_tokens * 3] = pos_0 + i;
|
||||
}
|
||||
for (int i = 0; i < batch.n_tokens; i++) {
|
||||
batch.n_seq_id[i] = 1;
|
||||
batch.seq_id [i] = seq_id_0.data();
|
||||
batch.logits [i] = false;
|
||||
}
|
||||
}
|
||||
|
||||
llama_batch get_view(int offset, int n_tokens) {
|
||||
GGML_ASSERT(offset >= 0 && n_tokens > 0 && offset + n_tokens <= batch.n_tokens);
|
||||
llama_pos * pos_ptr;
|
||||
pos_view.clear();
|
||||
pos_view.reserve(n_tokens * n_pos_per_embd);
|
||||
if (n_pos_per_embd > 1) {
|
||||
// mrope
|
||||
// for example, with layout of src: 1234...1234...1234...1234...
|
||||
// offset 2 will give us dst: 34...34...34...34...
|
||||
for (int i = 0; i < n_pos_per_embd; i++) {
|
||||
// assume n_tokens is less than or equal to batch.n_tokens
|
||||
// batch.n_tokens is number of **total** tokens
|
||||
// n_tokens is number of viewed token
|
||||
size_t src_idx = i * batch.n_tokens + offset;
|
||||
pos_view.insert(pos_view.end(),
|
||||
pos.data() + src_idx,
|
||||
pos.data() + src_idx + n_tokens);
|
||||
}
|
||||
pos_ptr = pos_view.data();
|
||||
} else {
|
||||
// normal
|
||||
pos_ptr = pos.data() + offset;
|
||||
}
|
||||
return {
|
||||
/*n_tokens =*/ n_tokens,
|
||||
/*tokens =*/ nullptr,
|
||||
/*embd =*/ batch.embd + offset * n_mmproj_embd,
|
||||
/*pos =*/ pos_ptr,
|
||||
/*n_seq_id =*/ batch.n_seq_id + offset,
|
||||
/*seq_id =*/ batch.seq_id + offset,
|
||||
/*logits =*/ batch.logits + offset,
|
||||
};
|
||||
}
|
||||
};
|
||||
@@ -0,0 +1,505 @@
|
||||
#include "mtmd.h"
|
||||
#include "mtmd-helper.h"
|
||||
#include "mtmd-helper-common.h"
|
||||
#include "llama.h"
|
||||
#include "../src/llama-ext.h"
|
||||
|
||||
#include <algorithm>
|
||||
#include <cstring>
|
||||
#include <memory>
|
||||
#include <string>
|
||||
#include <unordered_map>
|
||||
#include <vector>
|
||||
|
||||
#ifdef MTMD_INTERNAL_HEADER
|
||||
#error "mtmd-helper is a public library outside of mtmd. it must not include internal headers"
|
||||
#endif
|
||||
|
||||
//
|
||||
// Audio generation helpers
|
||||
//
|
||||
|
||||
// --tts-lang codes -> language names used by the codec_language special tokens
|
||||
static const std::unordered_map<std::string, std::string> tts_lang_codes = {
|
||||
{ "zh", "chinese" },
|
||||
{ "en", "english" },
|
||||
{ "de", "german" },
|
||||
{ "it", "italian" },
|
||||
{ "pt", "portuguese" },
|
||||
{ "es", "spanish" },
|
||||
{ "ja", "japanese" },
|
||||
{ "ko", "korean" },
|
||||
{ "fr", "french" },
|
||||
{ "ru", "russian" },
|
||||
};
|
||||
|
||||
static std::string tts_resolve_lang(const std::string & lang) {
|
||||
auto it = tts_lang_codes.find(lang);
|
||||
return it != tts_lang_codes.end() ? it->second : lang;
|
||||
}
|
||||
|
||||
static llama_token find_special_token(const llama_vocab * vocab, const std::string & piece) {
|
||||
const int32_t n = llama_vocab_n_tokens(vocab);
|
||||
for (llama_token t = 0; t < n; t++) {
|
||||
if (piece == llama_vocab_get_text(vocab, t)) {
|
||||
return t;
|
||||
}
|
||||
}
|
||||
return LLAMA_TOKEN_NULL;
|
||||
}
|
||||
|
||||
static bool write_wav16(std::vector<char> & buf, const std::vector<float> & pcm, int32_t rate) {
|
||||
// RIFF chunk sizes are 32-bit; refuse to emit a file with a truncated header
|
||||
if (pcm.size() > ((size_t) UINT32_MAX - 36) / 2) {
|
||||
return false;
|
||||
}
|
||||
const uint32_t data_sz = (uint32_t) (pcm.size() * 2);
|
||||
const uint32_t riff_sz = 36 + data_sz;
|
||||
const uint32_t fmt_sz = 16, byte_rate = (uint32_t) rate * 2;
|
||||
const uint16_t fmt = 1, ch = 1, align = 2, bits = 16;
|
||||
const uint32_t rate32 = (uint32_t) rate;
|
||||
auto put = [&](const void * p, size_t n) {
|
||||
const char * c = (const char *) p;
|
||||
buf.insert(buf.end(), c, c + n);
|
||||
};
|
||||
put("RIFF", 4); put(&riff_sz, 4); put("WAVE", 4);
|
||||
put("fmt ", 4); put(&fmt_sz, 4);
|
||||
put(&fmt, 2); put(&ch, 2); put(&rate32, 4);
|
||||
put(&byte_rate, 4); put(&align, 2); put(&bits, 2);
|
||||
put("data", 4); put(&data_sz, 4);
|
||||
for (float v : pcm) {
|
||||
int16_t s = (int16_t) (std::max(-1.0f, std::min(1.0f, v)) * 32767.0f);
|
||||
put(&s, 2);
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
class mtmd_gen_audio_pipeline {
|
||||
public:
|
||||
mtmd_gen_audio_pipeline(llama_context * lctx, mtmd_context * mctx)
|
||||
: lctx(lctx), mctx(mctx), model(llama_get_model(lctx)), vocab(llama_model_get_vocab(model)),
|
||||
n_embd(llama_model_n_embd(model)), info(mtmd_gen_audio_get_info(mctx)) {}
|
||||
virtual ~mtmd_gen_audio_pipeline() = default;
|
||||
|
||||
virtual void reset() = 0;
|
||||
virtual int32_t set_input(const mtmd_helper_gen_audio_inp * inp) = 0;
|
||||
// decodes at most n_batch prompt tokens; returns remaining count (0 = done), <0 on error
|
||||
virtual int32_t step_prompt(int32_t n_batch) = 0;
|
||||
// sampled can be LLAMA_TOKEN_NULL for pipelines with no discrete backbone token,
|
||||
// those read what they need from h_state_in instead
|
||||
virtual int32_t step_gen(llama_token sampled, const float * h_state_in, const float ** h_state_out) = 0;
|
||||
virtual int32_t get_output(int32_t * out_sample_rate, const char ** out_data, size_t * out_data_len, int64_t * out_n_samples) = 0;
|
||||
|
||||
protected:
|
||||
llama_context * lctx;
|
||||
mtmd_context * mctx;
|
||||
const llama_model * model;
|
||||
const llama_vocab * vocab;
|
||||
int n_embd;
|
||||
mtmd_gen_audio_info info;
|
||||
};
|
||||
|
||||
// Qwen3-TTS: backbone samples codec_0, code_predictor gives the other 15 codebooks,
|
||||
// then code2wav decodes them to PCM
|
||||
class qwen3tts_gen_audio_pipeline : public mtmd_gen_audio_pipeline {
|
||||
public:
|
||||
using mtmd_gen_audio_pipeline::mtmd_gen_audio_pipeline;
|
||||
|
||||
void reset() override {
|
||||
seq_id = 0;
|
||||
pos = 0;
|
||||
codes_buf.clear();
|
||||
c2w_state.clear();
|
||||
audio_pcm.clear();
|
||||
overlay.clear();
|
||||
overlay_idx = 0;
|
||||
h_state_buf.clear();
|
||||
out_buf.clear();
|
||||
prompt_embd_buf.clear();
|
||||
prompt_batch.reset();
|
||||
n_prompt = 0;
|
||||
prompt_pos = 0;
|
||||
}
|
||||
|
||||
int32_t set_input(const mtmd_helper_gen_audio_inp * inp) override {
|
||||
reset();
|
||||
seq_id = inp->seq_id;
|
||||
|
||||
if (!ensure_cache()) {
|
||||
return 1;
|
||||
}
|
||||
|
||||
const std::string lang = tts_resolve_lang((inp->lang && inp->lang[0]) ? inp->lang : "english");
|
||||
const llama_token c_lang = find_special_token(vocab, ("<|codec_language_" + lang + "|>").c_str());
|
||||
if (c_lang == LLAMA_TOKEN_NULL) {
|
||||
LOG_ERR("mtmd_helper_gen_audio: unknown language '%s'\n", lang.c_str());
|
||||
return 1;
|
||||
}
|
||||
|
||||
std::vector<float> speaker_embd;
|
||||
if (inp->speaker_ref) {
|
||||
if (!encode_speaker(inp->speaker_ref, speaker_embd)) {
|
||||
return 1;
|
||||
}
|
||||
}
|
||||
|
||||
const int n_e = n_embd;
|
||||
auto row = [&](llama_token t) {
|
||||
return std::vector<float>(tok_embd.begin() + (size_t) t * n_e,
|
||||
tok_embd.begin() + (size_t) (t + 1) * n_e);
|
||||
};
|
||||
auto sum_row = [&](llama_token a, llama_token b) {
|
||||
std::vector<float> va = row(a), vb = row(b);
|
||||
for (int i = 0; i < n_e; i++) va[(size_t) i] += vb[(size_t) i];
|
||||
return va;
|
||||
};
|
||||
auto sum_vec = [&](llama_token a, const std::vector<float> & vb) {
|
||||
std::vector<float> va = row(a);
|
||||
for (int i = 0; i < n_e; i++) va[(size_t) i] += vb[(size_t) i];
|
||||
return va;
|
||||
};
|
||||
|
||||
// upstream chat wrap, then slices: [0:3] role, [3:-5] utterance body
|
||||
const std::string full = "<|im_start|>assistant\n" + std::string(inp->prompt, inp->prompt_len) +
|
||||
"<|im_end|>\n<|im_start|>assistant\n";
|
||||
std::vector<llama_token> ids(full.size() + 16);
|
||||
int n_ids = llama_tokenize(vocab, full.c_str(), (int32_t) full.size(), ids.data(), (int32_t) ids.size(),
|
||||
false, true);
|
||||
if (n_ids < 8) {
|
||||
LOG_ERR("mtmd_helper_gen_audio: tokenization failed\n");
|
||||
return 1;
|
||||
}
|
||||
ids.resize((size_t) n_ids);
|
||||
|
||||
std::vector<std::vector<float>> prompt;
|
||||
for (int i = 0; i < 3; i++) prompt.push_back(row(ids[(size_t) i]));
|
||||
prompt.push_back(sum_row(tts_pad, c_think));
|
||||
prompt.push_back(sum_row(tts_pad, c_think_b));
|
||||
prompt.push_back(sum_row(tts_pad, c_lang));
|
||||
prompt.push_back(sum_row(tts_pad, c_think_e));
|
||||
if (!speaker_embd.empty()) prompt.push_back(sum_vec(tts_pad, speaker_embd));
|
||||
prompt.push_back(sum_row(tts_bos, codec_pad));
|
||||
for (int i = 3; i < n_ids - 5; i++) prompt.push_back(sum_row(ids[(size_t) i], codec_pad));
|
||||
prompt.push_back(sum_row(tts_eos, codec_pad));
|
||||
prompt.push_back(sum_row(tts_pad, codec_bos));
|
||||
|
||||
n_prompt = (int) prompt.size();
|
||||
|
||||
// the talker uses the qwen3vl interleaved mrope, all sections are equal for a text/codec stream
|
||||
mrope = llama_model_rope_type(model) == LLAMA_ROPE_TYPE_MROPE ||
|
||||
llama_model_rope_type(model) == LLAMA_ROPE_TYPE_IMROPE;
|
||||
const int n_pos_per_embd = mrope ? 4 : 1;
|
||||
|
||||
prompt_embd_buf.resize((size_t) n_prompt * (size_t) n_e);
|
||||
for (int i = 0; i < n_prompt; i++) {
|
||||
memcpy(prompt_embd_buf.data() + (size_t) i * n_e, prompt[(size_t) i].data(), (size_t) n_e * sizeof(float));
|
||||
}
|
||||
|
||||
prompt_batch.reset(new decode_embd_batch(prompt_embd_buf.data(), n_prompt, n_pos_per_embd, n_e));
|
||||
if (mrope) prompt_batch->set_position_mrope_1d(0, seq_id);
|
||||
else prompt_batch->set_position_normal (0, seq_id);
|
||||
prompt_pos = 0;
|
||||
|
||||
pos = 0;
|
||||
top_k = inp->top_k > 0 ? inp->top_k : 50;
|
||||
top_p = inp->top_p > 0 ? inp->top_p : 1.0f;
|
||||
out_type = inp->out_type;
|
||||
|
||||
// the text stream keeps flowing during generation: after frame k, the input adds
|
||||
// trailing text row k on top of the codes embedding, then tts_eos, then tts_pad
|
||||
for (int i = 3; i < n_ids - 5; i++) overlay.push_back(row(ids[(size_t) i]));
|
||||
overlay.push_back(row(tts_eos));
|
||||
overlay.push_back(row(tts_pad));
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
int32_t step_prompt(int32_t n_batch) override {
|
||||
GGML_ASSERT(n_batch > 0);
|
||||
if (prompt_pos >= n_prompt) {
|
||||
return 0;
|
||||
}
|
||||
const int32_t n_tokens_batch = std::min(n_batch, n_prompt - prompt_pos);
|
||||
llama_batch batch_view = prompt_batch->get_view(prompt_pos, n_tokens_batch);
|
||||
|
||||
const bool is_last_batch = (prompt_pos + n_tokens_batch) == n_prompt;
|
||||
if (is_last_batch) {
|
||||
batch_view.logits[n_tokens_batch - 1] = 1;
|
||||
}
|
||||
|
||||
if (llama_decode(lctx, batch_view) != 0) {
|
||||
LOG_ERR("mtmd_helper_gen_audio: prompt decode failed\n");
|
||||
return -1;
|
||||
}
|
||||
|
||||
pos += n_tokens_batch;
|
||||
prompt_pos += n_tokens_batch;
|
||||
|
||||
if (prompt_pos >= n_prompt) {
|
||||
// prompt fully processed, its embedding buffer is no longer needed
|
||||
prompt_batch.reset();
|
||||
prompt_embd_buf.clear();
|
||||
return 0;
|
||||
}
|
||||
return n_prompt - prompt_pos;
|
||||
}
|
||||
|
||||
int32_t step_gen(llama_token sampled, const float * h_state_in, const float ** h_state_out) override {
|
||||
mtmd_gen_inp inp{};
|
||||
inp.type = MTMD_GEN_PROCESS_TYPE_GEN_CODE;
|
||||
inp.code0 = sampled - codec_0;
|
||||
inp.embd = const_cast<float *>(h_state_in);
|
||||
inp.top_k = top_k;
|
||||
inp.top_p = top_p;
|
||||
mtmd_gen_out out{};
|
||||
if (mtmd_gen_audio_process(mctx, &inp, &out) != 0) {
|
||||
LOG_ERR("mtmd_helper_gen_audio: gen_code process failed\n");
|
||||
return 1;
|
||||
}
|
||||
|
||||
codes_buf.insert(codes_buf.end(), out.codes, out.codes + out.n_codes);
|
||||
if (out.n_codes > 0 && codes_buf.size() / out.n_codes >= window_frames) {
|
||||
if (!flush_gen_wav()) {
|
||||
return 1;
|
||||
}
|
||||
}
|
||||
|
||||
std::vector<float> fb(out.embd, out.embd + n_embd);
|
||||
const auto & ov = overlay[std::min(overlay_idx, overlay.size() - 1)];
|
||||
for (int i = 0; i < n_embd; i++) fb[(size_t) i] += ov[(size_t) i];
|
||||
overlay_idx++;
|
||||
|
||||
const int n_pos_per_embd = mrope ? 4 : 1;
|
||||
decode_embd_batch batch_embd(fb.data(), 1, n_pos_per_embd, n_embd);
|
||||
if (mrope) batch_embd.set_position_mrope_1d(pos, seq_id);
|
||||
else batch_embd.set_position_normal (pos, seq_id);
|
||||
batch_embd.batch.logits[0] = 1;
|
||||
pos++;
|
||||
|
||||
if (llama_decode(lctx, batch_embd.batch) != 0) {
|
||||
LOG_ERR("mtmd_helper_gen_audio: decode failed\n");
|
||||
return 1;
|
||||
}
|
||||
|
||||
const float * he = llama_get_embeddings_ith(lctx, -1);
|
||||
h_state_buf.assign(he, he + n_embd);
|
||||
*h_state_out = h_state_buf.data();
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
int32_t get_output(int32_t * out_sample_rate, const char ** out_data, size_t * out_data_len, int64_t * out_n_samples) override {
|
||||
if (!flush_gen_wav()) {
|
||||
return 1;
|
||||
}
|
||||
|
||||
*out_sample_rate = info.sample_rate;
|
||||
if (out_n_samples) {
|
||||
*out_n_samples = (int64_t) audio_pcm.size();
|
||||
}
|
||||
|
||||
if (out_type == MTMD_HELPER_GEN_AUDIO_OUTTYPE_PCM) {
|
||||
*out_data = (const char *) audio_pcm.data();
|
||||
*out_data_len = audio_pcm.size() * sizeof(float);
|
||||
return 0;
|
||||
}
|
||||
|
||||
out_buf.clear();
|
||||
if (!write_wav16(out_buf, audio_pcm, info.sample_rate)) {
|
||||
LOG_ERR("mtmd_helper_gen_audio: output too large for WAV\n");
|
||||
return 1;
|
||||
}
|
||||
*out_data = out_buf.data();
|
||||
*out_data_len = out_buf.size();
|
||||
return 0;
|
||||
}
|
||||
|
||||
private:
|
||||
bool ensure_cache() {
|
||||
if (specials_ok) {
|
||||
return true;
|
||||
}
|
||||
codec_0 = find_special_token(vocab, "<|codec_0|>");
|
||||
codec_bos = find_special_token(vocab, "<|codec_bos|>");
|
||||
codec_eos = find_special_token(vocab, "<|codec_eos_token|>");
|
||||
codec_pad = find_special_token(vocab, "<|codec_pad|>");
|
||||
c_think = find_special_token(vocab, "<|codec_think|>");
|
||||
c_think_b = find_special_token(vocab, "<|codec_think_bos|>");
|
||||
c_think_e = find_special_token(vocab, "<|codec_think_eos|>");
|
||||
tts_pad = find_special_token(vocab, "<tts_pad>");
|
||||
tts_bos = find_special_token(vocab, "<tts_text_bos>");
|
||||
tts_eos = find_special_token(vocab, "<tts_text_eod>");
|
||||
for (llama_token t : { codec_0, codec_bos, codec_eos, codec_pad,
|
||||
c_think, c_think_b, c_think_e,
|
||||
tts_pad, tts_bos, tts_eos }) {
|
||||
if (t == LLAMA_TOKEN_NULL) {
|
||||
LOG_ERR("mtmd_helper_gen_audio: missing a required special token in vocab\n");
|
||||
return false;
|
||||
}
|
||||
}
|
||||
const uint32_t n_tok_embd = llama_model_get_tok_embd(model, nullptr);
|
||||
if (n_tok_embd == 0) {
|
||||
LOG_ERR("mtmd_helper_gen_audio: model has no token embeddings\n");
|
||||
return false;
|
||||
}
|
||||
tok_embd.resize(n_tok_embd);
|
||||
if (llama_model_get_tok_embd(model, tok_embd.data()) != n_tok_embd) {
|
||||
LOG_ERR("mtmd_helper_gen_audio: token embedding copy failed\n");
|
||||
return false;
|
||||
}
|
||||
specials_ok = true;
|
||||
return true;
|
||||
}
|
||||
|
||||
// runs the reference wav through the speaker encoder, returns one x-vector embedding row
|
||||
bool encode_speaker(mtmd_bitmap * bitmap, std::vector<float> & out) {
|
||||
if (!mtmd_support_audio(mctx)) {
|
||||
LOG_ERR("mtmd_helper_gen_audio: mmproj has no speaker/audio encoder\n");
|
||||
return false;
|
||||
}
|
||||
const std::string marker = mtmd_default_marker();
|
||||
mtmd_input_text text{ marker.c_str(), marker.size(), false, true };
|
||||
mtmd_input_chunks * chunks = mtmd_input_chunks_init();
|
||||
const mtmd_bitmap * bptr = bitmap;
|
||||
bool ok = mtmd_tokenize(mctx, chunks, &text, &bptr, 1) == 0;
|
||||
if (ok) {
|
||||
ok = false;
|
||||
for (size_t i = 0; i < mtmd_input_chunks_size(chunks); i++) {
|
||||
const mtmd_input_chunk * chunk = mtmd_input_chunks_get(chunks, i);
|
||||
if (mtmd_input_chunk_get_type(chunk) != MTMD_INPUT_CHUNK_TYPE_AUDIO) {
|
||||
continue;
|
||||
}
|
||||
if (mtmd_encode_chunk(mctx, chunk) != 0) {
|
||||
LOG_ERR("mtmd_helper_gen_audio: speaker encode failed\n");
|
||||
break;
|
||||
}
|
||||
const float * embd = mtmd_get_output_embd(mctx);
|
||||
const size_t n = (size_t) llama_model_n_embd_inp(model) * mtmd_input_chunk_get_n_tokens(chunk);
|
||||
out.assign(embd, embd + n);
|
||||
ok = true;
|
||||
break;
|
||||
}
|
||||
}
|
||||
mtmd_input_chunks_free(chunks);
|
||||
return ok;
|
||||
}
|
||||
|
||||
// one GEN_WAV process() call over the buffered codes, state is carried across batches
|
||||
bool flush_gen_wav() {
|
||||
if (codes_buf.empty()) {
|
||||
return true;
|
||||
}
|
||||
mtmd_gen_inp inp{};
|
||||
inp.type = MTMD_GEN_PROCESS_TYPE_GEN_WAV;
|
||||
inp.codes = codes_buf.data();
|
||||
inp.n_codes = codes_buf.size();
|
||||
inp.state_data = c2w_state.empty() ? nullptr : (const char *) c2w_state.data();
|
||||
inp.state_size = c2w_state.size();
|
||||
mtmd_gen_out out{};
|
||||
if (mtmd_gen_audio_process(mctx, &inp, &out) != 0) {
|
||||
LOG_ERR("mtmd_helper_gen_audio: gen_wav process failed\n");
|
||||
return false;
|
||||
}
|
||||
audio_pcm.insert(audio_pcm.end(), out.audio, out.audio + out.n_samples);
|
||||
c2w_state.assign(out.state_data, out.state_data + out.state_size);
|
||||
codes_buf.clear();
|
||||
return true;
|
||||
}
|
||||
|
||||
// vocab specials fixed across the whole session, looked up once
|
||||
bool specials_ok = false;
|
||||
llama_token codec_0 = LLAMA_TOKEN_NULL;
|
||||
llama_token codec_bos = LLAMA_TOKEN_NULL;
|
||||
llama_token codec_eos = LLAMA_TOKEN_NULL;
|
||||
llama_token codec_pad = LLAMA_TOKEN_NULL;
|
||||
llama_token c_think = LLAMA_TOKEN_NULL;
|
||||
llama_token c_think_b = LLAMA_TOKEN_NULL;
|
||||
llama_token c_think_e = LLAMA_TOKEN_NULL;
|
||||
llama_token tts_pad = LLAMA_TOKEN_NULL;
|
||||
llama_token tts_bos = LLAMA_TOKEN_NULL;
|
||||
llama_token tts_eos = LLAMA_TOKEN_NULL;
|
||||
std::vector<float> tok_embd; // whole token embedding matrix, n_vocab * n_embd
|
||||
|
||||
// must match hparams.wav_tfm_swa hardcoded in clip.cpp
|
||||
size_t window_frames = 72;
|
||||
|
||||
// per-generation state, cleared by reset()
|
||||
llama_seq_id seq_id = 0;
|
||||
bool mrope = false;
|
||||
int pos = 0;
|
||||
// prompt decode state, consumed batch-by-batch by step_prompt()
|
||||
std::vector<float> prompt_embd_buf;
|
||||
std::unique_ptr<decode_embd_batch> prompt_batch;
|
||||
int n_prompt = 0;
|
||||
int prompt_pos = 0;
|
||||
int32_t top_k = 50;
|
||||
float top_p = 1.0f;
|
||||
std::vector<int32_t> codes_buf;
|
||||
std::vector<uint8_t> c2w_state;
|
||||
std::vector<float> audio_pcm;
|
||||
std::vector<std::vector<float>> overlay;
|
||||
size_t overlay_idx = 0;
|
||||
std::vector<float> h_state_buf;
|
||||
mtmd_helper_gen_audio_outtype out_type = MTMD_HELPER_GEN_AUDIO_OUTTYPE_WAV;
|
||||
std::vector<char> out_buf;
|
||||
};
|
||||
|
||||
static std::unique_ptr<mtmd_gen_audio_pipeline> make_pipeline(llama_context * lctx, mtmd_context * mctx) {
|
||||
switch (mtmd_gen_audio_get_info(mctx).type) {
|
||||
case MTMD_GEN_AUDIO_TYPE_QWEN3TTS:
|
||||
return std::unique_ptr<mtmd_gen_audio_pipeline>(new qwen3tts_gen_audio_pipeline(lctx, mctx));
|
||||
default:
|
||||
return nullptr;
|
||||
}
|
||||
}
|
||||
|
||||
struct mtmd_helper_gen_audio {
|
||||
std::unique_ptr<mtmd_gen_audio_pipeline> pipeline;
|
||||
};
|
||||
|
||||
mtmd_helper_gen_audio * mtmd_helper_gen_audio_init(struct llama_context * lctx, struct mtmd_context * mctx) {
|
||||
auto * ctx = new mtmd_helper_gen_audio();
|
||||
ctx->pipeline = make_pipeline(lctx, mctx);
|
||||
return ctx;
|
||||
}
|
||||
|
||||
void mtmd_helper_gen_audio_free(mtmd_helper_gen_audio * ctx) {
|
||||
delete ctx;
|
||||
}
|
||||
|
||||
void mtmd_helper_gen_audio_reset(mtmd_helper_gen_audio * ctx) {
|
||||
if (ctx->pipeline) {
|
||||
ctx->pipeline->reset();
|
||||
}
|
||||
}
|
||||
|
||||
int32_t mtmd_helper_gen_audio_set_input(mtmd_helper_gen_audio * ctx, const mtmd_helper_gen_audio_inp * inp) {
|
||||
if (!ctx->pipeline) {
|
||||
LOG_ERR("mtmd_helper_gen_audio: unsupported or missing gen-audio pipeline\n");
|
||||
return 1;
|
||||
}
|
||||
return ctx->pipeline->set_input(inp);
|
||||
}
|
||||
|
||||
int32_t mtmd_helper_gen_audio_step_prompt(mtmd_helper_gen_audio * ctx, int32_t n_batch) {
|
||||
if (!ctx->pipeline) {
|
||||
return -1;
|
||||
}
|
||||
return ctx->pipeline->step_prompt(n_batch);
|
||||
}
|
||||
|
||||
int32_t mtmd_helper_gen_audio_step_gen(mtmd_helper_gen_audio * ctx, llama_token sampled,
|
||||
const float * h_state_in, const float ** h_state_out) {
|
||||
if (!ctx->pipeline) {
|
||||
return 1;
|
||||
}
|
||||
return ctx->pipeline->step_gen(sampled, h_state_in, h_state_out);
|
||||
}
|
||||
|
||||
int32_t mtmd_helper_gen_audio_get_output(mtmd_helper_gen_audio * ctx, int32_t * out_sample_rate,
|
||||
const char ** out_data, size_t * out_data_len, int64_t * out_n_samples) {
|
||||
if (!ctx->pipeline) {
|
||||
return 1;
|
||||
}
|
||||
return ctx->pipeline->get_output(out_sample_rate, out_data, out_data_len, out_n_samples);
|
||||
}
|
||||
+16
-155
@@ -9,6 +9,7 @@
|
||||
|
||||
#include "mtmd.h"
|
||||
#include "mtmd-helper.h"
|
||||
#include "mtmd-helper-common.h"
|
||||
#include "llama.h"
|
||||
|
||||
#include <algorithm>
|
||||
@@ -45,50 +46,6 @@
|
||||
// internal logging functions
|
||||
//
|
||||
|
||||
struct mtmd_helper_logger {
|
||||
ggml_log_callback default_callback = [](ggml_log_level level, const char * text, void * user_data) {
|
||||
(void) level;
|
||||
(void) user_data;
|
||||
fputs(text, stderr);
|
||||
fflush(stderr);
|
||||
};
|
||||
|
||||
ggml_log_callback log_callback = default_callback;
|
||||
void * log_callback_user_data;
|
||||
|
||||
void log_v(enum ggml_log_level level, const char * format, va_list args) {
|
||||
if (format == NULL) {
|
||||
return;
|
||||
}
|
||||
va_list args_copy;
|
||||
va_copy(args_copy, args);
|
||||
char buffer[128];
|
||||
int len = vsnprintf(buffer, 128, format, args);
|
||||
if (len < 128) {
|
||||
log_callback(level, buffer, log_callback_user_data);
|
||||
} else {
|
||||
char * buffer2 = (char *) calloc(len + 1, sizeof(char));
|
||||
vsnprintf(buffer2, len + 1, format, args_copy);
|
||||
buffer2[len] = 0;
|
||||
log_callback(level, buffer2, log_callback_user_data);
|
||||
free(buffer2);
|
||||
}
|
||||
va_end(args_copy);
|
||||
}
|
||||
|
||||
void log(enum ggml_log_level level, const char * format, ...) {
|
||||
va_list args;
|
||||
va_start(args, format);
|
||||
log_v(level, format, args);
|
||||
va_end(args);
|
||||
}
|
||||
} g_logger;
|
||||
|
||||
#define LOG_DBG(...) g_logger.log(GGML_LOG_LEVEL_DEBUG, __VA_ARGS__)
|
||||
#define LOG_INF(...) g_logger.log(GGML_LOG_LEVEL_INFO, __VA_ARGS__)
|
||||
#define LOG_WRN(...) g_logger.log(GGML_LOG_LEVEL_WARN, __VA_ARGS__)
|
||||
#define LOG_ERR(...) g_logger.log(GGML_LOG_LEVEL_ERROR, __VA_ARGS__)
|
||||
|
||||
void mtmd_helper_log_set(ggml_log_callback log_callback, void * user_data) {
|
||||
if (log_callback == nullptr) {
|
||||
log_callback = g_logger.default_callback;
|
||||
@@ -127,117 +84,6 @@ void mtmd_helper_image_get_decoder_pos(const mtmd_image_tokens * chunks, llama_p
|
||||
}
|
||||
}
|
||||
|
||||
// helper struct to make working with embd batch easier
|
||||
// note: this will be removed after llama_batch_ext refactoring
|
||||
struct decode_embd_batch {
|
||||
int n_pos_per_embd;
|
||||
int n_mmproj_embd;
|
||||
std::vector<llama_pos> pos;
|
||||
std::vector<llama_pos> pos_view; // used by mrope
|
||||
std::vector<int32_t> n_seq_id;
|
||||
std::vector<llama_seq_id> seq_id_0;
|
||||
std::vector<llama_seq_id *> seq_ids;
|
||||
std::vector<int8_t> logits;
|
||||
llama_batch batch;
|
||||
decode_embd_batch(float * embd, int32_t n_tokens, int n_pos_per_embd, int n_mmproj_embd) : n_pos_per_embd(n_pos_per_embd), n_mmproj_embd(n_mmproj_embd) {
|
||||
GGML_ASSERT(n_tokens > 0 && n_pos_per_embd > 0 && n_mmproj_embd > 0);
|
||||
pos .resize(n_tokens * n_pos_per_embd);
|
||||
n_seq_id.resize(n_tokens);
|
||||
seq_ids .resize(n_tokens + 1);
|
||||
logits .resize(n_tokens);
|
||||
seq_id_0.resize(1);
|
||||
seq_ids [n_tokens] = nullptr;
|
||||
batch = {
|
||||
/*n_tokens =*/ n_tokens,
|
||||
/*tokens =*/ nullptr,
|
||||
/*embd =*/ embd,
|
||||
/*pos =*/ pos.data(),
|
||||
/*n_seq_id =*/ n_seq_id.data(),
|
||||
/*seq_id =*/ seq_ids.data(),
|
||||
/*logits =*/ logits.data(),
|
||||
};
|
||||
}
|
||||
|
||||
void set_position_normal(llama_pos pos_0, llama_seq_id seq_id) {
|
||||
seq_id_0[0] = seq_id;
|
||||
for (int i = 0; i < batch.n_tokens; i++) {
|
||||
batch.pos [i] = pos_0 + i;
|
||||
batch.n_seq_id[i] = 1;
|
||||
batch.seq_id [i] = seq_id_0.data();
|
||||
batch.logits [i] = false;
|
||||
}
|
||||
}
|
||||
|
||||
// M-RoPE for image
|
||||
void set_position_mrope_2d(const std::vector<mtmd_decoder_pos> & rel_pos, llama_seq_id seq_id) {
|
||||
GGML_ASSERT(n_pos_per_embd == 4);
|
||||
GGML_ASSERT(!rel_pos.empty() && (int32_t)rel_pos.size() == batch.n_tokens);
|
||||
seq_id_0[0] = seq_id;
|
||||
for (int32_t i = 0; i < batch.n_tokens; i++) {
|
||||
pos[i ] = rel_pos[i].t;
|
||||
pos[i + batch.n_tokens ] = rel_pos[i].y;
|
||||
pos[i + batch.n_tokens * 2] = rel_pos[i].x;
|
||||
pos[i + batch.n_tokens * 3] = rel_pos[i].z;
|
||||
}
|
||||
for (int i = 0; i < batch.n_tokens; i++) {
|
||||
batch.n_seq_id[i] = 1;
|
||||
batch.seq_id [i] = seq_id_0.data();
|
||||
batch.logits [i] = false;
|
||||
}
|
||||
}
|
||||
|
||||
// M-RoPE for audio
|
||||
void set_position_mrope_1d(llama_pos pos_0, llama_seq_id seq_id) {
|
||||
GGML_ASSERT(n_pos_per_embd == 4);
|
||||
seq_id_0[0] = seq_id;
|
||||
for (int i = 0; i < batch.n_tokens; i++) {
|
||||
pos[i ] = pos_0 + i;
|
||||
pos[i + batch.n_tokens ] = pos_0 + i;
|
||||
pos[i + batch.n_tokens * 2] = pos_0 + i;
|
||||
pos[i + batch.n_tokens * 3] = pos_0 + i;
|
||||
}
|
||||
for (int i = 0; i < batch.n_tokens; i++) {
|
||||
batch.n_seq_id[i] = 1;
|
||||
batch.seq_id [i] = seq_id_0.data();
|
||||
batch.logits [i] = false;
|
||||
}
|
||||
}
|
||||
|
||||
llama_batch get_view(int offset, int n_tokens) {
|
||||
GGML_ASSERT(offset >= 0 && n_tokens > 0 && offset + n_tokens <= batch.n_tokens);
|
||||
llama_pos * pos_ptr;
|
||||
pos_view.clear();
|
||||
pos_view.reserve(n_tokens * n_pos_per_embd);
|
||||
if (n_pos_per_embd > 1) {
|
||||
// mrope
|
||||
// for example, with layout of src: 1234...1234...1234...1234...
|
||||
// offset 2 will give us dst: 34...34...34...34...
|
||||
for (int i = 0; i < n_pos_per_embd; i++) {
|
||||
// assume n_tokens is less than or equal to batch.n_tokens
|
||||
// batch.n_tokens is number of **total** tokens
|
||||
// n_tokens is number of viewed token
|
||||
size_t src_idx = i * batch.n_tokens + offset;
|
||||
pos_view.insert(pos_view.end(),
|
||||
pos.data() + src_idx,
|
||||
pos.data() + src_idx + n_tokens);
|
||||
}
|
||||
pos_ptr = pos_view.data();
|
||||
} else {
|
||||
// normal
|
||||
pos_ptr = pos.data() + offset;
|
||||
}
|
||||
return {
|
||||
/*n_tokens =*/ n_tokens,
|
||||
/*tokens =*/ nullptr,
|
||||
/*embd =*/ batch.embd + offset * n_mmproj_embd,
|
||||
/*pos =*/ pos_ptr,
|
||||
/*n_seq_id =*/ batch.n_seq_id + offset,
|
||||
/*seq_id =*/ batch.seq_id + offset,
|
||||
/*logits =*/ batch.logits + offset,
|
||||
};
|
||||
}
|
||||
};
|
||||
|
||||
// Helper class to set non-causal attention via RAII
|
||||
class scope_non_causal {
|
||||
public:
|
||||
@@ -1084,3 +930,18 @@ int32_t mtmd_helper_video_read_next(mtmd_helper_video * ctx,
|
||||
GGML_ASSERT(false && "video is not supported in this build (MTMD_VIDEO is set to OFF)");
|
||||
#endif
|
||||
}
|
||||
|
||||
bool mtmd_helper_model_can_chat(llama_context * lctx, mtmd_context * mctx) {
|
||||
if (!mctx) {
|
||||
return true;
|
||||
}
|
||||
|
||||
auto * model = llama_get_model(lctx);
|
||||
auto * tmpl = llama_model_chat_template(model, nullptr);
|
||||
auto info = mtmd_gen_audio_get_info(mctx);
|
||||
|
||||
// tts-only model cannot be used for chat (no chat template)
|
||||
bool is_tts_only = info.type != MTMD_GEN_AUDIO_TYPE_NONE && tmpl == nullptr;
|
||||
|
||||
return !is_tts_only;
|
||||
}
|
||||
|
||||
@@ -157,6 +157,73 @@ MTMD_API int32_t mtmd_helper_video_read_next(mtmd_helper_video * ctx,
|
||||
mtmd_bitmap ** out_bitmap,
|
||||
char ** out_text);
|
||||
|
||||
// return true if model can be used for chat
|
||||
MTMD_API bool mtmd_helper_model_can_chat(struct llama_context * lctx, struct mtmd_context * mctx);
|
||||
|
||||
//
|
||||
// Audio generation helpers
|
||||
// (early-stage experimental, subjected to breaking changes)
|
||||
//
|
||||
|
||||
// audio generation helper context
|
||||
// contains accumulator for generated audio features and PCM audio
|
||||
struct mtmd_helper_gen_audio;
|
||||
typedef struct mtmd_helper_gen_audio mtmd_helper_gen_audio;
|
||||
|
||||
enum mtmd_helper_gen_audio_outtype {
|
||||
MTMD_HELPER_GEN_AUDIO_OUTTYPE_PCM, // raw PCM
|
||||
MTMD_HELPER_GEN_AUDIO_OUTTYPE_WAV, // WAV PCM 16-bit LE, mono
|
||||
};
|
||||
struct mtmd_helper_gen_audio_inp {
|
||||
llama_seq_id seq_id;
|
||||
|
||||
const char * prompt;
|
||||
size_t prompt_len;
|
||||
|
||||
mtmd_bitmap * speaker_ref; // optional, can be NULL
|
||||
const char * lang; // optional, can be NULL
|
||||
|
||||
int32_t top_k;
|
||||
float top_p;
|
||||
|
||||
enum mtmd_helper_gen_audio_outtype out_type;
|
||||
};
|
||||
|
||||
MTMD_API mtmd_helper_gen_audio * mtmd_helper_gen_audio_init(
|
||||
struct llama_context * lctx,
|
||||
struct mtmd_context * mctx);
|
||||
|
||||
MTMD_API void mtmd_helper_gen_audio_free(mtmd_helper_gen_audio * ctx);
|
||||
|
||||
MTMD_API void mtmd_helper_gen_audio_reset(mtmd_helper_gen_audio * ctx);
|
||||
|
||||
MTMD_API int32_t mtmd_helper_gen_audio_set_input(
|
||||
mtmd_helper_gen_audio * ctx,
|
||||
const struct mtmd_helper_gen_audio_inp * inp);
|
||||
|
||||
// processes at most n_batch prompt tokens per call
|
||||
// returns: >0 = number of prompt tokens remaining, 0 = done, <0 = error
|
||||
MTMD_API int32_t mtmd_helper_gen_audio_step_prompt(
|
||||
mtmd_helper_gen_audio * ctx,
|
||||
int32_t n_batch);
|
||||
|
||||
// generates one frame; must only be called after step_prompt() has returned 0
|
||||
// h_state_out is valid until next step_gen() or reset() call
|
||||
MTMD_API int32_t mtmd_helper_gen_audio_step_gen(
|
||||
mtmd_helper_gen_audio * ctx,
|
||||
llama_token sampled,
|
||||
const float * h_state_in,
|
||||
const float ** h_state_out);
|
||||
|
||||
// out_data valid until next get_output() or reset() call
|
||||
// out_n_samples (optional, can be NULL) receives the number of generated PCM samples
|
||||
MTMD_API int32_t mtmd_helper_gen_audio_get_output(
|
||||
mtmd_helper_gen_audio * ctx,
|
||||
int32_t * out_sample_rate,
|
||||
const char ** out_data,
|
||||
size_t * out_data_len,
|
||||
int64_t * out_n_samples);
|
||||
|
||||
#ifdef __cplusplus
|
||||
} // extern "C"
|
||||
#endif
|
||||
@@ -177,6 +244,31 @@ struct mtmd_helper_video_deleter {
|
||||
};
|
||||
using video_ptr = std::unique_ptr<mtmd_helper_video, mtmd_helper_video_deleter>;
|
||||
|
||||
// audio generation-related C++ wrappers
|
||||
struct mtmd_helper_gen_audio_deleter {
|
||||
void operator()(mtmd_helper_gen_audio * val) { mtmd_helper_gen_audio_free(val); }
|
||||
};
|
||||
using gen_audio_ptr = std::unique_ptr<mtmd_helper_gen_audio, mtmd_helper_gen_audio_deleter>;
|
||||
struct gen_audio {
|
||||
gen_audio_ptr ctx;
|
||||
gen_audio(struct llama_context * lctx, struct mtmd_context * mctx) : ctx(mtmd_helper_gen_audio_init(lctx, mctx)) {}
|
||||
void reset() {
|
||||
mtmd_helper_gen_audio_reset(ctx.get());
|
||||
}
|
||||
int32_t set_input(const struct mtmd_helper_gen_audio_inp * inp) {
|
||||
return mtmd_helper_gen_audio_set_input(ctx.get(), inp);
|
||||
}
|
||||
int32_t step_prompt(int32_t n_batch) {
|
||||
return mtmd_helper_gen_audio_step_prompt(ctx.get(), n_batch);
|
||||
}
|
||||
int32_t step_gen(llama_token sampled, const float * h_state, const float ** h_state_out) {
|
||||
return mtmd_helper_gen_audio_step_gen(ctx.get(), sampled, h_state, h_state_out);
|
||||
}
|
||||
int32_t get_output(int32_t * out_sample_rate, const char ** out_data, size_t * out_data_len, int64_t * out_n_samples = nullptr) {
|
||||
return mtmd_helper_gen_audio_get_output(ctx.get(), out_sample_rate, out_data, out_data_len, out_n_samples);
|
||||
}
|
||||
};
|
||||
|
||||
} // namespace mtmd_helper
|
||||
#endif
|
||||
|
||||
|
||||
@@ -262,6 +262,13 @@ struct mtmd_context {
|
||||
struct clip_ctx * ctx_a; // audio
|
||||
std::vector<float> out_embd; // image embedding vector
|
||||
|
||||
// generation context
|
||||
struct clip_ctx * ctx_gen_a; // audio
|
||||
std::vector<int32_t> gen_out_codes; // this frame's 16 sampled codes (GEN_CODE)
|
||||
std::vector<float> gen_out_embd; // next-step hidden state fed back to backbone (GEN_CODE)
|
||||
std::vector<float> gen_out_audio; // decoded PCM samples for the current frame (GEN_WAV)
|
||||
std::vector<uint8_t> gen_out_state; // state to feed into the next GEN_WAV call
|
||||
|
||||
bool print_timings;
|
||||
int n_threads;
|
||||
std::string media_marker;
|
||||
@@ -354,6 +361,7 @@ struct mtmd_context {
|
||||
auto res = clip_init(mmproj_fname, ctx_clip_params);
|
||||
ctx_v = res.ctx_v;
|
||||
ctx_a = res.ctx_a;
|
||||
ctx_gen_a = res.ctx_gen_a;
|
||||
if (!ctx_v && !ctx_a) {
|
||||
throw std::runtime_error(string_format("Failed to load CLIP model from %s\n", mmproj_fname));
|
||||
}
|
||||
@@ -378,6 +386,15 @@ struct mtmd_context {
|
||||
"hint: you may be using wrong mmproj\n",
|
||||
n_embd_text, n_embd_clip));
|
||||
}
|
||||
if (ctx_gen_a) {
|
||||
int n_embd_gen = clip_n_mmproj_embd(ctx_gen_a);
|
||||
if (n_embd_text > 0 && n_embd_text != n_embd_gen) {
|
||||
throw std::runtime_error(string_format(
|
||||
"mismatch between text model (n_embd = %d) and gen-audio mmproj (n_embd = %d)\n"
|
||||
"hint: you may be using wrong mmproj\n",
|
||||
n_embd_text, n_embd_gen));
|
||||
}
|
||||
}
|
||||
if (ctx_v) {
|
||||
init_vision();
|
||||
}
|
||||
@@ -740,6 +757,10 @@ struct mtmd_context {
|
||||
aud_end = "<|mimo_audio_end|>";
|
||||
audio_preproc = std::make_unique<mtmd_audio_preprocessor_mimo_audio>(ctx_a);
|
||||
} break;
|
||||
case PROJECTOR_TYPE_QWEN3TTS_SPKENC:
|
||||
{
|
||||
audio_preproc = std::make_unique<mtmd_audio_preprocessor_qwen3tts_spk>(ctx_a);
|
||||
} break;
|
||||
default:
|
||||
throw std::runtime_error(string_format("%s: unexpected audio projector type %d\n", __func__, proj));
|
||||
}
|
||||
@@ -780,6 +801,7 @@ struct mtmd_context {
|
||||
~mtmd_context() {
|
||||
clip_free(ctx_a);
|
||||
clip_free(ctx_v);
|
||||
clip_free(ctx_gen_a);
|
||||
}
|
||||
|
||||
private:
|
||||
@@ -1553,6 +1575,125 @@ float * mtmd_get_output_embd(mtmd_context * ctx) {
|
||||
return ctx->out_embd.data();
|
||||
}
|
||||
|
||||
//
|
||||
// audio generation
|
||||
//
|
||||
|
||||
mtmd_gen_audio_info mtmd_gen_audio_get_info(const mtmd_context * ctx) {
|
||||
mtmd_gen_audio_info info;
|
||||
if (!ctx->ctx_gen_a) {
|
||||
info.type = MTMD_GEN_AUDIO_TYPE_NONE;
|
||||
return info;
|
||||
}
|
||||
switch (clip_get_projector_type(ctx->ctx_gen_a)) {
|
||||
case PROJECTOR_TYPE_QWEN3TTS_GEN:
|
||||
info.type = MTMD_GEN_AUDIO_TYPE_QWEN3TTS;
|
||||
info.sample_rate = 24000;
|
||||
break;
|
||||
default:
|
||||
info.type = MTMD_GEN_AUDIO_TYPE_NONE;
|
||||
break;
|
||||
}
|
||||
return info;
|
||||
}
|
||||
|
||||
static int32_t mtmd_gen_audio_process_impl(mtmd_context * ctx, const mtmd_gen_inp * inp, mtmd_gen_out * out) {
|
||||
clip_ctx * ctx_clip = ctx->ctx_gen_a;
|
||||
if (!ctx_clip) {
|
||||
LOG_ERR("%s: model does not support audio generation\n", __func__);
|
||||
return 1;
|
||||
}
|
||||
|
||||
if (inp->type == MTMD_GEN_PROCESS_TYPE_GEN_CODE) {
|
||||
const size_t n_embd = (size_t) clip_n_mmproj_embd(ctx_clip);
|
||||
|
||||
clip_image_f32 hidden_state;
|
||||
hidden_state.set_size({(int) n_embd, 1}, false, true);
|
||||
hidden_state.cpy_buf(std::vector<float>(inp->embd, inp->embd + n_embd));
|
||||
|
||||
clip_image_f32_batch batch;
|
||||
batch.is_audio = true;
|
||||
batch.entries.push_back(std::move(hidden_state));
|
||||
|
||||
std::vector<float> out_embd(n_embd);
|
||||
std::vector<int32_t> out_codes;
|
||||
|
||||
clip_encode_params params;
|
||||
params.imgs = &batch;
|
||||
params.n_threads = ctx->n_threads;
|
||||
params.gen_process = CLIP_GEN_PROCESS_GEN_CODE;
|
||||
params.out_embd = &out_embd;
|
||||
params.out_codes = &out_codes;
|
||||
params.code0 = inp->code0;
|
||||
params.top_k = inp->top_k;
|
||||
params.top_p = inp->top_p;
|
||||
|
||||
if (!clip_encode(ctx_clip, ¶ms)) {
|
||||
LOG_ERR("%s: clip_encode failed (gen_code)\n", __func__);
|
||||
return 1;
|
||||
}
|
||||
|
||||
ctx->gen_out_embd = std::move(out_embd);
|
||||
ctx->gen_out_codes = std::move(out_codes);
|
||||
|
||||
out->embd = ctx->gen_out_embd.data();
|
||||
out->codes = ctx->gen_out_codes.data();
|
||||
out->n_codes = ctx->gen_out_codes.size();
|
||||
return 0;
|
||||
}
|
||||
|
||||
// MTMD_GEN_PROCESS_TYPE_GEN_WAV
|
||||
if (!inp->codes || inp->n_codes == 0) {
|
||||
LOG_ERR("%s: codes required for gen_wav\n", __func__);
|
||||
return 1;
|
||||
}
|
||||
std::vector<int32_t> in_codes(inp->codes, inp->codes + inp->n_codes);
|
||||
std::vector<uint8_t> in_state;
|
||||
if (inp->state_data) {
|
||||
in_state.assign(inp->state_data, inp->state_data + inp->state_size);
|
||||
}
|
||||
|
||||
// gen_wav has no hidden-state input, the batch entry is an unused placeholder
|
||||
// TODO @ngxson : some models in the future may require hidden-state input, need to update this code later
|
||||
clip_image_f32 dummy;
|
||||
dummy.set_size({1, 1}, false, true);
|
||||
dummy.cpy_buf(std::vector<float>(1, 0.0f));
|
||||
|
||||
clip_image_f32_batch batch;
|
||||
batch.is_audio = true;
|
||||
batch.entries.push_back(std::move(dummy));
|
||||
|
||||
clip_encode_params params;
|
||||
params.imgs = &batch;
|
||||
params.n_threads = ctx->n_threads;
|
||||
params.gen_process = CLIP_GEN_PROCESS_GEN_WAV;
|
||||
params.codes = &in_codes;
|
||||
params.out_audio = &ctx->gen_out_audio;
|
||||
params.state_in = inp->state_data ? &in_state : nullptr;
|
||||
params.state_out = &ctx->gen_out_state;
|
||||
|
||||
if (!clip_encode(ctx_clip, ¶ms)) {
|
||||
LOG_ERR("%s: clip_encode failed (code2wav)\n", __func__);
|
||||
return 1;
|
||||
}
|
||||
|
||||
out->audio = ctx->gen_out_audio.data();
|
||||
out->n_samples = ctx->gen_out_audio.size();
|
||||
out->state_data = (const char *) ctx->gen_out_state.data();
|
||||
out->state_size = ctx->gen_out_state.size();
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
int32_t mtmd_gen_audio_process(mtmd_context * ctx, const struct mtmd_gen_inp * inp, struct mtmd_gen_out * out) {
|
||||
try {
|
||||
return mtmd_gen_audio_process_impl(ctx, inp, out);
|
||||
} catch (const std::exception & e) {
|
||||
LOG_ERR("%s: error: %s\n", __func__, e.what());
|
||||
return 1;
|
||||
}
|
||||
}
|
||||
|
||||
mtmd_batch * mtmd_batch_init(mtmd_context * ctx) {
|
||||
return new mtmd_batch(ctx);
|
||||
}
|
||||
|
||||
@@ -327,6 +327,60 @@ struct mtmd_caps {
|
||||
};
|
||||
MTMD_API struct mtmd_caps mtmd_get_cap_from_file(const char * mmproj_fname);
|
||||
|
||||
/////////////////////////////////////////
|
||||
// EXPERIMENTAL API for audio generation, subjected to breaking changes
|
||||
|
||||
// represent the pipeline type
|
||||
enum mtmd_gen_audio_type {
|
||||
MTMD_GEN_AUDIO_TYPE_NONE, // not supported
|
||||
MTMD_GEN_AUDIO_TYPE_QWEN3TTS,
|
||||
};
|
||||
struct mtmd_gen_audio_info {
|
||||
enum mtmd_gen_audio_type type;
|
||||
int32_t sample_rate; // in Hz, for example 24000 for qwen3tts
|
||||
};
|
||||
MTMD_API struct mtmd_gen_audio_info mtmd_gen_audio_get_info(const mtmd_context * ctx);
|
||||
|
||||
enum mtmd_gen_process_type {
|
||||
MTMD_GEN_PROCESS_TYPE_GEN_CODE, // h_state to semantic (codes, mel-spectrogram, etc.)
|
||||
MTMD_GEN_PROCESS_TYPE_GEN_WAV, // convert semantic to PCM audio
|
||||
// for qwen3tts, this is code2wav
|
||||
};
|
||||
struct mtmd_gen_inp {
|
||||
enum mtmd_gen_process_type type;
|
||||
|
||||
// for MTMD_GEN_PROCESS_TYPE_GEN_CODE
|
||||
int32_t code0; // the sampled codebook 0 entry from backbone
|
||||
float * embd; // the hidden state from backbone, must have n_text_embd elements
|
||||
int32_t top_k;
|
||||
float top_p;
|
||||
|
||||
// for MTMD_GEN_PROCESS_TYPE_GEN_WAV
|
||||
int32_t * codes;
|
||||
size_t n_codes;
|
||||
const char * state_data;
|
||||
size_t state_size;
|
||||
};
|
||||
struct mtmd_gen_out {
|
||||
// note: output memory is allocated by the context, valid until next process() call
|
||||
|
||||
// for MTMD_GEN_PROCESS_TYPE_GEN_CODE
|
||||
const int32_t * codes;
|
||||
size_t n_codes;
|
||||
const float * embd; // the generated hidden state, to be fed back to backbone
|
||||
// it must have n_text_embd elements
|
||||
|
||||
// for MTMD_GEN_PROCESS_TYPE_GEN_WAV
|
||||
const float * audio;
|
||||
size_t n_samples;
|
||||
const char * state_data;
|
||||
size_t state_size;
|
||||
};
|
||||
// note: this API is stateless, caller must handle state management and audio frame accumulation
|
||||
MTMD_API int32_t mtmd_gen_audio_process(mtmd_context * ctx,
|
||||
const struct mtmd_gen_inp * inp,
|
||||
struct mtmd_gen_out * out);
|
||||
|
||||
/////////////////////////////////////////
|
||||
|
||||
// test function, to be used in test-mtmd-c-api.c
|
||||
|
||||
Reference in New Issue
Block a user