mtmd: drama3 voice cloning (#32586)

* mtmd: drama3 voice cloning

The Fish S2 codec encoder (fish_s2_spkenc) now also serves drama3:
- conversion: drama3 writes the codec encoder and the audio encoder KV, shared with fish_s2
- fish_s2_spkenc: outputs the codes of the clip (gen-side reference) and, in a drama3_gen mmproj,
  builds the drama3 backbone rows (E_lm[semantic] + E_cb[1][code 1]) / sqrt(2)
- drama3_gen tower: teacher forcing when the caller gives the codes of the next frame
- drama3 pipeline: reference-bracket-v1 prompt with one reference (vLLM-Omni fish_omni), the
  reference frames go through the backbone as codec rows and through the tower with their codes
- mtmd_gen_state_get_ref_codes(): read-only access to the reference codes of a gen state

Claude-Session: https://claude.ai/code/session_01PpX3DmaxNSmJbxGaxnUBrX

* fix correctness

* clean up

* rm redundant code

* fish_s2: rm unused import

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
This commit is contained in:
FeitengandXuan Son Nguyen authored and GitHub committed 2026-11-20 18:34:52 +01:00
1 parent 6f5c9ad217
commit 34ff2e047c
12 files changed
+414 -193

No files matched your search

+4
View File
@@ -1171,6 +1171,10 @@ struct mtmd_model_loader {
hparams.wav_tfm_n_head_kv = 16;
hparams.wav_tfm_eps = 1e-5f;
hparams.wav_tfm_rope_theta = 10000.0f;
// drama3 shares the codec, only the backbone rows of a frame differ
std::string gen_proj;
get_string(KEY_GEN_AUDIO_PROJ_TYPE, gen_proj, false);
hparams.spkenc_drama3_rows = projector_type_from_string(gen_proj) == PROJECTOR_TYPE_DRAMA3_GEN;
} break;
case PROJECTOR_TYPE_CHATTERBOX:
{