Files
llama.cpp/docs/tts-norm-patterns.md
T
Xuan-Son Nguyen 3574df84b7 mtmd: support tiny-voca (#32427)
* mtmd: support tiny-voca

* improve G2P
2026-11-18 00:04:15 +01:00

9.6 KiB

TTS normalization patterns

Text-to-speech models often need their input normalized before tokenization: numbers, currencies, dates and symbols spelled out, punctuation mapped, spacing fixed. The exact rules are model-specific (they must match what the model was trained on), so they live in the model's tts task template rather than in C++.

Jinja has no regex support, and exposing std::regex / Python re would not help: the two differ in many details (\d, \w, \b, lookbehind, case folding, backtracking limits), so a template could give different results in llama.cpp and in Python. Instead, task templates get two functions with a small matching language whose semantics are defined here:

  • tts_norm_scan(text, rules): split text into matched and unmatched segments
  • tts_norm_sub(text, pattern, repl): replace every match

A third function, tts_dict(name, key), gives the template access to dictionaries stored in the model (e.g. a pronunciation lexicon).

A fourth one, tts_g2p(word), gives the pronunciation of a word by the letter-to-sound model of the mmproj, when the caller provides one (see below).

Implementations:

  • C++: common/chat-tts-norm.cpp, injected into task templates by common_chat_template_render() in common/chat.cpp. They are not available to chat templates.
  • Python reference: scripts/chat_tts_norm.py, for use with jinja2 (env.globals["tts_norm_scan"] = chat_tts_norm.tts_norm_scan)
  • Both are checked against each other by tests/test-jinja.cpp (test-jinja -py "tts norm patterns.*")

Example

{%- for s in tts_norm_scan(text, [
    ['money',   '%$(%d+)(?:%.(%d%d))?'],
    ['percent', '(%d+)%%'],
    ['time',    '(%d%d?):(%d%d)(?!%d)'],
    ['neg',     '(?<!%w)-(%d+)'],
    ['number',  '%d+'],
]) -%}
  {%- if   s.rule == 'money'   -%}{%- set whole, cents = s.groups -%}{{ num(whole) }} dollars{{ ' and ' ~ num(cents) ~ ' cents' if cents else '' }}
  {%- elif s.rule == 'percent' -%}{{ num(s.groups[0]) }} percent
  {%- elif s.rule == 'time'    -%}{%- set h, m = s.groups -%}{{ num(h) }} {{ num(m) }}
  {%- elif s.rule == 'neg'     -%}minus {{ num(s.groups[0]) }}
  {%- elif s.rule == 'number'  -%}{{ num(s.text) }}
  {%- else -%}{{ s.text }}
  {%- endif -%}
{%- endfor -%}

It costs $3.50, 20% off, at 10:45, -5 degrees, 7 cats. becomes It costs three dollars and fifty cents, twenty percent off, at ten forty-five, minus five degrees, seven cats. (with a num macro that spells numbers, see the "mini text normalizer" test in tests/test-jinja.cpp).

Functions

tts_norm_scan(text, rules)

rules is one of:

  • a list of [name, pattern] pairs (recommended, the order is explicit)
  • a dict {name: pattern}, in insertion order
  • a single pattern string, the rule is named match

Scanning starts at the first codepoint. At each position, the rules are tried in order and the first one that produces a non-empty match wins; the scan continues after the match. If no rule matches, one codepoint is skipped. Matches never overlap.

The result is a list of segments covering the whole text, in order. Consecutive unmatched codepoints are merged into one segment. Each segment is a dict:

key matched segment unmatched segment
rule rule name none
text matched text unmatched text
groups list of capture groups, in order of their ( []

There are no named groups, unpack groups instead: {% set whole, cents = s.groups %}.

A group that did not participate in the match (e.g. inside a ? that matched nothing) is an empty string, so it renders as nothing and is falsy.

tts_norm_sub(text, pattern, repl)

Replaces every match of pattern, with the same scanning rules as tts_norm_scan. repl is a literal string with references:

reference value
%0 whole match
%1 .. %9 group
%% a literal %

Any other % sequence in repl, or a reference to a group that does not exist, is an error.

tts_dict(name, key)

Looks up key in the dictionary name of the model and returns its value, or none if the key is missing. An unknown dictionary name is an error. A model can ship any number of dictionaries, for example a pronunciation lexicon per language:

{%- set e = tts_dict('lexicon_en', word) -%}
{%- if e is none -%}{{ spell(word) }}
{%- elif e is mapping -%}{{ e[tag] if tag in e else e.DEFAULT }}
{%- else -%}{{ e }}
{%- endif -%}

The lookup is exact: no case folding and no normalization, do them in the template.

The GGUF lists the names of the dictionaries in tokenizer.tts_dicts. Each one is stored as two arrays of strings of the same length: tokenizer.tts_dict.<name>.keys, sorted by the UTF-8 bytes of the key, and tokenizer.tts_dict.<name>.values. A value is a JSON document (string, number, list, object, ...) and is returned decoded. GGUFWriter.add_tts_dicts(dicts) writes all of this.

tts_g2p(word)

Pronunciation of word by a letter-to-sound model, as phonemes separated by spaces, in the notation of the model (e.g. CMUdict with stress digits for tiny-voca), or none. The template converts it to the symbols of the model. It is none when the model has no letter-to-sound model, or when the caller gives none: common_chat_get_tts_prompt() takes it as an optional callback, llama-tts and llama-server pass mtmd_gen_audio_g2p(). A template must keep working without it, e.g. by spelling the word.

{%- set g = tts_g2p(word) -%}
{%- if g is none -%}{{ spell(word) }}{%- else -%}{{ from_cmudict(word, g.split(' ')) }}{%- endif -%}

There is no Python reference for it: a test passes its own function (env.globals["tts_g2p"]).

Pattern syntax

Patterns operate on Unicode codepoints. Invalid UTF-8 bytes (C++ only) are seen as U+FFFD, one per byte, and preserved in the output.

syntax meaning
x the literal codepoint x
%x escape: % followed by any non-alphanumeric codepoint is that literal codepoint (%., %$, %%, %(, ...)
. any codepoint, including newlines
%d %s %p %a %w %u %l %h %c character classes, see below; the uppercase form (%D, ...) is the negation
[...] set: literal codepoints, ranges a-z, classes %d; [^...] is the negation
(...) capture group
(?:...) group without capture
a|b ordered alternation
? * + {m} {m,} {m,n} quantifiers, greedy and possessive; m, n up to 1000
^ $ start / end of the whole text
%b boundary: between a %w codepoint and a non-%w codepoint (the edges of the text count as non-%w)
(?=...) (?!...) lookahead, positive / negative
(?<=c) (?<!c) lookbehind of exactly one codepoint: c is a literal, a set, a class or .
(?i) at the very start of the pattern only: ignore ASCII case

These characters must be escaped with % to be matched literally: ( ) [ ] { } . | ? * + ^ $ %. Inside a set, only ], %, - (between two codepoints) and ^ (first position) are special.

Matching semantics

The matcher is a PEG, not a backtracking regex:

  • Ordered alternation: alternatives are tried left to right and the first one that matches is committed to. If what follows fails, the other alternatives are not retried. (?:a|ab)c never matches abc; write (?:ab|a)c.
  • Possessive quantifiers: quantifiers match as many repetitions as possible and never give any back. %d+0 never matches; .*x never matches. Write patterns that describe exactly what to consume.
  • A repetition that matches the empty string ends the loop.
  • Captures inside lookaheads are discarded. A capture inside a repeated group holds its last repetition.

This gives a simple and linear-time implementation with identical results in C++ and Python.

Character classes

Classes are defined by explicit codepoint ranges, not by a Unicode database, so the result does not depend on the Unicode version of the platform.

class definition
%d ASCII digits 0-9 only
%s U+0009..U+000D, U+0020, U+0085, U+00A0, U+1680, U+2000..U+200A, U+2028, U+2029, U+202F, U+205F, U+3000
%p punctuation and symbols: ASCII punctuation, U+00A1..U+00BF (except U+00AA, U+00B5, U+00BA), U+00D7, U+00F7, U+2010..U+2027, U+2030..U+205E, U+20A0..U+20CF (currency), U+2100..U+2BFF (letterlike, number forms, arrows, math, technical, box drawing, shapes, misc symbols, dingbats), CJK punctuation (U+3001..U+3003, U+3008..U+3020, U+3030, U+303D, U+30FB), U+FE10..U+FE19, U+FE30..U+FE6B, fullwidth punctuation (U+FF01..U+FF0F, U+FF1A..U+FF20, U+FF3B..U+FF40, U+FF5B..U+FF65), U+1F000..U+1FAFF (emoji)
%c control and invisible format characters: U+0000..U+001F, U+007F..U+009F, U+200B..U+200F, U+2060..U+206F, U+FEFF
%a ASCII letters, U+00AA, U+00B5, U+00BA, and every codepoint from U+00C0 that is not in %s, %p or %c
%w %a or %d
%u / %l ASCII uppercase / lowercase letters only
%h Han: U+2E80..U+2FDF, U+3005, U+3007, U+3021..U+3029, U+3038..U+303B, U+3400..U+4DBF, U+4E00..U+9FFF, U+F900..U+FAFF, U+20000..U+323AF

Classes may overlap (e.g. U+0085 is both %s and %c). %a intentionally includes all scripts, combining marks and CJK, so (?<!%w) and %b work as "not inside a word" for any language.

Errors

Invalid patterns are rejected when the function is called, with a message and a position, for example: unbalanced parentheses, a quantifier with nothing to repeat, multiple quantifiers (a+?), repeating a zero-width assertion (%b+), unknown escapes (%q), unescaped ] or }, an empty set, a lookbehind wider than one codepoint and (?i) not at the start.