15 KiB
Parsing Model Output
The common library contains a PEG parser implementation suitable for parsing
model output.
Types with the prefix common_peg_* are intended for general use and may have
applications beyond parsing model output, such as parsing user-provided regex
patterns.
Types with the prefix common_chat_peg_* are specialized helpers for model
output.
The parser features:
- Partial parsing of streaming input
- Built-in JSON parsers
- AST generation with semantics via "tagged" nodes
Example
Below is a contrived example demonstrating how to use the PEG parser to parse output from a model that emits arguments as JSON.
auto parser = build_chat_peg_parser([&](common_chat_peg_builder & p) {
// Build a choice of all available tools
auto tool_choice = p.choice();
for (size_t i = 0; i < tools.size(); i++) {
const auto & tool = tools[i];
const auto & function = tool.at("function");
std::string name = function.at("name");
const auto schema = common_chat_tool_parameters(function);
auto tool_name = p.json_member("name", "\"" + p.literal(name) + "\"");
auto tool_args = p.json_member("arguments", p.schema(p.json(), "tool-" + std::to_string(i) + "-schema", schema));
tool_choice |= p.rule("tool-" + std::to_string(i), "{" << tool_name << "," << tool_args << "}");
}
// Define the tool call structure: <tool_call>[{tool}]</tool_call>
auto tool_call = p.trigger_rule("tool-call",
p.sequence({
p.literal("<tool_call>["),
tool_choice,
p.literal("]</tool_call>")
})
);
// Parser accepts content, optionally followed by a tool call
return p.sequence({
p.content(p.until("<tool_call>")),
p.optional(tool_call),
p.end()
});
});
For a more complete example, see test_example_native() in
tests/test-chat-peg-parser.cpp.
Parsers/Combinators
Basic Matchers
eps()- Matches nothing and always succeeds (epsilon/empty match)start()- Matches the start of input (anchor^)end()- Matches the end of input (anchor$)literal(string)- Matches an exact literal stringany()- Matches any single character (.)token(piece)- Matches a special token, or the literalpieceif it is not registered with the builder (see Special Tokens)token(pieces, negate)- Matches one of the tokens, or withnegateone token that is not one of them
Combinators
sequence(...)- Matches parsers in order; all must succeedchoice(...)- Matches the first parser that succeeds from alternatives (ordered choice)one_or_more(p)- Matches one or more repetitions (+)zero_or_more(p)- Matches zero or more repetitions (*)optional(p)- Matches zero or one occurrence (?)repeat(p, min, max)- Matches between min and max repetitions (use-1for unbounded)repeat(p, n)- Matches exactly n repetitions
Lookahead
peek(p)- Positive lookahead: succeeds if parser succeeds without consuming input (&)negate(p)- Negative lookahead: succeeds if parser fails without consuming input (!)
Character Classes & Utilities
chars(classes, min, max)- Matches repetitions of characters from a character classspace()- Matches zero or more whitespace characters (space, tab, newline)until(delimiter)- Matches characters until delimiter is found (delimiter not consumed)until_one_of(delimiters)- Matches characters until any delimiter in the list is founduntil(parser)- Matches characters until a delimiter built from literals and tokens is found (delimiter not consumed, see until(parser))through(content, branches, optional)- Matches content, then the first delimiter found and the rest of its branch (delimiter consumed). The grammar scans for the delimiters instead of following contentrest()- Matches everything remaining (.*)
JSON Parsers
json()- Complete JSON parser (objects, arrays, strings, numbers, booleans, null)json_object()- JSON object parserjson_array()- JSON array parserjson_string()- JSON string parserjson_number()- JSON number parserjson_bool()- JSON boolean parserjson_null()- JSON null parserjson_string_content()- JSON string content without surrounding quotesjson_member(key, p)- JSON object member with specific key and value parser
Grammar Building
ref(name)- Creates a lightweight reference to a named rule (for recursive grammars)rule(name, p, trigger)- Creates a named rule and returns a referencetrigger_rule(name, p)- Creates a trigger rule (entry point for lazy grammar generation)trigger_rule(name, start, rest)- Creates a trigger rule matchingstartthenrest, gating the lazy grammar on the startschema(p, name, schema, raw)- Wraps parser with JSON schema metadata for grammar generationschema(p, name, doc, node, raw)- Same, for a node of acommon_chat_schema_documentbuilt earlier, e.g. one tool parameter
AST Control
atomic(p)- Prevents AST node creation for partial parsestag(tag, p)- Creates AST nodes with semantic tags (multiple nodes can share tags)
Special Tokens
Matching a special token by its text alone is ambiguous: a model may emit
<|end|> as a single control token, or spell it out as plain text, such as
when quoting a chat template. The parser can tell them apart when the builder
knows the special tokens of the vocabulary, and the input carries the ids of
the tokens that produced it.
A common_peg_tokens table constructed from a llama_vocab registers every
control and user-defined token. Pass it to the builder:
common_peg_tokens specials(vocab);
auto parser = build_chat_peg_parser(specials, [&](common_chat_peg_builder & p) {
return p.content(p.until(p.token("<|end|>"))) + p.token("<|end|>");
});
Chat templates initialized with a model build this table from its vocabulary,
and format handlers receive it as inputs.specials.
token()
token(piece) matches only where the input holds the id of that token, and
consumes the whole piece. The same characters without the id do not match.
If piece is not registered with the builder, it falls back to
literal(piece).
token(pieces) matches any of the tokens, and token(pieces, true) matches one
token that is not in the set. Every piece of a negated set must be registered,
otherwise it throws std::invalid_argument.
In the grammar, a token emits <[id]> and a negated set emits !<[...]>:
p.token({ "<a>", "<b>" }) + p.token({ "<a>", "<b>" }, true)
root ::= (<[1]> | <[2]>) !<[1-2]>
until(parser)
until(delimiter) with a parser instead of a string stops at a delimiter that
may contain tokens, or several literals joined in a sequence:
auto end = p.token("<|end|>");
auto final = p.token("<|channel|>") + p.literal("final");
p.reasoning(p.until(end | final))
The delimiter may only be built from:
literal()andtoken()sequence()andchoice()of the abovetag()andatomic()wrapping any of the above
Any other parser throws std::invalid_argument. A choice inside a sequence
expands to every combination, so "a" + ("b" | "d") stops at both ab and
ad.
The grammar scans for the delimiters the same way as until_one_of():
p.until((p.literal("a") + p.literal("b")) | p.literal("cd"))
root ::= until-5
until-5 ::= | [a] until-5-01 | [c] until-5-03 | [^ac] until-5
until-5-01 ::= | [a] until-5-01 | [c] until-5-03 | [^abc] until-5
until-5-03 ::= | [a] until-5-01 | [c] until-5-03 | [^acd] until-5
A token never matches a character rule such as . or [^a], so the scan lets
the other tokens named elsewhere in the grammar through by name. Here <b> is
token 2:
p.until(p.token("<a>") + p.literal("x")) + p.token("<a>") + p.literal("x") + p.token("<b>")
root ::= until-3 <[1]> "x" <[2]>
until-3 ::= | <[1]> until-3-01 | (. | <[2]>) until-3
until-3-01 ::= | <[1]> until-3-01 | ([^x] | <[2]>) until-3
When the delimiters are made of tokens alone, the scan consumes whole tokens with a negated token set instead:
p.until(p.token("<a>")) + p.token("<a>")
root ::= until-1 <[1]>
until-1 ::= | !<[1]> until-1
GBNF Grammar Generation
The PEG parser also acts as a convenient DSL for generating GBNF grammars, with some exceptions.
data.grammar = build_grammar([&](const common_grammar_builder & builder) {
parser.build_grammar(builder, data.grammar_lazy);
});
The notable exception is the negate(p) lookahead parser, which cannot be
defined as a CFG grammar and therefore does not produce a rule. Its usage
should be limited and preferably hidden behind a schema() parser. In many
cases, until(delimiter) or until_one_of(delimiters) is a better choice.
Another limitation is that the PEG parser requires an unambiguous grammar. In
contrast, the llama-grammar implementation can support ambiguous grammars,
though they are difficult to parse.
Lazy Grammars
During lazy grammar generation, only rules reachable from a trigger_rule(p)
are emitted in the grammar. All trigger rules are added as alternations in the
root rule. It is still necessary to define trigger patterns, as the parser has
no interaction with the grammar sampling.
With trigger_rule(name, start, rest), the grammar instead scans for a start and
then constrains the rest. It runs from the first token, so grammar_lazy is false
and no trigger patterns are needed. For example:
auto ab = p.literal("ab");
auto cd = p.literal("cd");
return p.until(ab | cd) + (p.trigger_rule("one", ab, p.literal("x")) | p.trigger_rule("two", cd, p.literal("y")));
root ::= trigger
trigger ::= | [a] trigger-01 | [c] trigger-03 | [^ac] trigger
trigger-01 ::= | [b] one-rest | [a] trigger-01 | [c] trigger-03 | [^abc] trigger
trigger-03 ::= | [d] two-rest | [a] trigger-01 | [c] trigger-03 | [^acd] trigger
one-rest ::= "x"
two-rest ::= "y"
A start may only be a sequence of literals, tokens, and choices of them, possibly wrapped in tags and atomics.
JSON Schema
The schema(p, name, schema, raw) parser will use the json-schema-to-grammar
implementation to generate the grammar instead of the underlying parser.
The raw option emits a grammar suitable for a raw string instead of a JSON
string. In other words, it won't be wrapped in quotes or require escaping
quotes. It only takes effect when the schema may be a string, as reported by
common_chat_schema::may_be_string(), otherwise the JSON grammar is used.
The downside is that it can potentially lead to ambiguous grammars. For
example, if a user provides the pattern ^.*$, the following grammar may be
generated:
root ::= "<arg>" .* "</arg>"
This creates an ambiguous grammar that cannot be parsed by the PEG parser. To
help mitigate this, if .* is found in the pattern, the grammar from the
underlying parser will be emitted instead.
Common AST Shapes for Chat Parsing
Most model output can be placed in one of the following categories:
- Content only
- Tool calling with arguments emitted as a single JSON object
- Tool calling with arguments emitted as separate entities, either XML (Qwen3-Coder, MiniMax M2) or pseudo-function calls (LFM2)
To provide broad coverage,
common/chat-peg-parser.h contains builders and
mappers that help create parsers and visitors/extractors for these types. They
require parsers to tag nodes to conform to an AST "shape". This normalization
makes it easy to extract information and generalize parsing.
Simple
The common_chat_peg_builder builds a simple parser that supports
content-only models with optional reasoning.
reasoning(p)- Tag node for extractingreasoning_contentcontent(p)- Tag node for extractingcontent
build_chat_peg_parser([&](common_chat_peg_parser & p) {
return p.sequence({
p.optional("<think>" + p.reasoning(p.until("</think>")) + "</think>"),
p.content(p.until("<tool_call>")),
p.end()
});
});
Use common_chat_peg_mapper to extract the content. Note that this is already
done for you in common_chat_peg_parser when
chat_format == COMMON_CHAT_FORMAT_PEG_SIMPLE.
auto result = parser.parse(ctx);
common_chat_msg msg;
auto mapper = common_chat_peg_mapper(msg);
mapper.from_ast(ctx.ast, result);
Native
The common_chat_peg_builder builds a native parser suitable for
models that emit tool arguments as a direct JSON object.
reasoning(p)- Tag node forreasoning_contentcontent(p)- Tag node forcontenttool(p)- Tag entirety of a single tool calltool_open(p)- Tag start of a tool calltool_close(p)- Tag end of a tool calltool_id(p)- Tag the tool call ID (optional)tool_name(p)- Tag the tool nametool_args(p)- Tag the tool arguments
build_chat_peg_parser([&](common_chat_peg_builder & p) {
auto get_weather_tool = p.tool(p.sequence({
p.tool_open(p.literal("{")),
p.json_member("name", "\"" + p.tool_name(p.literal("get_weather")) + "\""),
p.literal(","),
p.json_member("arguments", p.tool_args(p.json())),
p.tool_close(p.literal("}"))
}));
return p.sequence({
p.content(p.until("<tool_call>")),
p.literal("<tool_call>"),
get_weather_tool,
p.literal("</tool_call>"),
p.end()
});
});
Constructed
The common_chat_peg_builder builds a constructed parser
suitable for models that emit tool arguments as separate entities, such as XML
tags.
reasoning(p)- Tag node forreasoning_contentcontent(p)- Tag node forcontenttool(p)- Tag entirety of a single tool calltool_open(p)- Tag start of a tool calltool_close(p)- Tag end of a tool calltool_name(p)- Tag the tool nametool_arg(p)- Tag a complete tool argument (name + value)tool_arg_open(p)- Tag start of a tool argumenttool_arg_close(p)- Tag end of a tool argumenttool_arg_name(p)- Tag the argument nametool_arg_string_value(p)- Tag string value for the argumenttool_arg_json_value(p)- Tag JSON value for the argument
build_chat_peg_parser([&](common_chat_peg_builder & p) {
auto location_arg = p.tool_arg(
p.tool_arg_open("<parameter name=\"" + p.tool_arg_name(p.literal("location")) + "\">"),
p.tool_arg_string_value(p.until("</parameter>")),
p.tool_arg_close(p.literal("</parameter>"))
);
auto get_weather_tool = p.tool(p.sequence({
p.tool_open("<function name=\"" + p.tool_name(p.literal("get_weather")) + "\">"),
location_arg,
p.tool_close(p.literal("</function>"))
}));
return p.sequence({
p.content(p.until("<tool_call>")),
p.literal("<tool_call>"),
get_weather_tool,
p.literal("</tool_call>"),
p.end()
});
});