test: move tools/parser to tests (#27548)

This commit is contained in:
Xuan-Son Nguyen
2026-08-23 18:38:51 +02:00
committed by GitHub
parent 56db501e73
commit 4a08fa2970
9 changed files with 465 additions and 504 deletions
+7 -7
View File
@@ -443,21 +443,21 @@ Each returned parser is wrapped by `wrap_for_generation_prompt()`, which prepend
| | `wrap_for_generation_prompt()`, string helpers |
| `common/chat-peg-parser.h/cpp` | `common_chat_peg_builder`, `common_chat_peg_mapper`, and helpers |
| `common/chat.cpp` | Entry point: `common_chat_templates_apply_jinja()` |
| `tools/parser/debug-template-parser.cpp` | Debug tool for template analysis |
| `tools/parser/template-analysis.cpp` | Template analysis tool |
| `tests/test-chat-auto-parser.cpp` | Auto-parser unit tests; also a debug tool when given a template path |
| `tests/test-chat-analysis.cpp` | Template differential analysis debug tool |
## Testing & Debugging
### Debug Tools
**Template Debugger**: `tools/parser/debug-template-parser.cpp`
**Template Debugger**: `tests/test-chat-auto-parser.cpp`
- Usage: `./bin/llama-debug-template-parser path/to/template.jinja`
- Usage: `./bin/test-chat-auto-parser path/to/template.jinja` (without a path, it runs the automated tests)
- Shows detected format, markers, generated parser, and GBNF grammar
**Template Analysis**: `tools/parser/template-analysis.cpp`
**Template Analysis**: `tests/test-chat-analysis.cpp`
- Usage: `./bin/llama-template-analysis path/to/template.jinja`
- Usage: `./bin/test-chat-analysis --template-file path/to/template.jinja` (without arguments, it runs on all templates from the test suite)
**Debug Logging**: Enable with `LLAMA_ARG_LOG_VERBOSITY=2`
@@ -519,7 +519,7 @@ The following templates have active tests in `tests/test-chat.cpp`:
To support a new template format:
1. **If it follows standard patterns** — The auto-parser should detect it automatically. Run `llama-debug-template-parser` to verify markers are correctly extracted.
1. **If it follows standard patterns** — The auto-parser should detect it automatically. Run `test-chat-auto-parser <template_path>` to verify markers are correctly extracted.
2. **If differential analysis extracts incorrect markers** — Add a workaround lambda to the `workarounds` vector in `common/chat-diff-analyzer.cpp`. Inspect the template source for a unique identifying substring.
3. **If it needs fundamentally different handling** — Add a dedicated handler function in `chat.cpp` before the auto-parser block (as done for GPT-OSS, Functionary v3.2, and Ministral).