Lumpiasty/llama.cpp
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

server: Adding spec-decode counters to /metrics endpoint (#26389)

Browse Source 
* * server: add spec-decode counters to /metrics endpoint

* server: fixed review comments and now aligned param names exactly with vLLM.
This commit is contained in:
Bhavik Sharda
2026-08-05 12:36:01 +02:00
committed by GitHub
parent 020760adfc
commit a035a88878
4 changed files with 58 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tools/server/README.md
+4
View File
@@ -1076,6 +1076,10 @@ In *router mode* the query param `?model={model_id}` has to be set. This endpoin
| `llamacpp:n_tokens_max` | Counter | High watermark of the context size observed. |
| `llamacpp:n_decode_total` | Counter | Total Number of llama_decode() calls. |
| `llamacpp:n_busy_slots_per_decode` | Gauge | Average number of busy slots per llama_decode() call. |
| `llamacpp:spec_decode_num_draft_tokens_total` | Counter | Total draft tokens generated (0 when spec-decode is off). |
| `llamacpp:spec_decode_num_accepted_tokens_total` | Counter | Total draft tokens accepted by the target model (0 when spec-decode is off). |
| `llamacpp:spec_decode_num_drafts_total` | Counter | Total speculative decoding verification steps (0 when spec-decode is off). |
| `llamacpp:spec_decode_num_accepted_tokens_per_pos_total` | Counter | Accepted tokens per draft position (labeled `position="N"`; absent when spec-decode is off or before the first completed speculative request). |
### POST `/slots/{id_slot}?action=save`: Save the prompt cache of the specified slot to a file.