server : pass id_slot through the Anthropic /v1/messages conversion (#28554)

* server : pass id_slot through the Anthropic /v1/messages conversion

server_chat_convert_anthropic_to_oai copies a small whitelist of common
params from the Anthropic request into the internal OAI-style body and
left id_slot out. Every other completion endpoint honours id_slot; a
proxy that pins a session to the slot holding its KV cache needs the
same on /v1/messages, or it silently falls back to automatic slot
selection and, on hybrid/recurrent models, pays a full re-prefill.

Adds a test asserting a request with id_slot set is served by that
slot, observed via GET /slots.

* server : shorten comments

* rm test

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
This commit is contained in:
Peter van VelzenandXuan Son Nguyen authored and GitHub committed 2026-10-20 13:23:41 +02:00
1 parent 704366d76f
commit d80619397c
1 file changed
+1 -1
+1 -1
View File
@@ -590,7 +590,7 @@ json server_chat_convert_anthropic_to_oai(const json & body) {
}
// Pass through common params
for (const auto & key : {"temperature", "top_p", "top_k", "stream", "chat_template_kwargs"}) {
for (const auto & key : {"temperature", "top_p", "top_k", "stream", "chat_template_kwargs", "id_slot"}) {
if (body.contains(key)) {
oai_body[key] = body.at(key);
}