server: use port 9931 by default (#30159)
* server: use port 9931 by default * revert unrelated changes
This commit is contained in:
1 parent
b013e56a71
commit
b42b7e6d30
40 files changed
+68
-66
No files matched your search
@@ -282,7 +282,7 @@ This table can be generated with:
|
||||
|
||||
# Usage - need tool-aware Jinja template
|
||||
|
||||
First, start a server with any model, but make sure it has a tools-enabled template: you can verify this by inspecting the `chat_template` or `chat_template_tool_use` properties in `http://localhost:8080/props`).
|
||||
First, start a server with any model, but make sure it has a tools-enabled template: you can verify this by inspecting the `chat_template` or `chat_template_tool_use` properties in `http://localhost:9931/props`).
|
||||
|
||||
Here are some models known to work (w/ chat template override when needed):
|
||||
|
||||
@@ -336,7 +336,7 @@ To get the official template from original HuggingFace repos, you can use [scrip
|
||||
Test in CLI (or with any library / software that can use OpenAI-compatible API backends):
|
||||
|
||||
```bash
|
||||
curl http://localhost:8080/v1/chat/completions -d '{
|
||||
curl http://localhost:9931/v1/chat/completions -d '{
|
||||
"model": "gpt-3.5-turbo",
|
||||
"tools": [
|
||||
{
|
||||
@@ -366,7 +366,7 @@ curl http://localhost:8080/v1/chat/completions -d '{
|
||||
}'
|
||||
|
||||
|
||||
curl http://localhost:8080/v1/chat/completions -d '{
|
||||
curl http://localhost:9931/v1/chat/completions -d '{
|
||||
"model": "gpt-3.5-turbo",
|
||||
"messages": [
|
||||
{"role": "system", "content": "You are a chatbot that uses tools/functions. Dont overthink things."},
|
||||
|
||||
Reference in new issue
Block a user