Lumpiasty/llama.cpp
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

server: support MCP stdio (#26062)

Browse Source 
* move server_pipe to common

* init impl

* vendor: update subprocess.h

* add server_mcp_stdio

* stderr drain

* server_mcp_transport

* server_mcp_stdio is now framing-only, no json

* internal/mcp-stdio: integration + tests + fixes (#26075)

* server-mcp: harden transport and wire up the tool integration

Builds on the transport/manager architecture (server_mcp_transport + server_pipe)
with the hardening and integration the draft did not yet have.

Hardening:
* Reader and stderr pumps are polled (running-aware) instead of blocking on a read
  that only ends at EOF. subprocess_terminate() SIGKILLs only the direct child, so a
  grandchild the MCP server spawned that inherited the pipe would otherwise keep the
  write end open and hang teardown (both warmup shutdown at startup and process
  shutdown). The writer is likewise non-blocking + polled.
* Windows: resolve the command through PATHEXT so "npx" (npm ships npx.cmd, never
  npx.exe) spawns, matching POSIX's PATH search; and enumerate the parent environment
  as UTF-8 (GetEnvironmentStringsW) instead of the active code page.
* server_pipe gains an opt-in max_size (default unbounded, so the router's streaming
  use is unchanged); the MCP reply queue uses it so a server that streams unsolicited
  notifications between requests cannot grow it without bound.

Integration:
* --mcp-servers-config / --mcp-servers-json flags; enabling MCP restricts default CORS
  to localhost, same as --tools.
* MCP tools are exposed through /tools (and chat-completions) as <server>_<tool>,
  skipping names that collide with a built-in or another MCP tool.
* Manager lifecycle wired into llama_server(): warmup at start, shutdown() from the
  signal handler before the HTTP server drains, blocking teardown in clean_up().
* SIGPIPE ignored so a child dying mid-write yields EPIPE rather than killing us.

Assisted-By: Claude Opus 4.8 <noreply@anthropic.com>

* server-mcp: add MCP test suite with grandchild deadlock regression test

21 tests over the /tools endpoint: tool discovery/invocation, timeouts, crash
recovery and respawn cooldown, warmup partial failure, malformed and batched
notification+response output, tool-definition shape, and prompt shutdown during a
slow call.

The last test spawns an MCP server that leaves a grandchild inheriting its
stdout/stderr and asserts the server both starts and stops promptly. Verified it
fails (5s SIGKILL fallback on a deadlocked reader-join) when the pump is made to
ignore the running flag, and passes with the polled reader.

Assisted-By: Claude Opus 4.8 <noreply@anthropic.com>

* clean up

* clean up 2

* even stricter life cycle

* nits

* nits 2

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>

* fix some edge cases

* fix last_error data race

* fix response schema + docs

* server: fix MCP zombie leak and timeout-induced transport teardown

join_pumps() never reaped the child, leaking one zombie per spawn:
call subprocess_join() before subprocess_destroy().

A per-call timeout permanently closed from_server and got a healthy
transport evicted: add close_on_stop to server_pipe::read() and pass
false from send_rpc(), where should_stop is a per-request deadline
and a late reply is already skipped on id mismatch.

Also drop the unreachable disconnect cancellation in
server_mcp_tool::invoke(): support_stream is false, st is always null.

(cherry picked from commit e6de1ec043174fd0570b1e60d47f06c7c19d620d)

Assisted-by: Claude Opus 4.8

* server: make MCP test fixtures JSON-RPC 2.0 compliant

Add the missing notification guard to mcp_malformed_server.py and
mcp_burst_server.py (the latter treated id 0 as a notification and
replied to unknown ones; its notification table is now unused).

Return -32602 instead of -32601 for unknown tools: tools/call is a
valid method, the tool name is the invalid parameter.

Also fix the test module docstring: tools are named <server>_<tool>.

(cherry picked from commit 74a08e8c311dabf3b49d06cc6d754b0097ae7a38)

Assisted-by: Claude Opus 4.8

---------

Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com>
Co-authored-by: Pascal <admin@serveurperso.com>
This commit is contained in:
Xuan-Son Nguyen
2026-07-26 01:08:49 +02:00
committed by GitHub
co-authored by Piotr Wilkin Pascal
parent 355303edab
commit 20455a4ad3
19 changed files with 2680 additions and 63 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tools/server/server.cpp
+29 -6
View File
@@ -88,6 +88,11 @@ static server_http_context::handler_t ex_wrapper(server_http_context::handler_t
int llama_server(int argc, char ** argv) {
std::setlocale(LC_NUMERIC, "C");
#ifndef _WIN32
// Ignore SIGPIPE so the server does not crash if an MCP child exits while we are writing to its stdin
signal(SIGPIPE, SIG_IGN);
#endif
// own arguments required by this example
common_params params;
@@ -157,6 +162,9 @@ int llama_server(common_params & params, int argc, char ** argv) {
params.model_alias.insert(model_name);
}
// note: this is guaranteed to out-live ctx_http and tools
server_mcp mcp_mgr;
// struct that contains llama context and inference
server_context ctx_server;
@@ -326,17 +334,28 @@ int llama_server(common_params & params, int argc, char ** argv) {
ctx_http.post("/cors-proxy", ex_wrapper(res_403));
}
// EXPERIMENTAL built-in tools
if (!params.server_tools.empty()) {
try {
mcp_mgr.start(params);
} catch (const std::exception & e) {
SRV_ERR("MCP starting failed: %s\n", e.what());
return 1;
}
if (!params.server_tools.empty() || !mcp_mgr.empty()) {
try {
tools.setup(params.server_tools);
tools.setup(params.server_tools, mcp_mgr);
} catch (const std::exception & e) {
SRV_ERR("tools setup failed: %s\n", e.what());
return 1;
}
ctx_http.get ("/tools", ex_wrapper(tools.handle_get));
ctx_http.post("/tools", ex_wrapper(tools.handle_post));
warn_names.push_back("built-in tools (experimental)");
if (!params.server_tools.empty()) {
warn_names.push_back("built-in tools (experimental)");
}
if (!mcp_mgr.empty()) {
warn_names.push_back("MCP servers (experimental)");
}
} else {
ctx_http.get ("/tools", ex_wrapper(res_403));
ctx_http.post("/tools", ex_wrapper(res_403));
@@ -378,7 +397,7 @@ int llama_server(common_params & params, int argc, char ** argv) {
if (is_router_server) {
SRV_INF("%s", "starting server in router mode. models will be automatically loaded on-demand\n");
clean_up = [&models_routes]() {
clean_up = [&models_routes, &mcp_mgr]() {
SRV_INF("%s: cleaning up before exit...\n", __func__);
// stop the session GC first, it finalizes live sessions and wakes pending readers
server_stream_session_manager_stop();
@@ -386,6 +405,7 @@ int llama_server(common_params & params, int argc, char ** argv) {
models_routes->stopping.store(true); // maybe redundant, but just to be safe
models_routes->models.unload_all();
}
mcp_mgr.shutdown();
llama_backend_free();
};
@@ -401,17 +421,19 @@ int llama_server(common_params & params, int argc, char ** argv) {
// important to disconnect any SSE clients
models_routes->stopping.store(true);
}
mcp_mgr.shutdown();
ctx_http.stop();
};
} else {
// setup clean up function, to be called before exit
clean_up = [&ctx_http, &ctx_server]() {
clean_up = [&ctx_http, &ctx_server, &mcp_mgr]() {
SRV_INF("%s: cleaning up before exit...\n", __func__);
// stop the session GC first, it finalizes live sessions and wakes pending readers
server_stream_session_manager_stop();
ctx_http.stop();
ctx_server.terminate();
mcp_mgr.shutdown();
llama_backend_free();
};
@@ -444,6 +466,7 @@ int llama_server(common_params & params, int argc, char ** argv) {
SRV_INF("%s", "model loaded\n");
shutdown_handler = [&](int) {
mcp_mgr.shutdown();
// this will unblock start_loop()
ctx_server.terminate();
};