Skip to content

[6/6] Fail exec client operations after disconnect#18027

Merged
aibrahim-oai merged 13 commits intomainfrom
dev/remote-mcp-exec-disconnect
Apr 20, 2026
Merged

[6/6] Fail exec client operations after disconnect#18027
aibrahim-oai merged 13 commits intomainfrom
dev/remote-mcp-exec-disconnect

Conversation

@aibrahim-oai
Copy link
Copy Markdown
Collaborator

@aibrahim-oai aibrahim-oai commented Apr 16, 2026

Summary

  • Reject new exec-server client operations once the transport has disconnected.
  • Convert pending RPC calls into closed errors instead of synthetic server errors.
  • Cover pending read and later write behavior after remote executor disconnect.

Verification

  • just fmt
  • cargo check -p codex-exec-server

Stack

@  #18027 [6/6] Fail exec client operations after disconnect
│
o  #18212 [5/6] Wire executor-backed MCP stdio
│
o  #18087 [4/6] Abstract MCP stdio server launching
│
o  #18020 [3/6] Add pushed exec process events
│
o  #18086 [2/6] Support piped stdin in exec process API
│
o  #18085 [1/6] Add MCP server environment config
│
o  main

@aibrahim-oai aibrahim-oai force-pushed the dev/remote-mcp-exec-disconnect branch from 26e6cb2 to cbf2f50 Compare April 16, 2026 05:10
@aibrahim-oai aibrahim-oai force-pushed the dev/remote-mcp-stdio-tests branch 2 times, most recently from 04edeeb to db17256 Compare April 16, 2026 06:20
@aibrahim-oai aibrahim-oai force-pushed the dev/remote-mcp-exec-disconnect branch from cbf2f50 to a5fb6ce Compare April 16, 2026 06:20
@aibrahim-oai aibrahim-oai force-pushed the dev/remote-mcp-stdio-tests branch from db17256 to aa58931 Compare April 16, 2026 13:55
@aibrahim-oai aibrahim-oai force-pushed the dev/remote-mcp-exec-disconnect branch from a5fb6ce to d73da80 Compare April 16, 2026 13:55
@aibrahim-oai aibrahim-oai force-pushed the dev/remote-mcp-stdio-tests branch from aa58931 to c4a27cd Compare April 16, 2026 13:59
@aibrahim-oai aibrahim-oai force-pushed the dev/remote-mcp-exec-disconnect branch from d73da80 to 82da696 Compare April 16, 2026 13:59
aibrahim-oai added a commit that referenced this pull request Apr 16, 2026
## Summary
- Add an MCP server environment setting with local as the default.
- Thread the default through config serialization, schema generation,
and existing config fixtures.

## Stack
```text
o  #18027 [8/8] Fail exec client operations after disconnect
│
o  #18025 [7/8] Cover MCP stdio tests with executor placement
│
o  #18089 [6/8] Wire remote MCP stdio through executor
│
o  #18088 [5/8] Add executor process transport for MCP stdio
│
o  #18087 [4/8] Abstract MCP stdio server launching
│
o  #18020 [3/8] Add pushed exec process events
│
o  #18086 [2/8] Support piped stdin in exec process API
│
@  #18085 [1/8] Add MCP server environment config
│
o  main
```

Co-authored-by: Codex <noreply@openai.com>
@aibrahim-oai aibrahim-oai force-pushed the dev/remote-mcp-stdio-tests branch from c4a27cd to 1e3bb61 Compare April 16, 2026 15:56
@aibrahim-oai aibrahim-oai force-pushed the dev/remote-mcp-exec-disconnect branch from 82da696 to 16f8737 Compare April 16, 2026 15:56
@aibrahim-oai aibrahim-oai force-pushed the dev/remote-mcp-stdio-tests branch from 1e3bb61 to fa90957 Compare April 16, 2026 16:13
@aibrahim-oai aibrahim-oai force-pushed the dev/remote-mcp-exec-disconnect branch from 16f8737 to 3967571 Compare April 16, 2026 16:13
@aibrahim-oai aibrahim-oai force-pushed the dev/remote-mcp-stdio-tests branch from fa90957 to e420153 Compare April 16, 2026 16:16
@aibrahim-oai aibrahim-oai force-pushed the dev/remote-mcp-exec-disconnect branch 3 times, most recently from b82920d to 418679b Compare April 16, 2026 16:29
@aibrahim-oai aibrahim-oai force-pushed the dev/remote-mcp-stdio-tests branch 2 times, most recently from 580771d to f694598 Compare April 16, 2026 16:42
@aibrahim-oai aibrahim-oai force-pushed the dev/remote-mcp-exec-disconnect branch 2 times, most recently from d2afcaa to d326394 Compare April 16, 2026 16:43
@aibrahim-oai aibrahim-oai force-pushed the dev/remote-mcp-stdio-tests branch from f694598 to c2713b6 Compare April 16, 2026 16:43
@aibrahim-oai aibrahim-oai force-pushed the dev/remote-mcp-exec-disconnect branch from d326394 to a7c7ede Compare April 16, 2026 16:44
@aibrahim-oai aibrahim-oai force-pushed the dev/remote-mcp-stdio-tests branch from c2713b6 to acd96bd Compare April 16, 2026 16:44
aibrahim-oai added a commit that referenced this pull request Apr 16, 2026
## Summary
- Add an explicit stdin mode to process/start.
- Keep normal non-interactive exec stdin closed while allowing
pipe-backed processes.

## Stack
```text
o  #18027 [8/8] Fail exec client operations after disconnect
│
o  #18025 [7/8] Cover MCP stdio tests with executor placement
│
o  #18089 [6/8] Wire remote MCP stdio through executor
│
o  #18088 [5/8] Add executor process transport for MCP stdio
│
o  #18087 [4/8] Abstract MCP stdio server launching
│
o  #18020 [3/8] Add pushed exec process events
│
@  #18086 [2/8] Support piped stdin in exec process API
│
o  #18085 [1/8] Add MCP server environment config
│
o  main
```

Co-authored-by: Codex <noreply@openai.com>
@aibrahim-oai aibrahim-oai force-pushed the dev/remote-mcp-exec-disconnect branch from a7c7ede to f1324a4 Compare April 16, 2026 18:21
@aibrahim-oai aibrahim-oai force-pushed the dev/remote-mcp-stdio-tests branch from acd96bd to 88e5a56 Compare April 16, 2026 18:21
@aibrahim-oai aibrahim-oai force-pushed the dev/remote-mcp-exec-disconnect branch from f1324a4 to f05fcee Compare April 16, 2026 18:32
@aibrahim-oai aibrahim-oai force-pushed the dev/remote-mcp-stdio-tests branch 2 times, most recently from adbbe17 to c970502 Compare April 16, 2026 18:40
@aibrahim-oai aibrahim-oai force-pushed the dev/remote-mcp-exec-disconnect branch from cab9201 to b53f34f Compare April 17, 2026 19:46
@aibrahim-oai aibrahim-oai force-pushed the dev/remote-mcp-executor-stdio-full-ci branch from be849e0 to 0ac58e2 Compare April 17, 2026 20:03
@aibrahim-oai aibrahim-oai force-pushed the dev/remote-mcp-exec-disconnect branch from b53f34f to 31c25bf Compare April 17, 2026 20:04
@aibrahim-oai aibrahim-oai force-pushed the dev/remote-mcp-executor-stdio-full-ci branch from 0ac58e2 to 5615470 Compare April 18, 2026 05:06
@aibrahim-oai aibrahim-oai force-pushed the dev/remote-mcp-exec-disconnect branch from 31c25bf to 7c6544f Compare April 18, 2026 05:06
@aibrahim-oai aibrahim-oai force-pushed the dev/remote-mcp-executor-stdio-full-ci branch from 5615470 to c219aeb Compare April 18, 2026 05:24
@aibrahim-oai aibrahim-oai force-pushed the dev/remote-mcp-exec-disconnect branch from 7c6544f to b8070ad Compare April 18, 2026 05:25
aibrahim-oai and others added 3 commits April 20, 2026 10:42
Reject new exec-server operations once the transport disconnects and convert pending RPC calls into closed errors. This lets remote MCP stdio calls surface executor loss immediately instead of waiting for the MCP tool timeout.

Co-authored-by: Codex <noreply@openai.com>
Preserve reader ordering for in-flight calls so responses queued before EOF are delivered before pending calls are failed as closed.

Co-authored-by: Codex <noreply@openai.com>
@aibrahim-oai
Copy link
Copy Markdown
Collaborator Author

@codex review

Copy link
Copy Markdown
Contributor

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

ExecServerError::Closed => {
io::Error::new(io::ErrorKind::BrokenPipe, "exec-server transport closed")
}
_ => io::Error::other(error.to_string()),

P2 Badge Treat disconnected transport as BrokenPipe in FS mapping

ExecServerClient::call now returns ExecServerError::Disconnected(...) after transport loss, but this mapper only special-cases Closed. As a result, post-disconnect remote filesystem calls fall through to _ and become io::ErrorKind::Other instead of BrokenPipe, changing transport-failure semantics and potentially breaking callers that rely on BrokenPipe handling.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread codex-rs/exec-server/src/client.rs Outdated
// with the same message. This process/filesystem-level latch prevents
// callers from waiting on request-specific timeouts after the environment
// is gone.
disconnected: std::sync::RwLock<Option<String>>,
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can this be a lighter sync primitive?

Use OnceLock for the one-way exec-server disconnect latch instead of a read/write lock.

Co-authored-by: Codex <noreply@openai.com>
Copy link
Copy Markdown
Contributor

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 38dc8ea086

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread codex-rs/exec-server/src/client.rs Outdated
// registered process session before returning this call
// error.
let message = disconnected_message(/*reason*/ None);
let message = mark_disconnected(&self.inner, message).await;
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Defer session drain until disconnect event ordering is preserved

ExecServerClient::call now invokes mark_disconnected on any transport-closed call error. That immediately drains sessions via take_all_sessions(). Since this runs concurrently with the event-reader loop, notifications already queued before Disconnected can be handled after the drain and be dropped by get_session, losing tail output/exit/closed events for active processes.

Useful? React with 👍 / 👎.

aibrahim-oai and others added 2 commits April 20, 2026 14:53
Record executor disconnects from call failures without draining sessions. Keep session failure publication in the reader task so queued process notifications are handled before disconnect.

Co-authored-by: Codex <noreply@openai.com>
@aibrahim-oai
Copy link
Copy Markdown
Collaborator Author

@codex review this

aibrahim-oai and others added 2 commits April 20, 2026 15:01
Treat the latched exec-server disconnect error the same as a closed transport in the remote filesystem adapter.

Co-authored-by: Codex <noreply@openai.com>
Clone the latched disconnect message before converting it into an ExecServerError.

Co-authored-by: Codex <noreply@openai.com>
Copy link
Copy Markdown
Contributor

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7d95be50bc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread codex-rs/exec-server/src/rpc.rs Outdated
Comment on lines +271 to +272
if *self.disconnected_rx.borrow() {
return Err(RpcCallError::Closed);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Delay closed errors until notification queue is drained

RpcClient::call now returns Closed as soon as disconnected_rx flips, but incoming_rx can still hold earlier process notifications. Session::read treats this as terminal and caches a synthetic failure, so a read in this window can report closed=true before queued output/exited/closed events are published. Polling clients may stop and miss tail output.

Useful? React with 👍 / 👎.

aibrahim-oai and others added 3 commits April 20, 2026 15:24
Serve disconnect-time process reads from the local event log so queued output is not hidden by a synthetic failure response.

Co-authored-by: Codex <noreply@openai.com>
Explain why remote reads use the local retained event log after transport close.

Co-authored-by: Codex <noreply@openai.com>
Annotate the new retained-read callsites for the argument comment lint.

Co-authored-by: Codex <noreply@openai.com>
Copy link
Copy Markdown
Contributor

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 53bee5f406

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread codex-rs/exec-server/src/rpc.rs Outdated
Comment on lines 271 to 277
if *self.disconnected_rx.borrow() {
return Err(RpcCallError::Closed);
}

let request_id = RequestId::Integer(self.next_request_id.fetch_add(1, Ordering::SeqCst));
let (response_tx, response_rx) = oneshot::channel();
self.pending
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Atomically gate new RPCs when transport disconnects

RpcClient::call reads disconnected_rx before registering the request in pending. If disconnect flips just after this check, the reader task can process Disconnected, run drain_pending, and exit before this call inserts its sender. write_tx.send may still succeed, leaving response_rx.await with no producer and the call hanging indefinitely.

Useful? React with 👍 / 👎.

aibrahim-oai and others added 2 commits April 20, 2026 15:58
Remove the retained-output recovery path for executor disconnects and return the normal synthesized failure response instead.

Co-authored-by: Codex <noreply@openai.com>
Check the disconnect latch while holding the pending request lock so calls cannot register after the reader drains pending requests.

Co-authored-by: Codex <noreply@openai.com>
@chatgpt-codex-connector
Copy link
Copy Markdown
Contributor

Codex Review: Something went wrong. Try again later by commenting “@codex review”.

An unknown error occurred
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

1 similar comment
@chatgpt-codex-connector
Copy link
Copy Markdown
Contributor

Codex Review: Something went wrong. Try again later by commenting “@codex review”.

An unknown error occurred
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants