`run_actor_network` hangs when a failing actor is not connected to all others

[Matt711]

When one actor in the network fails and shuts down its own channels, any actor that is not connected to those channels has no way to learn that the network is done. It stays blocked on its next channel->receive() or channel->send() waiting for a message or a consumer that will never come.

Reproducer

--- a/cpp/tests/streaming/test_error_handling.cpp
+++ b/cpp/tests/streaming/test_error_handling.cpp
+TEST_F(StreamingErrorHandling, UnconnectedActorCancelledOnFailure) {
+    // Actor A fails immediately.
+    auto ch_b = ctx->create_channel();
+    std::vector<Actor> actors;
+
+    // Actor A: fails immediately.
+    actors.push_back([](Context& ctx) -> Actor {
+        co_await ctx.executor()->schedule();
+        throw std::runtime_error("actor_a_failure");
+    }(*ctx));
+
+    // Actor B is unconnected to A but blocks
+    actors.push_back(
+        [](std::shared_ptr<Channel> ch_b) -> Actor {
+            std::ignore = co_await ch_b->receive();
+        }(ch_b)
+    );
+
+    try {
+        run_actor_network(std::move(actors), ctx);
+        FAIL() << "Expected std::runtime_error";
+    } catch (std::runtime_error const& e) {
+        EXPECT_THAT(e.what(), ::testing::HasSubstr("actor_a_failure"));
+    }
+}

One approach to a fix is to add a method to Context: cancel_network (for example) that broadcasts a shutdown to every channel and memory reservation the context has ever created, and to call it automatically whenever any actor in the network fails.

[wence-]

Unfortunately this is "as designed". Disconnected subgraphs today have no way of communicating to each other.

and to call it automatically whenever any actor in the network fails.

This is a chicken and egg situation today: we can't see that one actor has failed in coro::when_all until that completes (which it won't, because someone failed).

[Matt711]

This is a chicken and egg situation today: we can't see that one actor has failed in coro::when_all until that completes (which it won't, because someone failed).

I agree that is valid for any approach that tries to observe the failure from outside coro::when_all. coro::when_all won't resume until all tasks complete, so you can't react to a failure until it's already too late. But when I said

One approach to a fix is to add a method to Context: cancel_network (for example) that broadcasts a shutdown to every channel and memory reservation the context has ever created, and to call it automatically whenever any actor in the network fails.

What I had in mind was catching the exception from inside the failing actor's own coroutine, before the actor reports completion to coro::when_all. Like some sort of wrapper to do that.

[wence-]

OK, so you want fail-fast, I think.

One thing I would note is that the network you set up in this test is ill-formed (perhaps we should make it by-construction not possible to build). ch_b is received from, but never sent to.

Is this test a distillation of a real network?

[Matt711]

OK, so you want fail-fast, I think.

One thing I would note is that the network you set up in this test is ill-formed (perhaps we should make it by-construction not possible to build). ch_b is received from, but never sent to.

Is this test a distillation of a real network?

Yes and yes. I think a few queries (but definitely Q74). I found this by adding logs to all actors (starting and ending) and saw an exception in the log from a CppActor. Then the hang. That gave me the idea for this MRE.