Unobserved task exception when server is not reachable

[Callum-Shipton]

What version of gRPC and what language are you using?

Grpc.Net.Client = 2.43.0
Grpc.HealthCheck = 2.43.0

What operating system (Linux, Windows,...) and version?

Windows 10, Build 19041

What runtime / compiler are you using (e.g. .NET Core SDK version dotnet --info)

Version: 6.0.201

What did you do?

Ran Vigor health check example with no server running (only client), modified to capture UnobservedTaskException

static async Task Main(string[] args)
{
    TaskScheduler.UnobservedTaskException += (sender, e) => {
        Console.WriteLine("Detected unobserved task exception: " + e.Exception);
    };

    using var channel = GrpcChannel.ForAddress("https://localhost:5001");
    var client = new Health.HealthClient(channel);

    Console.WriteLine("Watching health status");

    var watchTask = Task.Run(async () =>
    {
        try
        {
            var response = await client.CheckAsync(new HealthCheckRequest { Service = "" });
            Console.WriteLine($"{DateTime.Now}: Service is {response.Status}");
        }
        catch (Exception)
        {
            Console.WriteLine("Detected expected exception");
        }
    });

    try
    {
        await watchTask;
    }
    catch (Exception)
    {
        Console.Write("Detected unexpected exception from watchTask");
    }


    Console.WriteLine();
    Console.WriteLine("Waiting for finalizers");

    // Provoke the garbage collector to find the unobserved exception.
    GC.Collect();
    // Wait for any failed tasks to be garbage collected
    GC.WaitForPendingFinalizers();

    Console.WriteLine("Finished");
}

What did you expect to see?

I expected to see the only exception thrown from awaiting client.CheckAsync when it was refused by the target machine

Watching health status
Detected expected exception

Waiting for finalizers
Finished

What did you see instead?

I saw the expected exception but also an UnobservedTaskException after GC had collected a task created within client.CheckAsync

Watching health status
Detected expected exception

Waiting for finalizers
Detected unobserved task exception: System.AggregateException: A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. (Status(StatusCode="Unavailable", Detail="Error connecting to subchannel.", DebugException="System.Net.Sockets.SocketException (10061): No connection could be made because the target machine actively refused it.
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|277_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
   at Grpc.Net.Client.Balancer.Internal.SocketConnectivitySubchannelTransport.TryConnectAsync(CancellationToken cancellationToken)"))
 ---> Grpc.Core.RpcException: Status(StatusCode="Unavailable", Detail="Error connecting to subchannel.", DebugException="System.Net.Sockets.SocketException (10061): No connection could be made because the target machine actively refused it.
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|277_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
   at Grpc.Net.Client.Balancer.Internal.SocketConnectivitySubchannelTransport.TryConnectAsync(CancellationToken cancellationToken)")
   at Grpc.Net.Client.Balancer.Internal.ConnectionManager.PickAsync(PickContext context, Boolean waitForReady, CancellationToken cancellationToken)
   at Grpc.Net.Client.Balancer.Internal.BalancerHttpHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Grpc.Net.Client.Internal.GrpcCall`2.RunCall(HttpRequestMessage request, Nullable`1 timeout)
   at Client.Program.<>c__DisplayClass0_0.<<Main>b__1>d.MoveNext() in C:\Users\Callum.Shipton\source\repos\grpc-dotnet\examples\Vigor\Client\Program.cs:line 48
   --- End of inner exception stack trace ---
Finished

Anything else we should know about?

I have ran this with both unary and streaming calls and they both exhibit the unexpected exception very repeatedly.

I was going to raise this in Grpc.Core but a similar issue was closed (grpc/grpc#22097) for only using Grpc.Net.Client which this also does.
Looks like the issue also affects Grpc.Core (grpc/grpc#24421) but has gone stale
Might also be related to (dotnet/runtime#61411) but I didn't dig deep enough to find out

[JamesNK]

Thanks for the reproduction. It took me a while but I was able to figure out what was going on and have a fix: #1660

[asynts]

I had the exact same error for a different reason, I thought I put a note here for the next person with a similar issue, since this is very high up in the search results.

In my case I was starting multiple tasks and then awaited them subsequently, here some simplified code:

async Task Example()
{
    var client = /* ... */;

    var task1 = client.ExampleAsync(new ExampleRequest { });
    var task2 = client.ExampleAsync(new ExampleRequest { });

    // ...

    await task1;
    await task2;
}

The problem is that if the server is unavailable, then both of the tasks will fail, but this means that only the await task1 is executed and the task2 will cause an unobserved task exception.

Unfortunately, the stack trace was truncated and it originated somewhere in the internals of dotnet-grpc, I was able to diagnose the issue by ignoring the unobserved exception which caused another exception to surface with a better stack trace.