Failed webRTC upgrade leads to dropped relay p2p-circuit connection

[julienmalard]

Version:

• "@libp2p/circuit-relay-v2": "^3.2.4",
• "@libp2p/webrtc": "^5.2.5",
• "libp2p": "^2.7.4",

Platform:

• macOS 12.5.1
• Firefox browser (135.0.1 (64 bits))
• Chrome browser (Version 133.0.6943.127 (Build officiel) (x86_64))

• Subsystem:
  webRTC and Circuit Relay

Severity:

High - The main functionality of the application does not work, API breakage, repo format breakage, etc.

Description:

When webRTC connections fail (e.g., because of a corporate firewall), the connection between the two peers is dropped instead of falling back to the relay server. This leads to an infinite loop, where both peers connect to the relay, connect to each other over their respective p2p-circuit addresses, attempt to upgrade to webRTC, and then drop the connection and start all over again.

I would expect that, if the upgrade fails, the peers would remain on the relayed connected (and potentially try to upgrade again eventually, but without dropping the relayed connection if webRTC fails).

I wonder if this is perhaps related to the fix for #2646.

Steps to reproduce the error:

Start two browser nodes (same machine, in my test) with webRTC and circuit relay enabled and a known circuit relay node in the bootstrap list over a network with a firewall that will prevent ICE from working.

[NiKrause]

I had the experience that dcutr (upgrade) fails. It reports missing multi addresses, when enabling libp2p:dcutr in the DEBUG log. In my case it does not lead to an infinite loop. It just drops the connection.

A relay connection via circuit-relay is only established for about 120 seconds and not longer. (limited connection)

[julienmalard]

In my case, webrtc failed due to an office firewall. However, I would expect the connection to fall back to the initial relay in such a case.

[NiKrause]

Your expectation is understandable and I would agree, on the other hand I'd like to add that in the past, in my opinion, a browser could operate over a relay connection for ever without upgrading to direct peer-to-peer via WebRTC.

It looks like to me now, and here I might be completely wrong, that for some reason with latest libp2p, the relay connection itself lives only for 120 seconds by default and if not upgraded, it will die.

[julienmalard]

@NiKrause That was also my experience in the past. I believe that this might be linked to the following fix for a different issue: #2646.

[NiKrause]

@julienmalard In my case, I completely forgot about the fact when using circuit-relay, I have at least two multiaddresses per transport and interface which I could dial to:
a) /circuit-relay/p2p and a
b) /p2p-circuit/webrtc/p2p/

In the past @libp2p/pubsub-peer-discovery package was automatically connecting the discovered peers and I didn't have to worry about such, but now I need to figure out which one to dial manually.

If browser b connects to variant a) it will drop after 120 seconds
variant b) when dialing it would upgrade to a direct webrtc connection to that peer but only when dcutr() is activated.

I could imagine a case where you are connecting to a /circuit-relay/p2p variant multiaddress instead of /p2p-circuit/webrtc/p2p/ and that is why it loops.

So all in all, I don't think that my case is necessarily related to your observation and your case, but remembering there are two different kind of multiaddresss solved my issue by just dialing the /p2p-circuit/webrtc/p2p/ addresses now in order to get an upgraded connection.

I have a network where I could test your case, but it will be earlier then in two weeks.

My questions to your question:

1. Falling back to other addresses when one transport fails, is that the PeerStore which stores all the multiaddresses of a peer?
2. Is maybe ConnectionManager responsible to dial addresses when connections fall below a certain treshold?
3. I am not sure if dcutr() is doing a fallback itself.

[achingbrain]

It looks like to me now, and here I might be completely wrong, that for some reason with latest libp2p, the relay connection itself lives only for 120 seconds by default and if not upgraded, it will die.

This is a feature of Circuit Relay v2 - every node on the network can act as a relay server, but to prevent abuse these relay connections are strictly limited in terms of data and duration, so the connection dying abruptly like this is the relay server closing it after the limit has been reached.

In the past @libp2p/pubsub-peer-discovery package was automatically connecting the discovered peers and I didn't have to worry about such, but now I need to figure out which one to dial manually.

If you are figuring this out in the peer:discovery event handler you should just dial the peer id rather than the multiaddrs, libp2p will order the addresses "best" to "worst" before dialing and will attempt to create a direct WebRTC connection before a relayed one.

is that the PeerStore which stores all the multiaddresses of a peer?

Yes, the peerstore has all of the known addresses but again, you should just dial the peer id.

Is maybe ConnectionManager responsible to dial addresses when connections fall below a certain threshold?

No, because the connection manager has no concept of which peers are important and which are not - individual protocols should ensure they are connected to the peers that matter to them.

I am not sure if dcutr() is doing a fallback itself.

DCUtR attempts to make a direct connection but doesn't use WebRTC, it's more for QUIC and TCP connections.

I would expect that, if the upgrade fails, the peers would remain on the relayed connected (and potentially try to upgrade again eventually, but without dropping the relayed connection if webRTC fails).

This should happen if you dial the peer id rather than individual addresses.

[NiKrause]

on my side the issue is clear. Dialing PeerIds was important info! Thanks.

[achingbrain]

I'm going to close this as I think dialling the peer id rather than individual addresses is what should be happening here.

@julienmalard please re-open if this is still broken for you.

[julienmalard]

Thanks @achingbrain @NiKrause
In our code, we were indeed dialing the peer id, though I have yet to reproduce this with the newest code. It was highly context-specific, though, and only occured on a work network with a firewall that didn't let direct p2p communication through.