[Pre2P] Refine the P2P module listening and IO behaviour

[Olshansk]

Objective

Implement and test RainTree's redundancy layer on top of the Pre2P module.

Origin Document

The Pocket Network V1 P2P Specification, supported by this explanation of RainTree is partially implemented in the pre2p/raintree branch atop the Pre2P module at the time of creating this issue.

While #80 is close to being done, #85 was opened to implement the redundancy layer.

Per offline discussions within the core team, we have decided to migrate Pre2P module to be the primary P2P module but transfer all the IO learnings from P2P` atop of it. The existing implementation can be found here: milestone/v1-prototype...integration/module/p2p-simplified-over-9000.

However, there are some major missing components which need to be added per a message from @derrandz:

Listening and IO behavior in Pre2P has consequences:

1. We accept connections sequentially, and we handle them in the same way, no async handling (not suitable for our use case)
2. We read off of connections with the expectation that the sender/writer will immediately close once he's done sending/that the sender is doing fire and dump style of communication (not connection pooling logic atm)

These observations mean:

we won't be able to achieve graceful shutdown with open connections, nor accept incoming connections concurrently or achieve ACK and waiting on ACKs behavior. (and make e2e tests difficult to write)

You can say that those were the main difference between P2P (albeit the approaches are different), as p2p does the following:

1. P2P Pools connections as default behavior and allows for graceful shutdowns.
2. P2P offers functionality to fire and forget with the Write method if you don't need the "connection pool". Send will pool or use a pooled connection. (e.g: the broadcast relies on Write)

The highlighted points are crucial behaviors that the p2p module has to have to allow for handshakes, sending messages and expecting ACKs (or syncing in the future) 

Goals

Deliverables

• Non-tangible deliverables
  • Transfer learnings from the initial P2P implementation to Pre2P
  • Identify and describe missing gaps in the Pre2P implementation when it comms IO and connection pooling in this PR
• List out the features which need to be transferred offered (e.g. P2P pooling, async message handling) and the difficulty of adding it to Pre2P
• Implement the list of features above
• Update the README.md (to be added in the pre2p/raintree soon) describing the added / modified code layout of Pre2P module with the new modifications

Testing Methodology

1. Use make test_pre2p to run existing tests
2. Update the test suite in raintree_utils_test.go and add raintree_redundancy_layer_test.go int he same package with new tests
3. [Optional] Use LocalNet as described in docs/development/README.md
4. [Optional] Using Telemetry (if ready) to validate the results from (2)

Non-goals

• Scaling LocalNet to many nodes
• Resolving tech debt or optimizing existing code
• Replacing P2P with Pre2P

Creator: @Olshansk
Co-Owners: @derrandz

[derrandz]

To elaborate on my message about the matter, here is the following breakdown:

Breakdown

What's missing

In the current form of Pre2P, the following the baseline peer functionality --which is summarized as the following bullet points, is missing:

• A peer should support accepting multiple connections concurrently
• A peer should support closing all open connections gracefully
• A peer should support consuming multiple connections concurrently:
  • A peer should support fire-and-forget types of communication (e.g: ping/pong, broadcasts)
  • A peer should support continuous types of communication: (handshake, syncing, large transfers and so on)

We can take a look at those missing gaps in p2p/pre2p/module.go in the Start method, specifically the m.listener.Read() function call, and it presents the following problems:

• the TCPAccept is called in the same routine as the net.Conn.Read and net.Conn.Close.

  • This means that we won't be able to accept new incoming connections until the current connection is finished transferring data. The net.Conn.Read (which is blocking) won't drop until the reader encounters an io.EOF, which is only emitted upon net.Conn.Close() being called from the other side. Meaning, even our node won't be able to process that incoming data until the sender has finished entirely with sending it.

  • Also, we have no way to achieve graceful shutdown since we have no control on the IO operation or the accepting operation. Example: we can't wait for them to wrap up before closing the process on shutdown, but rather just kill the process and hope nothing got interrupted while doing so.

  • This only works so far because the sender closes the connection as soon as it's done writing, leaving room for the other senders to do the same sequentially. Let's suppose the first sender decides to send chunks continually, this will block the rest of the senders indefinitely. An example scenario of this is when nodes will be required to perform two-part handshakes before establishing a connection: nodes will have to go sequentially until they are all authenticated, provided that they close the connection as soon as they are authenticated and send the data immediately. If a first sender authenticates and just decides to continually send data, no other sender will get to authenticate and send.

The process of accepting new connections should be independent and should run in parallel to the connection IO handling to avoid this.

The Fix and Its Impact:

To alleviate this issue, we will need to:

• Separate the connection-accepting process from the connection-handling process to achieve a concurrent p2p server
• Handle accepted connections IO in continually running separate go-routines that we can wait for in case of shutting down, and that can close on their on upon errors of EOF.

The impact will basically be touching three important methods in the pre2p module:

• The Read method of the Listener module will be refactored to accept connections concurrently.
• The Start method of the Pre2P module will be refactored to explicitly show the accepting and io-handling separation.
• the handleNetworkData method of the Network module will be deprecated in favor of new connection-IO handling methods that, depending on the communication type, either pools this connection or just instantly consumes it and drops it.
  • for fire-and-forget types of communication, similar logic from the Listener#Read method will take place such that we read off of the connection and drop it immediately.
  • for long-running types of communications, such as requesting a height and waiting for a block in response, or sending data and waiting for an ACK in response, new IO logic will take place that will kick Input and Output loops to continually consume the connection following "an event-poll" pattern, both closeable on-request or when finished.
• handleNetworkData will turn into a sink for all connections data, handling each data type accordingly regardless of where it came from.

Difficulty Assessement

We assess this to be of medium difficulty, however with a variable scope. This refactor is aimed to guarantee baseline functionality and health by transferring connections accepting, io and pooling logic from P2P, but also allow for more important upcoming features such as: Handshakes, Ackable Packets and Syncing.

In the long-running communication scenarios (such as the ones mentioned above), we will also have to deal with data-chunking, packet delimiting and packet ordering, or some type of fixed-size sliding-window to handle large chunks of data. This means that after implementing this refactor, we won't fully make use of the connection-pooling logic unless a wire byte-protocol is introduced.

With that being said, introducing the refactor as is will not break any existing behavior.