[P2P] Background gossip using libp2p-pubsub

[jessicadaugherty]

Objective

Add "random gossip" broadcast layer (i.e. in addition to raintree) in P2P module using go-libp2p-pubsub to support full-node participation and provide basic raintree redundancy.

Origin Document

Opportunity identified while integrating Pocket with LibP2P in #347

P2P currently only supports network broadcasting via raintree which in insufficient for full-node participation. The raintree algorithm requires trust in (and/or an effective disincentive against) the entire set of recipients in order to guarantee complete message propagation. For this reason an alternative broadcast mechanism must be used to support full-node participation in P2P (and, by extension, higher-level things like consensus).

Goals

• Determine a reliable, scalable, and customizable broadcasting protocol that has minimal network overhead
• Install and configure a broadcast protocol in the libp2p integration for full node random gossip

Deliverable

• Research the tradeoffs between gossipsub and randomsub (ADR)
• Integrate the background router with the P2P module
  • handle incoming messages from background router sources
  • include background router in P2P module broadcast implementation
• Wrap background router messages in new BackgroundMessage protobuf type

Integration / Architecture

Legends:

flowchart
subgraph Legend
    m[[`Method`]]
    c[Component]

    m -- "unconditional usage" --> c
    m -. "conditional usage" .-> c
end

Loading

classDiagram
class ConcreteType {
  +ExportedField
  -unexportedField
  +ExportedMethod(argType) returnType
  -unexportedMethod()
}

class InterfaceType {
    <<interface>>
    +Method(argType) (returnType1, returnType2)
}

ConcreteType --|> InterfaceType : Interface realization

ConcreteType --> OtherType : Direct usage
ConcreteType --o OtherType : Composition
ConcreteType --* OtherType : Aggregatation
ConcreteType ..*  "(cardinality)" OtherType : Indirect (via interface)

Loading

Outgoing messages:

flowchart
    subgraph p2p[P2P Module]
        ps[[`Send`]]
        pb[[`Broadcast`]]

        subgraph rt[RainTree Router]
            rtu[UnicastRouter]
        end

        subgraph bg[Background Router]
            bgu[UnicastRouter]
            bgg[Gossipsub Topic]
        end
    end
 
ps -.-> rtu
ps -.-> bgu
pb --> rtu
pb --> bgg

Loading

Incoming messages:

flowchart TD
    subgraph p2p[P2P Module]
        subgraph rt[RainTree Router]
            rth[[`RainTreeMessage` Handler]]
            rtu[UnicastRouter]
        end

        subgraph bg[Background Router]
            bgh[[`BackgroundMessage` Handler]]
            bgu[UnicastRouter]
            bgg[Gossipsub Subscription]
        end

        nd[Nonce Deduper]
    
        p2ph[[`PocketEnvelope` Handler]]
        bus
    end

p2ph --> bus
    
bgu --> bgh
bgg --> bgh
rtu --> rth

p2ph --> nd
rth --> p2ph
bgh --> p2ph

Loading

Message handling and deduplication:

classDiagram
    class RainTreeMessage {
        <<protobuf>>
        +Level uint32
        +Data []byte
    }

    class BackgroundMessage {
        <<protobuf>>
        +Data []byte
    }
    
    class PocketEnvelope {
        <<protobuf>>
        +Content *anypb.Any
        +Nonce uint64
    }

    RainTreeMessage --* PocketEnvelope : serialized as `Data`
    BackgroundMessage --* PocketEnvelope : serialized as `Data`
    
    
    class p2pModule {
        -handlePocketEnvelope([]byte) error
    }

    class P2PModule {
        <<interface>>
        GetAddress() (Address, error)
        HandleEvent(*anypb.Any) error
        Send([]byte, address Address) error
        Broadcast([]byte) error
    }
    p2pModule --|> P2PModule

    class RainTreeRouter {
        UnicastRouter
        -handler MessageHandler
        +Broadcast([]byte) error
        -handleRainTreeMsg([]byte) error
    }

    class NonceDeduper {
        
    }

    class BackgroundRouter {
        UnicastRouter
        -handler MessageHandler
        +Broadcast([]byte) error
        -handleBackgroundMsg([]byte) error
        -readSubscription(subscription *pubsub.Subscription)
    }

    class UnicastRouter {
        -messageHandler MessageHandler
        -peerHandler PeerHandler
        +Send([]byte, address Address) error
        -handleStream(stream libp2pNetwork.Stream)
        -readStream(stream libp2pNetwork.Stream)
    }
    RainTreeRouter --* UnicastRouter : (embedded)
    BackgroundRouter --* UnicastRouter : (embedded)
    %% UnicastRouter --> RainTreeRouter : via `messageHandler`
    %% UnicastRouter --> BackgroundRouter  : via `messageHandler`

    p2pModule ..* RainTreeRouter
    %% RainTreeRouter --> p2pModule : `handler` == `handlePocketEnvelope`
    RainTreeRouter --o RainTreeMessage
    
    p2pModule ..* BackgroundRouter
    %% BackgroundRouter --> p2pModule : `handler` == `handlePocketEnvelope`
    BackgroundRouter --o BackgroundMessage

    p2pModule --o PocketEnvelope
    p2pModule --* NonceDeduper

    %% class Router {
    %%     <<interface>>
    %%     +Send([]byte, address Address) error
    %%     +Broadcast([]byte) error
    %% }
    %% BackgroundRouter --|> Router
    %% RainTreeRouter --|> Router

Loading

P2P module / router decoupling:

classDiagram
    class p2pModule {
        -stakedActorRouter Router
        -unstakedActorRouter Router
        -handlePocketEnvelope([]byte) error
    }

    class P2PModule {
        <<interface>>
        GetAddress() (Address, error)
        HandleEvent(*anypb.Any) error
        Send([]byte, Address) error
        Broadcast([]byte) error
    }
    p2pModule --|> P2PModule

    class RainTreeRouter {
        UnicastRouter
        -handler MessageHandler
        +Broadcast([]byte) error
        -handleRainTreeMsg([]byte) error
    }

    class BackgroundRouter {
        UnicastRouter
        -handler MessageHandler
        +Broadcast([]byte) error
        -handleBackgroundMsg([]byte) error
        -readSubscription(subscription *pubsub.Subscription)
    }

    class UnicastRouter {
        -messageHandler MessageHandler
        -peerHandler PeerHandler
        +Send([]byte, Address) error
        -handleStream(libp2pNetwork.Stream)
        -readStream(libp2pNetwork.Stream)
    }
    RainTreeRouter --* UnicastRouter : (embedded)
    BackgroundRouter --* UnicastRouter : (embedded)

    p2pModule --o "2" Router
    p2pModule ..* RainTreeRouter : (`stakedActorRouter`)
    p2pModule ..* BackgroundRouter : (`unstakedActorRouter`)
    
    class Router {
        <<interface>>
        +Send([]byte, Address) error
        +Broadcast([]byte) error
    }
    BackgroundRouter --|> Router
    RainTreeRouter --|> Router

Loading

Non-goals / Non-deliverables

• Implementing raintree specific redundancy (see: [P2P] RainTree Redundancy Layer Implementation #85)
• Refactoring raintree in terms of go-libp2p-pubsub (see: [P2P] Integrate Raintree as Router in LibP2P Messaging #506)

General issue deliverables

• Update the appropriate CHANGELOG(s)
• Update any relevant local/global README(s)
• Update relevant source code tree explanations
• Add or update any relevant or supporting mermaid diagrams

Testing Methodology

• All tests: make test_all
• LocalNet: verify a LocalNet is still functioning correctly by following the instructions at docs/development/README.md

---

Creator: @jessicadaugherty
Co-creator: @bryanchriswhite

[jessicadaugherty]

Iteration 15 Planning: Revisit AC in case anything is stale or can be more relevant to latest info @bryanchriswhite