feat: GossipSub v1.4 by ufarooqstatus · Pull Request #1448 · vacp2p/nim-libp2p

ufarooqstatus · 2025-06-06T11:39:43Z

Added GossipSub v1.4 implementation. Specs proposal here.

Still need to complete the safety strategy (in next PR)

Requires the compile flag: -d:libp2p_gossipsub_1_4, although running nimble testpubsub does add the flag automatically.

ufarooqstatus · 2025-06-12T22:49:40Z

+  heartbeat "Gossipsub Peer Bandwidth Tracking", 1.seconds:
+    for (topic, peers) in g.gossipsub.pairs:
+      for peer in peers.items:
+        peer.bandwidthTracking.update()


If there's no traffic for a few seconds, bandwidth estimate moves to 0.

Actually, we are interested in estimating how fast a peer will transfer to us. Maybe we can use $\frac{messageSize}{downloadTime - preambleTime}$, and use EMA to maintain it
We won't need bandwithHeartbeat in that case

Maybe we can just skip updating the EMA if the sample is 0, altho that will leave you with a bandwidth estimation that is even less precise if the node is not chatty.

But, if going with this formula you suggest, what does the downloadTime and preambleTime refer to?

yes, sounds good. we just need a rough estimate.
Preamble time is whe we received preamble. download time is when we received complete message from the same peer

We just want to be able to configure that we only issue IMRECEIVING on receiving preamble from top X % of peers

ufarooqstatus · 2025-06-12T22:54:28Z

+
+  BandwidthTracking* = ref object
+    upload*: ExponentialMovingAverage
+    download*: ExponentialMovingAverage


We just need to know how fast a remote peer can transfer to us, so upload can suffice

ufarooqstatus · 2025-06-18T11:42:04Z

+    if ps.byId.hasKey(msgId):
+      ps.byId[msgId].deleted = true


This replaces the preamble info we are watching (for expiry with the new one). Potentially mesh may keep sending us preambles and we will keep updating this.

Although this never happens because we check for existing entries here

Yes, this is to keep the functionaliyu of PreambleStore to behave the same as a Table, (altho it internally also has a minheap). Except for the popExpired function, you'll notice that the API and behavior matches that of a Table.

ufarooqstatus · 2025-06-18T12:34:37Z

+          )
+        ),
+        isHighPriority = true,
+      )


we still dont know if peers[0] has the required message. what if we store 2-3 mesh members from here and 2-3 non-memsh members from here to preambleInfo (maybe as meshList and IHaveList). That can be more efficient, as we won't need to fetch mesh members, and we also won't need topicID in preambleInfo

codecov-commenter · 2025-06-24T22:09:11Z

Codecov Report

Attention: Patch coverage is 78.13620% with 61 lines in your changes missing coverage. Please review.

Project coverage is 84.45%. Comparing base (848fdde) to head (62b8c89).
Report is 14 commits behind head on master.

Files with missing lines	Patch %	Lines
libp2p/protocols/pubsub/gossipsub/behavior.nim	66.01%	35 Missing ⚠️
libp2p/protocols/pubsub/rpc/messages.nim	7.14%	13 Missing ⚠️
...ibp2p/protocols/pubsub/gossipsub/preamblestore.nim	91.17%	6 Missing ⚠️
libp2p/protocols/pubsub/rpc/protobuf.nim	88.37%	5 Missing ⚠️
libp2p/protocols/pubsub/gossipsub.nim	94.44%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1448      +/-   ##
==========================================
- Coverage   84.91%   84.45%   -0.46%     
==========================================
  Files         103      109       +6     
  Lines       18129    18341     +212     
==========================================
+ Hits        15394    15490      +96     
- Misses       2735     2851     +116

Files with missing lines	Coverage Δ
libp2p/protocols/pubsub/bandwidth.nim	`100.00% <100.00%> (ø)`
libp2p/protocols/pubsub/pubsubpeer.nim	`88.51% <100.00%> (+1.68%)`	⬆️
libp2p/protocols/pubsub/gossipsub.nim	`92.89% <94.44%> (+1.70%)`	⬆️
libp2p/protocols/pubsub/rpc/protobuf.nim	`85.83% <88.37%> (+0.55%)`	⬆️
...ibp2p/protocols/pubsub/gossipsub/preamblestore.nim	`91.54% <91.17%> (ø)`
libp2p/protocols/pubsub/rpc/messages.nim	`45.00% <7.14%> (-9.55%)`	⬇️
libp2p/protocols/pubsub/gossipsub/behavior.nim	`88.39% <66.01%> (-4.31%)`	⬇️

... and 11 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

gmelodie

I think this PR should be split into smaller ones as it was with AutoTLS

richard-ramos · 2025-07-10T11:50:07Z

@gmelodie: I think this PR should be split into smaller ones as it was with AutoTLS

yeah. that's a good idea. @ufarooqstatus I'll work on this if that's okay

richard-ramos · 2025-07-12T16:46:50Z

@gmelodie PR is now small enough for review after extracting part of it in #1513 and #1515

…d safety strategy Add preamble threshold Add bandwidth tracking fix: send preamble on publish only if using `sendIDontWantOnPublish` parameter adjusted bandwidth tracking feat: preamble store and heartbeat for expiration chore: send IWANT to mesh peer in expiry heartbeat chore: store peers in preambleinfo chore: hide 1_4 behind compile flag `-d:libp2p_gossipssub_1_4` added support for push operation Add behavior penalty for failed preamble commitment

ufarooqstatus requested a review from a team as a code owner June 6, 2025 11:39

ufarooqstatus requested review from richard-ramos and vladopajic June 6, 2025 11:39

github-project-automation Bot added this to nim-libp2p Jun 6, 2025

github-project-automation Bot moved this to new in nim-libp2p Jun 6, 2025

github-actions Bot assigned ufarooqstatus Jun 6, 2025

richard-ramos changed the title ~~Gossipsub v1 4~~ feat: GossipSub v1.4 Jun 10, 2025

richard-ramos force-pushed the gossipsub_v1_4 branch from 500331b to d287766 Compare June 10, 2025 14:52

richard-ramos marked this pull request as draft June 11, 2025 18:22

ufarooqstatus commented Jun 12, 2025

View reviewed changes

Comment thread libp2p/protocols/pubsub/gossipsub.nim Outdated

ufarooqstatus commented Jun 12, 2025

View reviewed changes

richard-ramos force-pushed the gossipsub_v1_4 branch 6 times, most recently from 255acb2 to 448e929 Compare June 17, 2025 21:52

ufarooqstatus commented Jun 18, 2025

View reviewed changes

Comment thread libp2p/protocols/pubsub/gossipsub/receivestore.nim Outdated

ufarooqstatus commented Jun 18, 2025

View reviewed changes

richard-ramos force-pushed the gossipsub_v1_4 branch 4 times, most recently from 12ebbcd to 62b8c89 Compare June 24, 2025 21:51

richard-ramos force-pushed the gossipsub_v1_4 branch from a36643b to 66ae07a Compare June 26, 2025 12:35

richard-ramos marked this pull request as ready for review July 8, 2025 20:58

gmelodie reviewed Jul 10, 2025

View reviewed changes

This was referenced Jul 10, 2025

feat(gossipsub1_4): preamble store #1513

Merged

feat(gossipsub1.4): adding new attributes and protobuffers #1515

Merged

richard-ramos force-pushed the gossipsub_v1_4 branch 3 times, most recently from fca0af7 to dd1be78 Compare July 12, 2025 16:45

richard-ramos force-pushed the gossipsub_v1_4 branch 2 times, most recently from b1f7cca to 491bb76 Compare July 12, 2025 19:36

gmelodie approved these changes Jul 14, 2025

View reviewed changes

richard-ramos moved this from new to In Progress in nim-libp2p Jul 18, 2025

vladopajic approved these changes Jul 18, 2025

View reviewed changes

Comment thread tests/pubsub/integration/testgossipsubcontrolmessages.nim

Comment thread libp2p/protocols/pubsub/gossipsub.nim Outdated

ufarooqstatus and others added 3 commits July 20, 2025 14:27

fix: code review

8e97f9b

fix: code review

1cc8351

richard-ramos force-pushed the gossipsub_v1_4 branch from ad1303f to 1cc8351 Compare July 20, 2025 18:27

richard-ramos enabled auto-merge (squash) July 20, 2025 18:28

richard-ramos merged commit 6576c5c into master Jul 20, 2025
22 of 23 checks passed

richard-ramos deleted the gossipsub_v1_4 branch July 20, 2025 18:47

github-project-automation Bot moved this from In Progress to done in nim-libp2p Jul 20, 2025

gmelodie mentioned this pull request Jul 21, 2025

chore(ci): initial flake.nix file with dev shell #1559

Merged

Conversation

ufarooqstatus commented Jun 6, 2025 • edited by richard-ramos Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

ufarooqstatus Jun 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

richard-ramos Jun 13, 2025

Choose a reason for hiding this comment

Uh oh!

ufarooqstatus Jun 15, 2025

Choose a reason for hiding this comment

Uh oh!

ufarooqstatus Jun 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ufarooqstatus Jun 12, 2025

Choose a reason for hiding this comment

Uh oh!

ufarooqstatus Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

richard-ramos Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ufarooqstatus Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov-commenter commented Jun 24, 2025

Codecov Report

Uh oh!

gmelodie left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

richard-ramos commented Jul 10, 2025

Uh oh!

richard-ramos commented Jul 12, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ufarooqstatus commented Jun 6, 2025 •

edited by richard-ramos

Loading

ufarooqstatus Jun 12, 2025 •

edited

Loading

ufarooqstatus Jun 15, 2025 •

edited

Loading

ufarooqstatus Jun 18, 2025 •

edited

Loading

gmelodie left a comment •

edited

Loading