Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement set reconciliation based sync (syncv2) #5769

Draft
wants to merge 30 commits into
base: develop
Choose a base branch
from

Conversation

ivan4th
Copy link
Contributor

@ivan4th ivan4th commented Mar 24, 2024

Motivation

Existing sync is causing a lot of resource usage, including memory consumption and network traffic, and can't replace gossip which can also be unreliable at times.

Description

This implements the new sync protocol as described here.
The algorithm is based on Range-Based Set Reconciliation paper by Aljoscha Meyer and also loosely based on SREP: Out-Of-Band Sync of Transaction Pools for Large-Scale Blockchains paper by Novak Boškov, Sevval Simsek, Ari Trachtenberg, and David Starobinski.

Test Plan

Add systests that verify new sync.
Verify on testnet, then on mainnet.

TODO

  • Explain motivation or link existing issue(s)
  • Sync implementation based on red-black monoid tree
  • P2P wire protocol for sync
  • Implement SREP-like multi-peer sync
  • Make code more efficient by getting rid of excess monoid abstraction
  • Implement syncv2 for ATXs. This may include adding an untrusted LayerID field to ATXs to make sync more efficient
  • Implement SQLite-based "pre-balanced" monoid tree for old data to reduce memory consumption
  • Harden the protocol by enforcing communication limits
  • Consider replacing red-black monoid tree with a BTree so it can also be used for atxsdata
  • Implement sync for other object types
  • Test changes and document test plan
  • Update documentation as needed
  • Update changelog as needed

Copy link

codecov bot commented Mar 24, 2024

Codecov Report

Attention: Patch coverage is 74.59252% with 265 lines in your changes are missing coverage. Please review.

Project coverage is 80.0%. Comparing base (f2770a4) to head (f8a021f).
Report is 22 commits behind head on develop.

Current head f8a021f differs from pull request most recent head ef02024

Please upload reports for the commit ef02024 to get more accurate results.

Files Patch % Lines
hashsync/sync_tree.go 81.1% 67 Missing and 21 partials ⚠️
hashsync/rangesync.go 58.3% 62 Missing and 23 partials ⚠️
hashsync/handler.go 70.6% 39 Missing and 24 partials ⚠️
hashsync/wire_types.go 52.2% 21 Missing ⚠️
hashsync/sync_trees_store.go 89.6% 4 Missing and 2 partials ⚠️
common/types/hashes.go 60.0% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff            @@
##           develop   #5769     +/-   ##
=========================================
- Coverage     80.7%   80.0%   -0.8%     
=========================================
  Files          288     292      +4     
  Lines        29894   30769    +875     
=========================================
+ Hits         24149   24620    +471     
- Misses        4158    4434    +276     
- Partials      1587    1715    +128     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@mathcrypto
Copy link

Did we agree already on adding the LayerID field to ATXs? @poszu @ivan4th. ATXs should only include what's need to be there

@ivan4th
Copy link
Contributor Author

ivan4th commented May 6, 2024

@mathcrypto the discussion was not finished yet, but I restarted it #5785 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants