Alleviate performance impact of Vote Extensions (v0.38)

[adizere]

Scope of this discussion

We'll continue here the conversation with the community regarding the performance impact of Vote Extensions (VE). The outcome of this discussion is to agree on concrete steps we'll take to alleviate the impact that VE have on performance.

Background

The full QA report for v0.38 can be found here. It includes an assessment of the perf. impact of VE. The concise conclusion of those measurements is that VE that exceed a certain size (16kb in our tests) can have significant impact on system latency and throughput.

Current status

Current status as discussed in June 8 at the community call (notes):

• The Comet team is continuing performance improvements, in particular, optimizations of bandwidth consumption, with the expectation that these will offset the impact of VE
• We plan to interview the Skip team for their specific VE use-case (the protocol they plan to run on top ABCI++). We would like to use their app in our e2e testing infrastructure to get benchmarks closer to reality in our next round of QA (for v0.39).

[IvoCrnkovic]

If helpful, we (dYdX trading) would also be happy to talk through our use case(s) for VE. Both of the main ones we have in mind are not viable with the current performance facts. We think we need 5kb per VE to be comfortable.

@ttl33 and @yang-dydx can speak to this with more detail if there is interest.

[sergio-mena]

Let me state a few points about the v0.38.x QA report I think should be part of this discussion (they were touched on during the community call where we presented the results):

• The network setup we use for our QA scale tests (200 full nodes) has all nodes co-located in the same data center
  • We strongly suspect this exacerbates message amplification (duplicates of votes, block parts, TXs)
  • Vote extensions are carried in one type of those amplified messages (precommit votes)
  • So, the performance presented in the QA is a lower bound, but we don't know for sure how tight that lower bound is
  • We are planning on improving the setup to include some kind of realistic latency between the nodes, but this will take some time to land in the codebase
• The application we used is a "demo app"
  • A kv store with ad hoc logic for snapshots, appHash calculation, valSet changes
  • On that app, we added demonstration code on how the proposer can make the necessary data available to other validators to prove it didn't misbehave during PrepProp
    • That (demo) code is clearly suboptimal (hex-encoding) and not necessary in all use cases (e.g., threshold encryption)
    • Moreover, further tests that @lasarojc ran (not included in the report) hint at that part of the app possibly being the bottleneck in our QA tests
  • So, changing the app (as @adizere suggests in his initial comment) to a real one might provide a tighter lower bound on the performance impact of vote extensions

[zmanian]

One of the biggest questions that surround the future of comet BFT is how long the completely unstructured and push based vote and block distribution system is tenable and if there is sufficient will to move beyond it .

In the short run there are hopefully enough wins to support the early set of VE use cases.

[hvanz]

These are some of the latest updates from the Comet team regarding the current status of the bandwidth improvement effort. In particular, this is the work that has already landed or will be released in a future version of Comet.

1. Consensus: reduce vote and block parts gossip with HasProposalBlockPartMessage and random sleeps (consensus: optimize vote and block part gossip with HasProposalBlockPartMessage and random sleeps #904)
   • ~20% bandwidth reduction on BlockPart messages on the usual 200-node testnet
   • To be released in v1
   • Nikhil from Skip helping port this to v0.38 (PR backport: Backport HasProposalBlockPart / HasProposal + PeerGossipIntraloopSleep #1706 )
2. Mempool: New experimental feature that limits the number of peers to which the node gossip txs (mempool: Limit gossip connections to persistent and non-persistent peers (experimental) #1584 + Experimental - Reduce # of connections effectively used to gossip transactions out #1558)
   • Two config parameters for the upper limit for persistent-peers and non-persistent peers.
   • The node initially connects up to these upper limits of peers, and when a peer disconnects, it tries to connect to the next available peer.
   • Experiments on 175 validators with default configuration and gossip limit set to 10 peers, show that sent/received mempool bytes is now ~1/3 of baseline.
   • Important. We have seen in the experiments that:
     • all txs are eventually included in a block
     • latency does not seem to be affected
     • FIFO ordering of txs is not affected
   • To be released in v1 and backported to v0.38, v0.37, v0.34

Next steps:

• we plan to tag alpha v1 and run our QA stress-testing using the kvstore app during late Q4, early Q1'24, then compare the results of these tests against the v0.38 QA and draw a conclusion of how much bandwidth efficiency overall was improved with the optimizations above.

[melekes]

The concise conclusion of those measurements is that VE that exceed a certain size (16kb in our tests) can have significant impact on system latency and throughput.

Does it make sense to cap the max size then? Is there an issue about it?

[zmanian]

I think there multiple use cases for vote extensions which exceed this size such as providing a price oracles with a large number of assets