abci: Giving applications access to the P2P layer

[thanethomson]

In thinking about future enabling enhancements to ABCI (e.g. ABCI 3.0), with #494 in mind, would it be useful to application developers to have access to the P2P layer? Specifically in validators being able to send other validators messages, piggybacking on the P2P network formed by the consensus engine. The chief use case here is to enable complex application-side protocols, like custom app-side mempool gossip protocols, but I'm interested to know whether there are other possible use cases folks could think of.

This whole part of the interface would necessarily be non-deterministic.

In this idea, the application would need to be able to initiate calls to several methods on the consensus engine at any time (i.e. in the opposite direction of the current ABCI calling mechanism):

• Multicast - Send a message through the best possible route(s)¹ to one or more specific peer(s) by way of their node ID. Optionally signed by the sending validator, if the sender is a validator.
• Broadcast - Send a message to all connected peers. Optionally signed by the sending validator, if the sender is a validator.
• ValidatorMulticast - Send a message through the best possible route(s)¹ to one or more specific validator(s) by way of their ID. Optionally signed by the sending validator, if the sender is a validator.
• ValidatorBroadcast - Send a message through the best possible route(s)¹ to all validators in the network. Optionally signed by the sending validator, if the sender is a validator.
• ConnectToPeer - Tell the consensus engine to attempt to connect to a specific peer.
• DisconnectFromPeer - Tell the consensus engine to disconnect from a specific peer.
• BanPeer - Tell the consensus engine to ban a specific peer, disconnecting from it if it is currently connected to it.

There would also need to be new methods exposed from the application's side that the consensus engine would call:

• ProcessMessage - Process an incoming message from a peer.
• PeerConnected - Called when a new connection with a peer is successfully established, along with that peer's details.
• PeerDisconnected - Called when a connection with a peer is terminated.

Would all or only some of the above methods be useful? Could some of these methods be coalesced into simpler primitives? Other/different methods perhaps? None at all? 🙂

¹ The "best possible route(s)" would potentially involve some form of DHT managed by the Comet node itself (currently not implemented, but under consideration as part of the bandwidth optimization work the team has been doing).

[alexanderbez]

Happy this discussion is coming into fruition!

I would say your ideas make sense and point to a direction we want to go towards in the SDK. Specifically, I foresee a few major themes in which the application would benefit greatly from in terms of having control or input over relative to the P2P stack:

• Having control or input over peer connectivity (e.g. I want to disconnect from a peer because of misbehavior or continuously creating bad proposals)
• Having control or input over transaction gossiping (huge win IMO and the biggest lacking feature of the app-side mempool)
• Having the ability to send messages to peer(s) without having to fork CometBFT and create custom channels (e.g. Skip would benefit heavily from this)

All in all, your points seem to line up with everything above 👍

@bpiv400 I wonder what your thoughts are here?

[BrendanChou]

To me, the minimum-viable-change would be to change the logic upon CheckTx response. Thecode uint32 could be overloaded (essentially turn it into an enum) instead of just using 0 as success and all other values as failure. Additional fields could also be added to the response, but this would be a larger change.

Brainstorming some possible behaviors (but would largely overlap with what @thanethomson has suggested above)

• BanPeer - this message is so malformed that an honest node cannot have sent this
• RetainSilent/EjectLoud/etc - Every combination of: (1) keep/eject from mempool pre-cache (2) keep/eject from mempool (3) gossip/do-not-gossip the message. For example I might want to gossip the message, keep it in the mempool pre-cache, but remove it from my own mempool

This obviously does not contain the full functionality of the original proposal, but could be a good-first-step with much less effort expended (and also be used as a stepping stone for more control by the application)

[adizere]

An incremental approach here with makes sense, and BanPeer could be a good start, + it aligns with SDK needs!

[thaodt]

Im wondering does this idea start any implementations yet? cc @thanethomson @adizere

[adizere]

not yet