Decouple ABCI from Consensus: Support for customizing the fields inside Block.Header

[S1nus]

Feature Request

Customizable parameters in the ABCI block header

Summary

Replace consenus-related parameters in the header with a generic blob byte array for increased customizability with Rollkit's various weird fork-choice schemes

Problem Definition

This increases separation between consensus and execution. We have some schemes where the block proposer does not matter and no signatures need to be verified, such as fork-choice based on ordering in a DA layer, or by gas consumed. We may also wish to add other things to the header, such as a ZK proof in a ZK rollup.

Proposal

The header currently looks like this:

type Header struct {
        .. fields omitted..
        // hashes from the app output from the prev block
        ValidatorsHash     tmbytes.HexBytes `json:"validators_hash"`      // validators for the current block
        NextValidatorsHash tmbytes.HexBytes `json:"next_validators_hash"` // validators for the next block
        ConsensusHash      tmbytes.HexBytes `json:"consensus_hash"`       // consensus params for current block
        ProposerAddress Address          `json:"proposer_address"` // original proposer of the block
}

We may wish to customize these fields and add more fields, one possibility is to replace it with a generic byte array and interpret serialized structs like so:

type Header struct {
        Payload []byte
}
// this can be put into something like a protobuf and inserted into the payload field of a header
type CometHeaderPayload {
        // hashes from the app output from the prev block
        ValidatorsHash     tmbytes.HexBytes `json:"validators_hash"`      // validators for the current block
        NextValidatorsHash tmbytes.HexBytes `json:"next_validators_hash"` // validators for the next block
        ConsensusHash      tmbytes.HexBytes `json:"consensus_hash"`       // consensus params for current block
        ProposerAddress Address          `json:"proposer_address"` // original proposer of the block
}

[lasarojc]

Thanks for you suggestions, @S1nus . The consensus parameters are something used by the consensus itself, not the application. If you could expand on what you are trying to achieve, we can work together on a solution.

I am changing this issue into a discussion for now.

[S1nus]

Ah i see.

I'm hoping to make it easier to use ABCI apps with different consensus engines besides CometBFT, e.g Rollkit.

[S1nus]

We think ABCI is useful for consensus engines besides CometBFT. It would be nice if the headers pertaining to consensus could be customized!

[lasarojc]

Thanks for clarifying. I see what you mean.
Would you like to present this idea at one of our community calls?

cc @adizere

[S1nus]

Yes!

[adizere]

Are you already part of the CometBFT community group? (If not, the details to join are here, see "If your work depends on CometBFT").

On a related note, another feedback from the Rollkit team is about modifying the ABCI interface, specifically to facilitate passing of proofs and app hash between app and Comet by adding 3 new methods.

If you can make it to the community call on April 13, we can kick off an initial discussion about both the change here and the one with the new methods. Thanks in advance for joining!

[tac0turtle]

This is a feature we would love to have as well. It would allow the sdk to work with different abci implementations.

A proto any message could suffice here and may be cleaner than bytes but bytes would also suffice

[adizere]

I'm curious, do you know if this change be done in a backwards-compatible way? Or does this change imply a block breaking change?

[tac0turtle]

I think it would be more of a api breaking change cause technically what is sent over the wire is the same just in a different form. I think we can make it not block breaking for current users

[adizere]

This would be fantastic!

[cason]

Just for a bit of context, because the name "consensus parameters" can be confusing.

Consensus parameters are not necessarily parameters for the consensus operation. In fact most of the parameters used by the consensus protocol are not "consensus parameters". For instance, consensus timeouts are local parameters, set in the configuration file. And here lays the distinction: "consensus parameters" are global parameters, shared by all nodes in a network; changes in "consensus parameters" requires consensus, namely they are "proposed", included in a block, then updated by all nodes at (or from) the same height of the blockchain.

[adizere]

Thanks for the clarification Daniel!

I dug into the codebase and found that the naming around consensus params is quite subtle indeed. We have these three names that would be good to disambiguate:

Consensus parameters

Defined in ConsensusParams. I guess this is what Daniel means by the "global consensus parameters". These can be changed only via agreement network-wide, updatable by ResponseFinalizeBlock.

Timeout parameters

• "Timeout parameters" as ADR 074 describes them. I think that's the right way to refer to these parameters that configure the timeouts of the consensus protocol.
  • Side note: we should consider at some point dealing with the technical debt here and implement ADR 074.

Consensus parameters and Timeout parameters are not relevant for this discussion.

Block.Header fields

• It seems like the present discussion focuses on all the fields within the Block.Header structure. I'll rename the discussion to prevent confusion.

---

Questions to further clarify & de-risk the scope

1. @S1nus @tac0turtle I understood the requirements is to have all of the fields in the header to be customizable. Does that sound right?

   • I am asking b/c some of the fields in the Header are tightly coupled and spread in the rest of the codebase (eg. Header.AppHash used in Light Client; Header.Height used in many places).
   • We can scope down the changes proposed here if we narrow the exact fields that you need to make customizable. It would be important to know which fields if any we can keep (such as AppHash and Height) as "canonical" fields of the Header.
   • Any thoughts?

2. re: "A proto any message could suffice here and may be cleaner than bytes but bytes would also suffice"

   • @S1nus any preference between Any or []byte, is this a constraint?

3. re: "we can make it not block breaking for current users"

   • How certain are we about this? I think we'd need to hack together a PoC/prototype to demonstrate this.

Finally, modifying/creating Headers seems to be relatively contained in the codebase (mostly in block.go, eg the Populate method). I'm curious how would the application customize these methods, any ideas or a prototype showing this would also be great!

[lasarojc]

I believe the idea here is a step towards decoupling ABCI from Comet by making the Header type a wrapper for actual headers. Then consensus implementations can unwrap the header assuming it has the right format for the specific consensus; CometBFT would unwrap it as a CometHeaderPayload and Avalanche would unwrap it as AvalancheHeaderPayload, for example.

[itsdevbear]

Bumping this @adizere, we need dis one as well

[adizere]

👨‍💻 🚀