Add finalized property to HTTP API responses

[nflaig]

Problem description

Latest beacon API spec requires finalized property in some HTTP API respones

• Add finalized flag to chain data requests ethereum/beacon-APIs#254

Solution description

Add finalized property to the following HTTP API responses:

• https://ethereum.github.io/beacon-APIs/#/Beacon/getBlockAttestations
• https://ethereum.github.io/beacon-APIs/#/Beacon/getBlindedBlock
• https://ethereum.github.io/beacon-APIs/#/Beacon/getBlockV2
• https://ethereum.github.io/beacon-APIs/#/Beacon/getBlockHeaders
• https://ethereum.github.io/beacon-APIs/#/Beacon/getBlockHeader
• https://ethereum.github.io/beacon-APIs/#/Beacon/getBlockRoot
• https://ethereum.github.io/beacon-APIs/#/Beacon/getEpochCommittees
• https://ethereum.github.io/beacon-APIs/#/Beacon/getStateFinalityCheckpoints
• https://ethereum.github.io/beacon-APIs/#/Beacon/getStateFork
• https://ethereum.github.io/beacon-APIs/#/Beacon/getStateRandao
• https://ethereum.github.io/beacon-APIs/#/Beacon/getStateRoot
• https://ethereum.github.io/beacon-APIs/#/Beacon/getEpochSyncCommittees
• https://ethereum.github.io/beacon-APIs/#/Beacon/getStateValidators
• https://ethereum.github.io/beacon-APIs/#/Beacon/getStateValidator
• https://ethereum.github.io/beacon-APIs/#/Beacon/getStateValidatorBalances
• https://ethereum.github.io/beacon-APIs/#/Debug/getStateV2

Additional context

No response

[philknows]

• state endpoints blocked on feat: add historical state regen #6033

[zilm13]

Any updates on this?
Testing Teku VC with Lodestar BN to fix Consensys/teku#8118
And getting error in parsing https://ethereum.github.io/beacon-APIs/#/Beacon/getStateValidators because of missing finalized field which is required by the spec https://github.com/ethereum/beacon-APIs/blob/master/apis/beacon/states/validators.yaml#L48

[nflaig]

I think @ensi321 wanted to look into this, right now Lodestar is pretty limited in serving finalized data derived from the state as we only keep the unfinalized state in-memory and only persist very few states to load from disk (will be addressed by #6033). We could just hard code those to finalized=false for now to fix the compatibility issue and be spec compliant.

On the note of cross-client compatibliy, we currently also only support json bodies for all requests but as per spec there are a few APIs that should also support ssz bodies. How does Teku handle this case, do you send the ssz body and if you receive a 415 (unsupported media type) do you resend the request with a json body?

I am working on a overhaul of the API to add first class support for ssz res/resp to (almost) all APIs so this will be addressed soon as well.

[zilm13]

@nflaig
Hmm if it's really not finalized it would be fair as a quick fix.
Yeah, we send it by default in ssz (could be overridden with command line flag), if we receive 415, we fallback to json storing that ssz is not supported, so on next requests we will use json until VC is restarted.

[nflaig]

if we receive 415, we fallback to json storing that ssz is not supported, so on next requests we will use json until VC is restarted

ah perfekt, this is the ideal behavior I also had in mind because we want to send ssz bodies for almost all APIs even those that don't have it (yet) defined in the spec

[nflaig]

• state endpoints blocked on feat: add historical state regen #6033

We can implement this already without the historical state regen, we already can serve some finalized data without it, e.g. finalized blocks or finalized checkpoints.