[LSP2] CompactBytesArray - complete technical overview with examples #172
Comments
[WIP].The encoding/decoding is implemented in Swift but is not yet properly mapped to something more simple for everyone to read, understand and discuss. Will update this comment soon. The main points are laid out already. Solution for encodingCode snippets are not written in any specific language General encodingHow the func encode(type, value, isSubtypeOfCBA = false)
if type is CBA:
// CBA must have a subtype otherwise even the syntax is not valid
subtype = type.subtype
encoded_data: bytes = '0x'
for subvalue in value:
// Also setting third parameter `isSubtypeOfCBA` as true
// The encoding of non-complex types assumes we do not add prefixes
// and suffixes when encoding for CBA.
data: bytes = encode(subtype, subvalue, true)
data = data.size.leftPadded(toBytes: 2) + data
encoded_data.append(data)
return encoded_data
// If type is not a CBA but has nested CBA, e.g. `bytes[CompactBytesArray][]`
// it must be custom encoded first before it is encoded as a part of the outer type
else if type.hasNestedCBA
// example: `type` is `bytes[CompactBytesArray][]`
// `value` is an array of CBAs
// `func preencode_CBAs` will replace each entry in `value`
// with encoded CBA and return new version of `value`
value = preencode_CBAs(type, value)
// each `some_type[CompactBytesArray]` from the perspective of Solidity ABI is just `bytes`
// `func replace_CBAs` will return new type where each `some_type[CompactBytesArray]` is replace
// with `bytes`.
// examples:
// - `bytes[CompactBytesArray][]` => `bytes[]`
// - `(address,uint32,int64,address[])[CompactBytesArray][]` => `bytes[]`
// - `(address[CompactBytesArray],uint256)` => `(bytes,uint256)`
type = replace_CBAs(type)
else if !type.isComplexType && isSubtypeOfCBA:
// A complex type is a type that has nested types.
// Complex types are: array, tuple, CompactBytesArray.
// The rest can be directly represented as bytes.
return convertToBytes(value)
return web3.eth.abi.encodeParameters(type, value)Explanation of the overall idea: CBA has a custom encoding format, but the values encoded by Encoding with a custom case for tuplesWe can add a custom case for a tuple that is the direct child of CBA: func encode(type, value)
if type is CBA:
// CBA must have a subtype otherwise even the syntax is not valid
subtype = type.subtype
encoded_data: bytes = '0x'
for subvalue in value:
if subtype is tuple:
// subvalue must be an array of arguments
tuple_data: bytes = '0x'
tuple_types = subtype.nested_types
for index in subvalue.indices():
// Also setting the third parameter `isSubtypeOfCBA` as true
// The encoding of non-complex types assumes we do not add prefixes
// and suffixes when encoding for CBA.
data: bytes = encode(tuple_types[index], subvalue[index], true)
// This line is what makes it different from LSP6 AllowedCalls:
// each value in a tuple has it's own size prefix because dynamic types
// can also be in a tuple.
data = data.size.leftPadded(toBytes: 2) + data
tuple_data.append(data)
tuple_data = tuple_data.size.leftPadded(toBytes: 2) + tuple_data
encoded_data.append(tuple_data)
else:
data: bytes = encode(subtype, subvalue)
data = data.size.leftPadded(toBytes: 2) + data
encoded_data.append(data)
return encoded_data
...CompactBytesArray of fixed sizeSince writing Example: |
CompactBytesArrayThanks @JeneaVranceanu for taking the time to write this detailed explanation! Although one can argue that the probability of someone writing 2D Arrays on the blockchain is minimal, but also it's worth having a general rule that applies to CBA regardless of what and how much will be written. We're not seeing much traction on this issue, as there are more high-priority discussions happening around features to be implemented in contracts that should not change, but we'll try to keep the discussion going until we reach to a proper solution. I don't have strong ideas and opinions about the topics mentioned, but it should be discussed whether the CBA format needs to be a general type of encoding or a limited type just for the sake of storing in an efficient way. Because that makes me think of a proposal mentioned by @skimaharvey , to have 1 byte that determines the length of the byte that determines the length of the element. In this way, CBA can have elements that have a very big length in terms of bytes. What made us go with 2 bytes as a fixed length for elements is that with 2 bytes you can store up to 2^16 (65K bytes), which is more than enough for the current blockchain capacity. If CBA ends up being a general type of encoding, then let's make it as generic as possible and start drafting rules for it. I guess this is where the discussion should start, because discussing supporting very nested types for an encoding scheme that is meant to be defined just for efficiency is a waste of time. So I would like to discuss here whether it makes sense to make CBA a general type of encoding or a limited one with support to a few elements. CompactBytesArray of fixed sizeI agree currently there is no way to know whether the array is fixed size or not so for instance, address[CBA] we cannot know whether it's fixed size or not. Not sure what is the use case but I agree about adding an optional semi-colon with the number of elements in case of a fixed-size Array. |
TL;DR - IMHO, it could be the general type of arrays used off-chain. Initially, I was thinking that it would make sense to make CBA a default type to use for keys that represent arrays. Still, it is convenient only if those keys are used roughly speaking off-chain because (and please correct me here if I'm wrong) the default arrays are able to be easily decoded from ABI back to representation usable in Solidity. Meaning: the ABI of type |
This is an interesting idea and I think it is worth implementing even though it will be probably rare that someone will try to exceed |
Unless I am misunderstanding, I think @skimaharvey proposal design is different than that. There would not be precisely 2 bytes anymore.
example 1: so for example 1, the full encoded entry would look like this: example 2: so for example 2, the full encoded entry would look like this: |
|
The one difference I see between your description and mine is that sections (A) and (B) are for some reason limited in length in your example. I do not see, at least it's not obvious to me, why we should limit the length. What I thought is that we could have the following:
Example: |
This issue is aimed at having a complete technical low-level overview of
CompactBytesArraytype with examples which combined should coverCompactBytesArraycompletely. LSP2 must have allvalueTypes described and covered the same way Solidity documentation has covered its data types and respective ABI encoding/decoding.TL;DR
Not enough documentation on how to encode/decode
[CompactBytesArray]. We have explanations forbytes[CompactBytesArray]andbytesN[CompactBytesArray]in LSP2, but in LSP6 we use(bytes4,address,bytes4)[CompactBytesArray]and both the type and the encoding don't match what is described in LSP2.The issue
Currently (06.02.2023) in LSP2 we have
bytes[CompactBytesArray]andbytesN[CompactBytesArray]value types. These are covered and well understood. But that documentation comes short of the explanation required with the introduction of LSP6's AllowedCalls. Both of these were released with v0.8.0 see CHANGELOG8.pdfLSP6's AllowedCalls scheme uses
valueTypeset as(bytes4,address,bytes4)[CompactBytesArray], and it has the custom case of encoding[CompactBytesArray]of tuples described in LSP6 (not the right place, IMO). This custom case, first of all, must not be custom and it must be covered by LSP2, and it's not.On top of that, since we have
(bytes4,address,bytes4)[CompactBytesArray]in LSP6 and it doesn't matchbytes[CompactBytesArray]andbytesN[CompactBytesArray]it implicitly allows developers to use any other type with[CompactBytesArray], likeaddress[CompactBytesArray][CompactBytesArray](2D CompactBytesArray). And the question is - how will it be encoded? There is no single source of truth that everyone can refer to at the moment to implement the correct encoding/decoding functions of CBAs.LSP2 is a standard. We must define low-level specifications that others will build upon their custom structures the same way we use Solidity and its specifications to encode and decode ABI of types it provides. These custom structures will use a specific, known and well-documented set of small blocks (value types) and everyone will be able to refer to LSP2 instead to understand how to create and disassemble these small blocks (encode and decode ABI).
Additional question: if the compact bytes array has 2 or more levels of nested types do we compact only the first layer or all of them? My answer is - only the first layer, the type declared directly next to the left side of the
[CompactBytesArray].Example:
(address[],(boolean,bytes4))[CompactBytesArray](a type that doesn't make sense but is technically possible).Do we encode
...,(boolean,bytes4)...tuple the default way or do we reduce/compact it as well?My understanding is that we encode it the default way. The same applies in my opinion to the question above about arrays in CompactBytesArray.
Encoding example of
(address[],(boolean,bytes4))[CompactBytesArray]:After a discussion with @b00ste on Friday (04.02.2023) it also came to mind that
[CompactBytesArray]could completely replace[]. Any thoughts on that idea are welcome.But
[CompactBytesArray]has two downsides in comparison to[]one of which can be fixed:[CompactBytesArray]. For the default array type, you can place a number between square brackets to define the array's size, e.g.[2]. As a solution it could be placed separated by a colon[CompactBytesArray:2];[CompactBytesArray]you have no way to tell if it's valid or not. The ABI of[](as of any other Solidity type) must have the number of bytes be a multiple of 32. One byte off and it's not valid. This is not the case with[CompactBytesArray]and cannot be.* if the solution mentioned below for the fixed-size CBAs will be applied there is a certain set of conditions that will allow us to produce a type that can be validated before the encoding attempt has happened. The set of conditions is: (1) all subtypes have fixed size and (2) CBA is itself fix-sized.
Example: the expected bytes count for
bytes8[CompactBytesArray:10]is(8 bytes + 2 bytes size prefix) * 10 = 100 bytes.The text was updated successfully, but these errors were encountered: