Fixing _sizeof() for variable-size fields.

[phire]

It would be really useful if _sizeof() could be made to work for variable sized fields, in cases when data was provided.

I've been experimenting with this wrapper that implements sizeof externally for the variably sized fields that I'm actually using. And apart from the fact I occasionally need to add new functionality, it works well.
It would be nice this functionality could moved upstream into each Construct class.

Proposal

So I'm proposing we add an additional parameter to the _sizeof function. Either as an optional arg def _sizeof(self, context, path, obj=None): or as a hidden _obj item in the context container. This obj will be expected to be whatever type the Construct returns on parsing or takes when building.

Once we have a way to pass the object in to _sizeof, actually implementing the variably sized _sizeof() seems to be pretty trivial for most Constructs. For Struct, it's just a matter of copying all the key/value pairs from obj into the new context Container that it passes to each subcon's _sizeof()

Public API

My favourite option for a public API is to extend Container (and ListContainer) to keep track of which Construct it was parsed from, and then add a .sizeof() method.

Then users will be able to do .sizeof() on the result of any parse operation that produces a Container.

Though we also need a way to query the size from the Construct before parsing. That would require either adding a second method (something like sizeof_object) or if we go with the hidden _obj item on the context container option, the current sizeof could be used like this:

my_struct = Struct(
    "Count" / Int32ul, 
    "Data" / Array(this.Count, Int32ul)
)

my_struct.sizeof(_obj=Container(Count=33))

[jpsnyder]

IMHO, sizeof() should just be able to accept a positional argument just like build() and parse() to be more symmetrical.

Since a Container object is returned whenever you parse a Struct, I don't understand why this shouldn't be able to work.

my_struct = Struct(
    "Count" / Int32ul, 
    "Data" / Array(this.Count, Int32ul)
)

obj = my_struct.parse(b"\x02\x00\x00\x00\x0A\x00\x00\x00\x0B\x00\x00\x00")
size = my_struct.sizeof(obj)
data = my_struct.build(obj)

sizeof() currently only takes keyword arguments, so adding an optional positional argument won't break any compatibility.

def sizeof(obj = None, /, **contextkw):
    ...

But I bet this would get shot down by the developer because it requires rethinking how sizeof is handled and would require updating a bunch of Construct classes that don't calculate sizeof correctly.