Determine byte size of construct containing a Switch() using sizeof() with context fails on parent construct

[alonbl]

Hello,

I would like to calculate the size of struct with Switch statement, much similar to #692.
Each statement is working separately but the sizeof() of the entire construct blows up.

In my case I would assume that the following will calculate the size of type==p2 packet size.

PACKET_STRUCT.sizeof(h={'type': 'p2'})}

The task is to determine the size without parse nor build with minimum requires context settings.
Maybe I am doing something wrong.

Version: construct-2.10.68

TEST

Calculate the size of type==p2 packet:

import construct

PACKET_STRUCT = construct.Struct(
    "h" / construct.Struct(
        "type" / construct.Enum(
            construct.Int32ul,
            p1=0x00000001,
            p2=0x00000002,
        ),
    ),
    "p" / construct.Switch(
        construct.this.h.type, {
            "p1": construct.Struct(
                "v1" / construct.Int32ul,
            ),
            "p2": construct.Struct(
                "v1" / construct.Int32ul,
                "v2" / construct.Int32ul,
            ),
        },
    ),
)

# calculat the size of type==p2
print(f"p2 size [components separate context]: {PACKET_STRUCT.h.sizeof() + PACKET_STRUCT.p.sizeof(h={'type': 'p2'})}")

# the following is the same as context does not bother the header
print(f"p2 size [components same context]:     {PACKET_STRUCT.h.sizeof(h={'type': 'p2'}) + PACKET_STRUCT.p.sizeof(h={'type': 'p2'})}")

# so we can do:
print(f"p2 size [components loop]:             {sum(x.sizeof(h={'type': 'p2'}) for x in PACKET_STRUCT.subcons)}")

# but this does not work:
print(f"p2 size [parent]:                      {PACKET_STRUCT.sizeof(h={'type': 'p2'})}")

EXPECTED BEHAVIOR

p2 size [components separate context]: 12
p2 size [components same context]:     12
p2 size [components loop]:             12
p2 size [parent]:                      12

ACTUAL BEHAVIOR

p2 size [components separate context]: 12
p2 size [components same context]:     12
p2 size [components loop]:             12

Traceback (most recent call last):
<snip>
  File ".env1\Lib\site-packages\construct\expr.py", line 188, in __call__
    return self.__parent(obj)[self.__field]
           ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
KeyError: 'h'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
<snip>
  File ".env1\Lib\site-packages\construct\core.py", line 3933, in _sizeof
    raise SizeofError("cannot calculate size, key not found in context", path=path)
construct.core.SizeofError: Error in path (sizeof) -> p
cannot calculate size, key not found in context

Thanks,

[arekbulski]

The problem from what I recall is that the kwarg passed to sizeof() is not accessible in the same way as context key obtained during parsing. In the switch, fiddle with this._.h.type or this._root.h.type and let me know?

[alonbl]

Hello @arekbulski,
Thank you for the prompt response.
Both options are not working, it raises KeyError unable to find the _ and the _root keys.
Regards,

[arekbulski]

Sorry, I dont know what is wrong with this parsing class, the Switch I mean. Can you use something like len(d.build(...)) for d being whatever that you want to measure? Its a meager workaround, I know.

[arekbulski]

[alonbl]

Hi @arekbulski,
WOW, nice!
Text would be nice so I can copy-paste :)
What version do you use? I use the latest pip one.
Thanks,

[arekbulski]

I used the latest version, 2.10.70. But Switch was not updated for years so that should not play a role. I think.

[alonbl]

I removed the construct-typing which kept this construct down, now I am in sync with you.

I cannot make this work consistently.

See below, please make sure the sizeof() and parse() both work with the same input, I tried with _ and with _root both fail with the parse().

import construct

PACKET_STRUCT = construct.Struct(
    "h" / construct.Struct(
        "type" / construct.Enum(
            construct.Int32ul,
            p1=0x00000001,
            p2=0x00000002,
        ),
    ),
    "p" / construct.Switch(
        construct.this.h.type, {
            "p1": construct.Struct(
                "v1" / construct.Int32ul,
            ),
            "p2": construct.Struct(
                "v1" / construct.Int32ul,
                "v2" / construct.Int32ul,
            ),
        },
    ),
)

print(PACKET_STRUCT.parse(bytearray((0x02, 0x00, 0x00, 0x00, 0x11, 0x11, 0x11, 0x11, 0x22, 0x22, 0x22, 0x22))))
print(f"p2 size [parent]:                      {PACKET_STRUCT.sizeof(h={'type': 'p2'})}")

[arekbulski]

Yes I know... the change I made, it made sizeof() work but I think it breaks parse() at same time. 😞 WOrking with the Switch is a bitch.

[alonbl]

Hi @arekbulski,

The Union also does not working correctly:

import construct

PACKET_STRUCT = construct.Struct(
    "l" / construct.Union(
        "raw",
        "raw" / construct.Int16ul,
        "flags" / construct.Bitwise(
            construct.Struct(
                "f" / construct.Flag,
                "p" / construct.Padding(15),
            )
        ),
    ),
)

print(PACKET_STRUCT.sizeof())

Output:

Traceback (most recent call last):
<snip>
  File ".env1\Lib\site-packages\construct\core.py", line 3744, in _sizeof
    raise SizeofError("Union builds depending on actual object dict, size is unknown", path=path)
construct.core.SizeofError: Error in path (sizeof) -> l
Union builds depending on actual object dict, size is unknown

While as far as I understand from documentation, if the 1st parameter of Union is not None the size of this integer (or field) is the size of the Union, even if it is None the size is specified to be zero as the stream is not seeked back.

Is this correct? Should I open a new bug?

Thanks,

[arekbulski]

It's midnight here. I will investigate tomorrow and get back to you. Okay?

[alonbl]

Sure, it is also midnight here... sorry... nothing urgent.... I appreciate you replied so fast :)

[arekbulski]

Hmm, now that I read it, the exception is different. It says explicitly that you cannot sizeof a Union. Its not a bug.

[alonbl]

Hmm, now that I read it, the exception is different. It says explicitly that you cannot sizeof a Union. Its not a bug.

but why? if it is a C like union, we can take the largest block, and construct added the 1st parameter to explicitly instruct how many bytes to advance in the stream, so in this explicit mode it is the size of the union in bytes or 0 if None.

If you think that the 1st parameter is not a good candidate to determine the size in bytes of the union, please let a context to be passed so that the developer may select the union variant to calculate the size based on, so that this will behave similar to the Switch issue we discuss.

End-state is to have outer_construct.sizeof(context) to be usable as size of the construct with minimum settings of context that is required to calculate the size.

[arekbulski]

I dont know why that is the case, I wrote this almost a decade ago. I will investigate it tomorrow in detail.

[alonbl]

Hi @arekbulski,
Any news regarding the Switch and Context/Sizeof issue?
Thanks,

[arekbulski]

None. I don't know how to fix this.

[alonbl]

None. I don't know how to fix this.

Oh, that is sad... any alternative to switch? I thought of using union but union does not have sizeof as well and I do not understand how it parse provided a field.

[arekbulski]

Yeah, the problem with Switch is actually several years old. You are not the first person who stumbled upon it. :|

[alonbl]

Maybe add Switchv2 that adds the required level?

Currently it omits the key which causes the issue, in the following example, the p1 and p2 are omitted from result while if we preserve them it will create a consistent structure which matches the context.

import construct

PACKET_STRUCT = construct.Struct(
    "h" / construct.Struct(
        "type" / construct.Enum(
            construct.Int32ul,
            p1=0x00000001,
            p2=0x00000002,
        ),
    ),
    "p" / construct.Switch(
        construct.this.h.type, {
            "p1": construct.Struct(
                "v1" / construct.Int32ul,
            ),
            "p2": construct.Struct(
                "v1" / construct.Int32ul,
                "v2" / construct.Int32ul,
            ),
        },
    ),
)

[arekbulski]

Here you are. I am afraif this is the closest you will ever get to a working Switch in this project:

[alonbl]

Thanks!
How do I know what key is when parsing? the entire idea was to perform the switch based on a value in the packet as shown in the issue description.
What about adding the nesting level as I suggested in my previous comment? I believe this will solve the issue.
Thanks,