Size of array is less than size of form with PHYSLITE schema

[alexander-held]

Describe the bug
I've ran into another bug that seems PHYSLITE schema related and occurs somewhat infrequently. Unfortunately I am not aware of suitable public files at the moment to reproduce, so I will point to the information relevant for finding them within ATLAS. If needed we can hopefully find a mechanism to share a relevant file. cc @nikoladze as PHYSLITE schema expert.

To Reproduce

from coffea.nanoevents import NanoEventsFactory, PHYSLITESchema
import uproot

fname = "DAOD_PHYSLITE.37021106._000087.pool.root.1"
treename = "CollectionTree"

# with PHYSLITE schema
events = NanoEventsFactory.from_root({fname: treename}, schemaclass=PHYSLITESchema).events()
events.Muons.topoetcone20_CloseByCorr.compute()  # this fails

# plain uproot
f = uproot.open({fname: treename})
f["AnalysisMuonsAuxDyn.topoetcone20_CloseByCorr"].array()  # this succeeds

The read when using the schema fails, it succeeds with plain uproot. The trace ends in

TypeError: size of array (61994) is less than size of form (61999)

This error occurred while calling

    ak.from_buffers(
        RecordForm-instance
        172020
        {'[/data/xTrigDecisionAux.](https://alheld-notebook-1.notebook.af.uchicago.edu/data/xTrigDecisionAux.)%2FxTrigDecisionAux.smk%2C%21load': <awkward...
        behavior = {'Systematic': <class 'coffea.nanoevents.methods.base.Syst...
        buffer_key = partial-instance
    )

with the full trace attached below.

Examples of files to test with, all are in the

data18_13TeV-data18_13TeV.periodAllYear.physics_Main.PhysCont.DAOD_PHYSLITE.grp18_v01_p6026

container:

DAOD_PHYSLITE.37020635._000006.pool.root.1  # no crash in coffea with this file
DAOD_PHYSLITE.37021106._000087.pool.root.1  # leads to crash in coffea

With plain uproot, both files work fine. I've also ran over many other files and have seen similar TypeError exceptions. I have not looked into their origin and tracked down whether the root cause may be similar.

Expected behavior
Successful branch reading.

Output

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
File [/venv/lib/python3.9/site-packages/awkward/_dispatch.py:39](https://alheld-notebook-1.notebook.af.uchicago.edu/venv/lib/python3.9/site-packages/awkward/_dispatch.py#line=38), in dispatch()
     38 with OperationErrorContext(name, args, kwargs):
---> 39     gen_or_result = func(*args, **kwargs)
     40     if isgenerator(gen_or_result):

File [/venv/lib/python3.9/site-packages/awkward/operations/ak_from_buffers.py:103](https://alheld-notebook-1.notebook.af.uchicago.edu/venv/lib/python3.9/site-packages/awkward/operations/ak_from_buffers.py#line=102), in from_buffers()
     39 """
     40 Args:
     41     form (#ak.forms.Form or str[/dict](https://alheld-notebook-1.notebook.af.uchicago.edu/dict) equivalent): The form of the Awkward
   (...)
    101 See #ak.to_buffers for examples.
    102 """
--> 103 return _impl(
    104     form,
    105     length,
    106     container,
    107     buffer_key,
    108     backend,
    109     byteorder,
    110     highlevel,
    111     behavior,
    112     attrs,
    113     allow_noncanonical_form,
    114 )

File [/venv/lib/python3.9/site-packages/awkward/operations/ak_from_buffers.py:149](https://alheld-notebook-1.notebook.af.uchicago.edu/venv/lib/python3.9/site-packages/awkward/operations/ak_from_buffers.py#line=148), in _impl()
    147 getkey = regularize_buffer_key(buffer_key)
--> 149 out = _reconstitute(form, length, container, getkey, backend, byteorder, simplify)
    151 return wrap_layout(out, highlevel=highlevel, attrs=attrs, behavior=behavior)

File [/venv/lib/python3.9/site-packages/awkward/operations/ak_from_buffers.py:403](https://alheld-notebook-1.notebook.af.uchicago.edu/venv/lib/python3.9/site-packages/awkward/operations/ak_from_buffers.py#line=402), in _reconstitute()
    402 elif isinstance(form, ak.forms.RecordForm):
--> 403     contents = [
    404         _reconstitute(
    405             content, length, container, getkey, backend, byteorder, simplify
    406         )
    407         for content in form.contents
    408     ]
    409     return ak.contents.RecordArray(
    410         contents,
    411         None if form.is_tuple else form.fields,
    412         length,
    413         parameters=form._parameters,
    414     )

File [/venv/lib/python3.9/site-packages/awkward/operations/ak_from_buffers.py:404](https://alheld-notebook-1.notebook.af.uchicago.edu/venv/lib/python3.9/site-packages/awkward/operations/ak_from_buffers.py#line=403), in <listcomp>()
    402 elif isinstance(form, ak.forms.RecordForm):
    403     contents = [
--> 404         _reconstitute(
    405             content, length, container, getkey, backend, byteorder, simplify
    406         )
    407         for content in form.contents
    408     ]
    409     return ak.contents.RecordArray(
    410         contents,
    411         None if form.is_tuple else form.fields,
    412         length,
    413         parameters=form._parameters,
    414     )

File [/venv/lib/python3.9/site-packages/awkward/operations/ak_from_buffers.py:381](https://alheld-notebook-1.notebook.af.uchicago.edu/venv/lib/python3.9/site-packages/awkward/operations/ak_from_buffers.py#line=380), in _reconstitute()
    380     next_length = 0 if len(offsets) == 1 else offsets[-1]
--> 381 content = _reconstitute(
    382     form.content, next_length, container, getkey, backend, byteorder, simplify
    383 )
    384 return ak.contents.ListOffsetArray(
    385     ak.index.Index(offsets),
    386     content,
    387     parameters=form._parameters,
    388 )

File [/venv/lib/python3.9/site-packages/awkward/operations/ak_from_buffers.py:403](https://alheld-notebook-1.notebook.af.uchicago.edu/venv/lib/python3.9/site-packages/awkward/operations/ak_from_buffers.py#line=402), in _reconstitute()
    402 elif isinstance(form, ak.forms.RecordForm):
--> 403     contents = [
    404         _reconstitute(
    405             content, length, container, getkey, backend, byteorder, simplify
    406         )
    407         for content in form.contents
    408     ]
    409     return ak.contents.RecordArray(
    410         contents,
    411         None if form.is_tuple else form.fields,
    412         length,
    413         parameters=form._parameters,
    414     )

File [/venv/lib/python3.9/site-packages/awkward/operations/ak_from_buffers.py:404](https://alheld-notebook-1.notebook.af.uchicago.edu/venv/lib/python3.9/site-packages/awkward/operations/ak_from_buffers.py#line=403), in <listcomp>()
    402 elif isinstance(form, ak.forms.RecordForm):
    403     contents = [
--> 404         _reconstitute(
    405             content, length, container, getkey, backend, byteorder, simplify
    406         )
    407         for content in form.contents
    408     ]
    409     return ak.contents.RecordArray(
    410         contents,
    411         None if form.is_tuple else form.fields,
    412         length,
    413         parameters=form._parameters,
    414     )

File [/venv/lib/python3.9/site-packages/awkward/operations/ak_from_buffers.py:197](https://alheld-notebook-1.notebook.af.uchicago.edu/venv/lib/python3.9/site-packages/awkward/operations/ak_from_buffers.py#line=196), in _reconstitute()
    196 real_length = length * math.prod(form.inner_shape)
--> 197 data = _from_buffer(
    198     backend.nplike,
    199     raw_array,
    200     dtype=dtype,
    201     count=real_length,
    202     byteorder=byteorder,
    203 )
    204 if form.inner_shape != ():

File [/venv/lib/python3.9/site-packages/awkward/operations/ak_from_buffers.py:174](https://alheld-notebook-1.notebook.af.uchicago.edu/venv/lib/python3.9/site-packages/awkward/operations/ak_from_buffers.py#line=173), in _from_buffer()
    173 if array.size < count:
--> 174     raise TypeError(
    175         f"size of array ({array.size}) is less than size of form ({count})"
    176     )
    178 return array[:count]

TypeError: size of array (61994) is less than size of form (61999)

The above exception was the direct cause of the following exception:

TypeError                                 Traceback (most recent call last)
Cell In[73], line 7
      5 # with PHYSLITE schema
      6 events = NanoEventsFactory.from_root({fname: treename}, schemaclass=PHYSLITESchema).events()
----> 7 events.Muons.topoetcone20_CloseByCorr.compute()

File [/venv/lib/python3.9/site-packages/dask/base.py:375](https://alheld-notebook-1.notebook.af.uchicago.edu/venv/lib/python3.9/site-packages/dask/base.py#line=374), in DaskMethodsMixin.compute(self, **kwargs)
    351 def compute(self, **kwargs):
    352     """Compute this dask collection
    353 
    354     This turns a lazy Dask collection into its in-memory equivalent.
   (...)
    373     dask.compute
    374     """
--> 375     (result,) = compute(self, traverse=False, **kwargs)
    376     return result

File [/venv/lib/python3.9/site-packages/dask/base.py:661](https://alheld-notebook-1.notebook.af.uchicago.edu/venv/lib/python3.9/site-packages/dask/base.py#line=660), in compute(traverse, optimize_graph, scheduler, get, *args, **kwargs)
    658     postcomputes.append(x.__dask_postcompute__())
    660 with shorten_traceback():
--> 661     results = schedule(dsk, keys, **kwargs)
    663 return repack([f(r, *a) for r, (f, a) in zip(results, postcomputes)])

File [/venv/lib/python3.9/site-packages/uproot/_dask.py:1343](https://alheld-notebook-1.notebook.af.uchicago.edu/venv/lib/python3.9/site-packages/uproot/_dask.py#line=1342), in __call__()
   1329     except self.allowed_exceptions as err:
   1330         return (
   1331             self.mock_empty(backend="cpu"),
   1332             _report_failure(
   (...)
   1340             ),
   1341         )
-> 1343 result, _ = self._call_impl(
   1344     file_path, object_path, i_step_or_start, n_steps_or_stop, is_chunk
   1345 )
   1346 return result

File [/venv/lib/python3.9/site-packages/uproot/_dask.py:1296](https://alheld-notebook-1.notebook.af.uchicago.edu/venv/lib/python3.9/site-packages/uproot/_dask.py#line=1295), in _call_impl()
   1290     start, stop = min((i_step_or_start * events_per_step), num_entries), min(
   1291         (i_step_or_start + 1) * events_per_step, num_entries
   1292     )
   1294 assert start <= stop
-> 1296 return self.read_tree(
   1297     ttree,
   1298     start,
   1299     stop,
   1300 )

File [/venv/lib/python3.9/site-packages/uproot/_dask.py:1017](https://alheld-notebook-1.notebook.af.uchicago.edu/venv/lib/python3.9/site-packages/uproot/_dask.py#line=1016), in read_tree()
   1009     # Otherwise, introduce a placeholder
   1010     else:
   1011         container[buffer_key] = awkward.typetracer.PlaceholderArray(
   1012             nplike=nplike,
   1013             shape=(awkward.typetracer.unknown_length,),
   1014             dtype=dtype,
   1015         )
-> 1017 out = awkward.from_buffers(
   1018     self.expected_form,
   1019     stop - start,
   1020     container,
   1021     behavior=self.form_mapping_info.behavior,
   1022     buffer_key=self.form_mapping_info.buffer_key,
   1023 )
   1024 assert tree.source  # we must be reading something here
   1025 return out, tree.source.performance_counters

File [/venv/lib/python3.9/site-packages/awkward/_dispatch.py:70](https://alheld-notebook-1.notebook.af.uchicago.edu/venv/lib/python3.9/site-packages/awkward/_dispatch.py#line=69), in dispatch()
     65     else:
     66         raise AssertionError(
     67             "high-level functions should only implement a single yield statement"
     68         )
---> 70 return gen_or_result

File [/venv/lib/python3.9/site-packages/awkward/_errors.py:85](https://alheld-notebook-1.notebook.af.uchicago.edu/venv/lib/python3.9/site-packages/awkward/_errors.py#line=84), in __exit__()
     78 try:
     79     # Handle caught exception
     80     if (
     81         exception_type is not None
     82         and issubclass(exception_type, Exception)
     83         and self.primary() is self
     84     ):
---> 85         self.handle_exception(exception_type, exception_value)
     86 finally:
     87     # Step out of the way so that another ErrorContext can become primary.
     88     if self.primary() is self:

File [/venv/lib/python3.9/site-packages/awkward/_errors.py:95](https://alheld-notebook-1.notebook.af.uchicago.edu/venv/lib/python3.9/site-packages/awkward/_errors.py#line=94), in handle_exception()
     93     self.decorate_exception(cls, exception)
     94 else:
---> 95     raise self.decorate_exception(cls, exception)

TypeError: size of array (61994) is less than size of form (61999)

This error occurred while calling

    ak.from_buffers(
        RecordForm-instance
        172020
        {'[/data/xTrigDecisionAux.](https://alheld-notebook-1.notebook.af.uchicago.edu/data/xTrigDecisionAux.)%2FxTrigDecisionAux.smk%2C%21load': <awkward...
        behavior = {'Systematic': <class 'coffea.nanoevents.methods.base.Syst...
        buffer_key = partial-instance
    )

Desktop (please complete the following information):

awkward: 2.6.2
dask-awkward: 2024.3.0
uproot: 5.3.2
coffea: 2024.3.0

Additional context
n/a

[alexander-held]

Some more examples that might be useful when debugging: AnalysisPhotonsAuxDyn.ptcone20_CloseByCorr in root://192.170.240.143:1094//root://fax.mwt2.org:1094//pnfs/uchicago.edu/atlaslocalgroupdisk/rucio/data18_13TeV/1f/87/DAOD_PHYSLITE.37020635._000031.pool.root.1, AnalysisPhotonsAuxDyn.topoetcone40_CloseByCorr in root://192.170.240.143:1094//root://fax.mwt2.org:1094//pnfs/uchicago.edu/atlaslocalgroupdisk/rucio/data18_13TeV/1f/87/DAOD_PHYSLITE.37020635._000031.pool.root.1.