Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Running Producer with empty dataset fails #361

Open
mafrahm opened this issue Nov 24, 2023 · 0 comments
Open

Running Producer with empty dataset fails #361

mafrahm opened this issue Nov 24, 2023 · 0 comments
Labels
bug Something isn't working

Comments

@mafrahm
Copy link
Contributor

mafrahm commented Nov 24, 2023

Running a producer on an (post-selection) empty dataset seems to not work anymore. I suspect this happened due to the PR where we changed the strategy of our ChunkedIOReader (#345)

iterate through 0 events in 0 chunks ...
failed (took 0.49 seconds)
ERROR: luigi-interface - [pid 530] Worker Worker(salt=6786987636, workers=1, host=naf-cms11.desy.de, username=frahmmat, pid=40574) failed    cf.ProduceColumns(effective_workflow=local, branch=0, analysis=hbw.analysis.hbw_sl.hbw_sl, version=prod1, config=c17, shift=nominal, local_shift=nominal, dataset=qcd_mu_pt15to20_pythia, calibrators=skip_jecunc, selector=sl, producer=ml_inputs)
Traceback (most recent call last):
  File "/afs/desy.de/user/f/frahmmat/Projects/hh2bbww/data/software/venvs/venv_columnar_dev_3cbb5aff/lib/python3.9/site-packages/luigi/worker.py", line 203, in run
    new_deps = self._run_get_new_deps()
  File "/afs/desy.de/user/f/frahmmat/Projects/hh2bbww/data/software/venvs/venv_columnar_dev_3cbb5aff/lib/python3.9/site-packages/luigi/worker.py", line 138, in _run_get_new_deps
    task_gen = self.task.run()
  File "/afs/desy.de/user/f/frahmmat/Projects/hh2bbww/modules/columnflow/modules/law/law/decorator.py", line 244, in wrapper
    return decorator(fn, _opts, *args, **kwargs)
  File "/afs/desy.de/user/f/frahmmat/Projects/hh2bbww/modules/columnflow/modules/law/law/decorator.py", line 272, in log
    return fn(task, *args, **kwargs)
  File "/afs/desy.de/user/f/frahmmat/Projects/hh2bbww/modules/columnflow/modules/law/law/decorator.py", line 244, in wrapper
    return decorator(fn, _opts, *args, **kwargs)
  File "/afs/desy.de/user/f/frahmmat/Projects/hh2bbww/modules/columnflow/modules/law/law/decorator.py", line 540, in localize
    return fn(task, *args, **kwargs)
  File "/afs/desy.de/user/f/frahmmat/Projects/hh2bbww/modules/columnflow/modules/law/law/decorator.py", line 231, in wrapper
    result = call(state)
  File "/afs/desy.de/user/f/frahmmat/Projects/hh2bbww/modules/columnflow/modules/law/law/decorator.py", line 303, in call
    return fn(task, *args, **kwargs)
  File "/afs/desy.de/user/f/frahmmat/Projects/hh2bbww/modules/columnflow/columnflow/tasks/production.py", line 111, in run
    for (events, *cols), pos in self.iter_chunked_io(
  File "/afs/desy.de/user/f/frahmmat/Projects/hh2bbww/modules/columnflow/columnflow/tasks/framework/mixins.py", line 1865, in iter_chunked_io
    for obj in self.iter_progress(handler, max(handler.n_chunks, 1), msg=msg):
  File "/afs/desy.de/user/f/frahmmat/Projects/hh2bbww/modules/columnflow/modules/law/law/task/base.py", line 565, in iter_progress
    for i, val in enumerate(iterable):
  File "/afs/desy.de/user/f/frahmmat/Projects/hh2bbww/modules/columnflow/columnflow/columnar_util.py", line 3372, in __iter__
    yield from self._iter_impl()
  File "/afs/desy.de/user/f/frahmmat/Projects/hh2bbww/modules/columnflow/columnflow/columnar_util.py", line 3303, in _iter_impl
    result_obj = result.get()
  File "/afs/desy.de/user/f/frahmmat/Projects/hh2bbww/data/software/conda/lib/python3.9/multiprocessing/pool.py", line 771, in get
    raise self._value
  File "/afs/desy.de/user/f/frahmmat/Projects/hh2bbww/data/software/conda/lib/python3.9/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/afs/desy.de/user/f/frahmmat/Projects/hh2bbww/modules/columnflow/columnflow/columnar_util.py", line 3267, in read
    chunks.append(read_func(chunk_pos))
  File "/afs/desy.de/user/f/frahmmat/Projects/hh2bbww/modules/columnflow/columnflow/columnar_util.py", line 3158, in read_awkward_parquet
    return source_object.materialize(
  File "/afs/desy.de/user/f/frahmmat/Projects/hh2bbww/modules/columnflow/columnflow/columnar_util.py", line 2510, in materialize
    return self._materialize_via_slices(*args, **kwargs)
  File "/afs/desy.de/user/f/frahmmat/Projects/hh2bbww/modules/columnflow/columnflow/columnar_util.py", line 2527, in _materialize_via_slices
    return self.dak_array[entry_start:entry_stop].compute()
  File "/afs/desy.de/user/f/frahmmat/Projects/hh2bbww/data/software/venvs/venv_columnar_dev_3cbb5aff/lib/python3.9/site-packages/dask/base.py", line 342, in compute
    (result,) = compute(self, traverse=False, **kwargs)
  File "/afs/desy.de/user/f/frahmmat/Projects/hh2bbww/data/software/venvs/venv_columnar_dev_3cbb5aff/lib/python3.9/site-packages/dask/base.py", line 628, in compute
    results = schedule(dsk, keys, **kwargs)
  File "/afs/desy.de/user/f/frahmmat/Projects/hh2bbww/data/software/venvs/venv_columnar_dev_3cbb5aff/lib/python3.9/site-packages/dask_awkward/lib/io/parquet.py", line 124, in __call__
    return ak.Array(unproject_layout(self.original_form, array.layout))
  File "/afs/desy.de/user/f/frahmmat/Projects/hh2bbww/data/software/venvs/venv_columnar_dev_3cbb5aff/lib/python3.9/site-packages/dask_awkward/lib/unproject_layout.py", line 411, in unproject_layout
    return _unproject_layout(form, layout, layout.length, layout.backend)
  File "/afs/desy.de/user/f/frahmmat/Projects/hh2bbww/data/software/venvs/venv_columnar_dev_3cbb5aff/lib/python3.9/site-packages/dask_awkward/lib/unproject_layout.py", line 269, in _unproject_layout
    _unproject_layout(
  File "/afs/desy.de/user/f/frahmmat/Projects/hh2bbww/data/software/venvs/venv_columnar_dev_3cbb5aff/lib/python3.9/site-packages/dask_awkward/lib/unproject_layout.py", line 257, in _unproject_layout
    content=_unproject_layout(
  File "/afs/desy.de/user/f/frahmmat/Projects/hh2bbww/data/software/venvs/venv_columnar_dev_3cbb5aff/lib/python3.9/site-packages/dask_awkward/lib/unproject_layout.py", line 377, in _unproject_layout
    raise AssertionError(f"unexpected combination: {type(form)} and {type(layout)}")
AssertionError: unexpected combination: <class 'awkward.forms.recordform.RecordForm'> and <class 'awkward.contents.bitmaskedarray.BitMaskedArray'>
@mafrahm mafrahm added the bug Something isn't working label Nov 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant