v2024.4.1

What's Changed

New Features

• feat: Request all arrays from uproot at once inside dask task by @nsmith- in #1076
• feat: use tree reduction to aggregate files in preprocessing by @alexander-held in #1079

Fixes

• fix: update Applying corrections notebook by @Saransh-cpp in #1070

Misc

• ci(pre-commit): pre-commit autoupdate by @pre-commit-ci in #1071
• ci(pre-commit): pre-commit autoupdate by @pre-commit-ci in #1081

Full Changelog: v2024.4.0...v2024.4.1

v2024.4.0

What's Changed

New Features

Fixes

• fix: Do not add trailing slash in xrootd urls by @valsdav in #1054

Misc

• ci(pre-commit): pre-commit autoupdate by @pre-commit-ci in #1060
• ci(pre-commit): pre-commit autoupdate by @pre-commit-ci in #1062
• ci(pre-commit): pre-commit autoupdate by @pre-commit-ci in #1067
• build: bump awkward for numpy 2 compatibility by @lgray in #1068
• ci(pre-commit): pre-commit autoupdate by @pre-commit-ci in #1069

Full Changelog: v2024.3.0...v2024.4.0

v2024.3.0

What's Changed

New Features

Fixes

• fix: properly retain user metadata in preprocess by @lgray in #1051
• fix(build): unpin distributed as well! by @lgray in #1057

Misc.

• ci: bump pypa/gh-action-pypi-publish from 1.8.11 to 1.8.12 by @dependabot in #1049
• build: unpin dask and skooch dask-awkward pin by @lgray in #1055
• build: pin dask-histogram to >=2024.3.0 by @lgray in #1056
• ci: bump pypa/gh-action-pypi-publish from 1.8.12 to 1.8.14 by @dependabot in #1058
• build: for recent python use dask 2024.3.0 by @lgray in #1059

Full Changelog: v2024.2.2...v2024.3.0

v2024.2.2

What's Changed

This release is to pin to uproot 5.3.0 and adapt to the interface changes therein.

New Features

• feat: modifier argument for partial_weights method by @rkansal47 in #1042

Fixes

• fix: use ak.merge_union_of_records to generate input data format by @lgray in #1017
• fix: make apply_to_fileset be able to handle tuple outputs of data_manipulation. by @iasonkrom in #1038
• fix: adjust to breaking changes in form mapping interface by @lgray in #1047

Other

• ci(pre-commit): pre-commit autoupdate by @pre-commit-ci in #1035
• chore: test in python 3.12 by @lgray in #1040
• perf: use an lru_cache for weight, partial_weight, require by @lgray in #1043
• ci(pre-commit): pre-commit autoupdate by @pre-commit-ci in #1044

Full Changelog: v2024.2.1...v2024.2.2

v2024.2.1

What's Changed

New Features

• feat: allowing unmatched files in rucio utils by @valsdav in #1021

Fixes

• fix: updateDataDiscoveryCLI usage of dataset_tools.preprocess by @iasonkrom in #1024
• fix: remove branches that aren't interpretable as nanoevents in preprocess by @lgray in #1031
• fix(build): pin dask < 2024.2.0 until a fix in awkward is in place by @lgray in #1033

Other

• ci(pre-commit): pre-commit autoupdate by @pre-commit-ci in #1026
• ci: bump ts-graphviz/setup-graphviz from 1 to 2 by @dependabot in #1028
• ci: bump pre-commit/action from 3.0.0 to 3.0.1 by @dependabot in #1029

Full Changelog: v2024.2.0...v2024.2.1

v2024.2.0

What's Changed

New Features:

• feat: round target step multiplicity so we end up with more even steps on average by @lgray in #1011
• feat: deprecate coffea.nanoevents.methods.vector by @lgray in #997

Fixes:

• fix!: improve user interface for preprocess by @lgray in #1008
• fix: use public uproot interfaces when extracting form by @lgray in #1013
• fix: force application of to_packed in awkward binnings by @lgray in #1016
• fix: convert sys.exit(1) to ValueError in DataDiscoveryCLI by @iasonkrom in #1020

Documentation:

• fix(docs): update Running inference tools notebook by @Saransh-cpp in #1004
• fix(docs): Update dataset_discovery.ipynb by @nsmith- in #1006
• fix(docs): rewrite Coffea Processors notebook by @Saransh-cpp in #1005

Other:

• build(binder): update binder install to include distributed by @lgray in #1009
• build: pin to uproot 5.2.2 by @lgray in #1012
• ci(pre-commit): pre-commit autoupdate by @pre-commit-ci in #1015
• chore: get ruff to show fixes when it applies them by @lgray in #1018
• ci: bump codecov/codecov-action from 3 to 4 by @dependabot in #1022
• build: bump dask-awkward and dask-histogram by @lgray in #1023

Full Changelog: v2024.1.2...v2024.2.0

v2024.1.2

What's Changed

• refactor: significantly simplify dask version of LumiList by @lgray in #993
• fix: better typing in manipulations by @lgray in #994
• fix: better empty files filter by @lgray in #995
• fix(docs): use correct source links in docs by @lgray in #996
• chore: cleanup VCS versioning by @Saransh-cpp in #999
• fix: update Reading data with coffea NanoEvents notebook by @Saransh-cpp in #1000
• fix: test delta-r + metric return with and without optimization by @lgray in #1001
• feat: warn user about abnormally large step sizes when align_clusters==True by @lgray in #1003
• fix: make dask layer name shorthands easier to read by @lgray in #1002

New Contributors

• @Saransh-cpp made their first contribution in #999

Full Changelog: v2024.1.1...v2024.1.2

v2024.1.1

What's Changed

• feat!: dask-based LumiList and LumiData by @lgray in #990

Full Changelog: v2024.1.0...v2024.1.1

v2024.1.0

What's Changed

• test: add skip for missing dependency, fixture for local tests directory by @lobis in #933
• docs: add packedselection notebook to coffea by example by @lgray in #970
• ci(pre-commit): pre-commit autoupdate by @pre-commit-ci in #971
• ci: triton changed their extras by @lgray in #974
• ci(pre-commit): pre-commit autoupdate by @pre-commit-ci in #976
• chore: remove now-defunct servicex tests by @lgray in #977
• build: bump dask-awkward requirement by @lgray in #979
• feat: max_files by @lgray in #981
• fix: remove hack to try to make correction serialization faster, it is too flaky by @lgray in #982
• ci(pre-commit): pre-commit autoupdate by @pre-commit-ci in #983
• feat: preprocess a dataset's base form and hint it to uproot by @lgray in #978
• fix: form_hash_md5 is not needed in FilesetSpec by @lgray in #984
• fix: remove conversion to dask arrays by @lgray in #985
• fix: adapt to read-only dask_awkward attrs by @lgray in #986
• build(deps): bump awkward version to 2.5.2 by @lgray in #987
• feat: preprocess returns [[0, 0]] for empty ttrees, add filter_files tool by @lgray in #989
• fix: use dask.delayed to reference objects like ml models or corrections by @lgray in #988

New Contributors

• @lobis made their first contribution in #933

Full Changelog: v2023.12.0...v2024.1.0

v2023.12.0

This release is a significant, backwards incompatible change with respect to coffea 0.7 analysis code and patterns. We have made sure the translation from coffea 0.7 to this version of coffea is not unendurably difficult.

The coffea 202x series is a major upgrade to coffea that incorporates uproot5, awkward2, dask-awkward, dask-histogram, and fsspec. It uses all of these packages to create a cohesive, scalable, and fluidly evolving analysis and analysis-framework design platform that is convenient, well performing, and robust. Using this release, analysis design and execution is made significantly more smooth and manageable at all stages from initial ideation and data exploration to producing publication quality results.

It includes significant new features like: failed file handling and partial job completion, skimming, high-level dataset handling, streamlined interfaces to machine learning inference at scale, access to columnar jet-clustering, ... .

Many thanks to everyone who made this release possible including: @jpivarski @agoose77 @nsmith- @douglasdavis @yimuchen @iasonkrom @valsdav

What's Changed

• build(deps): bump pypa/gh-action-pypi-publish from 1.7.1 to 1.8.1 by @dependabot in #774
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #781
• build(deps): bump pypa/gh-action-pypi-publish from 1.8.1 to 1.8.4 by @dependabot in #780
• Awkward v2 transition by @lgray in #736
• build(deps): bump pypa/gh-action-pypi-publish from 1.8.4 to 1.8.5 by @dependabot in #782
• Get tests in passing state, mark heisenbugs by @lgray in #789
• update awkward2 pin by @lgray in #785
• Use codecov action now by @lgray in #791
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #787
• update awkward, dask_awkward, dask_histogram pins by @lgray in #792
• Use uproot chunking for uproot.dask, repin uproot by @lgray in #793
• Pin dask < 2023.4.0 while dask_awkward isn't compatible by @lgray in #794
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #799
• pins for coffea 2023.4.0rc3 by @lgray in #796
• Anticipate form.length_zero_array() highlevel deprecation by @lgray in #801
• Check for Weight.partial_weight() by @rkansal47 in #803
• build(deps): bump pypa/gh-action-pypi-publish from 1.8.5 to 1.8.6 by @dependabot in #804
• next coffea will be 2023.5.0.rc0 by @lgray in #802
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #805
• Repin for ak + dak + dhi by @lgray in #806
• The next coffea release will be 2023.5.0.rc1 by @lgray in #808
• fix: Allow lookup_base args to be in any order by @lgray in #811
• Use maintained action for conda and graphviz by @lgray in #812
• chore: remove ci based release by @lgray in #813
• chore: Update CONTRIBUTING.md to indicate calver, release process by @lgray in #815
• feat: pr labeller by @lgray in #816
• fix: use correct secret name by @lgray in #817
• fix(ci): no need to run outdated PR hashes by @lgray in #818
• build: dask-awkward/histogram 2023.5.1 by @lgray in #820
• feat: added rapidity getter in coffea/nanoevents/methods/vector.py by @raeubaen in #825
• build: move to pyproject style setup by @lgray in #819
• docs: Added example code for correctionlib_wrapper by @yimuchen in #790
• build: use hatch to build release instead of setup.py by @lgray in #828
• chore: update to empty_if_typetracer by @lgray in #821
• ci(pre-commit.ci): pre-commit autoupdate by @pre-commit-ci in #831
• fix: Added proper behavior for PFNanoAODSchema by @yimuchen in #832
• fix: serialize dask_awkward.Array to None and store hard ref in closure in behavior by @lgray in #824
• refactor: use length_zero/one_if_typetracer by @lgray in #834
• build: repin dask_awkward >= 2023.6.1 by @lgray in #836
• fix: PFnano format for dask by @yimuchen in #835
• feat: ML inference tools by @yimuchen in #798
• ci: [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #838
• fix: remove now-incorrect documentation from ml_tools by @lgray in #839
• feat: Pass list of files to NanoEventsFactory by @chrispap95 in #837
• build: pin to awkward 2.2.3, dask-histogram 2023.6.0 by @lgray in #840
• build(pre-commit.ci): pre-commit autoupdate by @pre-commit-ci in #841
• ci: get CI jobs completing again by @lgray in #842
• fix: adjustments to callable wrap to deal with typetracers in nested python structures by @lgray in #843
• ci: don't use url in test_fix823 by @lgray in #844
• chore: remove references to old awkward1 classes by @lgray in #846
• docs: update readme versions by @lgray in #847
• feat: New features to PackedSelection by @iasonkrom in #797
• ci: Bump pypa/gh-action-pypi-publish from 1.8.6 to 1.8.7 by @dependabot in #849
• build: pin awkward up to 2.2.4 by @lgray in #850
• ci(pre-commit): pre-commit autoupdate by @pre-commit-ci in #851
• ci(pre-commit): pre-commit autoupdate by @pre-commit-ci in #864
• Bump pypa/gh-action-pypi-publish from 1.8.7 to 1.8.8 by @dependabot in #866
• build: pin adjustments by @lgray in #852
• docs: Create CITATION.cff by @nsmith- in #853
• chore: coffea uses pyproject.toml now update in readme by @lgray in #868
• chore: Update README.rst with citation by @nsmith- in #869
• build: ignore dask-awkward 2023.8.0 by @lgray in #873
• ci: Bump pypa/gh-action-pypi-publish from 1.8.8 to 1.8.10 by @dependabot in #877
• fix: Make LumiMask compatible with dask-awkward by @jrueb in #879
• build: bump awkward pin to >= 2.3.3 by @lgray in #880
• feat: Dataset querying features using rucio by @valsdav in #883
• feat: dataset discovery CLI by @valsdav in #884
• ci: Bump actions/checkout from 3 to 4 by @dependabot in #889
• build: repin to latest fixed awkward by @lgray in #890
• feat: also pass uproot_options to uproot.dask in from_root() by @iasonkrom in #887
• fix: Daskify Elementlinks in PHYSLITE schema by @nikoladze in #872
• fix: allow for collections that contain non-jagged arrays in PHYSLITE schema by @nikoladze in #888
• ci: Bump crazy-max/ghaction-github-pages from 3 to 4 by @dependabot in #893
• ci(pre-commit): pre-commit autoupdate by @pre-commit-ci in #894
• ci(pre-commit): pre-commit autoupdate by @pre-commit-ci in #896
• fix: comply with boost-histogram 1.4.0 by @iasonkrom in #895
• ci: Bump amannn/action-semantic-pull-request from 5.2.0 to 5.3.0 by @dependabot in #897
• fix: Make rochester_lookup compatible with dask_awkward by @jrueb in #875
• ci(pre-commit): pre-commit autoupdate by @pre-commit-ci in #898
• ci(pre-commit): pre-commit autoupdate by @pre-commit-ci in #905
• feat: support shape_touched from Dask by @agoose77 in #900
• fix: added hasattr check to NanoAODEvents repr by @pvisco...