Releases: pola-rs/polars
Python Polars 0.20.22-rc.1
🚀 Performance improvements
- improve filter parallelism (#15686)
🐞 Bug fixes
- Add the missing feature flag for
ewm_mean_by
(#15687) - 8/16-bits int could also apply in place for log expr (#15680)
prepare_expression_for_context
shouldn't panic if exceptions raised from optimizer (#15681)
📖 Documentation
- Various deprecation docstring improvements (#15648)
🛠️ Other improvements
- Make
json_path_match
expr non-anonymous (#15682)
Thank you to all our contributors for making this release possible!
@henryharbeck, @reswqa and @ritchie46
Rust Polars 0.39.1
🚀 Performance improvements
- Fix regression that led to using only a single thread (#15667)
✨ Enhancements
- add ewm_mean_by (#15638)
🐞 Bug fixes
- Ensure profile of simple-projection only take own runtime (#15671)
- Panic if invalid array in object (#15664)
- Ensure 'CachedSchema' doesn't get synced between plans (#15661)
group_by
multiple null columns produce phantom row (#15659)- rolling_* aggs were behaving as if they return scalars in group-by (#15657)
- Correct the unsoundness slice range of
arr.min/max
(#15654) list.mean
fast path shouldn't produce NaN (#15652)- Fix Display implementation of Duration (#15647)
📖 Documentation
🛠️ Other improvements
- Remove the remaining usage of deprecated
numpy
crate APIs (#15668) - make Duration.is_constant_duration less strict for non-timezone-aware case (#15639)
- Fix some typos in comments (#15665)
- remove unnecessary unsafe in list mean/sum (#15660)
Thank you to all our contributors for making this release possible!
@MarcoGorelli, @Priyansh4444, @StevenMia, @itamarst, @mcrumiller, @orlp, @reswqa, @ritchie46 and @stinodego
Python Polars 0.20.21
🚀 Performance improvements
- Fix regression that led to using only a single thread (#15667)
✨ Enhancements
- add ewm_mean_by (#15638)
🐞 Bug fixes
- Ensure profile of simple-projection only take own runtime (#15671)
- Panic if invalid array in object (#15664)
- Ensure 'CachedSchema' doesn't get synced between plans (#15661)
group_by
multiple null columns produce phantom row (#15659)- rolling_* aggs were behaving as if they return scalars in group-by (#15657)
- Correct the unsoundness slice range of
arr.min/max
(#15654) list.mean
fast path shouldn't produce NaN (#15652)
📖 Documentation
- Add missing deprecation warning to
DataFrame.replace
(#15612) - Fix typo in legacy install instructions (#15662)
🛠️ Other improvements
- make Duration.is_constant_duration less strict for non-timezone-aware case (#15639)
- Fix some typos in comments (#15665)
- remove unnecessary unsafe in list mean/sum (#15660)
- fixup failing test due to
offset
deprecation inupsample
(#15636)
Thank you to all our contributors for making this release possible!
@MarcoGorelli, @Priyansh4444, @StevenMia, @eitsupi, @itamarst, @mcrumiller, @orlp, @reswqa, @ritchie46 and @stinodego
Rust Polars 0.39.0
🏆 Highlights
- Full plan CSE (#15264)
💥 Breaking changes
- rename
memmap
->memory_map
as like Python (#15642) - pref(rust!, python): Unify
sort
withSortOptions
andSortMultipleOptions
(#15590) - Update the argument name from
dims
todimensions
inreshape
(#15561) - Allow specifying Hive schema in
read/scan_parquet
(#15434) - Raise error when
schema_overrides
contains nonexistent column name (#15290) - Rename
Chunk
toRecordBatch
(#15298) - Refactor AnyValue supertype logic (#15280)
- Rename
group_by_rolling
torolling
and improve related error messages (#14765) - Rename
ChunkedArray.try_apply
totry_apply_values
(#14947) - Rename parameter
by
togroup_by
inDataFrame.upsample/group_by_dynamic/rolling
(#14840)
🚀 Performance improvements
- Fix cross join batch size when one of the DataFrames is tiny (#14347)
- Fix binview growable complexity O(n*m) -> O(n) (#15628)
- Remove extra thread spawn from row group fetcher (#15626)
- Use vertical parallelism if input is chunked for
Filter
,Select
,WithColumns
(#15608) - read_ipc memory usage tests, and writing fix (#15599)
- Refactor CSV serialization to not go thorough
AnyValue
(#15576) - don't use dynamic dispatch in visitors (#15607)
- Improve Bitmap construction performance (#15570)
- join by row-encoding (#15559)
- Replace std::thread spawn with tokio block_in_place (#15517)
- speed up offset_by when a single offset is passed (#15493)
- Avoid allocation in the hot path for struct JSON serialization (#15449)
- avoid double-allocation in rolling_apply_agg_window (#15423)
- Make LogicalPlan immutable (#15416)
- Add non-order preserving variable row-encoding (#15414)
- Use row-encoding for multiple key group by (#15392)
- load bits one word at a time for BitmapIter (#15333)
- Ipc exec multiple paths (#15040)
- add SIMD support for if-then-else kernels (#15131)
✨ Enhancements
- add Expr.dt.add_business_days and Series.dt.add_business_days (#15595)
- Add
str.head
andstr.tail
(#14425) - Extended
BytecodeParser
to handle additional math functions, and imports from the global namespace (#15627) - Push down
is_between
expressions to Arrow (#15180) - add holidays argument to business_day_count (#15580)
- change default to write parquet statistics (#15597)
- Expressify
to_integer
(#15604) - Optimizer; remove double SORT and redundant projections (#15573)
- Add
null_on_oob
parameter toexpr.array.get
(#15426) - support weekend argument in business_day_count (#15544)
- Enable
is_first/last_distinct
for not nested non-numeric list (#15552) - Turn off cse if cache node found (#15554)
- Tag concat list as elementwise (#15545)
- Support list group-by of non numeric lists (#15540)
- add business_day_count function (#15512)
- Add SQL support for
MEDIAN
aggfunc (#15519) - Implement
string
,boolean
andbinary
dtype intop_k
(#15488) - Add SQL support for
TRUNCATE TABLE
command (#15513) - Add SQL support for
GREATEST
andLEAST
(#15511) - Allow specifying Hive schema in
read/scan_parquet
(#15434) - Implements
agg_list
forNullChunked
(#15439) - Supports
explode_by_offsets
for decimal (#15417) - Add
null_on_oob
parameter toexpr.list.get
(#15395) - CSV-writer escape carriage return (#15399)
- Remove 'FileCacher' optimization (#15357)
- check input type in entropy (#15351)
- Implements
arr.n_unique
(#15296) - CSE don't scan share if predicate pushdown predicates don't match (#15328)
- Remove cached nodes when finished (#15310)
- Full plan CSE (#15264)
- Add IR for expressions. (#15168)
- Warn if
map_elements
is called withoutreturn_dtype
specified (#15188) - Rename
group_by_rolling
torolling
and improve related error messages (#14765) - Rename
ChunkedArray.try_apply
totry_apply_values
(#14947) - Implement strict AnyValue construction for temporal types (#15146)
🐞 Bug fixes
- Return appropriate data type for time
mean
andmedian
(#14471) - Support index upsampling (#13621)
- Fix issue in
write_excel
that could lead to incorrect spanning range determination (#15631) - Output correct dtype for
mean_horizontal
on a single column (#15118) - Recompute RowIndex schema after projection pd (#15625)
- Mean of boolean in streaming group_by incorrectly always gave NULL (#15616)
- Include cloud creds in cache key (#15609)
- Fix elementwise-apply if any input is
AggregatedScalar
(#15606) - Explode list should take validity into account (#15572)
- use larger recursive stack in debug mode (#15593)
- SQL interface "off-by-one' indexing error with
GROUP BY
clauses that use position ordinals (#15584) - Enable missing features in polars-time (#15558)
- Handle quoted identifiers when registering CTEs in the SQL engine (#15564)
- Decompress moved out of schema initialization (#15550)
- Turn off cse if cache node found (#15554)
- Resolve function names and prune all aliases. (#15522)
list.get
should take validity into account (#15516)- block decimal in streaming (#15520)
group_by
partitioned with literalSeries
panic (#15487)- Initialize validity for
GroupsProxy::Slice
windows (#15509) - Fix struct name resolving (#15507)
pow
return type evaluation (#15506)- Allow selectors inside frame-level
.filter()
(#15445) - Don't prune alias in AnonymousFunction subtree (#15453)
- Fix deadlock in async parquet scan (#15440)
- datetime operations (e.g. .dt.year) were raising when null values were backed by out-of-range integers (#15420)
- Ensure Binary -> Binview cast doesn't overflow the buffer size (#15408)
- Don't prune alias in function subtree (#15406)
- Return 0 for
n_unique()
in group-by context when group is empty (#15289) - Unset UpdateGroups after group-sensitive expression (#15400)
to_any_value
should supports all LiteralValue type (#15387)- Hash failure combining hash of two numeric columns containing equal values (#15397)
- Add FixedSizeBinary to arrow field conversion (#15389)
- Conversion of expr_ir in partition fast path (#15388)
sort
for series with unsupported dtype should raise instead of panic (#15385)- Return correct dtype for
s.clear()
when dtype isObject
(#15315) - ensure first datapoint is always included in group_by_dynamic (#15312)
- Non-exhaustive patterns: arrow-schema::DataType in polars-arrow (#15250)
- use dynamic stacks for problematic recursive functions (#15355)
- Raise error when
schema_overrides
contains nonexistent column name (#15290) - Fix cache dot visualization (#15311)
- Properly propagate
strict
flag when constructing a Struct Series from any values (#15302) - ensure
eq
forBinaryViewArray
checks all elements (#15268) - Raise when join projects name with suffix that doesn't exist (#15256)
- fix kurtosis/skew (#15137)
- Ensure ooc_start is set (#15255)
- Fix bug where rolling operations were ignoring
check_sorted
in some cases (#15227) - Fix lazy schema for
rle
expression (#15248) - incorrect negative offset in multi-byte string slicing (#15140)
- do not clamp negative offsets to start of array prematurely (#15242)
- allow null index in list.get and array.get (#15239)
- properly support nulls_last + descending (#15212)
- Block rounding/truncating to negative durations (#15175)
- Make parse_url work on windows with object_store (#15191)
- divide by zero in download speed computation (#15182)
📖 Documentation
- Add legacy CPU install instructions in user guide (#13676)
- Various minor updates to User Guide's SQL intro section (#15557)
- Add
outer_coalesce
join strategy in the user guide (#15405) - Improve docs for
Series::new
withAnyValue
input (#15306) - Fix formatting in
Series::from_any_values_and_dtype
docs (#15244) - Correct the definition of an expression in the user guide (#14750)
📦 Build system
🛠️ Other improvements
- rename
memmap
->memory_map
as like Python (#15642) - fixup failing test due to
offset
deprecation inupsample
(#15636) - use bound api (#15630)
- Don't run streaming group-by in partitionable gb (#15611)
- pref(rust!, python): Unify
sort
withSortOptions
andSortMultipleOptions
(#15590) - remove try_binary_elementwise_values (#15592)
- remove raw pointers from visitors. (#15579)
- rename to IR (#15571)
- Update the argument name from
dims
todimensions
inreshape
(#15561) - Rename ALogicalPlan to FullAccessIR (#15553)
- Set up CodSpeed (#15537)
- make dsl immutable and cheap to clone (#15394)
- use recursive crate, add missing recursive tag (#15393)
- Update CODEOWNERS (polars-sql) (#15384)
- Update Rust toolchain (#15353)
- Update CODEOWNERS (#15352)
- remove try_apply_values (#15336)
- always use non-legacy float_sum for mean (#15343)
- remove legacy bitmap module (#15335)
- More clippy in Makefile (#15340)
- Rename
Cache[count]
toCache[cache_hits]
(#15300) - Cleanup file_caching optimization call (#15299)
- Rename
Chunk
toRecordBatch
(#15298) - Refactor AnyValue supertype logic (#15280)
- reuse message parsing in IPC (#15265)
- remove 'fast-projection' node (#15253)
- cleanup column names in optimizer (#15252)
- remove left_most_input_name from expr ir (#15251)
- add AlignedBitmapSlice (#15171)
- Refactor AnyValue construction for Categorical/Enum dtype (#15220)
- Move ConsecutiveCountState into support module (#15186)
- Run non-benchmark tests in benchmark workflow (#15207)
- Add
wrapping_abs
to arithmetic kernel (#15210) - remove raw buffers from BinViewArray (#15206)
- Enable
RUST_BACKTRACE=1
in the CI test suite (#15204) - Rename parameter
by
togroup_by
inDataFrame.upsample/group_by_dynamic/rolling
(#14840) - Set dual license for
polars-arrow
andpolars-parquet
(#15173)
...
Python Polars 0.20.20
🚀 Performance improvements
- Fix cross join batch size when one of the DataFrames is tiny (#14347)
- Fix binview growable complexity O(n*m) -> O(n) (#15628)
- Remove extra thread spawn from row group fetcher (#15626)
- Use vertical parallelism if input is chunked for
Filter
,Select
,WithColumns
(#15608) - Refactor CSV serialization to not go thorough
AnyValue
(#15576) - don't use dynamic dispatch in visitors (#15607)
- Improve Bitmap construction performance (#15570)
- join by row-encoding (#15559)
✨ Enhancements
- add Expr.dt.add_business_days and Series.dt.add_business_days (#15595)
- Add
str.head
andstr.tail
(#14425) - Add
union
/or
operator forpl.Enum
(#14965) - Extended
BytecodeParser
to handle additional math functions, and imports from the global namespace (#15627) - Push down
is_between
expressions to Arrow (#15180) - add holidays argument to business_day_count (#15580)
- change default to write parquet statistics (#15597)
- Expressify
to_integer
(#15604) - Optimizer; remove double SORT and redundant projections (#15573)
- Add
null_on_oob
parameter toexpr.array.get
(#15426) - support weekend argument in business_day_count (#15544)
- Enable
is_first/last_distinct
for not nested non-numeric list (#15552) - Turn off cse if cache node found (#15554)
- Tag concat list as elementwise (#15545)
🐞 Bug fixes
- Return appropriate data type for time
mean
andmedian
(#14471) - Fix issue in
write_excel
that could lead to incorrect spanning range determination (#15631) - Output correct dtype for
mean_horizontal
on a single column (#15118) - Recompute RowIndex schema after projection pd (#15625)
- Mean of boolean in streaming group_by incorrectly always gave NULL (#15616)
- Include cloud creds in cache key (#15609)
- Fix elementwise-apply if any input is
AggregatedScalar
(#15606) - Explode list should take validity into account (#15572)
- use larger recursive stack in debug mode (#15593)
- SQL interface "off-by-one' indexing error with
GROUP BY
clauses that use position ordinals (#15584) - Enable missing features in polars-time (#15558)
- Handle quoted identifiers when registering CTEs in the SQL engine (#15564)
- Decompress moved out of schema initialization (#15550)
- Turn off cse if cache node found (#15554)
📖 Documentation
- Add legacy CPU install instructions in user guide (#13676)
- Examples for errors (#13724)
- Add docstring examples for reading json (#14481)
- Add security warning in LazyFrame.deserialize() docstring (#15282)
- Various minor updates to User Guide's SQL intro section (#15557)
🛠️ Other improvements
- Replace most deprecated calls with bounded version (#15632)
- use bound api (#15630)
- Initial PyO3 0.21 support (#15622)
- Don't run streaming group-by in partitionable gb (#15611)
- pref(rust!, python): Unify
sort
withSortOptions
andSortMultipleOptions
(#15590) - Set up CodSpeed (#15537)
Thank you to all our contributors for making this release possible!
@CanglongCl, @ChayimFriedman2, @Fokko, @JamesCE2001, @MarcoGorelli, @NedJWestern, @TrevorWinstral, @alexander-beedie, @deanm0000, @douglas-raillard-arm, @eitsupi, @filabrazilska, @i-aki-y, @itamarst, @leoforney, @mcrumiller, @nameexhaustion, @orlp, @ozgrakkurt, @reswqa, @ritchie46 and @stinodego
Python Polars 0.20.19
🚀 Performance improvements
- Replace std::thread spawn with tokio block_in_place (#15517)
- speed up offset_by when a single offset is passed (#15493)
- Avoid allocation in the hot path for struct JSON serialization (#15449)
✨ Enhancements
- Support list group-by of non numeric lists (#15540)
- add business_day_count function (#15512)
- Add SQL support for
MEDIAN
aggfunc (#15519) - Implement
string
,boolean
andbinary
dtype intop_k
(#15488) - Add SQL support for
TRUNCATE TABLE
command (#15513) - Add SQL support for
GREATEST
andLEAST
(#15511) - Allow specifying Hive schema in
read/scan_parquet
(#15434) - Implements
agg_list
forNullChunked
(#15439)
🐞 Bug fixes
- dot product of two integer series is cast to float (#15502)
- Resolve function names and prune all aliases. (#15522)
- Pass
skip_rows_after_header
to pyarrow csv reader (#15533) - No longer error when
schema_overrides
contains nonexistent columns (#15528) list.get
should take validity into account (#15516)- block decimal in streaming (#15520)
group_by
partitioned with literalSeries
panic (#15487)- Initialize validity for
GroupsProxy::Slice
windows (#15509) - Fix struct name resolving (#15507)
pow
return type evaluation (#15506)- Address issue with
read_database
draining iter_batches early (#15504) - Allow selectors inside frame-level
.filter()
(#15445) - Don't prune alias in AnonymousFunction subtree (#15453)
- Raise if pass a negative
n
intoclear
(#15432) - Fix deadlock in async parquet scan (#15440)
📖 Documentation
- Update leftover references of
by
parameter togroup_by
inDataFrame/LazyFrame.upsample/group_by_dynamic/rolling
(#15527) - Add
make docs
command, DataType docs/layout tweak, minor README updates (#15386) - Add example for
Series.list.median
. (#15451)
🛠️ Other improvements
- Remove unused code paths in
read_parquet
(#15532) - Organize utils for I/O functionality (#15529)
- Remove private
DataFrame._read
classmethods (#15521) - Move dedicated inference code out of
io.database
executor module (#15526) - Add unstable warning to
hive_schema
functionality (#15508)
Thank you to all our contributors for making this release possible!
@CanglongCl, @ChayimFriedman2, @MarcoGorelli, @alexander-beedie, @cmdlineluser, @dependabot, @dependabot[bot], @henryharbeck, @mbuhidar, @nameexhaustion, @reswqa, @ritchie46, @rob-sil and @stinodego
Python Polars 0.20.18
🚀 Performance improvements
- CSV reading memory usage tests and fixes (#15422)
- avoid double-allocation in rolling_apply_agg_window (#15423)
- Make LogicalPlan immutable (#15416)
- Add non-order preserving variable row-encoding (#15414)
- Use row-encoding for multiple key group by (#15392)
✨ Enhancements
- Supports
explode_by_offsets
for decimal (#15417) - Add
read_clipboard
andDataFrame.write_clipboard
(#15272) - Add
null_on_oob
parameter toexpr.list.get
(#15395) - make Series.__bool__ error message Rusttier (#15407)
- CSV-writer escape carriage return (#15399)
🐞 Bug fixes
- datetime operations (e.g. .dt.year) were raising when null values were backed by out-of-range integers (#15420)
- Ensure Binary -> Binview cast doesn't overflow the buffer size (#15408)
- Don't prune alias in function subtree (#15406)
- Return 0 for
n_unique()
in group-by context when group is empty (#15289) - Unset UpdateGroups after group-sensitive expression (#15400)
to_any_value
should supports all LiteralValue type (#15387)- Hash failure combining hash of two numeric columns containing equal values (#15397)
- Add FixedSizeBinary to arrow field conversion (#15389)
- Conversion of expr_ir in partition fast path (#15388)
- fix panic when doing a scan_parquet with hive partioning (#15381)
sort
for series with unsupported dtype should raise instead of panic (#15385)
📖 Documentation
- Added example for
explode
mapping strategy inpl.Expr.over
(#15402) - Add
outer_coalesce
join strategy in the user guide (#15405) - Change the example to series for
series/array.py
(#15383) - Add "See Also" for
arg_sort
andarg_sort_by
(#15348)
🛠️ Other improvements
- make dsl immutable and cheap to clone (#15394)
- use recursive crate, add missing recursive tag (#15393)
- Update CODEOWNERS (polars-sql) (#15384)
Thank you to all our contributors for making this release possible!
@CanglongCl, @JamesCE2001, @MarcoGorelli, @Sol-Hee, @alexander-beedie, @dependabot, @dependabot[bot], @itamarst, @kszlim, @mcrumiller, @nameexhaustion, @orlp, @reswqa, @ritchie46, @rob-sil and @thomaslin2020
Python Polars 0.20.17
🏆 Highlights
- Full plan CSE (#15264)
⚠️ Deprecations
- Rename parameter
by
togroup_by
inDataFrame.upsample/group_by_dynamic/rolling
(#14840) - Rename
from_repr
parameter fromtbl
todata
(#15156)
🚀 Performance improvements
- load bits one word at a time for BitmapIter (#15333)
- Ipc exec multiple paths (#15040)
- add SIMD support for if-then-else kernels (#15131)
✨ Enhancements
- Remove 'FileCacher' optimization (#15357)
- check input type in entropy (#15351)
- Implements
arr.n_unique
(#15296) - CSE don't scan share if predicate pushdown predicates don't match (#15328)
- Add
read_database
support forSurrealDB
("ws" and "http") (#15269) - Only allow inputs of type
Sequence
infrom_records
(#15329) - In hypothesis testing strategies, enable Decimal strategy by default (#15321)
- Remove cached nodes when finished (#15310)
- Full plan CSE (#15264)
- More robust handling of
async
database calls (#15202) - Add
name
parameter toGroupBy.len
method (#15235) - Add IR for expressions. (#15168)
- Improve
read_database
when reading from Kùzu graph database (#15218) - Warn if
map_elements
is called withoutreturn_dtype
specified (#15188) - Add support for
async
SQLAlchemy connections toread_database
(#15162) - Infer
time_unit
inpl.duration
when nanoseconds is specified (#14987) - Add
strict
parameter tofrom_dict/from_records
(#15158)
🐞 Bug fixes
- Return correct dtype for
s.clear()
when dtype isObject
(#15315) - ensure first datapoint is always included in group_by_dynamic (#15312)
- Non-exhaustive patterns: arrow-schema::DataType in polars-arrow (#15250)
- use dynamic stacks for problematic recursive functions (#15355)
- Adding default ddof for
Series.list.std
andSeries.list.var
(#15267) - Raise properly for slices not supported by
LazyFrame
(#15331) - Propagate strictness in
from_dicts
(#15344) - Raise error when
schema_overrides
contains nonexistent column name (#15290) - Enforce integer
dtype
input forint_range
andint_ranges
(#15339) - Preserve Decimal precision when constructing empty Series (#15320)
- Fix cache dot visualization (#15311)
- Handle special case correctly when slicing a
LazyFrame
(#15297) - Properly propagate
strict
flag when constructing a Struct Series from any values (#15302) - Consistent expansion of nested struct data during
DataFrame
init from dict (#15217) - Raise when join projects name with suffix that doesn't exist (#15256)
- Ensure ooc_start is set (#15255)
- Fix bug where rolling operations were ignoring
check_sorted
in some cases (#15227) - Fix lazy schema for
rle
expression (#15248) - incorrect negative offset in multi-byte string slicing (#15140)
- do not clamp negative offsets to start of array prematurely (#15242)
- allow null index in list.get and array.get (#15239)
- Avoid loading all columns in
read_parquet
whencolumns
parameter is specified (#15229) - properly support nulls_last + descending (#15212)
- fix nested runtime panic (#15216)
- Block rounding/truncating to negative durations (#15175)
- Ensure the
cs.temporal()
selector uses wildcard time zone matching forDatetime
(#13683) - Consistently raise
TypeError
on constructor failure (#15178) - Properly propagate strictness in some constructor cases (#15166)
- Fix constructing a Series from a list of Series with given dtype (#15144)
📖 Documentation
- Fix time unit in
timestamp
example (#15281) - Fix link to renamed method (.list.lengths -> .list.len) (#15228)
- Update Excel and database pages in user guide (#14721)
- Add examples for
Series.search_sorted
(#14737) - Correct the definition of an expression in the user guide (#14750)
- Add a note about the behaviour of lower/upper bounds for
is_between
, and add an example (#15197)
📦 Build system
- Update Cargo lock (#15370)
🛠️ Other improvements
- Memory usage test infrastructure, plus a test for #15098 (#15285)
- Update CODEOWNERS (#15352)
- remove try_apply_values (#15336)
- always use non-legacy float_sum for mean (#15343)
- remove legacy bitmap module (#15335)
- Fix test not writing to temporary directory (#15318)
- Reorganize tests for
clear
operation (#15304) - Rename
Cache[count]
toCache[cache_hits]
(#15300) - Cleanup file_caching optimization call (#15299)
- Minor refactor of
PyDataFrame.from_dicts
(#15274) - remove 'fast-projection' node (#15253)
- cleanup column names in optimizer (#15252)
- remove left_most_input_name from expr ir (#15251)
- add AlignedBitmapSlice (#15171)
- Run non-benchmark tests in benchmark workflow (#15207)
- Add
wrapping_abs
to arithmetic kernel (#15210) - remove raw buffers from BinViewArray (#15206)
- Enable
RUST_BACKTRACE=1
in the CI test suite (#15204) - Split
read_database
functionality into cleaner module structure (#15201) - Clean up some of the AnyValue conversion logic (#15190)
- remove parts of legacy bit_util (#15169)
- remove legacy arrow compute (#15164)
- Split up
dataframe
module in PyO3 bindings (#15165) - Remove unused private constructors (#15160)
Thank you to all our contributors for making this release possible!
@MarcoGorelli, @alexander-beedie, @braaannigan, @c-peters, @cojmeister, @deanm0000, @dependabot, @dependabot[bot], @itamarst, @kszlim, @mbuhidar, @mcrumiller, @mickvangelderen, @orlp, @petrosbar, @reswqa, @ritchie46, @rob-sil, @sportfloh, @stinodego and @yutannihilation
Rust Polars 0.38.3
🚀 Performance improvements
- add new when-then-otherwise kernels (#15089)
- Coerce sorted flag of unit arrays during concat (#15104)
- Use sorted flag for
(first|last)_non_null
(#15050) - OOC sort improvements (#14994)
✨ Enhancements
- raise if both
closed
andby
are passed torolling_*
aggregations (#15108) - raise informative error for rolling_* aggs with
by
of invalid dtype (#15088) - add
non_existent
arg toreplace_time_zone
(#15062) - Support single nested row encodings (#15105)
- make ooc sort configurable (#15084)
- Make
register_plugin
a standalone function and include shared lib discovery (#14804) - Async parquet: Decode parquet on a blocking thread pool (#15083)
- let "ambiguous" take "null" value (#14961)
- Raise informative error message when join would introduce duplicate column name (#15042)
- Allow cast of decimal to boolean (#15015)
- Return error when no supertype can be determined in AnyValue constructor when
strict=false
(#15025) - Implement IpcReaderAsync (#14984)
- Support Array statistics in parquet (#15031)
- Support decimal groupby (#15000)
- Add thread names to rayon thread pool (#15024)
- Support decimal uniq (#15001)
- expose timings in verbose state of OOC sort (#14979)
🐞 Bug fixes
- Fix Series construction from nested list with mixed data types (#15046)
- Support BinaryView in row decoder to prevent a panic in streaming group by (#15117)
- Binview chunked gather; don't modify inlined view (#15124)
- Fix chunked_id gather for binview buffers (#15123)
- Don't cache HTTP object stores as they maintain URL state (#15121)
- use wrapping_add in csv line snooping (#15109)
- Output
u32
whensum_horizontal
provided with single boolean column (#15114) - Ensure
eprintln!
is only called within debug/verbose context (#15100) - Propagate error instead of panicking when calling
product
on an invalid type (#15093) - Raise error when casting Array to different width (#14995)
- Fix file scan bugs for ipc, csv and parquet that occur with combinations of glob paths, row indices and predicates (#15065)
- Incorrectly preserved sorted flag when concatenating sorted series containing nulls (#15082)
- Return largest non-NaN value for
max()
on sorted float arrays if it exists instead of NaN (#15060) - return NaN for all-NaN min/max (#15066)
- Prevent "index out of range for slice" error in parquet reader (#15021)
- Respect
nulls_last
in streaming sort (#15061) - Fix Series construction from nested list with mixed data types (#15046)
- Don't count nulls in streaming
count
agg (#15051) - agg_list on decimal lost scale (#15054)
- Block predicate pushdown on equality that are use in join (#15055)
- Enum equality based on categories (#15053)
- Strict cast in when/then/otherwise operation (#15052)
- Don't panic in
string_addition_to_linear_concat
(#15006) - CSV do utf8-validation after escaping fields (#15004)
- Use primitive constructors to create a Series of lists when dtype is provided (#15002)
- replace_time_zone with single-null-element "ambiguous" was panicking (#14971)
📖 Documentation
- Fix typo in comment (#14997)
🛠️ Other improvements
- Extend and speed up scan tests (#15127)
- always assert on ChunkedArray::get (#15120)
- Use ObjectStore instead of AsyncRead in parquet get metadata (#15069)
- Minor refactor of Rust any value constructors (#15077)
- Simplify streaming execution (#15039)
- Ensure we hit the spilled source path in ooc sort test (#15010)
- Refactor constructor code (#15009)
- Apply
clippy:assigning_clones
lint (#14999) - fix features (#14977)
Thank you to all our contributors for making this release possible!
@JackRolfe, @MKisilyov, @MarcoGorelli, @alexander-beedie, @c-peters, @flisky, @jqnatividad, @mcrumiller, @mickvangelderen, @nameexhaustion, @orlp, @petrosbar, @ritchie46, @stinodego and @trueb2
Python Polars 0.20.16
🚀 Performance improvements
- add new when-then-otherwise kernels (#15089)
- Coerce sorted flag of unit arrays during concat (#15104)
- Use sorted flag for
(first|last)_non_null
(#15050) - OOC sort improvements (#14994)
✨ Enhancements
- improved dtype inference/refinement for
read_database
results (#15126) - raise if both
closed
andby
are passed torolling_*
aggregations (#15108) - raise informative error for rolling_* aggs with
by
of invalid dtype (#15088) - add
non_existent
arg toreplace_time_zone
(#15062) - Support single nested row encodings (#15105)
- make ooc sort configurable (#15084)
- Make
register_plugin
a standalone function and include shared lib discovery (#14804) - Expose
infer_schema_length
parameter onread_database
(#15076) - Async parquet: Decode parquet on a blocking thread pool (#15083)
- let "ambiguous" take "null" value (#14961)
- Raise informative error message when join would introduce duplicate column name (#15042)
- Allow cast of decimal to boolean (#15015)
- Add
strict
parameter toDataFrame
constructor to allow non-strict construction (#15034) - Support Array statistics in parquet (#15031)
- Support decimal groupby (#15000)
- Add thread names to rayon thread pool (#15024)
- Support decimal uniq (#15001)
- expose timings in verbose state of OOC sort (#14979)
🐞 Bug fixes
- Support BinaryView in row decoder to prevent a panic in streaming group by (#15117)
- Binview chunked gather; don't modify inlined view (#15124)
- Fix chunked_id gather for binview buffers (#15123)
- Don't cache HTTP object stores as they maintain URL state (#15121)
- Output
u32
whensum_horizontal
provided with single boolean column (#15114) - Propagate error instead of panicking when calling
product
on an invalid type (#15093) - Raise error when casting Array to different width (#14995)
- Fix file scan bugs for ipc, csv and parquet that occur with combinations of glob paths, row indices and predicates (#15065)
- Incorrectly preserved sorted flag when concatenating sorted series containing nulls (#15082)
- Return largest non-NaN value for
max()
on sorted float arrays if it exists instead of NaN (#15060) - return NaN for all-NaN min/max (#15066)
- Prevent "index out of range for slice" error in parquet reader (#15021)
- Respect
nulls_last
in streaming sort (#15061) - Fix Series construction from nested list with mixed data types (#15046)
- Don't count nulls in streaming
count
agg (#15051) - agg_list on decimal lost scale (#15054)
- Block predicate pushdown on equality that are use in join (#15055)
- Enum equality based on categories (#15053)
- Don't panic in
string_addition_to_linear_concat
(#15006) - CSV do utf8-validation after escaping fields (#15004)
- Use primitive constructors to create a Series of lists when dtype is provided (#15002)
- replace_time_zone with single-null-element "ambiguous" was panicking (#14971)
📖 Documentation
- Update write_database code blocks in user guide (#15106)
- Add missing docstring examples in the Struct namespace (#15071)
- Improve API reference landing page (#14888)
- improve join_asof example (#14993)
- Fix inadvertent swap of
new
andold
parameters inreplace
description (#15019)
🛠️ Other improvements
- Extend and speed up scan tests (#15127)
- Add parameterized-scan-tests (#15057)
- Simplify streaming execution (#15039)
- Ensure we hit the spilled source path in ooc sort test (#15010)
- Refactor constructor code (#15009)
- fix features (#14977)
- Revert pinning PyPI publish action (#14975)
Thank you to all our contributors for making this release possible!
@JackRolfe, @MKisilyov, @MarcoGorelli, @alexander-beedie, @c-peters, @flisky, @jqnatividad, @mcrumiller, @mickvangelderen, @nameexhaustion, @orlp, @petrosbar, @ritchie46, @stinodego and @trueb2