Polars: py-1.0.0-rc.2 Release

Release date:
June 24, 2024
Previous version:
py-1.0.0-rc.1 (released June 23, 2024)
Magnitude:
2,196 Diff Delta
Contributors:
10 total committers
Data confidence:
Commits:

18 Commits in this Release

Ordered by the degree to which they evolved the repo in this version.

Authored June 24, 2024
Authored June 24, 2024
Authored June 24, 2024

Top Contributors in py-1.0.0-rc.2

coastalwhite
alexander-beedie
adamreeve
ritchie46
datapythonista
orlp
lukeshingles
eitsupi
nameexhaustion
mcrumiller

Directory Browser for py-1.0.0-rc.2

All files are compared to previous version, py-1.0.0-rc.1. Click here to browse diffs between other versions.

Loading File Browser...

Release Notes Published

πŸ’₯ Breaking changes

  • Make hive_partitioning parameter default to None, which is automatically enabled for single directory inputs, and disabled otherwise (#17106)
  • Split replace functionality into two separate functions (#16921)
  • Default to writing binview data to IPC (#17084)
  • Remove re-export of type aliases (#17032)
  • Add strict parameter to DataFrame/LazyFrame.drop and fix behavior to default to True (#17044)
  • Rename ModuleUpgradeRequired and PolarsPanicError error, remove InvalidAssert error (#17033)
  • Change data orientation inference logic for DataFrame construction and warn when row orientation is inferred (#16976)
  • Properly apply strict parameter in Series constructor (#16939)
  • Remove supertype definition of List and non-List types (#16918)
  • Consistently convert to given time zone in Series constructor (#16828)
  • Update reshape to return Array types instead of List types (#16825)
  • Default to raising on out-of-bounds indices in all get/gather operations (#16841)
  • Native selector XOR set operation, guarantee consistent selector column-order (#16833)
  • Set infer_schema_length as keyword-only argument in str.json_decode (#16835)
  • Update set_sorted to only accept a single column (#16800)
  • Remove deprecated parameters in Series.cut/qcut and update struct field names (#16741)
  • Expedited removal of certain deprecated functionality (#16754)
  • Update some error types to more appropriate variants (#15030)
  • Scheduled removal of deprecated functionality (#16715)
  • Change default offset in group_by_dynamic from 'negative every' to 'zero' (#16658)
  • Constrain access to globals from DataFrame.sql in favor of top-level pl.sql (#16598)
  • Read 2D NumPy arrays as Array type instead of List (#16710)
  • Update clip to no longer propagate nulls in the given bounds (#14413)
  • Change str.to_datetime to default to microsecond precision for format specifiers "%f" and "%.f" (#13597)
  • Update resulting column names in pivot when pivoting by multiple values (#16439)
  • Preserve nulls in ewm_mean, ewm_std, and ewm_var (#15503)
  • Restrict casting for temporal data types (#14142)
  • Support Decimal types by default when converting from Arrow (#15324)
  • Remove serde functionality from pl.read_json and DataFrame.write_json (#16550)
  • Update function signature of nth to allow positional input of indices, remove columns parameter (#16510)
  • Rename struct fields of rle output to len/value and update data type of len field (#15249)
  • Remove class variables from some DataTypes (#16524)
  • Add check_names parameter to Series.equals and default to False (#16610)

⚠️ Deprecations

  • Deprecate size parameter in parametric testing strategies in favor of min_size/max_size (#17128)
  • Split replace functionality into two separate functions (#16921)
  • Rename DataFrame.melt to unpivot and make parameters consistent with pivot (#17095)
  • Remove re-export of exceptions at top-level (#17059)
  • Deprecate dt.mean/dt.median in favor of mean/median (#16888)
  • Deprecate LazyFrame.with_context in favor of horizontal concatenation (#16860)
  • Rename parameter descending to reverse in top_k methods (#16817)
  • Rename str.concat to str.join and update default delimiter (#16790)
  • Deprecate arctan2d in favor of arctan2(...).degrees() (#16786)

πŸš€ Performance improvements

  • create UniqueKernel and improve bool implementation (#17160)
  • parallel linearize in new streaming engine (#17050)
  • Default to writing binview data to IPC (#17084)
  • Parallelize arrow conversion if binview -> large_bin (#17083)
  • GC buffers in if_then_else view kernel (#16993)
  • Desugar AND filter into multiple nodes (#16992)
  • Optimize generic argsort of row-encoding (#16894)
  • Improve rle_id iteration perf and set sorted flags (#16893)
  • Optimize string/binary sort (#16871)
  • Use split_at in split (#16865)
  • Use split_at instead of double slice in chunk splits. (#16856)
  • Don't rechunk in align_ if arrays are aligned (#16850)
  • Don't create small chunks in parallel collect. (#16845)
  • Add dedicated no-null branch in arg_sort (#16808)
  • Speed up dt.offset_by 2x for constant durations (#16728)
  • Toggle coalesce in join if non-coalesced key isn't projected (#16677)
  • Make dt.truncate 1.5x faster when every is just a single duration (and not an expression) (#16666)
  • Always prune unused columns in semi/anti join (#16665)

✨ Enhancements

  • Support reading byte stream split encoded floats and doubles in parquet (#17099)
  • Add float_scientific option to write_csv/sink_csv (#17111)
  • Support Struct field selection in the SQL engine, RENAME and REPLACE select wildcard options (#17109)
  • Update DataFrame.pivot to allow index=None when values is set (#17126)
  • Make hive_partitioning parameter default to None, which is automatically enabled for single directory inputs, and disabled otherwise (#17106)
  • Improve ipython autocomplete for LazyFrame and DataFrame (#17091)
  • Split replace functionality into two separate functions (#16921)
  • Improve schema inference for hive partitions (#17079)
  • Rename DataFrame.melt to unpivot and make parameters consistent with pivot (#17095)
  • print row index in explain + dot (#17074)
  • Support top-level pl.col autocompletion for iPython (#17080)
  • Remove re-export of exceptions at top-level (#17059)
  • predicate + projection pushdown in NDJson (#17068)
  • Allow (non-)coalescing in join_asof (#17066)
  • Turn of coalescing and fix mutation of join on expressions (#17061)
  • Expand NDJson glob into one SCAN (#17063)
  • Do not parse hive partitions from user provided base directory path (#17055)
  • Support directory paths in scans for Parquet, IPC and CSV (#17017)
  • Implement general array equality checks (#17043)
  • Add strict parameter to DataFrame/LazyFrame.drop and fix behavior to default to True (#17044)
  • Rename ModuleUpgradeRequired and PolarsPanicError error, remove InvalidAssert error (#17033)
  • Add rechunk parameter to read_delta (#16991)
  • allow experimental metadata use on release (#17005)
  • first working prototype of new streaming engine (#16970)
  • Add simple version of json_normalize (#17015)
  • Change data orientation inference logic for DataFrame construction and warn when row orientation is inferred (#16976)
  • Desugar AND filter into multiple nodes (#16992)
  • Handle textio even if not correct (#16971)
  • Properly apply strict parameter in Series constructor (#16939)
  • Add SQL support for INTERSECT and EXCEPT ops (#16960)
  • Add PerformanceWarning to LazyFrame properties (#16964)
  • Add collect_schema method to LazyFrame and DataFrame (#16929)
  • Allow setting file cache TTL on a per-file basis (#16891)
  • Support Decimal inputs for lit (#16950)
  • Implement multiply and division for lhs duration (#16948)
  • Raise on invalid temporal arithmetic (#16934)
  • Always end with a in-memory sink on collect (#16928)
  • add style namespace (which defers to Great Tables) (#16809)
  • Add Schema class (#16873)
  • Normalize value_counts (#16917)
  • add eq/ne for more FixedSizeLists (#16902)
  • setup skeleton (#16900)
  • add fundamentals for new async-based streaming execution engine (#16884)
  • Cache downloaded cloud IPC files (#16892)
  • Consistently convert to given time zone in Series constructor (#16828)
  • Improve read_csv SQL table reading function defaults (better handle dates) (#16866)
  • Support SQL VALUES clause and inline renaming of columns in CTE & derived table definitions (#16851)
  • Support Python Enum values in lit (#16858)
  • convert to give time zone in .str.to_datetime when values are offset-aware (#16742)
  • Update reshape to return Array types instead of List types (#16825)
  • Default to raising on out-of-bounds indices in all get/gather operations (#16841)
  • Support SQL "SELECT" with no tables, optimise registration of globals (#16836)
  • Native selector XOR set operation, guarantee consistent selector column-order (#16833)
  • Extend recognised EXTRACT and DATE_PART SQL part abbreviations (#16767)
  • Improve error message when raising integers to negative integers, improve docs (#16827)
  • Return datetime for mean/median of Date colum (#16795)
  • Update set_sorted to only accept a single column (#16800)
  • Expose overflowing cast (#16805)
  • Update group_by iteration and partition_by to always return tuple keys (#16793)
  • Support array arithmetic for equally sized shapes (#16791)
  • Expedited removal of certain deprecated functionality (2) (#16779)
  • Removal of read_database_uri passthrough from read_database (#16783)
  • Remove pyxlsb engine from read_database (#16784)
  • Add check_order parameter to assert_series_equal (#16778)
  • Enforce deprecation of keyword arguments as positional (#16755)
  • Support cloud storage in scan_csv (#16674)
  • Streamline SQL INTERVAL handling and improve related error messages, update sqlparser-rs lib (#16744)
  • Support use of ordinal values in SQL ORDER BY clause (#16745)
  • Support executing polars SQL against pandas and pyarrow objects (#16746)
  • Remove deprecated parameters in Series.cut/qcut and update struct field names (#16741)
  • Expedited removal of certain deprecated functionality (#16754)
  • Remove deprecated functionality from rolling methods (#16750)
  • Update date_range to no longer produce datetime ranges (#16734)
  • Mark min_periods as keyword-only for rolling methods (#16738)
  • Remove deprecated top_k parameters nulls_last, maintain_order, and multithreaded (#16599)
  • Support order-by in window functions (#16743)
  • Add SQL support for NULLS FIRST/LAST ordering (#16711)
  • Update some error types to more appropriate variants (#15030)
  • Initial SQL support for INTERVAL strings (#16732)
  • Scheduled removal of deprecated functionality (2) (#16724)
  • Scheduled removal of deprecated functionality (#16715)
  • Enforce deprecation of offset arg in truncate and round (#16655)
  • Change default offset in group_by_dynamic from 'negative every' to 'zero' (#16658)
  • Constrain access to globals from DataFrame.sql in favor of top-level pl.sql (#16598)
  • Read 2D NumPy arrays as Array type instead of List (#16710)
  • Update clip to no longer propagate nulls in the given bounds (#14413)
  • Change str.to_datetime to default to microsecond precision for format specifiers "%f" and "%.f" (#13597)
  • Update resulting column names in pivot when pivoting by multiple values (#16439)
  • Preserve nulls in ewm_mean, ewm_std, and ewm_var (#15503)
  • Restrict casting for temporal data types (#14142)
  • Add many more auto-inferable datetime formats for str.to_datetime (#16634)
  • Support Decimal types by default when converting from Arrow (#15324)
  • Remove serde functionality from pl.read_json and DataFrame.write_json (#16550)
  • Update function signature of nth to allow positional input of indices, remove columns parameter (#16510)
  • Rename struct fields of rle output to len/value and update data type of len field (#15249)
  • Remove class variables from some DataTypes (#16524)
  • Add check_names parameter to Series.equals and default to False (#16610)
  • Dedicated SQLInterface and SQLSyntax errors (#16635)
  • Add DIV function support to the SQL interface (#16678)
  • Support non-coalescing streaming left join (#16672)
  • Allow wildcard and exclude before struct expansions (#16671)

🐞 Bug fixes

  • Use explicit turbofish to help rustc (#17159)
  • Raise on invalid set dtypes (#17157)
  • Fix corrupted reads for hive parts from cloud and projection pushdown failure on hive parts (#17152)
  • Set intersection supertype (#17154)
  • ChainedWhen should not inherit Expr (#17142)
  • Fix decompress_impl for csv with n_rows set (#17118)
  • adds "polars-ops/timezones" dependency for "timezones" feature (#17115)
  • Fix incorrect window std for chunked series (#17110)
  • make GetOutput::get_field fallible (#17114)
  • Fix melt panic (#17088)
  • Fix expression autocomplete in ipython (#17072)
  • Exclude index from expansion in rolling/group_by_dynamic (#17086)
  • Update some Series dunder method type signatures (#17053)
  • Fix oob of join with literals and empty table (#17047)
  • Don't silently accept multi-table FROM clauses (implicit JOIN syntax) (#17028)
  • Don't split up ANDed filters that are group-aware (#17031)
  • Harden "async" check for users with out-of-date sqlalchemy libraries (#17029)
  • error when sort_by of unequal length (#17026)
  • properly catch not found explode cols (#17020)
  • Correctly convert data frames to NumPy for C index order (#17000)
  • Raise on invalid arithmetic shapes (#16986)
  • Don't pushdown predicates in cross join if the refer to both tables (#16983)
  • Fix projection pushdown with literal joins (#16981)
  • Fix edge case in DataFrame constructor data orientation inference (#16975)
  • Raise on list of objects (#16959)
  • Handle strictness for Decimal Series construction (#15309)
  • Don't panic in object to anyvalue (#16957)
  • properly set FAST_EXPLODE_LIST metadata (#16951)
  • Raise informative error when writing object to file (#16954)
  • Remove supertype definition of List and non-List types (#16918)
  • Remove unwrap in extend() (#16890)
  • Fix should_rechunk check (#16852)
  • Ensure read_excel and read_ods return identical frames across all engines when given empty spreadsheet tables (#16802)
  • Consistent behaviour when "infer_schema_length=0" for read_excel (#16840)
  • Standardised additional SQL interface errors (#16829)
  • Ensure that splitted ChunkedArray also flattens chunks (#16837)
  • Reduce needless panics in comparisons (#16831)
  • Reset if next caller clones inner series (#16812)
  • Raise on non-positive json schema inference (#16770)
  • Rewrite implementation of top_k/bottom_k and fix a variety of bugs (#16804)
  • Fix comparison of UInt64 with zero (#16799)
  • Fix incorrect parquet statistics written for UInt64 values > Int64::MAX (#16766)
  • Fix boolean distinct (#16765)
  • DATE_PART SQL syntax/parsing, improve some error messages (#16761)
  • Include pl. qualifier for inner dtypes in to_init_repr (#16235)
  • Column selection wasn't applied when reading CSV with no rows (#16739)
  • Panic on empty df / null List(Categorical) (#16730)
  • Only flush if operator can flush in streaming outer join (#16723)
  • Raise unsupported cat array (#16717)
  • Assert SQLInterfaceError is raised (#16713)
  • Restrict casting for temporal data types (#14142)
  • Handle nested categoricals in assert_series_equal when categorical_as_str=True (#16700)
  • Improve read_database check for SQLAlchemy async Session objects (#16680)
  • Reduce scope of multi-threaded numpy conversion (#16686)
  • Full null on dyn int (#16679)
  • Fix filter shape on empty null (#16670)

πŸ“– Documentation

  • Add doc examples to concat_list (#17127)
  • Add "coming from pandas" note to DataFrame.unique docstring (#17119)
  • Fix some warnings during doc build (#17077)
  • Properly expose InProcessQuery in docs, mark as unstable (#17097)
  • Add upgrade guide for Python Polars 1.0.0 (#16914)
  • Lots of additions to the SQL reference docs (#16990)
  • Minor doctest fixes (#17002)
  • Include a doc entry for every exception type (#17001)
  • fixup bullet points in write_parquet (#16909)
  • Update version switcher for 1.0.0 prereleases (#16847)
  • Update link from Python API reference to user guide (#16849)
  • Update docstring/test/etc usage of select and with_columns to idiomatic form (#16801)
  • Update versioning docs for 1.0.0 (#16757)
  • Add docstring example for DataFrame.limit (#16753)
  • Fix incorrect stated value of include_nulls in DataFrame.update docstring (#16701)
  • Update deprecation docs in the user guide (#14315)
  • Add example for index count in DataFrame.rolling (#16600)
  • Improve docstring of Expr/Series.map_elements (#16079)
  • Add missing polars.sql docs entry and small docstring update (#16656)

πŸ“¦ Build system

  • Do not change environment on import (#17101)
  • Fix config flag for Tracemalloc (#17098)
  • Pin optional NumPy dependency to < 2.0.0 for now (#17060)

πŸ› οΈ Other improvements

  • Add missing spaces in cargo.toml (#17145)
  • Update rustc 2024-06-23 (#17135)
  • Minor test refactor for concat_list (#17120)
  • Remove re-export of data type groups (#17073)
  • Add pivot test #17081 (#17090)
  • Minor cleanup to better define boundaries of public API (#17051)
  • Support directory paths in scans for Parquet, IPC and CSV (#17017)
  • Remove re-export of type aliases (#17032)
  • Remove file cache test (#17038)
  • Update exception imports in test suite (#17035)
  • Point polars-stream to crates/ again (#17024)
  • Fix failing file cache test in CI (#17014)
  • Add some parametric tests for sort functionality (#17008)
  • Pin NumPy to <2.0 for now (#16999)
  • Use proper join type in test (#16994)
  • Fix file cache verbose logging leakage during pytest (#16984)
  • Skip another intermitently failing AWS test (#16980)
  • Update test suite to explicitly use orient="row" in DataFrame constructor when applicable (#16977)
  • Remove redundant projection attribute in IR::DataFrameScan (#16952)
  • Factor out some apply calls in duration namespace (#16941)
  • extend new streaming engine with some initial nodes (#16940)
  • Skip intermittently failing AWS test (#16908)
  • Refactor expression parsing utils (#16906)
  • setup skeleton (#16900)
  • Refactor parts of IR. (#16899)
  • Move around some existing tests (#16877)
  • Remove inner Arc from FileCacheEntry (#16870)
  • Do not update stable API reference on prerelease (#16846)
  • Update links to API references (#16843)
  • Prepare update of API reference URLs (#16816)
  • Rename allow_overflow to wrap_numerical (#16807)
  • Set infer_schema_length as keyword-only argument in str.json_decode (#16835)
  • Don't enter streaming engine for groupby-> agg mean/median … (#16810)
  • Improve safety of amortized_iter (#16820)
  • Remove needless inner type clone (#16718)
  • Fix incorrect debug assertion in ChunkedArray::from_chunks_and_dtype (#16697)
  • Update version resolver for 1.0.0 release (#16705)
  • Avoid AWS pinning to outdated crc32c version (#16681)

Thank you to all our contributors for making this release possible! @JulianCologne, @KDruzhkin, @Kylea650, @MarcoGorelli, @Mottl, @Object905, @adamreeve, @alexander-beedie, @bertiewooster, @borchero, @c-peters, @coastalwhite, @datapythonista, @datenzauberai, @dependabot, @dependabot[bot], @eitsupi, @henryharbeck, @itamarst, @lukeshingles, @machow, @marenwestermann, @mcrumiller, @montanarograziano, @nameexhaustion, @orlp, @p3i0t, @ritchie46, @sherlockbeard, @stinodego, @tkellogg, @universalmind303 and @wence-