polars

💥 Unstable Breaking changes

These API's were marked unstable and are allowed to change.

• Use Altair in DataFrame.plot (#17995)

🚀 Performance improvements

• Parquet do not copy uncompressed pages (#18441)
• Several large parquet optimizations (#18437)
• Batch Plain Parquet UTF-8 verification (#18397)
• Partition metadata for parquet statistic loading (#18343)
• Fix accidental quadratic parquet metadata (#18327)
• Lazy decompress Parquet pages (#18326)
• Don't rechunk aligned chunks in owned_binary_chunk_align (#18314)
• Batch DELTA_LENGTH_BYTE_ARRAY decoding (#18299)
• Slice pushdown for SimpleProjection (#18296)
• Use direct path for time/timedelta literals (#18223)
• Speedup ndjson reader ~40% (#18197)
• Skip parquet page when unneeded (#18192)

✨ Enhancements

• Use Altair in DataFrame.plot (#17995)
• Allow mapping as syntactic sugar in str.replace_many (#18214)
• Respect input time zone if input is pandas Timestamp (#18346)
• Improve Schema and DataType interop with Python types (#18308)
• Add POLARS_BACKTRACE_IN_ERR for debugging (#18333)
• IR serde (#18298)
• Improve decimal_comma error message (#18269)
• Support pre-signed URLs for cloud scan (#18274)
• Support the most recent version of "duckdb_engine" connections via read_database (#18277)
• Support empty structs (#18249)
• Allow float in interpolate_by by column (#18015)
• Make show_versions more responsive (#18208)

🐞 Bug fixes

• Enable CSE in eager if struct are expanded (#18426)
• Treat explode as gather (#18431)
• Parquet nested values that span several pages (#18407)
• Support reading empty parquet files (#18392)
• Recurse on map field during type conversion (#15075)
• Allow search_sorted on boolean series (#18387)
• Mark Expr.(lower|upper)_bound as returning scalar (#18383)
• Fix compressed ndjson row count (#18371)
• Use correct column names when there are no value columns in unpivot (#18340)
• Parquet several smaller issues (#18325)
• Fix group-by slice on all keys (#18324)
• Compute joint null mask before calling rolling corr/cov stats (#18246)
• Several scan_parquet(parallel='prefiltered') problems (#18278)
• Json feature flag missing imports (#18305)
• Check groups in group-by filter (#18300)
• Parquet delta encoding for 0-bitwidth miniblocks (#18289)
• Arguments for upsample only have to be sorted within groups (#18264)
• Use appropriate bins in hist when bin_count specified (#16942)
• Raise suitable error on unsupported SQL set op syntax (#18205)
• Fix invalid state due to cached IR (#18262)
• Fix failed AWS credential load from '~/.aws/credentials' due to formatting (#18259)
• Fix panic streaming parquet scan from cloud with slice (#18202)
• Consistently round half-way points down in dt.round (#18245)
• Fix duplicate column output and panic for include_file_paths (#18255)
• Fix unit null rank (#18252)
• Use physical for row-encoding (#18251)
• Convert date and datetime in literal construction (#16018)
• Fix gather str as lit (#18207)

📖 Documentation

• Add date_range and datetime_ranges examples without eager=True (#18379)
• Fix incorrect comments in group_by_dynamic (#18415)
• Alphabetise methods in Python API reference (#18380)
• Document POLARS_BACKTRACE_IN_ERR env var (#18354)
• Add missing aggregation entries (#18334) (#18341)
• Add missing Series methods to API reference (#18312)
• Document DataFrame.__getitem__ and Series.__getitem__ (#18309)
• Fix typos and add see also links to struct name expressions (#18282)
• Improve decimal_comma error message (#18269)
• Clarify coalesce behaviour in join_asof (#18273)
• Add note to Expr.shuffle differentiating from df method (#18266)
• Improve formatting and consistency of various docstrings (#18237)
• Add missing "Parameters" section to bin.size expr docstring (#18222)
• Fix column name output in example of DataFrame.map_rows (#18227)

📦 Build system

• Bump Rust toolchain to nightly-2024-08-26 (#18370)

🛠️ Other improvements

• Address spurious hypothesis test failure (#18434)
• Turn all Binary/Utf8 into BinaryView/Utf8View in Parquet (#18331)
• Fix the required version of rust in README.md (#18357)
• Remove unused Parquet indexes (#18329)
• Deprecate serialize json for LazyFrame (#18283)
• Don't add sink node to cloud query (#18280)
• Split py-polars crate (#18204)
• Fix test for new deltalake release (#18211)
• Update the required version of rust in README.md (#18203)
• Fix version bifurcation for test_read_database_cx_credentials (#18220)
• Use or_else for raising (#18206)
• Remove unused Parquet source files (#18193)

Thank you to all our contributors for making this release possible!
@BartSchuurmans, @ChayimFriedman2, @MarcoGorelli, @StepfenShawn, @agossard, @alexander-beedie, @cgbur, @coastalwhite, @corwinjoy, @deanm0000, @henryharbeck, @ion-elgreco, @jqnatividad, @krasnobaev, @liufeimath, @markxwang, @mcrumiller, @nameexhaustion, @orlp, @ritchie46, @stinodego, @sunadase, @thomascamminady and @wence-

🚀 Performance improvements

• Improve binview extend/ifthenelse (#18164)
• Start on better Parquet delta decoding (#18049)
• Rechunk group-by __iter__ (#18162)
• Tune jemalloc to not create muzzy pages (#18148)
• Reduce default async thread count (#18142)
• Make expensive selector expansion lazy (#18118)
• Use single threaded algorithms if only 1 core given (#18101)
• Use Arc<Vec<_>> instead of Arc<[_]> for paths and hive partitions (#18066)
• SIMD View from FixedSizeBinary (#18059)
• Use bitmask to filter Parquet predicate-pushdown items (#17993)
• Zerocopy buffers for FixedSizeBinary to BinaryView cast (#18043)

✨ Enhancements

• Create literals for datetime/date expressions (#18184)
• Create literals in 'datetime' expression (#18182)
• Expose top-level "has_header" param for read_excel and read_ods (#18078)
• Raise on invalid 'is_between' and improve error message quality (#18147)

🐞 Bug fixes

• Fix struct shift and list builder (#18189)
• Don't load Parquet nested metadata (#18183)
• Throw bigidx error for Parquet row-count (#18154)
• Fix unpivot on empty df (#18179)
• Don't vertically parallelize cse contexts (#18177)
• Ensure default values are included when saving/restoring the current Config state (#18151)
• Properly handle empty Parquet row groups with no dictionary (#18161)
• Struct outer nullabillity (#18156)
• Fix pyarrow predicate pushdown regression (#18145)
• Prevent unwanted supertype cast in 'search_sorted' (#18143)
• Parquet with filter=None (#18139)
• Don't raise when converting from pandas if index contains duplicate names when include_index=False (the default) (#18133)
• Fix cast Float to String where Float is not turn to Integer before turning to String (#18123)
• Don't remove leading whitespace in read_csv (#18131)
• Py-polars compilation with no features (#18129)
• String transform to_titlecase was too narrowly defined (#18122)
• Reading Parquet with Null dictionary page (#18112)
• When setting write_excel column totals, don't forget to include any row-total cols (#18042)
• Incorrect lazy CSV select(len()) for compressed files (#18067)
• Fix sink_ipc_cloud panicking with runtime error (#18091)
• Properly write Parquet for sliced lists (#18073)
• Panic reading multiple CSV files from cloud (#18056)
• Fix CloudWriter to use buffer before making requests (#18027)
• Fix typos and remove trailing whitespace (#18024)
• Handle cfg(feature) for shrink_dtype (#18038)

📖 Documentation

• Fix references to old methods in lazy docstring (#18178)
• Include PyCapsule Interface in DataFrame and Series API docs (#18174)
• Corrected example result in group_by docs (#18169)
• Mention 'Array' in data types overview (#18060)
• Correct concat rechunk in user guide (#18080)
• Fix typo in title of Hugging Face docs page (#18097)
• Update pivot docstring for clarity (#18000)

🛠️ Other improvements

• Remove unneeded growable (#18165)
• Update Cargo.lock to fix build error on Linux (#18153)
• Remove Nth,Wildcard from ExprIR and make conversion falllible (#18115)

Thank you to all our contributors for making this release possible!
@EricTulowetzke, @KDruzhkin, @MarcoGorelli, @Vincenthays, @alexander-beedie, @coastalwhite, @davanstrien, @deanm0000, @ember91, @kylebarron, @mcrumiller, @nameexhaustion, @orlp, @philss, @ritchie46 and @rosstitmarsh

💥 Breaking changes

• Reject literal input in sort_by_exprs() (#17606)

🚀 Performance improvements

• Skip parquet page when unneeded (#18192)
• Improve binview extend/ifthenelse (#18164)
• Start on better Parquet delta decoding (#18049)
• Tune jemalloc to not create muzzy pages (#18148)
• Reduce default async thread count (#18142)
• Use single threaded algorithms if only 1 core given (#18101)
• Use Arc<Vec<_>> instead of Arc<[_]> for paths and hive partitions (#18066)
• SIMD View from FixedSizeBinary (#18059)
• Use bitmask to filter Parquet predicate-pushdown items (#17993)
• Zerocopy buffers for FixedSizeBinary to BinaryView cast (#18043)
• Integer fast path Parquet dict encoding (#18030)
• Speedup writing of Parquet primitive values (#18020)
• Remove temporary allocations in Parquet (#18013)
• Delay selection expansion (#18011)
• Optimize strings slices (#17996)
• Make .dt.weekday 20x faster (#17992)
• Shrink MemSliceInner enum (#17991)
• Push down slice with non-zero offset to Parquet (#17972)
• Reduce copy in MemSlice (#17983)
• Ensure metadata flags are maintained on vertical parallelization (#17804)
• Ensure only nodes that are not changed are cached in collapse optimizer (#17791)
• Use bitflags for OptState (#17788)
• Remove async directory auto-detection (#17779)
• Fix accidental quadratic horizontal concat (#17783)
• Batch parquet integer decoding (#17734)
• Use mmap-ed memory if possible in Parquet reader (#17725)
• Use bitflags for function options (#17723)
• Introduce MemReader to file buffer in Parquet reader (#17712)
• Better GC and push_view for binviews (#17627)
• Fix pathological perf issue in window-order-by (#17650)
• Cache path resolving of scan functions (#17616)
• Add ArrayChunks to optimize codegen of BatchDecoder (#17632)
• Rechunk before we go into grouped gathers (#17623)
• Cache schema resolve back to DSL (#17610)
• Add fastpath for when rounding by single constant durations (#17580)
• Improve parallelism in writing hive parquet (#17512)
• Support datetime in predicate during hive partition pruning (#17545)
• Batch nested embed parquet decoding (#17549)
• Batch nested Parquet decoding (#17542)
• Collect Parquet dictionary binary as view (#17475)
• Keep more parallelism when CSE plan cache hits (#17463)
• Batch parquet primitive decoding (#17462)
• Respect allow_threading in some more operators (#17450)
• Parallelize parquet metadata deserialization (#17399)

✨ Enhancements

• Create literals for datetime/date expressions (#18184)
• Create literals in 'datetime' expression (#18182)
• Add missing impl for Series (#18166)
• Raise on invalid 'is_between' and improve error message quality (#18147)
• Add boolean Parquet HybridRle encoding (#18022)
• Add nested SQL join support (#18006)
• Push down slice with non-zero offset to Parquet (#17972)
• Add support for binary size method to Expr and Series "bin" namespace (#17924)
• Add SQL interface support for PostgreSQL dollar-quoted string literals (#17940)
• Allow for parsing parquet file where the time zone is stored as lowercase "utc" (#17925)
• Expose binary_elementwise_into_string_amortized for plugin authors, recommend apply_into_string_amortized instead of apply_to_buffer (#17903)
• Decompress in CSV / NDJSON scan (#17841)
• Ensure unique names in HConcat (#17884)
• Support authentication with HuggingFace login (#17881)
• Support "BY NAME" qualifier for SQL "INTERSECT" and "EXCEPT" set ops (#17835)
• Raise informative error instead of panicking when passing invalid directives to to_string for Date dtype (#17670)
• Implement forward/backward fill for all types (#17861)
• Implement is_in operation on decimal type (#17832)
• Support hf:// in read_(csv|ipc|ndjson) functions (#17785)
• Allow literals in sort (#17780)
• Cloud support for NDJSON (#17717)
• Support API token for scanning hf:// (#17682)
• Raise error instead of panic in unsupported serde (#17679)
• Include file path option for NDJSON (#17681)
• Hugging Face path expansion (#17665)
• Add DSL validation for cloud eligible check (#17287)
• Raise informative error message if non-IntoExpr is passed by name in *Frame.group_by (#17654)
• Change API for writing partitioned Parquet to reduce code duplication (#17586)
• Cache schema resolve back to DSL (#17610)
• Expose returns_scalar to map_elements (#17613)
• Add option to include file path for Parquet, IPC, CSV scans (#17563)
• Support describe on decimal (#15092)
• Support datetime in predicate during hive partition pruning (#17545)
• Raise more informative error message for directories containing files with mixed extensions (#17480)
• Exclude empty files from directory/glob expansion (#17478)
• Add "future" versioning (#17421)
• Apply slice pushdown immediately to in-memory frames (#17459)
• Support writing hive partitioned parquet (#17324)
• Add right join support (#17441)
• Support hive partitioning in scan_ipc (#17434)

🐞 Bug fixes

• Fix struct shift and list builder (#18189)
• Don't load Parquet nested metadata (#18183)
• Throw bigidx error for Parquet row-count (#18154)
• Fix unpivot on empty df (#18179)
• Don't vertically parallelize cse contexts (#18177)
• Properly handle empty Parquet row groups with no dictionary (#18161)
• Struct outer nullabillity (#18156)
• Fix pyarrow predicate pushdown regression (#18145)
• Prevent unwanted supertype cast in 'search_sorted' (#18143)
• Parquet with filter=None (#18139)
• Don't raise when converting from pandas if index contains duplicate names when include_index=False (the default) (#18133)
• Don't remove leading whitespace in read_csv (#18131)
• Py-polars compilation with no features (#18129)
• String transform to_titlecase was too narrowly defined (#18122)
• Reading Parquet with Null dictionary page (#18112)
• Incorrect lazy CSV select(len()) for compressed files (#18067)
• Fix sink_ipc_cloud panicking with runtime error (#18091)
• Properly write Parquet for sliced lists (#18073)
• Panic reading multiple CSV files from cloud (#18056)
• Fix CloudWriter to use buffer before making requests (#18027)
• Fix typos and remove trailing whitespace (#18024)
• Handle cfg(feature) for shrink_dtype (#18038)
• Subtraction with overflow on negative slice offset in Parquet (#18036)
• Add nested SQL join support (#18006)
• Allow read_csv schema to take unparsable types (#17765)
• Multi-output column expressions in frame sort method (#17947)
• Fix Asof join by schema (#17988)
• Fix glob resolution for Hugging Face (#17958)
• Several parquet reader/writer regressions (#17941)
• Incorrect filter on categorical columns from parquet files (#17950)
• SQL COUNT(DISTINCT x) should not include NULL values (#17930)
• Scanning '%' from cloud (#17890)
• Respect glob=False for cloud reads (#17860)
• Properly write nest-nulled values in Parquet (#17845)
• Allow full-null Object series to be built (#17870)
• Fix from_arrow for struct type (#17839)
• Infer decimal scales on mixed scale input (#17840)
• Raise on unsupported fill strategy dtype (#17837)
• Properly write nested NullArray in Parquet (#17807)
• Check input type on list.to_struct (#17834)
• Fix right join schema (#17833)
• Non-compliant Parquet list element name (#17803)
• Correctly set should_broadcast flag in HStack CSE rewrite (#17784)
• Fix projection pusdhown of literals without names (#17778)
• Don't expand HTTP paths (#17774)
• Check funtion input len at expansion (#17763)
• Don't panic in invalid agg_groups (#17762)
• Raise empty struct (#17736)
• Fix GC logic in write_ipc (#17752)
• Panic in pl.concat_list and list.concat on empty inputs (#17742)
• Fix out nullability for structs coming from arrow (#17738)
• Percent encode for Hugging Face paths (#17718)
• Use bytemuck in slice reinterpret for Parquet ArrayChunks (#17700)
• Propagate struct outer nullability eagerly (#17697)
• Use ETag for HTTP file cache invalidation (#17684)
• Fix type inference failure caused by double transpose (#17663)
• Interpret %y consistently with Chrono in to_date/to_datetime/strptime (#17661)
• Fix explode invalid check (#17651)
• Tighten up error checking on join keys (#17517)
• Expand brackets in async glob expansion (#17630)
• Fix row index disappearing after projection pushdown in NDJSON (#17631)
• Fix struct -> enum is_in (#17622)
• Don't needlessly unwrap in pivot_schema (#17611)
• Reject literal input in sort_by_exprs() (#17606)
• Bitmap collect into safety (#17588)
• Method dt.truncate was sometimes returning incorrect results for pre-1970 datetimes (#17582)
• Defer path expansion until collect in file scan methods (#17532)
• Correct logic for descending sort of BooleanChunked (#17558)
• Don't unwrap send attempt to oneshot channel (#17566)
• Fix scanning from HTTP cloud paths (#17571)
• Properly implement struct (#17522)
• Add missing commas in python IR interchange (#17518)
• Fix predicate pushdown for .list.(get|gather) (#17511)
• Turn panic into error when serializing Object types (#17353)
• Fix struct expansion and raise on exclude (#17489)
• Fix decimal dyn float supertype (#17464)
• Don't rechunk on phys_repr (#17461)
• Harden alchemy session for old sqlalchemy versions (#17366)
• Fix swapping rename schema (#17458)
• Raise on oob decimal precision (#17445)
• Don't allow json inference method to be chunked/streaming (#17396)
• avoid panic when projecting solitary count into empty frame (#17393)
• Set literal nesting to 0 (#17392)
• Fix scanning cloud paths with spaces (#17379)
• Fix slice length no longer allowing None (#17372)
• Cull row index in scan if projection pushdown removes it (#17363)
• Fix typo in SchemaError exception message (#17350)

📖 Documentation

• Mention 'Array' in data types overview (#18060)
• Correct concat rechunk in user guide (#18080)
• Fix typo in title of Hugging Face docs page (#18097)
• Update pivot docstring for clarity (#18000)
• Fix aggregation guide discrepancies (#18003)
• Updating the join example input for rust for consistency with python example (#17898)
• Rename str.lengths to str.len_bytes in description text (#11577) (#17626)
• Create example for polars.Expr.bin.decode (#17508)
• Add right join in the user guide (#17608)
• Update SQL examples in README (#17568)
• Fix typo in Getting Started section of user guide (#17465)

📦 Build system

• Update Rust toolchain to nightly-2024-07-26 (#17891)

🛠️ Other improvements

• Consistently use PlRandomState, PlHashMap/Set (#18190)
• Remove unneeded growable (#18165)
• Allow non-scoped tasks to be spawned (#18163)
• Update Cargo.lock to fix build error on Linux (#18153)
• Remove Nth,Wildcard from ExprIR and make conversion falllible (#18115)
• Allow zip node to extend with nulls or broadcast inputs (#18077)
• Add @coastalwhite to parquet codeowners (#18032)
• Minor bump to comfy-table version (#18028)
• Add unit tests for str.contains_any and str.replace_many (#17961)
• Remove apply_generic, use unary_elementwise (#17902)
• Add general filters in Parquet (#17910)
• Fix broken typos command in make pre-commit for py-polars folder (#17897)
• Remove HybridRLE iter / batch nested parquet decoding (#17889)
• Deal with re-entrant expressions locally (#17885)
• Unify Parquet nested decoders (#17816)
• Add zip node to streaming engine (#17866)
• Fix simple projection in streaming engine (#17871)
• Make better use of typos configuration features (#17800)
• Remove complicated Parquet traits (#17767)
• Rename Unit to Plain in Parquet reader (#17751)
• Add missing feature flag for prefetching on arm (#17740)
• Add the capability to run new streaming engine on test suite (#17706)
• Update CODEOWNERS (#17707)
• Allow compute nodes in streaming engine to decide whether they want serial or parallel input/output (#17699)
• Name tests so they actually run (#17690)
• Add reduce ComputeNode in new streaming engine (#17389)
• Add DSL validation for cloud eligible check (#17287)
• Rename Pipe to Connector (#17655)
• Document and simplify MutableBinView::push_ignore_validity (#17645)
• Remove has_validity, use has_nulls (#17519)
• Add ArrayChunks to optimize codegen of BatchDecoder (#17632)
• Move path logic to from utils to path_utils in polars-io (#17635)
• Fix struct gather (#17621)
• Back to StructChunked name (#17609)
• Re-enable struct related tests (#17597)
• Completely redo structure of Parquet decoder (#17589)
• Fix struct outer validity;fmt;is_in;cast;cmp (#17590)
• Make no-default-features compile (#17516)
• No longer blanket allow unused items in polars-stream (#17551)
• Smaller filter / valid in Parquet (#17569)
• Move path expansion utils to polars-io crate (#17521)
• Add streaming slice node to new streaming engine (#17451)
• Add ordered union to new streaming engine (#17359)
• Add automated check for PR title formatting (#17412)
• Remove transmute for object store path (#17395)
• Fix Python version resolver in release drafter (#17390)
• Remove dead code (#17378)

Thank you to all our contributors for making this release possible!
@5j9, @ByteNybbler, @EricTulowetzke, @JamesCE2001, @Julian-J-S, @KDruzhkin, @MarcoGorelli, @Object905, @SandroCasagrande, @Vincenthays, @alexander-beedie, @anergictcell, @arnabanimesh, @atigbadr, @brandon-b-miller, @brunobbaraujo, @cmdlineluser, @coastalwhite, @davanstrien, @deanm0000, @deepyaman, @delsner, @dependabot, @dependabot[bot], @diegoglozano, @eitsupi, @ember91, @flisky, @henryharbeck, @implicit-apparatus, @itamarst, @jonaylor89, @jparag, @knl, @kylebarron, @lukapeschke, @mcrumiller, @moritzwilksch, @nameexhaustion, @orlp, @phi-friday, @philss, @r-brink, @ragyabraham, @rcorty, @ritchie46, @rosstitmarsh, @ruihe774, @sherlockbeard, @stinodego, @szepeviktor, @tylerriccio33, @wangxiaoying and @wence-

🚀 Performance improvements

• Integer fast path Parquet dict encoding (#18030)
• Speedup writing of Parquet primitive values (#18020)
• Remove temporary allocations in Parquet (#18013)

✨ Enhancements

• Add boolean Parquet HybridRle encoding (#18022)
• Support passing Worksheet objects to the write_excel method (#18031)

🐞 Bug fixes

• Subtraction with overflow on negative slice offset in Parquet (#18036)
• Fix drop selector (#18034)

📖 Documentation

• Update map_batches docstring (#18001)

🛠️ Other improvements

• Add @coastalwhite to parquet codeowners (#18032)
• Minor bump to comfy-table version (#18028)

Thank you to all our contributors for making this release possible!
@alexander-beedie, @coastalwhite, @deanm0000, @nameexhaustion and @ritchie46

🚀 Performance improvements

• Delay selection expansion (#18011)
• Optimize strings slices (#17996)
• Make .dt.weekday 20x faster (#17992)
• Shrink MemSliceInner enum (#17991)
• Push down slice with non-zero offset to Parquet (#17972)
• Reduce copy in MemSlice (#17983)

✨ Enhancements

• Add nested SQL join support (#18006)
• Push down slice with non-zero offset to Parquet (#17972)
• Add support for binary size method to Expr and Series "bin" namespace (#17924)
• IO plugins (#17939)
• Add SQL interface support for PostgreSQL dollar-quoted string literals (#17940)
• Allow for parsing parquet file where the time zone is stored as lowercase "utc" (#17925)

🐞 Bug fixes

• Add nested SQL join support (#18006)
• Respect strict argument (#17990)
• Multi-output column expressions in frame sort method (#17947)
• Fix Asof join by schema (#17988)
• Set default flags for FFI plugin (#17984)
• Fix glob resolution for Hugging Face (#17958)
• Several parquet reader/writer regressions (#17941)
• Incorrect filter on categorical columns from parquet files (#17950)
• SQL COUNT(DISTINCT x) should not include NULL values (#17930)
• Default to None in pycapsule interface export (#17922)

📖 Documentation

• Fix aggregation guide discrepancies (#18003)
• Ensure last is never ambiguous with max (#17962)
• Documentation for Arrow PyCapsule interface integration (#17935)
• Fix Hugging Face link in user guide (#17943)

🛠️ Other improvements

• Add unit tests for str.contains_any and str.replace_many (#17961)
• Suggest allow_null as replacement (#17969)
• Remove apply_generic, use unary_elementwise (#17902)
• Add general filters in Parquet (#17910)

Thank you to all our contributors for making this release possible!
@JamesCE2001, @MarcoGorelli, @alexander-beedie, @coastalwhite, @deanm0000, @deepyaman, @dependabot, @dependabot[bot], @henryharbeck, @kylebarron, @nameexhaustion, @ritchie46 and @wangxiaoying

🚀 Performance improvements

• Ensure metadata flags are maintained on vertical parallelization (#17804)
• Ensure only nodes that are not changed are cached in collapse optimizer (#17791)
• Use bitflags for OptState (#17788)
• Remove async directory auto-detection (#17779)
• Fix accidental quadratic horizontal concat (#17783)
• Batch parquet integer decoding (#17734)
• Use mmap-ed memory if possible in Parquet reader (#17725)
• Use bitflags for function options (#17723)
• Also set target features and tune cpu for CC (#17716)
• Introduce MemReader to file buffer in Parquet reader (#17712)

✨ Enhancements

• Expose binary_elementwise_into_string_amortized for plugin authors, recommend apply_into_string_amortized instead of apply_to_buffer (#17903)
• Expose allocator to capsule (#17817)
• Decompress in CSV / NDJSON scan (#17841)
• Ensure unique names in HConcat (#17884)
• Support authentication with HuggingFace login (#17881)
• Enable collection with gpu engine (#17550)
• Support "BY NAME" qualifier for SQL "INTERSECT" and "EXCEPT" set ops (#17835)
• Write data at table level in write_excel (#17757)
• Support PyCapsule Interface in DataFrame & Series constructors (#17693)
• Implement Arrow PyCapsule Interface for Series/DataFrame export (#17676)
• Raise informative error instead of panicking when passing invalid directives to to_string for Date dtype (#17670)
• Implement forward/backward fill for all types (#17861)
• Implement is_in operation on decimal type (#17832)
• Optimise read_excel when using "calamine" engine with the latest fastexcel (#17735)
• Support hf:// in read_(csv|ipc|ndjson) functions (#17785)
• Allow literals in sort (#17780)
• Expose 'strict' argument to 'is_in' (#17776)
• Release the GIL in collect_schema (#17761)
• Cloud support for NDJSON (#17717)
• Support API token for scanning hf:// (#17682)

🐞 Bug fixes

• Scanning '%' from cloud (#17890)
• Raise suitable error when invalid column passed to get_column_index (#17868)
• Respect glob=False for cloud reads (#17860)
• Properly write nest-nulled values in Parquet (#17845)
• Improve default write_excel int/float format when using a dark "table_style" (#17869)
• Fix from_arrow for struct type (#17839)
• Fix bool/string usage of "column_totals" parameter in write_excel (#17846)
• Infer decimal scales on mixed scale input (#17840)
• Don't ignore timezones in list of dicts constructor (#14211)
• Raise on unsupported fill strategy dtype (#17837)
• Properly write nested NullArray in Parquet (#17807)
• Check input type on list.to_struct (#17834)
• Fix right join schema (#17833)
• Simultaneous usage of named_expr and schema in pl.struct (#17768)
• Fix projection pusdhown of literals without names (#17778)
• Don't expand HTTP paths (#17774)
• Check funtion input len at expansion (#17763)
• Don't panic in invalid agg_groups (#17762)
• Raise empty struct (#17736)
• Fix GC logic in write_ipc (#17752)
• Panic in pl.concat_list and list.concat on empty inputs (#17742)
• Fix out nullability for structs coming from arrow (#17738)
• Percent encode for Hugging Face paths (#17718)

📖 Documentation

• Updating the join example input for rust for consistency with python example (#17898)
• Improve filter documentation (#17755)
• Reword "how" param docstring entry for 'semi' and 'anti' join types for clarity (#17843)
• Mention read_* functions in Hugging Face section in user guide (#17799)
• Show return type for Series attributes in API reference (#17759)
• Add function with multiple arguments example to Expr.map_batches (#17789)
• Add Hugging Face section to user guide (#17721)

📦 Build system

• Update Rust toolchain to nightly-2024-07-26 (#17891)
• Correctly reference released package in optional dependencies (#17691)

🛠️ Other improvements

• On Python release, trigger docs build after API reference build (#17904)
• Set uv pip install to verbose (#17901)
• Fix broken typos command in make pre-commit for py-polars folder (#17897)
• Remove HybridRLE iter / batch nested parquet decoding (#17889)
• Add version field for python IR (#17876)
• Pass through missing rolling and stringfunction information in pyir (#17702)
• Make better use of typos configuration features (#17800)
• Better deprecate message for _import_from_c (#17753)
• Rename Unit to Plain in Parquet reader (#17751)
• Unpin setuptools (#17726)
• Update CODEOWNERS (#17707)

Thank you to all our contributors for making this release possible!
@MarcoGorelli, @Object905, @SandroCasagrande, @alexander-beedie, @atigbadr, @coastalwhite, @deanm0000, @delsner, @dependabot, @dependabot[bot], @henryharbeck, @implicit-apparatus, @jparag, @knl, @kylebarron, @lukapeschke, @mcrumiller, @nameexhaustion, @orlp, @ritchie46, @ruihe774, @stinodego, @szepeviktor and @wence-

🚀 Performance improvements

• Specify tune-cpu & add more features (#17615)
• Better GC and push_view for binviews (#17627)

✨ Enhancements

• Raise error instead of panic in unsupported serde (#17679)
• Expose Arrow C interface directly on Polars (#17696)
• Include file path option for NDJSON (#17681)

🐞 Bug fixes

• Use bytemuck in slice reinterpret for Parquet ArrayChunks (#17700)
• Remove non-existing names from __all__ (#17494)
• Fix return type hint for LazyFrame sink methods (#17698)
• Propagate struct outer nullability eagerly (#17697)
• Address read_database issue with batched reads from Snowflake (#17688)
• Use ETag for HTTP file cache invalidation (#17684)

📖 Documentation

• Fixed default name for value_counts methods based on normalize parameter (#17685)

📦 Build system

• Pin setuptools to fix failing CI (#17695)

🛠️ Other improvements

• Fix return type hint for LazyFrame sink methods (#17698)
• Pin setuptools to fix failing CI (#17695)
• Name tests so they actually run (#17690)
• Add reduce ComputeNode in new streaming engine (#17389)

Thank you to all our contributors for making this release possible!
@5j9, @ByteNybbler, @MarcoGorelli, @alexander-beedie, @coastalwhite, @diegoglozano, @eitsupi, @nameexhaustion, @orlp, @ragyabraham, @ritchie46 and @ruihe774

🚀 Performance improvements

• Fix pathological perf issue in window-order-by (#17650)
• Cache path resolving of scan functions (#17616)
• Add ArrayChunks to optimize codegen of BatchDecoder (#17632)
• Rechunk before we go into grouped gathers (#17623)
• Cache schema resolve back to DSL (#17610)
• Add fastpath for when rounding by single constant durations (#17580)
• Improve parallelism in writing hive parquet (#17512)
• Support datetime in predicate during hive partition pruning (#17545)
• Batch nested embed parquet decoding (#17549)
• Batch nested Parquet decoding (#17542)
• Collect Parquet dictionary binary as view (#17475)

✨ Enhancements

• Hugging Face path expansion (#17665)
• Add DSL validation for cloud eligible check (#17287)
• Raise informative error message if non-IntoExpr is passed by name in *Frame.group_by (#17654)
• Add infer_schema parameter to read_csv / scan_csv (#17617)
• Change API for writing partitioned Parquet to reduce code duplication (#17586)
• Cache schema resolve back to DSL (#17610)
• Expose returns_scalar to map_elements (#17613)
• Add option to include file path for Parquet, IPC, CSV scans (#17563)
• Support describe on decimal (#15092)
• Support datetime in predicate during hive partition pruning (#17545)
• Raise more informative error message for directories containing files with mixed extensions (#17480)
• Exclude empty files from directory/glob expansion (#17478)
• Support use of SQLAlchemy "Connectable" in write_database (#17470)

🐞 Bug fixes

• Support duplicate expression names when calling ufuncs (#17641)
• Interpret %y consistently with Chrono in to_date/to_datetime/strptime (#17661)
• Fix explode invalid check (#17651)
• Raise for overlapping index/column names in pandas dataframes post string coercion (#17628)
• Expand brackets in async glob expansion (#17630)
• Fix row index disappearing after projection pushdown in NDJSON (#17631)
• Fix struct -> enum is_in (#17622)
• Don't needlessly unwrap in pivot_schema (#17611)
• Reject literal input in sort_by_exprs() (#17606)
• Don't enforce row order in join test results where not guaranteed (#17596)
• Bitmap collect into safety (#17588)
• Make schema picklable (#17524)
• Handle current position of file objects (#17543)
• Set O_CLOEXEC on duplicated file descriptor (#17537)
• Method dt.truncate was sometimes returning incorrect results for pre-1970 datetimes (#17582)
• Defer path expansion until collect in file scan methods (#17532)
• Fix retries parameter in scan functions not taking effect when it was set to 0 (#17564)
• Don't unwrap send attempt to oneshot channel (#17566)
• Fix scanning from HTTP cloud paths (#17571)
• Properly implement struct (#17522)
• Add right to lazyframe join docstring (#17529)
• Fix predicate pushdown for .list.(get|gather) (#17511)
• Make sure scan_ipc does not go through fsspec (#17495)
• Turn panic into error when serializing Object types (#17353)
• Fix struct expansion and raise on exclude (#17489)
• Normalize path in sink_csv (#17476)

📖 Documentation

• Update plot docs to refer to docstrings (#17504)
• Rename str.lengths to str.len_bytes in description text (#11577) (#17626)
• Create example for polars.Expr.bin.decode (#17508)
• Add right join in the user guide (#17608)
• Adjust rendering of links in read_database_uri docstring (#17536)
• Update SQL examples in README (#17568)
• Fixup "deprecated" directive for DataFrame.melt and LazyFrame.melt (#17530)
• Add write_parquet_partitioned (#17488)
• Add example for writing hive partitioned parquet to user guide (#17483)
• Fix typo in Getting Started section of user guide (#17465)

🛠️ Other improvements

• Add DSL validation for cloud eligible check (#17287)
• Add ArrayChunks to optimize codegen of BatchDecoder (#17632)
• Move path logic to from utils to path_utils in polars-io (#17635)
• Fix struct gather (#17621)
• Back to StructChunked name (#17609)
• Remove unused with_column method of PyLazyFrame (#17607)
• Re-enable struct related tests (#17597)
• Completely redo structure of Parquet decoder (#17589)
• Fix struct outer validity;fmt;is_in;cast;cmp (#17590)
• Add/fix version-gating in some SQLAlchemy and Pandas tests (#17538)
• Add style accessor to DataFrame (#17502)
• Remove unused is_supported_cloud util (#17493)

Thank you to all our contributors for making this release possible!
@Julian-J-S, @MarcoGorelli, @alexander-beedie, @anergictcell, @arnabanimesh, @brandon-b-miller, @cmdlineluser, @coastalwhite, @deanm0000, @eitsupi, @flisky, @henryharbeck, @itamarst, @jonaylor89, @moritzwilksch, @nameexhaustion, @orlp, @phi-friday, @r-brink, @rcorty, @ritchie46, @ruihe774, @stinodego, @tylerriccio33 and @wence-

🚀 Performance improvements

• Keep more parallelism when CSE plan cache hits (#17463)
• Batch parquet primitive decoding (#17462)
• Respect allow_threading in some more operators (#17450)
• Parallelize parquet metadata deserialization (#17399)
• Use underlying fileno for Python files when possible (#17315)
• Add future arg to Series.to_arrow (#17371)

✨ Enhancements

• add "future" versioning (#17421)
• Apply slice pushdown immediately to in-memory frames (#17459)
• Support writing hive partitioned parquet (#17324)
• Add right join support (#17441)
• Support hive partitioning in scan_ipc (#17434)
• Improve error message when passing string key to Series.__getitem__ (#17408)

🐞 Bug fixes

• Handle DB cursor descriptions that contain more fields than the DBAPI2 standard (#17468)
• Fix decimal dyn float supertype (#17464)
• Verify the integrity of pandas column names before implied string conversion (#17433)
• Don't rechunk on phys_repr (#17461)
• Harden alchemy session for old sqlalchemy versions (#17366)
• Fix swapping rename schema (#17458)
• Make boolean reads consistent across all read_excel engines (#17448)
• Raise on oob decimal precision (#17445)
• Fix handling of TextIOWrapper in write_csv (#17328)
• Support sa session (#17435)
• Fix from_pandas for string columns with missing values (#17397)
• Fix a global variable table-discovery edge case for the SQL interface (#17400)
• Don't allow json inference method to be chunked/streaming (#17396)
• Set literal nesting to 0 (#17392)
• Fix scanning cloud paths with spaces (#17379)
• Fix slice length no longer allowing None (#17372)
• Fix typo in SchemaError exception message (#17350)
• Raise proper error for mismatching parquet schema instead of panicking (#17321)

📖 Documentation

• Add examples for scanning hive datasets to user guide (#17431)
• Update partition_by docstring to match new behavior (#17394)
• Update GroupBy.__iter__ docstring to match new behavior (#17383)

📦 Build system

• Add support for NumPy 2.0 (#17384)

🛠️ Other improvements

• Add automated check for PR title formatting (#17412)
• Remove transmute for object store path (#17395)
• Fix Python version resolver in release drafter (#17390)
• Avoid use of np.trapz in tests to prepare for NumPy 2.0 (#17387)
• Avoid writing to disk when running sink_csv test (#17386)

Thank you to all our contributors for making this release possible!
@alexander-beedie, @brunobbaraujo, @cmdlineluser, @coastalwhite, @dependabot, @dependabot[bot], @nameexhaustion, @orlp, @phi-friday, @ritchie46, @ruihe774, @sherlockbeard, @stinodego, @tylerriccio33 and @wence-

🚀 Performance improvements

• Improve unique performance by adding RangedUniqueKernel for primitive arrays (#17166)
• faster decode on Parquet HybridRLE (#17208)

✨ Enhancements

• Add SQL support for NATURAL joins and the COLUMNS function (#17295)
• Add str.extract_many expression (#17304)
• Support '%' in pathnames for async scan (#17271)
• Support SQL Struct/JSON field access operators (#17226)
• Exclude directories from glob expansion result (#17174)
• Support SQL ORDER BY ALL syntax (#17212)
• Support PostgreSQL ^@ ("starts with"), and ~~,~~*,!~~,!~~* ("like", "ilike") string-matching operators (#17251)
• Support SQL SELECT * ILIKE wildcard syntax (#17169)
• Support SQL temporal functions STRFTIME and STRPTIME, and typed literal syntax (#17245)
• Support date/datetime for hive parts (#17256)
• Expose some more information in translated expression IR to python (#17209)
• Allow no-op round/ceil/floor on integer types (#17241)
• Support loading from datasets where the hive columns are also stored in the file (#17203)
• Implement serde for Null columns (#17218)
• Support Decimal types in write_csv/write_json (#14209)
• Improve SQL support for array indexing, increase test coverage (#16972)
• Support reading byte stream split encoded floats and doubles in parquet (#17099)
• Add float_scientific option to write_csv/sink_csv (#17111)

🐞 Bug fixes

• Raise proper error for mismatching parquet schema instead of panicking (#17321)
• Raise on invalid shape dataframe arithmetic (#17322)
• Fix panic in window case (#17320)
• Raise errors instead of panicking when sink_csv fails (#17313)
• Raise if join keys are passed to cross join (#17305)
• Don't null on oob in list.get for column index (#17276)
• Fix issue where sliced PyArrow record batches were not handled correctly (#17058)
• Don't oob on nulls in list.get (#17262)
• Fix list getter with nulls (#17261)
• Respect nulls_last parameter in aggregate sort_by (#17249)
• Fix literal slice in group by (#17242)
• Fix DataFrame.top_k not handling nulls correctly (#17239)
• Avoid using the regex dependency when the regex feature is not used (#17206)
• properly check the BMI2 uleb128 (#17191)

📖 Documentation

• Minor layout/terminology improvement for selector set ops (#17299)
• Fix polars-plan docs.rs build (#17266)
• Add SQL docs for the CAST and TRY_CAST functions (#17214)

🛠️ Other improvements

• Prefer ParquetError::oos to ParquetError::OutOfSpec (#17314)
• remove seqmacro and u8,u16 bitpack (#17290)
• Fix typo in join validation error message (#17296)
• Use typed iter in list.get (#17286)
• add ability to have pipeline blockers in new streaming engine (#17247)
• Support date/datetime for hive parts (#17256)
• Add elementwise select and with_columns to new streaming engine (#17185)
• chrono's ParseErrorKind is now public (#17201)

Thank you to all our contributors for making this release possible!
@IvanIsCoding, @JamesCE2001, @MarcoGorelli, @SeanTater, @adamreeve, @alexander-beedie, @coastalwhite, @datapythonista, @flisky, @itamarst, @jqnatividad, @lukeshingles, @mcrumiller, @nameexhaustion, @orlp, @ritchie46, @stinodego and @wence-

This is the first major release for Python Polars. Please check out the upgrade guide for help navigating the breaking changes when upgrading to this version.

💥 Breaking changes

• Change default engine for read_excel to "calamine" (#17263)
• Implement binary serialization of LazyFrame/DataFrame/Expr and set it as the default format (#17223)
• Streamline optional dependency definitions in pyproject.toml (#17168)
• Update read/scan_parquet to disable Hive partitioning by default for file inputs (#17106)
• Split replace functionality into two separate methods (#16921)
• Default to writing binview data to IPC, mark compression argument as keyword-only (#17084)
• Remove re-export of type aliases (#17032)
• Rename ModuleUpgradeRequired and PolarsPanicError error, remove InvalidAssert error (#17033)
• Change data orientation inference logic for DataFrame construction and warn when row orientation is inferred (#16976)
• Properly apply strict parameter in Series constructor (#16939)
• Remove supertype definition of List and non-List types (#16918)
• Consistently convert to given time zone in Series constructor (#16828)
• Update reshape to return Array types instead of List types (#16825)
• Default to raising on out-of-bounds indices in all get/gather operations (#16841)
• Native selector XOR set operation, guarantee consistent selector column-order (#16833)
• Set infer_schema_length as keyword-only argument in str.json_decode (#16835)
• Update set_sorted to only accept a single column (#16800)
• Remove deprecated parameters in Series.cut/qcut and update struct field names (#16741)
• Expedited removal of certain deprecated functionality (#16754)
• Update some error types to more appropriate variants (#15030)
• Scheduled removal of deprecated functionality (#16715)
• Change default offset in group_by_dynamic from 'negative every' to 'zero' (#16658)
• Constrain access to globals from DataFrame.sql in favor of top-level pl.sql (#16598)
• Read 2D NumPy arrays as Array type instead of List (#16710)
• Update clip to no longer propagate nulls in the given bounds (#14413)
• Change str.to_datetime to default to microsecond precision for format specifiers "%f" and "%.f" (#13597)
• Update resulting column names in pivot when pivoting by multiple values (#16439)
• Preserve nulls in ewm_mean, ewm_std, and ewm_var (#15503)
• Restrict casting for temporal data types (#14142)
• Support Decimal types by default when converting from Arrow (#15324)
• Remove serde functionality from pl.read_json and DataFrame.write_json (#16550)
• Update function signature of nth to allow positional input of indices, remove columns parameter (#16510)
• Rename struct fields of rle output to len/value and update data type of len field (#15249)
• Remove class variables from some DataTypes (#16524)
• Add check_names parameter to Series.equals and default to False (#16610)

⚠️ Deprecations

• Deprecate LazyFrame.fetch (#17278)
• Deprecate size parameter in parametric testing strategies in favor of min_size/max_size (#17128)
• Split replace functionality into two separate methods (#16921)
• Rename DataFrame.melt to unpivot and make parameters consistent with pivot (#17095)
• Remove re-export of exceptions at top-level (#17059)
• Deprecate dt.mean/dt.median in favor of mean/median (#16888)
• Deprecate LazyFrame.with_context in favor of horizontal concatenation (#16860)
• Rename parameter descending to reverse in top_k methods (#16817)
• Rename str.concat to str.join and update default delimiter (#16790)
• Deprecate arctan2d in favor of arctan2(...).degrees() (#16786)

🚀 Performance improvements

• Rechunk before group_by `iteration (#17302)
• Improve unique performance by adding RangedUniqueKernel for primitive arrays (#17166)
• Improve unique performance by creating UniqueKernel and improve bool implementation (#17160)
• Default to writing binview data to IPC, mark compression argument as keyword-only (#17084)
• Parallelize arrow conversion if binview -> large_bin (#17083)
• Garbage collect buffers in if-then-else view kernel (#16993)
• Desugar AND filter into multiple nodes (#16992)
• Optimize generic arg_sort of row-encoding (#16894)
• Improve rle_id iteration performance and set sorted flags (#16893)
• Optimize sort for String and Binary types (#16871)
• Use split_at in split (#16865)
• Use split_at instead of double slice in chunk splits. (#16856)
• Don't rechunk in align_ if arrays are aligned (#16850)
• Don't create small chunks in parallel collect. (#16845)
• Add dedicated no-null branch in arg_sort (#16808)
• Speed up dt.offset_by 2x for constant durations (#16728)
• Toggle coalesce in join if non-coalesced key isn't projected (#16677)
• Make dt.truncate 1.5x faster when every is just a single duration (and not an expression) (#16666)
• Always prune unused columns in semi/anti join (#16665)

✨ Enhancements

• Add SQL support for NATURAL joins and the COLUMNS function (#17295)
• Add str.extract_many expression (#17304)
• Change default engine for read_excel to "calamine" (#17263)
• Deprecate LazyFrame.fetch (#17278)
• Support '%' in pathnames for async scan (#17271)
• Support SQL Struct/JSON field access operators (#17226)
• Exclude directories from glob expansion result (#17174)
• Support SQL ORDER BY ALL syntax (#17212)
• Support PostgreSQL ^@ ("starts with"), and ~~,~~*,!~~,!~~* ("like", "ilike") string-matching operators (#17251)
• Support SQL SELECT * ILIKE wildcard syntax (#17169)
• Support SQL temporal functions STRFTIME and STRPTIME, and typed literal syntax (#17245)
• Support date/datetime for hive parts (#17256)
• Implement binary serialization of LazyFrame/DataFrame/Expr and set it as the default format (#17223)
• Allow no-op round/ceil/floor on integer types (#17241)
• Support loading from datasets where the hive columns are also stored in the file (#17203)
• Implement serde for Null columns (#17218)
• Support Decimal types in write_csv/write_json (#14209)
• Add optional "default" to get_column DataFrame method (#17176)
• Improve SQL support for array indexing, increase test coverage (#16972)
• Support reading byte stream split encoded floats and doubles in parquet (#17099)
• Add float_scientific option to write_csv/sink_csv (#17111)
• Support Struct field selection in the SQL engine, RENAME and REPLACE select wildcard options (#17109)
• Update DataFrame.pivot to allow index=None when values is set (#17126)
• Update read/scan_parquet to disable Hive partitioning by default for file inputs (#17106)
• Improve ipython autocomplete for LazyFrame and DataFrame (#17091)
• Split replace functionality into two separate methods (#16921)
• Improve schema inference for hive partitions (#17079)
• Rename DataFrame.melt to unpivot and make parameters consistent with pivot (#17095)
• Print row index in explain and show_graph (#17074)
• Support top-level pl.col autocompletion for iPython (#17080)
• Remove re-export of exceptions at top-level (#17059)
• Implement predicate and projection pushdown for read_ndjson (#17068)
• Allow (non-)coalescing in join_asof (#17066)
• Turn of coalescing and fix mutation of join on expressions (#17061)
• Expand NDJson glob into one SCAN (#17063)
• Do not parse hive partitions from user provided base directory path (#17055)
• Support directory paths in scans for Parquet, IPC and CSV (#17017)
• Implement general array equality checks (#17043)
• Add strict parameter to DataFrame/LazyFrame.drop and fix behavior to default to True (#17044)
• Rename ModuleUpgradeRequired and PolarsPanicError error, remove InvalidAssert error (#17033)
• Add rechunk parameter to read_delta (#16991)
• allow experimental metadata use on release (#17005)
• Add simple version of json_normalize (#17015)
• Change data orientation inference logic for DataFrame construction and warn when row orientation is inferred (#16976)
• Desugar AND filter into multiple nodes (#16992)
• Handle textio even if not correct (#16971)
• Properly apply strict parameter in Series constructor (#16939)
• Add SQL support for INTERSECT and EXCEPT ops (#16960)
• Add PerformanceWarning to LazyFrame properties (#16964)
• Add collect_schema method to LazyFrame and DataFrame (#16929)
• Allow setting file cache TTL on a per-file basis (#16891)
• Support Decimal inputs for lit (#16950)
• Implement multiply and division for lhs duration (#16948)
• Raise on invalid temporal arithmetic (#16934)
• Always end with a in-memory sink on collect (#16928)
• Add DataFrame.style namespace (#16809)
• Add Schema class (#16873)
• Normalize value_counts (#16917)
• Implement equality for more Array types (#16902)
• Set up some of the infrastructure for new streaming engine (#16900)
• Cache downloaded cloud IPC files (#16892)
• Consistently convert to given time zone in Series constructor (#16828)
• Improve read_csv SQL table reading function defaults (better handle dates) (#16866)
• Support SQL VALUES clause and inline renaming of columns in CTE & derived table definitions (#16851)
• Support Python Enum values in lit (#16858)
• Convert to given time zone in .str.to_datetime when values are offset-aware (#16742)
• Update reshape to return Array types instead of List types (#16825)
• Default to raising on out-of-bounds indices in all get/gather operations (#16841)
• Support SQL "SELECT" with no tables, optimise registration of globals (#16836)
• Native selector XOR set operation, guarantee consistent selector column-order (#16833)
• Extend recognised EXTRACT and DATE_PART SQL part abbreviations (#16767)
• Improve error message when raising integers to negative integers, improve docs (#16827)
• Return datetime for mean/median of Date colum (#16795)
• Update set_sorted to only accept a single column (#16800)
• Expose overflowing cast (#16805)
• Update group_by iteration and partition_by to always return tuple keys (#16793)
• Support array arithmetic for equally sized shapes (#16791)
• Expedited removal of certain deprecated functionality (2) (#16779)
• Removal of read_database_uri passthrough from read_database (#16783)
• Remove pyxlsb engine from read_excel (#16784)
• Add check_order parameter to assert_series_equal (#16778)
• Enforce deprecation of keyword arguments as positional (#16755)
• Support cloud storage in scan_csv (#16674)
• Streamline SQL INTERVAL handling and improve related error messages, update sqlparser-rs lib (#16744)
• Support use of ordinal values in SQL ORDER BY clause (#16745)
• Support executing polars SQL against pandas and pyarrow objects (#16746)
• Remove deprecated parameters in Series.cut/qcut and update struct field names (#16741)
• Expedited removal of certain deprecated functionality (#16754)
• Remove deprecated functionality from rolling methods (#16750)
• Update date_range to no longer produce datetime ranges (#16734)
• Mark min_periods as keyword-only for rolling methods (#16738)
• Remove deprecated top_k parameters nulls_last, maintain_order, and multithreaded (#16599)
• Support order-by in window functions (#16743)
• Add SQL support for NULLS FIRST/LAST ordering (#16711)
• Update some error types to more appropriate variants (#15030)
• Initial SQL support for INTERVAL strings (#16732)
• Scheduled removal of deprecated functionality (2) (#16724)
• Scheduled removal of deprecated functionality (#16715)
• Enforce deprecation of offset arg in truncate and round (#16655)
• Change default offset in group_by_dynamic from 'negative every' to 'zero' (#16658)
• Constrain access to globals from DataFrame.sql in favor of top-level pl.sql (#16598)
• Read 2D NumPy arrays as Array type instead of List (#16710)
• Update clip to no longer propagate nulls in the given bounds (#14413)
• Change str.to_datetime to default to microsecond precision for format specifiers "%f" and "%.f" (#13597)
• Update resulting column names in pivot when pivoting by multiple values (#16439)
• Preserve nulls in ewm_mean, ewm_std, and ewm_var (#15503)
• Restrict casting for temporal data types (#14142)
• Add many more auto-inferable datetime formats for str.to_datetime (#16634)
• Support Decimal types by default when converting from Arrow (#15324)
• Remove serde functionality from pl.read_json and DataFrame.write_json (#16550)
• Update function signature of nth to allow positional input of indices, remove columns parameter (#16510)
• Rename struct fields of rle output to len/value and update data type of len field (#15249)
• Remove class variables from some DataTypes (#16524)
• Add check_names parameter to Series.equals and default to False (#16610)
• Dedicated SQLInterface and SQLSyntax errors (#16635)
• Add DIV function support to the SQL interface (#16678)
• Support non-coalescing streaming left join (#16672)
• Allow wildcard and exclude before struct expansions (#16671)

🐞 Bug fixes

• Raise on invalid shape dataframe arithmetic (#17322)
• Fix panic in window case (#17320)
• Raise errors instead of panicking when sink_csv fails (#17313)
• Raise if join keys are passed to cross join (#17305)
• Ensure we don't close extant adbc connections in write_database (#17298)
• Don't null on oob in list.get for column index (#17276)
• Fix issue where sliced PyArrow record batches were not handled correctly (#17058)
• Don't oob on nulls in list.get (#17262)
• Fix list getter with nulls (#17261)
• Respect nulls_last parameter in aggregate sort_by (#17249)
• Fix literal slice in group by (#17242)
• Fix DataFrame.top_k not handling nulls correctly (#17239)
• Update implementation of Enum support in lit to address spurious test failure (#17187)
• Use explicit turbofish to help rustc (#17159)
• Raise on invalid set dtypes (#17157)
• Fix corrupted reads for hive parts from cloud and projection pushdown failure on hive parts (#17152)
• Set intersection supertype (#17154)
• ChainedWhen should not inherit Expr (#17142)
• Fix decompress_impl for csv with n_rows set (#17118)
• Fix incorrect window std for chunked series (#17110)
• Fix panic when using fold in certain situations (#17114)
• Fix melt panic (#17088)
• Fix expression autocomplete in IPython (#17072)
• Exclude index from expansion in rolling/group_by_dynamic (#17086)
• Update some Series dunder method type signatures (#17053)
• Fix oob of join with literals and empty table (#17047)
• Don't silently accept multi-table FROM clauses (implicit JOIN syntax) (#17028)
• Don't split up ANDed filters that are group-aware (#17031)
• Harden "async" check for users with out-of-date sqlalchemy libraries (#17029)
• Error when sort_by of unequal length (#17026)
• Properly catch not found explode cols (#17020)
• Correctly convert data frames to NumPy for C index order (#17000)
• Raise on invalid arithmetic shapes (#16986)
• Don't pushdown predicates in cross join if the refer to both tables (#16983)
• Fix projection pushdown with literal joins (#16981)
• Fix edge case in DataFrame constructor data orientation inference (#16975)
• Raise on list of objects (#16959)
• Handle strictness for Decimal Series construction (#15309)
• Don't panic in object to anyvalue (#16957)
• Properly set FAST_EXPLODE_LIST metadata (#16951)
• Raise informative error when writing object to file (#16954)
• Remove supertype definition of List and non-List types (#16918)
• Remove unwrap in extend() (#16890)
• Fix should_rechunk check (#16852)
• Ensure read_excel and read_ods return identical frames across all engines when given empty spreadsheet tables (#16802)
• Consistent behaviour when "infer_schema_length=0" for read_excel (#16840)
• Standardised additional SQL interface errors (#16829)
• Ensure that splitted ChunkedArray also flattens chunks (#16837)
• Reduce needless panics in comparisons (#16831)
• Reset if next caller clones inner series (#16812)
• Raise on non-positive json schema inference (#16770)
• Rewrite implementation of top_k/bottom_k and fix a variety of bugs (#16804)
• Fix comparison of UInt64 with zero (#16799)
• Fix incorrect parquet statistics written for UInt64 values > Int64::MAX (#16766)
• Fix boolean distinct (#16765)
• DATE_PART SQL syntax/parsing, improve some error messages (#16761)
• Include pl. qualifier for inner dtypes in to_init_repr (#16235)
• Column selection wasn't applied when reading CSV with no rows (#16739)
• Panic on empty df / null List(Categorical) (#16730)
• Only flush if operator can flush in streaming outer join (#16723)
• Raise unsupported cat array (#16717)
• Assert SQLInterfaceError is raised (#16713)
• Restrict casting for temporal data types (#14142)
• Handle nested categoricals in assert_series_equal when categorical_as_str=True (#16700)
• Improve read_database check for SQLAlchemy async Session objects (#16680)
• Reduce scope of multi-threaded numpy conversion (#16686)
• Full null on dyn int (#16679)
• Fix filter shape on empty null (#16670)

📖 Documentation

• Update version switcher for 1.0.0 final release (#16848)
• Finish upgrade guide for 1.0.0 (#17257)
• Minor layout/terminology improvement for selector set ops (#17299)
• Mark hypothesis testing functionality as unstable (#17258)
• Add SQL docs for the CAST and TRY_CAST functions (#17214)
• Mark plot namespace as unstable (#17205)
• Bump docs dependencies (#17199)
• More accurate and helpful docs for user defined functions (#15194)
• Add doc examples to concat_list (#17127)
• Add "coming from pandas" note to DataFrame.unique docstring (#17119)
• Fix some warnings during doc build (#17077)
• Properly expose InProcessQuery in docs, mark as unstable (#17097)
• Add upgrade guide for Python Polars 1.0.0 (#16914)
• Lots of additions to the SQL reference docs (#16990)
• Minor doctest fixes (#17002)
• Include a doc entry for every exception type (#17001)
• Fixup bullet points in write_parquet docstring (#16909)
• Update version switcher for 1.0.0 prereleases (#16847)
• Update link from Python API reference to user guide (#16849)
• Update docstring/test/etc usage of select and with_columns to idiomatic form (#16801)
• Update versioning docs for 1.0.0 (#16757)
• Add docstring example for DataFrame.limit (#16753)
• Fix incorrect stated value of include_nulls in DataFrame.update docstring (#16701)
• Update deprecation docs in the user guide (#14315)
• Add example for index count in DataFrame.rolling (#16600)
• Improve docstring of Expr/Series.map_elements (#16079)
• Add missing polars.sql docs entry and small docstring update (#16656)

📦 Build system

• Update Cargo.lock (#17284)
• Streamline optional dependency definitions in pyproject.toml (#17168)
• Update rustc 2024-06-23 (#17135)
• Do not set environment variable on import (#17101)
• Fix config flag for Tracemalloc (#17098)
• Pin optional NumPy dependency to < 2.0.0 for now (#17060)

🛠️ Other improvements

• Fix typo in join validation error message (#17296)
• Fix linting issue in docs (#17292)
• Use typed iter in list.get (#17286)
• Rename type_aliases module to _typing (#17282)
• add ability to have pipeline blockers in new streaming engine (#17247)
• Support date/datetime for hive parts (#17256)
• Refactor serde tests, add hypothesis tests (#17216)
• Refactor parsing of data type inputs to Polars data types (#17164)
• Skip all moto AWS tests for now (#17178)
• Add missing spaces in cargo.toml (#17145)
• Minor test refactor for concat_list (#17120)
• Remove re-export of data type groups (#17073)
• Add pivot test #17081 (#17090)
• Minor cleanup to better define boundaries of public API (#17051)
• Support directory paths in scans for Parquet, IPC and CSV (#17017)
• Remove re-export of type aliases (#17032)
• Remove file cache test (#17038)
• Update exception imports in test suite (#17035)
• Point polars-stream to crates/ again (#17024)
• Fix failing file cache test in CI (#17014)
• Add some parametric tests for sort functionality (#17008)
• Pin NumPy to <2.0 for now (#16999)
• Use proper join type in test (#16994)
• Fix file cache verbose logging leakage during pytest (#16984)
• Skip another intermitently failing AWS test (#16980)
• Update test suite to explicitly use orient="row" in DataFrame constructor when applicable (#16977)
• Remove redundant projection attribute in IR::DataFrameScan (#16952)
• Factor out some apply calls in duration namespace (#16941)
• Skip intermittently failing AWS test (#16908)
• Refactor expression parsing utils (#16906)
• Set up some of the infrastructure for new streaming engine (#16900)
• Refactor parts of IR. (#16899)
• Add fundamentals for new async-based streaming execution engine (#16884)
• Move around some existing tests (#16877)
• Remove inner Arc from FileCacheEntry (#16870)
• Do not update stable API reference on prerelease (#16846)
• Update links to API references (#16843)
• Prepare update of API reference URLs (#16816)
• Rename allow_overflow to wrap_numerical (#16807)
• Set infer_schema_length as keyword-only argument in str.json_decode (#16835)
• Don't enter streaming engine for groupby-> agg mean/median … (#16810)
• Improve safety of amortized_iter (#16820)
• Remove needless inner type clone (#16718)
• Fix incorrect debug assertion in ChunkedArray::from_chunks_and_dtype (#16697)
• Update version resolver for 1.0.0 release (#16705)
• Avoid AWS pinning to outdated crc32c version (#16681)

Thank you to all our contributors for making this release possible!
@IvanIsCoding, @JamesCE2001, @JulianCologne, @KDruzhkin, @Kylea650, @MarcoGorelli, @Mottl, @Object905, @SeanTater, @adamreeve, @alexander-beedie, @bertiewooster, @borchero, @c-peters, @coastalwhite, @datapythonista, @datenzauberai, @dependabot, @dependabot[bot], @eitsupi, @flisky, @henryharbeck, @itamarst, @jqnatividad, @lukeshingles, @machow, @marenwestermann, @mcrumiller, @montanarograziano, @nameexhaustion, @orlp, @p3i0t, @ritchie46, @sherlockbeard, @stinodego, @tkellogg, @universalmind303 and @wence-

💥 Breaking changes

• Make hive_partitioning parameter default to None, which is automatically enabled for single directory inputs, and disabled otherwise (#17106)
• Split replace functionality into two separate functions (#16921)
• Default to writing binview data to IPC (#17084)
• Remove re-export of type aliases (#17032)
• Add strict parameter to DataFrame/LazyFrame.drop and fix behavior to default to True (#17044)
• Rename ModuleUpgradeRequired and PolarsPanicError error, remove InvalidAssert error (#17033)
• Change data orientation inference logic for DataFrame construction and warn when row orientation is inferred (#16976)
• Properly apply strict parameter in Series constructor (#16939)
• Remove supertype definition of List and non-List types (#16918)
• Consistently convert to given time zone in Series constructor (#16828)
• Update reshape to return Array types instead of List types (#16825)
• Default to raising on out-of-bounds indices in all get/gather operations (#16841)
• Native selector XOR set operation, guarantee consistent selector column-order (#16833)
• Set infer_schema_length as keyword-only argument in str.json_decode (#16835)
• Update set_sorted to only accept a single column (#16800)
• Remove deprecated parameters in Series.cut/qcut and update struct field names (#16741)
• Expedited removal of certain deprecated functionality (#16754)
• Update some error types to more appropriate variants (#15030)
• Scheduled removal of deprecated functionality (#16715)
• Change default offset in group_by_dynamic from 'negative every' to 'zero' (#16658)
• Constrain access to globals from DataFrame.sql in favor of top-level pl.sql (#16598)
• Read 2D NumPy arrays as Array type instead of List (#16710)
• Update clip to no longer propagate nulls in the given bounds (#14413)
• Change str.to_datetime to default to microsecond precision for format specifiers "%f" and "%.f" (#13597)
• Update resulting column names in pivot when pivoting by multiple values (#16439)
• Preserve nulls in ewm_mean, ewm_std, and ewm_var (#15503)
• Restrict casting for temporal data types (#14142)
• Support Decimal types by default when converting from Arrow (#15324)
• Remove serde functionality from pl.read_json and DataFrame.write_json (#16550)
• Update function signature of nth to allow positional input of indices, remove columns parameter (#16510)
• Rename struct fields of rle output to len/value and update data type of len field (#15249)
• Remove class variables from some DataTypes (#16524)
• Add check_names parameter to Series.equals and default to False (#16610)

⚠️ Deprecations

• Deprecate size parameter in parametric testing strategies in favor of min_size/max_size (#17128)
• Split replace functionality into two separate functions (#16921)
• Rename DataFrame.melt to unpivot and make parameters consistent with pivot (#17095)
• Remove re-export of exceptions at top-level (#17059)
• Deprecate dt.mean/dt.median in favor of mean/median (#16888)
• Deprecate LazyFrame.with_context in favor of horizontal concatenation (#16860)
• Rename parameter descending to reverse in top_k methods (#16817)
• Rename str.concat to str.join and update default delimiter (#16790)
• Deprecate arctan2d in favor of arctan2(...).degrees() (#16786)

🚀 Performance improvements

• create UniqueKernel and improve bool implementation (#17160)
• parallel linearize in new streaming engine (#17050)
• Default to writing binview data to IPC (#17084)
• Parallelize arrow conversion if binview -> large_bin (#17083)
• GC buffers in if_then_else view kernel (#16993)
• Desugar AND filter into multiple nodes (#16992)
• Optimize generic argsort of row-encoding (#16894)
• Improve rle_id iteration perf and set sorted flags (#16893)
• Optimize string/binary sort (#16871)
• Use split_at in split (#16865)
• Use split_at instead of double slice in chunk splits. (#16856)
• Don't rechunk in align_ if arrays are aligned (#16850)
• Don't create small chunks in parallel collect. (#16845)
• Add dedicated no-null branch in arg_sort (#16808)
• Speed up dt.offset_by 2x for constant durations (#16728)
• Toggle coalesce in join if non-coalesced key isn't projected (#16677)
• Make dt.truncate 1.5x faster when every is just a single duration (and not an expression) (#16666)
• Always prune unused columns in semi/anti join (#16665)

✨ Enhancements

• Support reading byte stream split encoded floats and doubles in parquet (#17099)
• Add float_scientific option to write_csv/sink_csv (#17111)
• Support Struct field selection in the SQL engine, RENAME and REPLACE select wildcard options (#17109)
• Update DataFrame.pivot to allow index=None when values is set (#17126)
• Make hive_partitioning parameter default to None, which is automatically enabled for single directory inputs, and disabled otherwise (#17106)
• Improve ipython autocomplete for LazyFrame and DataFrame (#17091)
• Split replace functionality into two separate functions (#16921)
• Improve schema inference for hive partitions (#17079)
• Rename DataFrame.melt to unpivot and make parameters consistent with pivot (#17095)
• print row index in explain + dot (#17074)
• Support top-level pl.col autocompletion for iPython (#17080)
• Remove re-export of exceptions at top-level (#17059)
• predicate + projection pushdown in NDJson (#17068)
• Allow (non-)coalescing in join_asof (#17066)
• Turn of coalescing and fix mutation of join on expressions (#17061)
• Expand NDJson glob into one SCAN (#17063)
• Do not parse hive partitions from user provided base directory path (#17055)
• Support directory paths in scans for Parquet, IPC and CSV (#17017)
• Implement general array equality checks (#17043)
• Add strict parameter to DataFrame/LazyFrame.drop and fix behavior to default to True (#17044)
• Rename ModuleUpgradeRequired and PolarsPanicError error, remove InvalidAssert error (#17033)
• Add rechunk parameter to read_delta (#16991)
• allow experimental metadata use on release (#17005)
• first working prototype of new streaming engine (#16970)
• Add simple version of json_normalize (#17015)
• Change data orientation inference logic for DataFrame construction and warn when row orientation is inferred (#16976)
• Desugar AND filter into multiple nodes (#16992)
• Handle textio even if not correct (#16971)
• Properly apply strict parameter in Series constructor (#16939)
• Add SQL support for INTERSECT and EXCEPT ops (#16960)
• Add PerformanceWarning to LazyFrame properties (#16964)
• Add collect_schema method to LazyFrame and DataFrame (#16929)
• Allow setting file cache TTL on a per-file basis (#16891)
• Support Decimal inputs for lit (#16950)
• Implement multiply and division for lhs duration (#16948)
• Raise on invalid temporal arithmetic (#16934)
• Always end with a in-memory sink on collect (#16928)
• add style namespace (which defers to Great Tables) (#16809)
• Add Schema class (#16873)
• Normalize value_counts (#16917)
• add eq/ne for more FixedSizeLists (#16902)
• setup skeleton (#16900)
• add fundamentals for new async-based streaming execution engine (#16884)
• Cache downloaded cloud IPC files (#16892)
• Consistently convert to given time zone in Series constructor (#16828)
• Improve read_csv SQL table reading function defaults (better handle dates) (#16866)
• Support SQL VALUES clause and inline renaming of columns in CTE & derived table definitions (#16851)
• Support Python Enum values in lit (#16858)
• convert to give time zone in .str.to_datetime when values are offset-aware (#16742)
• Update reshape to return Array types instead of List types (#16825)
• Default to raising on out-of-bounds indices in all get/gather operations (#16841)
• Support SQL "SELECT" with no tables, optimise registration of globals (#16836)
• Native selector XOR set operation, guarantee consistent selector column-order (#16833)
• Extend recognised EXTRACT and DATE_PART SQL part abbreviations (#16767)
• Improve error message when raising integers to negative integers, improve docs (#16827)
• Return datetime for mean/median of Date colum (#16795)
• Update set_sorted to only accept a single column (#16800)
• Expose overflowing cast (#16805)
• Update group_by iteration and partition_by to always return tuple keys (#16793)
• Support array arithmetic for equally sized shapes (#16791)
• Expedited removal of certain deprecated functionality (2) (#16779)
• Removal of read_database_uri passthrough from read_database (#16783)
• Remove pyxlsb engine from read_database (#16784)
• Add check_order parameter to assert_series_equal (#16778)
• Enforce deprecation of keyword arguments as positional (#16755)
• Support cloud storage in scan_csv (#16674)
• Streamline SQL INTERVAL handling and improve related error messages, update sqlparser-rs lib (#16744)
• Support use of ordinal values in SQL ORDER BY clause (#16745)
• Support executing polars SQL against pandas and pyarrow objects (#16746)
• Remove deprecated parameters in Series.cut/qcut and update struct field names (#16741)
• Expedited removal of certain deprecated functionality (#16754)
• Remove deprecated functionality from rolling methods (#16750)
• Update date_range to no longer produce datetime ranges (#16734)
• Mark min_periods as keyword-only for rolling methods (#16738)
• Remove deprecated top_k parameters nulls_last, maintain_order, and multithreaded (#16599)
• Support order-by in window functions (#16743)
• Add SQL support for NULLS FIRST/LAST ordering (#16711)
• Update some error types to more appropriate variants (#15030)
• Initial SQL support for INTERVAL strings (#16732)
• Scheduled removal of deprecated functionality (2) (#16724)
• Scheduled removal of deprecated functionality (#16715)
• Enforce deprecation of offset arg in truncate and round (#16655)
• Change default offset in group_by_dynamic from 'negative every' to 'zero' (#16658)
• Constrain access to globals from DataFrame.sql in favor of top-level pl.sql (#16598)
• Read 2D NumPy arrays as Array type instead of List (#16710)
• Update clip to no longer propagate nulls in the given bounds (#14413)
• Change str.to_datetime to default to microsecond precision for format specifiers "%f" and "%.f" (#13597)
• Update resulting column names in pivot when pivoting by multiple values (#16439)
• Preserve nulls in ewm_mean, ewm_std, and ewm_var (#15503)
• Restrict casting for temporal data types (#14142)
• Add many more auto-inferable datetime formats for str.to_datetime (#16634)
• Support Decimal types by default when converting from Arrow (#15324)
• Remove serde functionality from pl.read_json and DataFrame.write_json (#16550)
• Update function signature of nth to allow positional input of indices, remove columns parameter (#16510)
• Rename struct fields of rle output to len/value and update data type of len field (#15249)
• Remove class variables from some DataTypes (#16524)
• Add check_names parameter to Series.equals and default to False (#16610)
• Dedicated SQLInterface and SQLSyntax errors (#16635)
• Add DIV function support to the SQL interface (#16678)
• Support non-coalescing streaming left join (#16672)
• Allow wildcard and exclude before struct expansions (#16671)

🐞 Bug fixes

• Use explicit turbofish to help rustc (#17159)
• Raise on invalid set dtypes (#17157)
• Fix corrupted reads for hive parts from cloud and projection pushdown failure on hive parts (#17152)
• Set intersection supertype (#17154)
• ChainedWhen should not inherit Expr (#17142)
• Fix decompress_impl for csv with n_rows set (#17118)
• adds "polars-ops/timezones" dependency for "timezones" feature (#17115)
• Fix incorrect window std for chunked series (#17110)
• make GetOutput::get_field fallible (#17114)
• Fix melt panic (#17088)
• Fix expression autocomplete in ipython (#17072)
• Exclude index from expansion in rolling/group_by_dynamic (#17086)
• Update some Series dunder method type signatures (#17053)
• Fix oob of join with literals and empty table (#17047)
• Don't silently accept multi-table FROM clauses (implicit JOIN syntax) (#17028)
• Don't split up ANDed filters that are group-aware (#17031)
• Harden "async" check for users with out-of-date sqlalchemy libraries (#17029)
• error when sort_by of unequal length (#17026)
• properly catch not found explode cols (#17020)
• Correctly convert data frames to NumPy for C index order (#17000)
• Raise on invalid arithmetic shapes (#16986)
• Don't pushdown predicates in cross join if the refer to both tables (#16983)
• Fix projection pushdown with literal joins (#16981)
• Fix edge case in DataFrame constructor data orientation inference (#16975)
• Raise on list of objects (#16959)
• Handle strictness for Decimal Series construction (#15309)
• Don't panic in object to anyvalue (#16957)
• properly set FAST_EXPLODE_LIST metadata (#16951)
• Raise informative error when writing object to file (#16954)
• Remove supertype definition of List and non-List types (#16918)
• Remove unwrap in extend() (#16890)
• Fix should_rechunk check (#16852)
• Ensure read_excel and read_ods return identical frames across all engines when given empty spreadsheet tables (#16802)
• Consistent behaviour when "infer_schema_length=0" for read_excel (#16840)
• Standardised additional SQL interface errors (#16829)
• Ensure that splitted ChunkedArray also flattens chunks (#16837)
• Reduce needless panics in comparisons (#16831)
• Reset if next caller clones inner series (#16812)
• Raise on non-positive json schema inference (#16770)
• Rewrite implementation of top_k/bottom_k and fix a variety of bugs (#16804)
• Fix comparison of UInt64 with zero (#16799)
• Fix incorrect parquet statistics written for UInt64 values > Int64::MAX (#16766)
• Fix boolean distinct (#16765)
• DATE_PART SQL syntax/parsing, improve some error messages (#16761)
• Include pl. qualifier for inner dtypes in to_init_repr (#16235)
• Column selection wasn't applied when reading CSV with no rows (#16739)
• Panic on empty df / null List(Categorical) (#16730)
• Only flush if operator can flush in streaming outer join (#16723)
• Raise unsupported cat array (#16717)
• Assert SQLInterfaceError is raised (#16713)
• Restrict casting for temporal data types (#14142)
• Handle nested categoricals in assert_series_equal when categorical_as_str=True (#16700)
• Improve read_database check for SQLAlchemy async Session objects (#16680)
• Reduce scope of multi-threaded numpy conversion (#16686)
• Full null on dyn int (#16679)
• Fix filter shape on empty null (#16670)

📖 Documentation

• Add doc examples to concat_list (#17127)
• Add "coming from pandas" note to DataFrame.unique docstring (#17119)
• Fix some warnings during doc build (#17077)
• Properly expose InProcessQuery in docs, mark as unstable (#17097)
• Add upgrade guide for Python Polars 1.0.0 (#16914)
• Lots of additions to the SQL reference docs (#16990)
• Minor doctest fixes (#17002)
• Include a doc entry for every exception type (#17001)
• fixup bullet points in write_parquet (#16909)
• Update version switcher for 1.0.0 prereleases (#16847)
• Update link from Python API reference to user guide (#16849)
• Update docstring/test/etc usage of select and with_columns to idiomatic form (#16801)
• Update versioning docs for 1.0.0 (#16757)
• Add docstring example for DataFrame.limit (#16753)
• Fix incorrect stated value of include_nulls in DataFrame.update docstring (#16701)
• Update deprecation docs in the user guide (#14315)
• Add example for index count in DataFrame.rolling (#16600)
• Improve docstring of Expr/Series.map_elements (#16079)
• Add missing polars.sql docs entry and small docstring update (#16656)

📦 Build system

• Do not change environment on import (#17101)
• Fix config flag for Tracemalloc (#17098)
• Pin optional NumPy dependency to < 2.0.0 for now (#17060)

🛠️ Other improvements

• Add missing spaces in cargo.toml (#17145)
• Update rustc 2024-06-23 (#17135)
• Minor test refactor for concat_list (#17120)
• Remove re-export of data type groups (#17073)
• Add pivot test #17081 (#17090)
• Minor cleanup to better define boundaries of public API (#17051)
• Support directory paths in scans for Parquet, IPC and CSV (#17017)
• Remove re-export of type aliases (#17032)
• Remove file cache test (#17038)
• Update exception imports in test suite (#17035)
• Point polars-stream to crates/ again (#17024)
• Fix failing file cache test in CI (#17014)
• Add some parametric tests for sort functionality (#17008)
• Pin NumPy to <2.0 for now (#16999)
• Use proper join type in test (#16994)
• Fix file cache verbose logging leakage during pytest (#16984)
• Skip another intermitently failing AWS test (#16980)
• Update test suite to explicitly use orient="row" in DataFrame constructor when applicable (#16977)
• Remove redundant projection attribute in IR::DataFrameScan (#16952)
• Factor out some apply calls in duration namespace (#16941)
• extend new streaming engine with some initial nodes (#16940)
• Skip intermittently failing AWS test (#16908)
• Refactor expression parsing utils (#16906)
• setup skeleton (#16900)
• Refactor parts of IR. (#16899)
• Move around some existing tests (#16877)
• Remove inner Arc from FileCacheEntry (#16870)
• Do not update stable API reference on prerelease (#16846)
• Update links to API references (#16843)
• Prepare update of API reference URLs (#16816)
• Rename allow_overflow to wrap_numerical (#16807)
• Set infer_schema_length as keyword-only argument in str.json_decode (#16835)
• Don't enter streaming engine for groupby-> agg mean/median … (#16810)
• Improve safety of amortized_iter (#16820)
• Remove needless inner type clone (#16718)
• Fix incorrect debug assertion in ChunkedArray::from_chunks_and_dtype (#16697)
• Update version resolver for 1.0.0 release (#16705)
• Avoid AWS pinning to outdated crc32c version (#16681)

Thank you to all our contributors for making this release possible!
@JulianCologne, @KDruzhkin, @Kylea650, @MarcoGorelli, @Mottl, @Object905, @adamreeve, @alexander-beedie, @bertiewooster, @borchero, @c-peters, @coastalwhite, @datapythonista, @datenzauberai, @dependabot, @dependabot[bot], @eitsupi, @henryharbeck, @itamarst, @lukeshingles, @machow, @marenwestermann, @mcrumiller, @montanarograziano, @nameexhaustion, @orlp, @p3i0t, @ritchie46, @sherlockbeard, @stinodego, @tkellogg, @universalmind303 and @wence-

🚀 Performance improvements

• create UniqueKernel and improve bool implementation (#17160)
• parallel linearize in new streaming engine (#17050)

✨ Enhancements

• Support Struct field selection in the SQL engine, RENAME and REPLACE select wildcard options (#17109)
• POC metadata reading and writing (#17112)
• Use AsRef<Path> instead of PathBuf in sink_ methods (#17150)
• Update DataFrame.pivot to allow index=None when values is set (#17126)

🐞 Bug fixes

• Use explicit turbofish to help rustc (#17159)
• Raise on invalid set dtypes (#17157)
• Fix corrupted reads for hive parts from cloud and projection pushdown failure on hive parts (#17152)
• Set intersection supertype (#17154)
• Fix feature gates (#17141)
• Fix feature gate (#17134)

🛠️ Other improvements

• MinMaxKernel in primitive/binary parquet stats (#17158)
• Add a test for AnonymousScan options (projection and slice pushdown) (#17149)
• MinMaxKernel in prim parquet stats (#17153)
• Add missing spaces in cargo.toml (#17145)
• Update rustc 2024-06-23 (#17135)

Thank you to all our contributors for making this release possible!
@MarcoGorelli, @alexander-beedie, @coastalwhite, @datapythonista, @eitsupi, @mcrumiller, @nameexhaustion, @orlp, @ritchie46 and @stinodego

💥 Breaking changes

• Make hive_partitioning parameter default to None, which is automatically enabled for single directory inputs, and disabled otherwise (#17106)
• Split replace functionality into two separate functions (#16921)
• Default to writing binview data to IPC (#17084)
• Do not parse hive partitions from user provided directory/glob path (#17055)
• Remove re-export of type aliases (#17032)
• Add strict parameter to DataFrame/LazyFrame.drop and fix behavior to default to True (#17044)
• Rename ModuleUpgradeRequired and PolarsPanicError error, remove InvalidAssert error (#17033)
• Change data orientation inference logic for DataFrame construction and warn when row orientation is inferred (#16976)
• Properly apply strict parameter in Series constructor (#16939)
• Remove supertype definition of List and non-List types (#16918)
• Consistently convert to given time zone in Series constructor (#16828)
• Update reshape to return Array types instead of List types (#16825)
• Default to raising on out-of-bounds indices in all get/gather operations (#16841)
• Native selector XOR set operation, guarantee consistent selector column-order (#16833)
• Set infer_schema_length as keyword-only argument in str.json_decode (#16835)
• Update set_sorted to only accept a single column (#16800)
• Remove deprecated parameters in Series.cut/qcut and update struct field names (#16741)
• Expedited removal of certain deprecated functionality (#16754)
• Update some error types to more appropriate variants (#15030)
• Scheduled removal of deprecated functionality (#16715)
• Change default offset in group_by_dynamic from 'negative every' to 'zero' (#16658)
• Constrain access to globals from DataFrame.sql in favor of top-level pl.sql (#16598)
• Read 2D NumPy arrays as Array type instead of List (#16710)
• Update clip to no longer propagate nulls in the given bounds (#14413)
• Change str.to_datetime to default to microsecond precision for format specifiers "%f" and "%.f" (#13597)
• Update resulting column names in pivot when pivoting by multiple values (#16439)
• Preserve nulls in ewm_mean, ewm_std, and ewm_var (#15503)
• Restrict casting for temporal data types (#14142)
• Support Decimal types by default when converting from Arrow (#15324)
• Remove serde functionality from pl.read_json and DataFrame.write_json (#16550)
• Update function signature of nth to allow positional input of indices, remove columns parameter (#16510)
• Rename struct fields of rle output to len/value and update data type of len field (#15249)
• Remove class variables from some DataTypes (#16524)
• Add check_names parameter to Series.equals and default to False (#16610)

⚠️ Deprecations

• Deprecate size parameter in parametric testing strategies in favor of min_size/max_size (#17128)
• Split replace functionality into two separate functions (#16921)
• Rename DataFrame.melt to unpivot and make parameters consistent with pivot (#17095)
• Remove re-export of exceptions at top-level (#17059)
• Deprecate dt.mean/dt.median in favor of mean/median (#16888)
• Deprecate LazyFrame.with_context in favor of horizontal concatenation (#16860)
• Rename parameter descending to reverse in top_k methods (#16817)
• Rename str.concat to str.join and update default delimiter (#16790)
• Deprecate arctan2d in favor of arctan2(...).degrees() (#16786)

🚀 Performance improvements

• Default to writing binview data to IPC (#17084)
• Parallelize arrow conversion if binview -> large_bin (#17083)
• GC buffers in if_then_else view kernel (#16993)
• Desugar AND filter into multiple nodes (#16992)
• Optimize generic argsort of row-encoding (#16894)
• Improve rle_id iteration perf and set sorted flags (#16893)
• Optimize string/binary sort (#16871)
• Use split_at in split (#16865)
• Use split_at instead of double slice in chunk splits. (#16856)
• Don't rechunk in align_ if arrays are aligned (#16850)
• Don't create small chunks in parallel collect. (#16845)
• Add dedicated no-null branch in arg_sort (#16808)
• Speed up dt.offset_by 2x for constant durations (#16728)
• Toggle coalesce in join if non-coalesced key isn't projected (#16677)
• Make dt.truncate 1.5x faster when every is just a single duration (and not an expression) (#16666)
• Always prune unused columns in semi/anti join (#16665)

✨ Enhancements

• Update DataFrame.pivot to allow index=None when values is set (#17126)
• Make hive_partitioning parameter default to None, which is automatically enabled for single directory inputs, and disabled otherwise (#17106)
• Improve ipython autocomplete for LazyFrame and DataFrame (#17091)
• Split replace functionality into two separate functions (#16921)
• Improve schema inference for hive partitions (#17079)
• Rename DataFrame.melt to unpivot and make parameters consistent with pivot (#17095)
• print row index in explain + dot (#17074)
• Support top-level pl.col autocompletion for iPython (#17080)
• Remove re-export of exceptions at top-level (#17059)
• predicate + projection pushdown in NDJson (#17068)
• Allow (non-)coalescing in join_asof (#17066)
• Turn of coalescing and fix mutation of join on expressions (#17061)
• Expand NDJson glob into one SCAN (#17063)
• Do not parse hive partitions from user provided directory/glob path (#17055)
• Support directory paths in scans for Parquet, IPC and CSV (#17017)
• Implement general array equality checks (#17043)
• Add strict parameter to DataFrame/LazyFrame.drop and fix behavior to default to True (#17044)
• Rename ModuleUpgradeRequired and PolarsPanicError error, remove InvalidAssert error (#17033)
• Add rechunk parameter to read_delta (#16991)
• allow experimental metadata use on release (#17005)
• first working prototype of new streaming engine (#16970)
• Add simple version of json_normalize (#17015)
• Change data orientation inference logic for DataFrame construction and warn when row orientation is inferred (#16976)
• Desugar AND filter into multiple nodes (#16992)
• Handle textio even if not correct (#16971)
• Properly apply strict parameter in Series constructor (#16939)
• Add SQL support for INTERSECT and EXCEPT ops (#16960)
• Add PerformanceWarning to LazyFrame properties (#16964)
• Add collect_schema method to LazyFrame and DataFrame (#16929)
• Allow setting file cache TTL on a per-file basis (#16891)
• Support Decimal inputs for lit (#16950)
• Implement multiply and division for lhs duration (#16948)
• Raise on invalid temporal arithmetic (#16934)
• Always end with a in-memory sink on collect (#16928)
• add style namespace (which defers to Great Tables) (#16809)
• Add Schema class (#16873)
• Normalize value_counts (#16917)
• add eq/ne for more FixedSizeLists (#16902)
• setup skeleton (#16900)
• add fundamentals for new async-based streaming execution engine (#16884)
• Cache downloaded cloud IPC files (#16892)
• Consistently convert to given time zone in Series constructor (#16828)
• Improve read_csv SQL table reading function defaults (better handle dates) (#16866)
• Support SQL VALUES clause and inline renaming of columns in CTE & derived table definitions (#16851)
• Support Python Enum values in lit (#16858)
• convert to give time zone in .str.to_datetime when values are offset-aware (#16742)
• Update reshape to return Array types instead of List types (#16825)
• Default to raising on out-of-bounds indices in all get/gather operations (#16841)
• Support SQL "SELECT" with no tables, optimise registration of globals (#16836)
• Native selector XOR set operation, guarantee consistent selector column-order (#16833)
• Extend recognised EXTRACT and DATE_PART SQL part abbreviations (#16767)
• Improve error message when raising integers to negative integers, improve docs (#16827)
• Return datetime for mean/median of Date colum (#16795)
• Update set_sorted to only accept a single column (#16800)
• Expose overflowing cast (#16805)
• Update group_by iteration and partition_by to always return tuple keys (#16793)
• Support array arithmetic for equally sized shapes (#16791)
• Expedited removal of certain deprecated functionality (2) (#16779)
• Removal of read_database_uri passthrough from read_database (#16783)
• Remove pyxlsb engine from read_database (#16784)
• Add check_order parameter to assert_series_equal (#16778)
• Enforce deprecation of keyword arguments as positional (#16755)
• Support cloud storage in scan_csv (#16674)
• Streamline SQL INTERVAL handling and improve related error messages, update sqlparser-rs lib (#16744)
• Support use of ordinal values in SQL ORDER BY clause (#16745)
• Support executing polars SQL against pandas and pyarrow objects (#16746)
• Remove deprecated parameters in Series.cut/qcut and update struct field names (#16741)
• Expedited removal of certain deprecated functionality (#16754)
• Remove deprecated functionality from rolling methods (#16750)
• Update date_range to no longer produce datetime ranges (#16734)
• Mark min_periods as keyword-only for rolling methods (#16738)
• Remove deprecated top_k parameters nulls_last, maintain_order, and multithreaded (#16599)
• Support order-by in window functions (#16743)
• Add SQL support for NULLS FIRST/LAST ordering (#16711)
• Update some error types to more appropriate variants (#15030)
• Initial SQL support for INTERVAL strings (#16732)
• Scheduled removal of deprecated functionality (2) (#16724)
• Scheduled removal of deprecated functionality (#16715)
• Enforce deprecation of offset arg in truncate and round (#16655)
• Change default offset in group_by_dynamic from 'negative every' to 'zero' (#16658)
• Constrain access to globals from DataFrame.sql in favor of top-level pl.sql (#16598)
• Read 2D NumPy arrays as Array type instead of List (#16710)
• Update clip to no longer propagate nulls in the given bounds (#14413)
• Change str.to_datetime to default to microsecond precision for format specifiers "%f" and "%.f" (#13597)
• Update resulting column names in pivot when pivoting by multiple values (#16439)
• Preserve nulls in ewm_mean, ewm_std, and ewm_var (#15503)
• Restrict casting for temporal data types (#14142)
• Add many more auto-inferable datetime formats for str.to_datetime (#16634)
• Support Decimal types by default when converting from Arrow (#15324)
• Remove serde functionality from pl.read_json and DataFrame.write_json (#16550)
• Update function signature of nth to allow positional input of indices, remove columns parameter (#16510)
• Rename struct fields of rle output to len/value and update data type of len field (#15249)
• Remove class variables from some DataTypes (#16524)
• Add check_names parameter to Series.equals and default to False (#16610)
• Dedicated SQLInterface and SQLSyntax errors (#16635)
• Add DIV function support to the SQL interface (#16678)
• Support non-coalescing streaming left join (#16672)
• Allow wildcard and exclude before struct expansions (#16671)

🐞 Bug fixes

• Fix decompress_impl for csv with n_rows set (#17118)
• adds "polars-ops/timezones" dependency for "timezones" feature (#17115)
• Fix incorrect window std for chunked series (#17110)
• make GetOutput::get_field fallible (#17114)
• Fix melt panic (#17088)
• Fix expression autocomplete in ipython (#17072)
• Exclude index from expansion in rolling/group_by_dynamic (#17086)
• Update some Series dunder method type signatures (#17053)
• Fix oob of join with literals and empty table (#17047)
• Don't silently accept multi-table FROM clauses (implicit JOIN syntax) (#17028)
• Don't split up ANDed filters that are group-aware (#17031)
• Harden "async" check for users with out-of-date sqlalchemy libraries (#17029)
• error when sort_by of unequal length (#17026)
• properly catch not found explode cols (#17020)
• Correctly convert data frames to NumPy for C index order (#17000)
• Raise on invalid arithmetic shapes (#16986)
• Don't pushdown predicates in cross join if the refer to both tables (#16983)
• Fix projection pushdown with literal joins (#16981)
• Fix edge case in DataFrame constructor data orientation inference (#16975)
• Raise on list of objects (#16959)
• Handle strictness for Decimal Series construction (#15309)
• Don't panic in object to anyvalue (#16957)
• properly set FAST_EXPLODE_LIST metadata (#16951)
• Raise informative error when writing object to file (#16954)
• Remove supertype definition of List and non-List types (#16918)
• Remove unwrap in extend() (#16890)
• Fix should_rechunk check (#16852)
• Ensure read_excel and read_ods return identical frames across all engines when given empty spreadsheet tables (#16802)
• Consistent behaviour when "infer_schema_length=0" for read_excel (#16840)
• Standardised additional SQL interface errors (#16829)
• Ensure that splitted ChunkedArray also flattens chunks (#16837)
• Reduce needless panics in comparisons (#16831)
• Reset if next caller clones inner series (#16812)
• Raise on non-positive json schema inference (#16770)
• Rewrite implementation of top_k/bottom_k and fix a variety of bugs (#16804)
• Fix comparison of UInt64 with zero (#16799)
• Fix incorrect parquet statistics written for UInt64 values > Int64::MAX (#16766)
• Fix boolean distinct (#16765)
• DATE_PART SQL syntax/parsing, improve some error messages (#16761)
• Include pl. qualifier for inner dtypes in to_init_repr (#16235)
• Column selection wasn't applied when reading CSV with no rows (#16739)
• Panic on empty df / null List(Categorical) (#16730)
• Only flush if operator can flush in streaming outer join (#16723)
• Raise unsupported cat array (#16717)
• Assert SQLInterfaceError is raised (#16713)
• Restrict casting for temporal data types (#14142)
• Handle nested categoricals in assert_series_equal when categorical_as_str=True (#16700)
• Improve read_database check for SQLAlchemy async Session objects (#16680)
• Reduce scope of multi-threaded numpy conversion (#16686)
• Full null on dyn int (#16679)
• Fix filter shape on empty null (#16670)

📖 Documentation

• Add doc examples to concat_list (#17127)
• Add "coming from pandas" note to DataFrame.unique docstring (#17119)
• Fix some warnings during doc build (#17077)
• Properly expose InProcessQuery in docs, mark as unstable (#17097)
• Add upgrade guide for Python Polars 1.0.0 (#16914)
• Lots of additions to the SQL reference docs (#16990)
• Minor doctest fixes (#17002)
• Include a doc entry for every exception type (#17001)
• fixup bullet points in write_parquet (#16909)
• Update version switcher for 1.0.0 prereleases (#16847)
• Update link from Python API reference to user guide (#16849)
• Update docstring/test/etc usage of select and with_columns to idiomatic form (#16801)
• Update versioning docs for 1.0.0 (#16757)
• Add docstring example for DataFrame.limit (#16753)
• Fix incorrect stated value of include_nulls in DataFrame.update docstring (#16701)
• Update deprecation docs in the user guide (#14315)
• Add example for index count in DataFrame.rolling (#16600)
• Improve docstring of Expr/Series.map_elements (#16079)
• Add missing polars.sql docs entry and small docstring update (#16656)

📦 Build system

• Do not change environment on import (#17101)
• Fix config flag for Tracemalloc (#17098)
• Pin optional NumPy dependency to < 2.0.0 for now (#17060)

🛠️ Other improvements

• Update rustc 2024-06-23 (#17135)
• Minor test refactor for concat_list (#17120)
• Remove re-export of data type groups (#17073)
• Add pivot test #17081 (#17090)
• Minor cleanup to better define boundaries of public API (#17051)
• Support directory paths in scans for Parquet, IPC and CSV (#17017)
• Remove re-export of type aliases (#17032)
• Remove file cache test (#17038)
• Update exception imports in test suite (#17035)
• Point polars-stream to crates/ again (#17024)
• Fix failing file cache test in CI (#17014)
• Add some parametric tests for sort functionality (#17008)
• Pin NumPy to <2.0 for now (#16999)
• Use proper join type in test (#16994)
• Fix file cache verbose logging leakage during pytest (#16984)
• Skip another intermitently failing AWS test (#16980)
• Update test suite to explicitly use orient="row" in DataFrame constructor when applicable (#16977)
• Remove redundant projection attribute in IR::DataFrameScan (#16952)
• Factor out some apply calls in duration namespace (#16941)
• extend new streaming engine with some initial nodes (#16940)
• Skip intermittently failing AWS test (#16908)
• Refactor expression parsing utils (#16906)
• setup skeleton (#16900)
• Refactor parts of IR. (#16899)
• Move around some existing tests (#16877)
• Remove inner Arc from FileCacheEntry (#16870)
• Do not update stable API reference on prerelease (#16846)
• Update links to API references (#16843)
• Prepare update of API reference URLs (#16816)
• Rename allow_overflow to wrap_numerical (#16807)
• Set infer_schema_length as keyword-only argument in str.json_decode (#16835)
• Don't enter streaming engine for groupby-> agg mean/median … (#16810)
• Improve safety of amortized_iter (#16820)
• Remove needless inner type clone (#16718)
• Fix incorrect debug assertion in ChunkedArray::from_chunks_and_dtype (#16697)
• Update version resolver for 1.0.0 release (#16705)
• Avoid AWS pinning to outdated crc32c version (#16681)

Thank you to all our contributors for making this release possible!
@JulianCologne, @KDruzhkin, @Kylea650, @MarcoGorelli, @Mottl, @Object905, @alexander-beedie, @bertiewooster, @borchero, @c-peters, @coastalwhite, @datenzauberai, @dependabot, @dependabot[bot], @henryharbeck, @itamarst, @machow, @marenwestermann, @mcrumiller, @montanarograziano, @nameexhaustion, @orlp, @p3i0t, @ritchie46, @sherlockbeard, @stinodego, @tkellogg, @universalmind303 and @wence-

💥 Breaking changes

• Make hive_partitioning parameter default to None, which is automatically enabled for single directory inputs, and disabled otherwise (#17106)
• Split replace functionality into two separate functions (#16921)
• Rename DataFrame.melt to unpivot and make parameters consistent with pivot (#17095)
• Default to writing binview data to IPC (#17084)
• Do not parse hive partitions from user provided directory/glob path (#17055)
• Add strict parameter to DataFrame/LazyFrame.drop and fix behavior to default to True (#17044)
• Remove supertype definition of List and non-List types (#16918)
• Native selector XOR set operation, guarantee consistent selector column-order (#16833)
• move offset_by implementation from polars-plan to polars-time, rename feature from DateOffset to OffsetBy (#16796)
• Rename str.concat to str.join and update default delimiter (#16790)
• Remove deprecated parameters in Series.cut/qcut and update struct field names (#16741)
• Expedited removal of certain deprecated functionality (#16754)
• Update some error types to more appropriate variants (#15030)
• Change default offset in group_by_dynamic from 'negative every' to 'zero' (#16658)
• Update clip to no longer propagate nulls in the given bounds (#14413)
• Change str.to_datetime to default to microsecond precision for format specifiers "%f" and "%.f" (#13597)
• Update resulting column names in pivot when pivoting by multiple values (#16439)
• Preserve nulls in ewm_mean, ewm_std, and ewm_var (#15503)
• Restrict casting for temporal data types (#14142)
• Rename struct fields of rle output to len/value and update data type of len field (#15249)
• Add check_names parameter to Series.equals and default to False (#16610)
• Deprecate str.explode in favor of str.split("").explode() (#16508)
• Deprecate how="outer" join type in favour of how="full" (left/right are *also* outer joins) (#16417)
• Change DataFrame.is_empty() to check height == 0 instead of width == 0 (#16351)

🚀 Performance improvements

• Default to writing binview data to IPC (#17084)
• Parallelize arrow conversion if binview -> large_bin (#17083)
• GC buffers in if_then_else view kernel (#16993)
• Desugar AND filter into multiple nodes (#16992)
• Optimize generic argsort of row-encoding (#16894)
• Improve rle_id iteration perf and set sorted flags (#16893)
• Optimize string/binary sort (#16871)
• Use split_at in split (#16865)
• Use split_at instead of double slice in chunk splits. (#16856)
• Don't rechunk in align_ if arrays are aligned (#16850)
• Don't create small chunks in parallel collect. (#16845)
• Add dedicated no-null branch in arg_sort (#16808)
• Speed up dt.offset_by 2x for constant durations (#16728)
• Toggle coalesce in join if non-coalesced key isn't projected (#16677)
• Make dt.truncate 1.5x faster when every is just a single duration (and not an expression) (#16666)
• Always prune unused columns in semi/anti join (#16665)
• make truncate 4x faster in simple cases (#16615)
• Cache arena's (and conversion) in SQL context (#16566)
• Partial schema cache. (#16549)
• improved numeric fill_(forward/backward) (#16475)
• only rechunk once per aggregate (#16469)
• Fix pathological small chunk parquet writing (#16433)

✨ Enhancements

• Make hive_partitioning parameter default to None, which is automatically enabled for single directory inputs, and disabled otherwise (#17106)
• Split replace functionality into two separate functions (#16921)
• Improve schema inference for hive partitions (#17079)
• Rename DataFrame.melt to unpivot and make parameters consistent with pivot (#17095)
• print row index in explain + dot (#17074)
• Support top-level pl.col autocompletion for iPython (#17080)
• predicate + projection pushdown in NDJson (#17068)
• Allow (non-)coalescing in join_asof (#17066)
• Turn of coalescing and fix mutation of join on expressions (#17061)
• Expand NDJson glob into one SCAN (#17063)
• Do not parse hive partitions from user provided directory/glob path (#17055)
• Support directory paths in scans for Parquet, IPC and CSV (#17017)
• Implement general array equality checks (#17043)
• Add strict parameter to DataFrame/LazyFrame.drop and fix behavior to default to True (#17044)
• allow experimental metadata use on release (#17005)
• first working prototype of new streaming engine (#16970)
• Desugar AND filter into multiple nodes (#16992)
• use min/max metadata on debug builds with POLARS_METADATA_FLAGS=extensive (#16963)
• Add SQL support for INTERSECT and EXCEPT ops (#16960)
• Allow setting file cache TTL on a per-file basis (#16891)
• Implement multiply and division for lhs duration (#16948)
• Raise on invalid temporal arithmetic (#16934)
• Always end with a in-memory sink on collect (#16928)
• Normalize value_counts (#16917)
• add eq/ne for more FixedSizeLists (#16902)
• setup skeleton (#16900)
• add fundamentals for new async-based streaming execution engine (#16884)
• Cache downloaded cloud IPC files (#16892)
• Improve read_csv SQL table reading function defaults (better handle dates) (#16866)
• Support SQL VALUES clause and inline renaming of columns in CTE & derived table definitions (#16851)
• convert to give time zone in .str.to_datetime when values are offset-aware (#16742)
• Support SQL "SELECT" with no tables, optimise registration of globals (#16836)
• Native selector XOR set operation, guarantee consistent selector column-order (#16833)
• Extend recognised EXTRACT and DATE_PART SQL part abbreviations (#16767)
• Improve error message when raising integers to negative integers, improve docs (#16827)
• Return datetime for mean/median of Date colum (#16795)
• Expose overflowing cast (#16805)
• Expose a few more expression nodes in the expression IR (#16781)
• Support array arithmetic for equally sized shapes (#16791)
• Support cloud storage in scan_csv (#16674)
• Streamline SQL INTERVAL handling and improve related error messages, update sqlparser-rs lib (#16744)
• Support use of ordinal values in SQL ORDER BY clause (#16745)
• Support executing polars SQL against pandas and pyarrow objects (#16746)
• add env locked metadata functions (#16719)
• Remove deprecated parameters in Series.cut/qcut and update struct field names (#16741)
• Expedited removal of certain deprecated functionality (#16754)
• Update date_range to no longer produce datetime ranges (#16734)
• Remove deprecated top_k parameters nulls_last, maintain_order, and multithreaded (#16599)
• Support order-by in window functions (#16743)
• Add SQL support for NULLS FIRST/LAST ordering (#16711)
• Update some error types to more appropriate variants (#15030)
• Initial SQL support for INTERVAL strings (#16732)
• Enforce deprecation of offset arg in truncate and round (#16655)
• eliminate ProjectionExprs and handle CSE by stacking extra columns (#16682)
• Change default offset in group_by_dynamic from 'negative every' to 'zero' (#16658)
• Update clip to no longer propagate nulls in the given bounds (#14413)
• Change str.to_datetime to default to microsecond precision for format specifiers "%f" and "%.f" (#13597)
• Update resulting column names in pivot when pivoting by multiple values (#16439)
• Preserve nulls in ewm_mean, ewm_std, and ewm_var (#15503)
• Restrict casting for temporal data types (#14142)
• Add many more auto-inferable datetime formats for str.to_datetime (#16634)
• Rename struct fields of rle output to len/value and update data type of len field (#15249)
• Add check_names parameter to Series.equals and default to False (#16610)
• Dedicated SQLInterface and SQLSyntax errors (#16635)
• Add DIV function support to the SQL interface (#16678)
• add additional control to write_parquet::statistics parameter (#16575)
• Support non-coalescing streaming left join (#16672)
• Allow wildcard and exclude before struct expansions (#16671)
• Support per-column nulls_last on sort operations (#16639)
• Add split_at method to arrow Array (#16620)
• Initial support for SQL ARRAY literals and the UNNEST table function (#16330)
• Don't allow struct.with_fields in grouping (#16629)
• Add SQL support for TRY_CAST function (#16589)
• add fuzzer for expressions (#16581)
• handle CSE dtypes in NodeTraverser.get_dtype (#16552)
• check if by column is sorted, rather than just checking sorted flag, in group_by_dynamic, upsample, and rolling (#16494)
• Add general metadata structure to ChunkedArray (#16399)
• Add is_column_selection() to expression meta, enhance expand_selector (#16479)
• NDarray/Tensor support (#16466)
• Allow designation of a custom name for the value_counts "count" column (#16434)
• Default rechunk=False for read_parquet (#16427)
• Add field expression as selector with an struct scope (#16402)
• Field expansion renaming (#16397)
• add cluster_with_columns plan optimization (#16274)
• Change DataFrame.is_empty() to check height == 0 instead of width == 0 (#16351)
• add Expr.interpolate_by (#16313)

🐞 Bug fixes

• Expand i128 primitive type match (#17076)
• Fix decompress_impl for csv with n_rows set (#17118)
• adds "polars-ops/timezones" dependency for "timezones" feature (#17115)
• Fix incorrect window std for chunked series (#17110)
• make GetOutput::get_field fallible (#17114)
• bubble error when no available bitrepr (#17116)
• Fix melt panic (#17088)
• Exclude index from expansion in rolling/group_by_dynamic (#17086)
• fix #17043 binary compare (#17052)
• Fix oob of join with literals and empty table (#17047)
• Don't silently accept multi-table FROM clauses (implicit JOIN syntax) (#17028)
• fix get categories on multiple row groups (#17041)
• Don't split up ANDed filters that are group-aware (#17031)
• Harden "async" check for users with out-of-date sqlalchemy libraries (#17029)
• error when sort_by of unequal length (#17026)
• properly catch not found explode cols (#17020)
• Correctly convert data frames to NumPy for C index order (#17000)
• Raise on invalid arithmetic shapes (#16986)
• Don't pushdown predicates in cross join if the refer to both tables (#16983)
• Fix projection pushdown with literal joins (#16981)
• Handle strictness for Decimal Series construction (#15309)
• properly set FAST_EXPLODE_LIST metadata (#16951)
• Raise informative error when writing object to file (#16954)
• Remove supertype definition of List and non-List types (#16918)
• Reject non-integral offset and length in AExpr::Slice (#16874)
• properly read/write fixed-sized lists from/to parquet files (#16747)
• Remove unwrap in extend() (#16890)
• Fix should_rechunk check (#16852)
• Standardised additional SQL interface errors (#16829)
• Ensure that splitted ChunkedArray also flattens chunks (#16837)
• Reduce needless panics in comparisons (#16831)
• Reset if next caller clones inner series (#16812)
• Raise on non-positive json schema inference (#16770)
• describe / explain streaming plan (#16771)
• Rewrite implementation of top_k/bottom_k and fix a variety of bugs (#16804)
• Fix comparison of UInt64 with zero (#16799)
• properly set boolean distinct count (#16782)
• Fix incorrect parquet statistics written for UInt64 values > Int64::MAX (#16766)
• Fix boolean distinct (#16765)
• DATE_PART SQL syntax/parsing, improve some error messages (#16761)
• Column selection wasn't applied when reading CSV with no rows (#16739)
• Only flush if operator can flush in streaming outer join (#16723)
• Raise unsupported cat array (#16717)
• Restrict casting for temporal data types (#14142)
• Improve read_database check for SQLAlchemy async Session objects (#16680)
• Full null on dyn int (#16679)
• Fix filter shape on empty null (#16670)
• Potentially deal with empty range (#16650)
• Use of SQL ORDER BY should not cause reordering of SELECT cols (#16579)
• ensure df in empty parquet (#16621)
• get_dtype handles input node schema and CSE (#16582)
• Crash using empty Series in LazyFrame.select() (#16592)
• small safety issue in CWC filtermap (#16591)
• bail on bool floordiv (#16578)
• Resolve multiple SQL JOIN issues (#16507)
• solve panic in cluster_with_columns, found with small fuzzer (#16562)
• Project last column if count query (#16569)
• Properly split struct columns (#16563)
• Ensure strict chunking in chunked partitioned group by (#16561)
• Error selecting columns after non-coalesced join (multiple join keys) (#16559)
• Don't panic on hashing nested list types (#16555)
• deal with realiases in cluster_with_columns (#16548)
• Ensure deduced join key names are unique (#16551)
• Crash selecting columns after non-coalesced join (#16541)
• Fix group gather of single literal (#16539)
• throw an invalid operation exception on performing a sum over a list of strs (#16521)
• properly fill null into unknown struct fields (#16518)
• Fix df.chunked for struct (#16504)
• Mix of column and field expansion (#16502)
• Fix split_chunks for nested dtypes (#16493)
• Handle struct.fields as special case of alias (#16484)
• Correct schema for list.sum (#16483)
• properly display other length in simple project and display more columns up to a maximum length (#16464)
• allow search_sorted directly on multiple chunks, and fix behavior around nulls (#16447)
• Fix use of COUNT(*) in SQL GROUP BY operations (#16465)
• properly set schemas in cluster_with_columns when reassigning to columns (#16463)
• Fix panic when computing min() of Duration series. (#16455)
• keep track of removed indices in CWC (#16443)
• Fix struct 'with_fields' schema for update dtypes (#16428)
• Fix error reading lists of CSV files that contain comments (#16426)
• Fix struct arithmetic schema (#16396)
• Fix don't panic on chunked to_numpy conversion (#16393)
• Don't check nulls before conversion (#16392)
• fix a printing issue in IR::Filter (#16378)
• correct AExpr.to_field for bitwise and logical and/or (#16360)

📖 Documentation

• Fix some warnings during doc build (#17077)
• Lots of additions to the SQL reference docs (#16990)
• Update the Overview section of the contributing guide (#15674)
• Update outdated performance section (#16409)

📦 Build system

• Bump Rust toolchain version to nightly-2024-06-03 (#16669)
• bump brotli from 5.0.0 to 6.0.0 (#16643)

🛠️ Other improvements

• Split replace functionality into two separate functions (#16921)
• Rename DataFrame.melt to unpivot and make parameters consistent with pivot (#17095)
• Add pivot test #17081 (#17090)
• Make Series and ChunkedArray ops fallible (#16965)
• Support directory paths in scans for Parquet, IPC and CSV (#17017)
• expose all metadata to polars-expr (#17018)
• Move in-memory engine to its own crate (#17039)
• Remove file cache test (#17038)
• Point polars-stream to crates/ again (#17024)
• Fix failing file cache test in CI (#17014)
• Use proper join type in test (#16994)
• Fix file cache verbose logging leakage during pytest (#16984)
• Remove redundant projection attribute in IR::DataFrameScan (#16952)
• Follow-up changes to polars-parquet (#16949)
• Factor out some apply calls in duration namespace (#16941)
• extend new streaming engine with some initial nodes (#16940)
• setup skeleton (#16900)
• Refactor parts of IR. (#16899)
• Remove inner Arc from FileCacheEntry (#16870)
• Update links to API references (#16843)
• Prepare update of API reference URLs (#16816)
• Rename allow_overflow to wrap_numerical (#16807)
• Remove unneeded code (#16838)
• Don't enter streaming engine for groupby-> agg mean/median … (#16810)
• move offset_by implementation from polars-plan to polars-time, rename feature from DateOffset to OffsetBy (#16796)
• Improve safety of amortized_iter (#16820)
• start further use of polars-compute in polars-parquet (#16788)
• Remove deprecated MutableBitmap.null_count method (#16797)
• Rename str.concat to str.join and update default delimiter (#16790)
• add distinct_non_null_count method (#16789)
• Remove needless inner type clone (#16718)
• Simplify NodeTraverser.get_dtype (#16712)
• Fix incorrect debug assertion in ChunkedArray::from_chunks_and_dtype (#16697)
• separate Aggregation evaluation PhysicalExpr from conversion (#16688)
• Update version resolver for 1.0.0 release (#16705)
• Branch earlier in binary type resolving (#16685)
• Avoid AWS pinning to outdated crc32c version (#16681)
• use binary try_{add, mul, ...} ops in borrowed dispatch (#16580)
• Split dates_times into separate date and time modules (#16667)
• Add test for 16642 (#16646)
• Remove duplicate tag in CODEOWNERS (#16625)
• CWC prealloc pushable and potential_pushable (#16626)
• Update dprint hook versions and enable JSON linting (#16611)
• defensively invalidate metadata and start on copying of min_value, max_value and distinct_count (#16593)
• Deprecate str.explode in favor of str.split("").explode() (#16508)
• make Parquet Statistics into enum instead of trait (#16485)
• small change on CWC where we can reuse the old pushable_set_bits and reserve space for input_exprs (#16468)
• Deprecate how="outer" join type in favour of how="full" (left/right are *also* outer joins) (#16417)
• Include license file in polars-expr crate (#16421)

Thank you to all our contributors for making this release possible!
@BGR360, @JulianCologne, @KDruzhkin, @Kylea650, @MarcoGorelli, @Mottl, @Object905, @alexander-beedie, @ankane, @bertiewooster, @borchero, @c-peters, @cmdlineluser, @coastalwhite, @dangotbanned, @datenzauberai, @dependabot, @dependabot[bot], @hattajr, @henryharbeck, @itamarst, @machow, @marenwestermann, @mcrumiller, @mdavis-xyz, @messense, @montanarograziano, @nameexhaustion, @orlp, @p3i0t, @r-brink, @ritchie46, @siddharth-gulia, @stinodego, @tkellogg, @twoertwein, @universalmind303 and @wence-

💥 Breaking changes

• Change data orientation inference logic for DataFrame construction and warn when row orientation is inferred (#16976)
• Properly apply strict parameter in Series constructor (#16939)
• Remove supertype definition of List and non-List types (#16918)
• Consistently convert to given time zone in Series constructor (#16828)
• Update reshape to return Array types instead of List types (#16825)
• Default to raising on out-of-bounds indices in all get/gather operations (#16841)
• Native selector XOR set operation, guarantee consistent selector column-order (#16833)
• Set infer_schema_length as keyword-only argument in str.json_decode (#16835)
• Update set_sorted to only accept a single column (#16800)
• Remove deprecated parameters in Series.cut/qcut and update struct field names (#16741)
• Expedited removal of certain deprecated functionality (#16754)
• Update some error types to more appropriate variants (#15030)
• Scheduled removal of deprecated functionality (#16715)
• Change default offset in group_by_dynamic from 'negative every' to 'zero' (#16658)
• Constrain access to globals from DataFrame.sql in favor of top-level pl.sql (#16598)
• Read 2D NumPy arrays as Array type instead of List (#16710)
• Update clip to no longer propagate nulls in the given bounds (#14413)
• Change str.to_datetime to default to microsecond precision for format specifiers "%f" and "%.f" (#13597)
• Update resulting column names in pivot when pivoting by multiple values (#16439)
• Preserve nulls in ewm_mean, ewm_std, and ewm_var (#15503)
• Restrict casting for temporal data types (#14142)
• Support Decimal types by default when converting from Arrow (#15324)
• Remove serde functionality from pl.read_json and DataFrame.write_json (#16550)
• Update function signature of nth to allow positional input of indices, remove columns parameter (#16510)
• Rename struct fields of rle output to len/value and update data type of len field (#15249)
• Remove class variables from some DataTypes (#16524)
• Add check_names parameter to Series.equals and default to False (#16610)

⚠️ Deprecations

• Deprecate dt.mean/dt.median in favor of mean/median (#16888)
• Deprecate LazyFrame.with_context in favor of horizontal concatenation (#16860)
• Rename parameter descending to reverse in top_k methods (#16817)
• Rename str.concat to str.join and update default delimiter (#16790)
• Deprecate arctan2d in favor of arctan2(...).degrees() (#16786)

🚀 Performance improvements

• GC buffers in if_then_else view kernel (#16993)
• Desugar AND filter into multiple nodes (#16992)
• Optimize generic argsort of row-encoding (#16894)
• Improve rle_id iteration perf and set sorted flags (#16893)
• Optimize string/binary sort (#16871)
• Use split_at in split (#16865)
• Use split_at instead of double slice in chunk splits. (#16856)
• Don't rechunk in align_ if arrays are aligned (#16850)
• Don't create small chunks in parallel collect. (#16845)
• Add dedicated no-null branch in arg_sort (#16808)
• Speed up dt.offset_by 2x for constant durations (#16728)
• Toggle coalesce in join if non-coalesced key isn't projected (#16677)
• Make dt.truncate 1.5x faster when every is just a single duration (and not an expression) (#16666)
• Always prune unused columns in semi/anti join (#16665)

✨ Enhancements

• allow experimental metadata use on release (#17005)
• first working prototype of new streaming engine (#16970)
• Add simple version of json_normalize (#17015)
• Change data orientation inference logic for DataFrame construction and warn when row orientation is inferred (#16976)
• Desugar AND filter into multiple nodes (#16992)
• Handle textio even if not correct (#16971)
• Properly apply strict parameter in Series constructor (#16939)
• Add SQL support for INTERSECT and EXCEPT ops (#16960)
• Add PerformanceWarning to LazyFrame properties (#16964)
• Add collect_schema method to LazyFrame and DataFrame (#16929)
• Allow setting file cache TTL on a per-file basis (#16891)
• Support Decimal inputs for lit (#16950)
• Implement multiply and division for lhs duration (#16948)
• Raise on invalid temporal arithmetic (#16934)
• Always end with a in-memory sink on collect (#16928)
• add style namespace (which defers to Great Tables) (#16809)
• Add Schema class (#16873)
• Normalize value_counts (#16917)
• add eq/ne for more FixedSizeLists (#16902)
• setup skeleton (#16900)
• add fundamentals for new async-based streaming execution engine (#16884)
• Cache downloaded cloud IPC files (#16892)
• Consistently convert to given time zone in Series constructor (#16828)
• Improve read_csv SQL table reading function defaults (better handle dates) (#16866)
• Support SQL VALUES clause and inline renaming of columns in CTE & derived table definitions (#16851)
• Support Python Enum values in lit (#16858)
• convert to give time zone in .str.to_datetime when values are offset-aware (#16742)
• Update reshape to return Array types instead of List types (#16825)
• Default to raising on out-of-bounds indices in all get/gather operations (#16841)
• Support SQL "SELECT" with no tables, optimise registration of globals (#16836)
• Native selector XOR set operation, guarantee consistent selector column-order (#16833)
• Extend recognised EXTRACT and DATE_PART SQL part abbreviations (#16767)
• Improve error message when raising integers to negative integers, improve docs (#16827)
• Return datetime for mean/median of Date colum (#16795)
• Update set_sorted to only accept a single column (#16800)
• Expose overflowing cast (#16805)
• Update group_by iteration and partition_by to always return tuple keys (#16793)
• Support array arithmetic for equally sized shapes (#16791)
• Expedited removal of certain deprecated functionality (2) (#16779)
• Removal of read_database_uri passthrough from read_database (#16783)
• Remove pyxlsb engine from read_database (#16784)
• Add check_order parameter to assert_series_equal (#16778)
• Enforce deprecation of keyword arguments as positional (#16755)
• Support cloud storage in scan_csv (#16674)
• Streamline SQL INTERVAL handling and improve related error messages, update sqlparser-rs lib (#16744)
• Support use of ordinal values in SQL ORDER BY clause (#16745)
• Support executing polars SQL against pandas and pyarrow objects (#16746)
• Remove deprecated parameters in Series.cut/qcut and update struct field names (#16741)
• Expedited removal of certain deprecated functionality (#16754)
• Remove deprecated functionality from rolling methods (#16750)
• Update date_range to no longer produce datetime ranges (#16734)
• Mark min_periods as keyword-only for rolling methods (#16738)
• Remove deprecated top_k parameters nulls_last, maintain_order, and multithreaded (#16599)
• Support order-by in window functions (#16743)
• Add SQL support for NULLS FIRST/LAST ordering (#16711)
• Update some error types to more appropriate variants (#15030)
• Initial SQL support for INTERVAL strings (#16732)
• Scheduled removal of deprecated functionality (2) (#16724)
• Scheduled removal of deprecated functionality (#16715)
• Enforce deprecation of offset arg in truncate and round (#16655)
• Change default offset in group_by_dynamic from 'negative every' to 'zero' (#16658)
• Constrain access to globals from DataFrame.sql in favor of top-level pl.sql (#16598)
• Read 2D NumPy arrays as Array type instead of List (#16710)
• Update clip to no longer propagate nulls in the given bounds (#14413)
• Change str.to_datetime to default to microsecond precision for format specifiers "%f" and "%.f" (#13597)
• Update resulting column names in pivot when pivoting by multiple values (#16439)
• Preserve nulls in ewm_mean, ewm_std, and ewm_var (#15503)
• Restrict casting for temporal data types (#14142)
• Add many more auto-inferable datetime formats for str.to_datetime (#16634)
• Support Decimal types by default when converting from Arrow (#15324)
• Remove serde functionality from pl.read_json and DataFrame.write_json (#16550)
• Update function signature of nth to allow positional input of indices, remove columns parameter (#16510)
• Rename struct fields of rle output to len/value and update data type of len field (#15249)
• Remove class variables from some DataTypes (#16524)
• Add check_names parameter to Series.equals and default to False (#16610)
• Dedicated SQLInterface and SQLSyntax errors (#16635)
• Add DIV function support to the SQL interface (#16678)
• Support non-coalescing streaming left join (#16672)
• Allow wildcard and exclude before struct expansions (#16671)

🐞 Bug fixes

• properly catch not found explode cols (#17020)
• Correctly convert data frames to NumPy for C index order (#17000)
• Raise on invalid arithmetic shapes (#16986)
• Don't pushdown predicates in cross join if the refer to both tables (#16983)
• Fix projection pushdown with literal joins (#16981)
• Fix edge case in DataFrame constructor data orientation inference (#16975)
• Raise on list of objects (#16959)
• Handle strictness for Decimal Series construction (#15309)
• Don't panic in object to anyvalue (#16957)
• properly set FAST_EXPLODE_LIST metadata (#16951)
• Raise informative error when writing object to file (#16954)
• Remove supertype definition of List and non-List types (#16918)
• Remove unwrap in extend() (#16890)
• Fix should_rechunk check (#16852)
• Ensure read_excel and read_ods return identical frames across all engines when given empty spreadsheet tables (#16802)
• Consistent behaviour when "infer_schema_length=0" for read_excel (#16840)
• Standardised additional SQL interface errors (#16829)
• Ensure that splitted ChunkedArray also flattens chunks (#16837)
• Reduce needless panics in comparisons (#16831)
• Reset if next caller clones inner series (#16812)
• Raise on non-positive json schema inference (#16770)
• Rewrite implementation of top_k/bottom_k and fix a variety of bugs (#16804)
• Fix comparison of UInt64 with zero (#16799)
• Fix incorrect parquet statistics written for UInt64 values > Int64::MAX (#16766)
• Fix boolean distinct (#16765)
• DATE_PART SQL syntax/parsing, improve some error messages (#16761)
• Include pl. qualifier for inner dtypes in to_init_repr (#16235)
• Column selection wasn't applied when reading CSV with no rows (#16739)
• Panic on empty df / null List(Categorical) (#16730)
• Only flush if operator can flush in streaming outer join (#16723)
• Raise unsupported cat array (#16717)
• Assert SQLInterfaceError is raised (#16713)
• Restrict casting for temporal data types (#14142)
• Handle nested categoricals in assert_series_equal when categorical_as_str=True (#16700)
• Improve read_database check for SQLAlchemy async Session objects (#16680)
• Reduce scope of multi-threaded numpy conversion (#16686)
• Full null on dyn int (#16679)
• Fix filter shape on empty null (#16670)

📖 Documentation

• Add upgrade guide for Python Polars 1.0.0 (#16914)
• Lots of additions to the SQL reference docs (#16990)
• Minor doctest fixes (#17002)
• Include a doc entry for every exception type (#17001)
• fixup bullet points in write_parquet (#16909)
• Update version switcher for 1.0.0 prereleases (#16847)
• Update link from Python API reference to user guide (#16849)
• Update docstring/test/etc usage of select and with_columns to idiomatic form (#16801)
• Update versioning docs for 1.0.0 (#16757)
• Add docstring example for DataFrame.limit (#16753)
• Fix incorrect stated value of include_nulls in DataFrame.update docstring (#16701)
• Update deprecation docs in the user guide (#14315)
• Add example for index count in DataFrame.rolling (#16600)
• Improve docstring of Expr/Series.map_elements (#16079)
• Add missing polars.sql docs entry and small docstring update (#16656)

🛠️ Other improvements

• Fix failing file cache test in CI (#17014)
• Add some parametric tests for sort functionality (#17008)
• Pin NumPy to <2.0 for now (#16999)
• Use proper join type in test (#16994)
• Fix file cache verbose logging leakage during pytest (#16984)
• Skip another intermitently failing AWS test (#16980)
• Update test suite to explicitly use orient="row" in DataFrame constructor when applicable (#16977)
• Remove redundant projection attribute in IR::DataFrameScan (#16952)
• Factor out some apply calls in duration namespace (#16941)
• extend new streaming engine with some initial nodes (#16940)
• Skip intermittently failing AWS test (#16908)
• Refactor expression parsing utils (#16906)
• setup skeleton (#16900)
• Refactor parts of IR. (#16899)
• Move around some existing tests (#16877)
• Remove inner Arc from FileCacheEntry (#16870)
• Do not update stable API reference on prerelease (#16846)
• Update links to API references (#16843)
• Prepare update of API reference URLs (#16816)
• Rename allow_overflow to wrap_numerical (#16807)
• Set infer_schema_length as keyword-only argument in str.json_decode (#16835)
• Don't enter streaming engine for groupby-> agg mean/median … (#16810)
• Improve safety of amortized_iter (#16820)
• Remove needless inner type clone (#16718)
• Fix incorrect debug assertion in ChunkedArray::from_chunks_and_dtype (#16697)
• Update version resolver for 1.0.0 release (#16705)
• Avoid AWS pinning to outdated crc32c version (#16681)

Thank you to all our contributors for making this release possible!
@JulianCologne, @KDruzhkin, @MarcoGorelli, @Object905, @alexander-beedie, @bertiewooster, @borchero, @coastalwhite, @datenzauberai, @dependabot, @dependabot[bot], @henryharbeck, @itamarst, @machow, @marenwestermann, @mcrumiller, @montanarograziano, @nameexhaustion, @orlp, @ritchie46, @siddharth-gulia, @stinodego, @tkellogg, @universalmind303 and @wence-

💥 Breaking changes

• Consistently convert to given time zone in Series constructor (#16828)
• Update reshape to return Array types instead of List types (#16825)
• Default to raising on out-of-bounds indices in all get/gather operations (#16841)
• Native selector XOR set operation, guarantee consistent selector column-order (#16833)
• Set infer_schema_length as keyword-only argument in str.json_decode (#16835)
• Update set_sorted to only accept a single column (#16800)
• Update group_by iteration and partition_by to always return tuple keys (#16793)
• Default to coalesce=False in left outer join (#16769)
• Remove pyxlsb engine from read_database (#16784)
• Remove deprecated parameters in Series.cut/qcut and update struct field names (#16741)
• Expedited removal of certain deprecated functionality (#16754)
• Remove deprecated top_k parameters nulls_last, maintain_order, and multithreaded (#16599)
• Update some error types to more appropriate variants (#15030)
• Scheduled removal of deprecated functionality (#16715)
• Enforce deprecation of offset arg in truncate and round (#16655)
• Change default offset in group_by_dynamic from 'negative every' to 'zero' (#16658)
• Constrain access to globals from DataFrame.sql in favor of top-level pl.sql (#16598)
• Read 2D NumPy arrays as multidimensional Array instead of List (#16710)
• Update clip to no longer propagate nulls in the given bounds (#14413)
• Change str.to_datetime to default to microsecond precision for format specifiers "%f" and "%.f" (#13597)
• Update resulting column names in pivot when pivoting by multiple values (#16439)
• Preserve nulls in ewm_mean, ewm_std, and ewm_var (#15503)
• Restrict casting for temporal data types (#14142)
• Support Decimal types by default when converting from Arrow (#15324)
• Remove serde functionality from pl.read_json and DataFrame.write_json (#16550)
• Update function signature of nth to allow positional input of indices, remove columns parameter (#16510)
• Rename struct fields of rle output to len/value and update data type of len field (#15249)
• Remove class variables from some DataTypes (#16524)
• Add check_names parameter to Series.equals and default to False (#16610)

⚠️ Deprecations

• Deprecate LazyFrame.with_context (#16860)
• Rename parameter descending to reverse in top_k methods (#16817)
• Rename str.concat to str.join (#16790)
• Deprecate arctan2d (#16786)

🚀 Performance improvements

• Optimize string/binary sort (#16871)
• Use split_at in split (#16865)
• Use split_at instead of double slice in chunk splits. (#16856)
• Don't rechunk in align_ if arrays are aligned (#16850)
• Don't create small chunks in parallel collect. (#16845)
• Add dedicated no-null branch in arg_sort (#16808)
• Speed up dt.offset_by 2x for constant durations (#16728)
• Toggle coalesce if non-coalesced key isn't projected (#16677)
• Make dt.truncate 1.5x faster when every is just a single duration (and not an expression) (#16666)
• Always prune unused columns in semi/anti join (#16665)

✨ Enhancements

• Consistently convert to given time zone in Series constructor (#16828)
• Improve read_csv SQL table reading function defaults (better handle dates) (#16866)
• Support SQL VALUES clause and inline renaming of columns in CTE & derived table definitions (#16851)
• Support Python Enum values in lit (#16858)
• convert to give time zone in .str.to_datetime when values are offset-aware (#16742)
• Update reshape to return Array types instead of List types (#16825)
• Default to raising for oob on all get/gather operations (#16841)
• Support SQL "SELECT" with no tables, optimise registration of globals (#16836)
• Native selector XOR set operation, guarantee consistent selector column-order (#16833)
• Extend recognised EXTRACT and DATE_PART SQL part abbreviations (#16767)
• Improve error message when raising integers to negative integers, improve docs (#16827)
• Return datetime for mean/median of Date colum (#16795)
• Only accept a single column in set_sorted (#16800)
• Expose overflowing cast (#16805)
• Update group-by iteration to always return tuple keys (#16793)
• Support array arithmetic for equally sized shapes (#16791)
• Default to coalesce=False in left outer join (#16769)
• More removal of deprecated functionality (#16779)
• Removal of read_database_uri passthrough from read_database (#16783)
• Remove pyxlsb engine from read_database (#16784)
• Add check_order parameter to assert_series_equal (#16778)
• Enforce deprecation of keyword arguments as positional (#16755)
• Support cloud storage in scan_csv (#16674)
• Streamline SQL INTERVAL handling and improve related error messages, update sqlparser-rs lib (#16744)
• Support use of ordinal values in SQL ORDER BY clause (#16745)
• Support executing polars SQL against pandas and pyarrow objects (#16746)
• Remove deprecated parameters in Series.cut/qcut (#16741)
• Expedited removal of certain deprecated functionality (#16754)
• Remove deprecated functionality from rolling methods (#16750)
• Update date_range to no longer produce datetime ranges (#16734)
• Mark min_periods as keyword-only for rolling methods (#16738)
• Remove deprecated top_k parameters (#16599)
• Support order-by in window functions (#16743)
• Add SQL support for NULLS FIRST/LAST ordering (#16711)
• Update some error types to more appropriate variants (#15030)
• Initial SQL support for INTERVAL strings (#16732)
• More scheduled removal of deprecated functionality (#16724)
• Scheduled removal of deprecated functionality (#16715)
• Enforce deprecation of offset arg in truncate and round (#16655)
• Change default of offset in group_by_dynamic from "negative every" to "zero" (#16658)
• Constrain access to globals from df.sql in favour of top-level pl.sql (#16598)
• Read 2D numpy arrays as Array[dt, shape] instead of Listst[dt] (#16710)
• Activate decimal by default (#16709)
• Do not propagate nulls in clip bounds (#14413)
• Change .str.to_datetime to default to microsecond precision for format specifiers "%f" and "%.f" (#13597)
• Remove redundant column name when pivoting by multiple values (#16439)
• Preserve nulls in ewm_mean, ewm_std, and ewm_var (#15503)
• Restrict casting for temporal data types (#14142)
• Add many more auto-inferable datetime formats for str.to_datetime (#16634)
• Support decimals by default when converting from Arrow (#15324)
• Remove serde functionality from pl.read_json and DataFrame.write_json (#16550)
• Update function signature of nth to allow positional input of indices, remove columns parameter (#16510)
• Rename struct fields of rle output to len/value and update data type of len field (#15249)
• Remove default class variable values on DataTypes (#16524)
• Add check_names parameter to Series.equals and default to False (#16610)
• Dedicated SQLInterface and SQLSyntax errors (#16635)
• Add DIV function support to the SQL interface (#16678)
• Support non-coalescing streaming left join (#16672)
• Allow wildcard and exclude before struct expansions (#16671)

🐞 Bug fixes

• Fix should_rechunk check (#16852)
• Ensure read_excel and read_ods return identical frames across all engines when given empty spreadsheet tables (#16802)
• Consistent behaviour when "infer_schema_length=0" for read_excel (#16840)
• Standardised additional SQL interface errors (#16829)
• Ensure that splitted ChunkedArray also flattens chunks (#16837)
• Reduce needless panics in comparisons (#16831)
• Reset if next caller clones inner series (#16812)
• Raise on non-positive json schema inference (#16770)
• Rewrite implementation of top_k/bottom_k and fix a variety of bugs (#16804)
• Fix comparison of UInt64 with zero (#16799)
• Fix incorrect parquet statistics written for UInt64 values > Int64::MAX (#16766)
• Fix boolean distinct (#16765)
• DATE_PART SQL syntax/parsing, improve some error messages (#16761)
• Include pl. qualifier for inner dtypes in to_init_repr (#16235)
• Column selection wasn't applied when reading CSV with no rows (#16739)
• Panic on empty df / null List(Categorical) (#16730)
• Only flush if operator can flush in streaming outer join (#16723)
• Raise unsupported cat array (#16717)
• Assert SQLInterfaceError is raised (#16713)
• Restrict casting for temporal data types (#14142)
• Handle nested categoricals in assert_series_equal when categorical_as_str=True (#16700)
• Improve read_database check for SQLAlchemy async Session objects (#16680)
• Reduce scope of multi-threaded numpy conversion (#16686)
• Full null on dyn int (#16679)
• Fix filter shape on empty null (#16670)

📖 Documentation

• Update version switcher for 1.0.0 prereleases (#16847)
• Update link from Python API reference to user guide (#16849)
• Update docstring/test/etc usage of select and with_columns to idiomatic form (#16801)
• Update versioning docs for 1.0.0 (#16757)
• Add docstring example for DataFrame.limit (#16753)
• Fix incorrect stated value of include_nulls in DataFrame.update docstring (#16701)
• Update deprecation docs in the user guide (#14315)
• Add example for index count in DataFrame.rolling (#16600)
• Improve docstring of Expr/Series.map_elements (#16079)
• Add missing polars.sql docs entry and small docstring update (#16656)

🛠️ Other improvements

• Remove inner Arc from FileCacheEntry (#16870)
• Do not update stable API reference on prerelease (#16846)
• Update links to API references (#16843)
• Prepare update of API reference URLs (#16816)
• Rename allow_overflow to wrap_numerical (#16807)
• Set infer_schema_length as keyword-only for str.json_decode (#16835)
• Don't enter streaming engine for groupby-> agg mean/median … (#16810)
• Improve safety of amortized_iter (#16820)
• Remove needless inner type clone (#16718)
• Fix incorrect debug assertion in ChunkedArray::from_chunks_and_dtype (#16697)
• Update version resolver for 1.0.0 release (#16705)
• Avoid AWS pinning to outdated crc32c version (#16681)

Thank you to all our contributors for making this release possible!
@JulianCologne, @KDruzhkin, @MarcoGorelli, @Object905, @alexander-beedie, @bertiewooster, @coastalwhite, @datenzauberai, @dependabot, @dependabot[bot], @henryharbeck, @marenwestermann, @mcrumiller, @montanarograziano, @nameexhaustion, @orlp, @ritchie46, @siddharth-gulia, @stinodego, @universalmind303 and @wence-

[!IMPORTANT]
The decision to change the default coalesce behavior of left join has been reversed.
You can ignore the associated deprecation warning.

⚠️ Deprecations

• Rename dtypes parameter to schema_overrides for read_csv/scan_csv/read_csv_batched (#16628)
• Deprecate nulls_last/maintain_order/multithreaded parameters for top_k methods (#16597)
• Rename SQLContext "eager_execution" param to "eager" (#16595)
• Rename Series.equals parameter strict to check_dtypes and rename assertion utils parameter check_dtype to check_dtypes (#16573)
• Add DataFrame.serialize/deserialize (#16545)
• Deprecate str.explode in favor of str.split("").explode() (#16508)
• Deprecate default coalesce behavior of left join (#16532) - !! Reversed in 1.0.0 - see message above !!

🚀 Performance improvements

• make truncate 4x faster in simple cases (#16615)
• Cache arena's (and conversion) in SQL context (#16566)
• Partial schema cache. (#16549)

✨ Enhancements

• Support per-column nulls_last on sort operations (#16639)
• Initial support for SQL ARRAY literals and the UNNEST table function (#16330)
• Don't allow struct.with_fields in grouping (#16629)
• improve support for user-defined functions that return scalars (#16556)
• Add SQL support for TRY_CAST function (#16589)
• Add top-level pl.sql function (#16528)
• Expose temporal function expression ops to expr ir (#16546)
• Add DataFrame.serialize/deserialize (#16545)
• check if by column is sorted, rather than just checking sorted flag, in group_by_dynamic, upsample, and rolling (#16494)

🐞 Bug fixes

• Potentially deal with empty range (#16650)
• Use of SQL ORDER BY should not cause reordering of SELECT cols (#16579)
• ensure df in empty parquet (#16621)
• Fix Array constructor when inner type is another Array (#16622)
• Fix parsing of shape in Array constructor and deprecate width parameter (#16567)
• Crash using empty Series in LazyFrame.select() (#16592)
• improve support for user-defined functions that return scalars (#16556)
• Resolve multiple SQL JOIN issues (#16507)
• Project last column if count query (#16569)
• Properly split struct columns (#16563)
• Ensure strict chunking in chunked partitioned group by (#16561)
• Error selecting columns after non-coalesced join (multiple join keys) (#16559)
• Don't panic on hashing nested list types (#16555)
• Crash selecting columns after non-coalesced join (#16541)
• Fix group gather of single literal (#16539)
• throw an invalid operation exception on performing a sum over a list of strs (#16521)
• Fix DataFrame.__getitem__ for empty list input - df[[]] (#16520)
• Fix issue in DataFrame.__getitem__ with 2 column inputs (#16517)

📖 Documentation

• Overview of available SQL functions (#16268)
• Update filter description to clarify that null evaluations are removed (#16632)
• Include warning in docstrings that accessing LazyFrame properties may be expensive (#16618)
• Add a few versionadded tags, and add is_column_selection to the Expr meta docs (#16590)
• Fix bullet points not rendering correctly in DataFrame.join docstring (#16576)
• Remove erroneous implode reference from the user guide section on window functions (#16544)

📦 Build system

• Run cargo update (#16574)

🛠️ Other improvements

• Add test for 16642 (#16646)
• Remove duplicate tag in CODEOWNERS (#16625)
• Update dprint hook versions and enable JSON linting (#16611)
• Fewer typing.no_type_check (#16497)

Thank you to all our contributors for making this release possible!
@MarcoGorelli, @alexander-beedie, @coastalwhite, @hattajr, @itamarst, @mcrumiller, @nameexhaustion, @r-brink, @ritchie46, @stinodego, @twoertwein and @wence-

⚠️ Deprecations

• Add Series/Expr.has_nulls and deprecate Series.has_validity (#16488)
• Deprecate tree_format parameter for LazyFrame.explain in favor of format (#16486)

🚀 Performance improvements

• improved numeric fill_(forward/backward) (#16475)
• only rechunk once per aggregate (#16469)

✨ Enhancements

• Minor DataFrame.__getitem__ improvements (#16495)
• Add is_column_selection() to expression meta, enhance expand_selector (#16479)
• Add Series/Expr.has_nulls and deprecate Series.has_validity (#16488)
• NDarray/Tensor support (#16466)

🐞 Bug fixes

• Fix df.chunked for struct (#16504)
• Mix of column and field expansion (#16502)
• Fix split_chunks for nested dtypes (#16493)
• Fix handling NaT values when creating Series from NumPy ndarray (#16490)
• Fix boolean trap issue in top_k/bottom_k (#16489)
• Handle struct.fields as special case of alias (#16484)
• Correct schema for list.sum (#16483)
• allow search_sorted directly on multiple chunks, and fix behavior around nulls (#16447)
• Fix use of COUNT(*) in SQL GROUP BY operations (#16465)
• respect nan_to_null when using multi-thread in pl.from_pandas (#16459)
• write_delta() apparently does support Categorical columns (#16454)

📖 Documentation

• Update the Overview section of the contributing guide (#15674)
• Use pl.field inside with_fields examples. (#16451)
• Change ordering of values in example for cum_max (#16456)

🛠️ Other improvements

• Refactor Series/DataFrame.__getitem__ logic (#16482)

Thank you to all our contributors for making this release possible!
@BGR360, @alexander-beedie, @cmdlineluser, @coastalwhite, @itamarst, @marenwestermann, @mdavis-xyz, @messense, @orlp, @ritchie46 and @stinodego

⚠️ Deprecations

• Deprecate how="outer" join type in favour of how="full" (left/right are *also* outer joins) (#16417)

🚀 Performance improvements

• Fix pathological small chunk parquet writing (#16433)

✨ Enhancements

• Support zero-copy conversion for temporal types in DataFrame.to_numpy (#16429)
• Allow designation of a custom name for the value_counts "count" column (#16434)
• Default rechunk=False for read_parquet (#16427)
• Add "ignore_spaces" to alpha and alphanumeric selectors, add "ascii_only" to digit (#16362)
• Update __array__ method for Series and DataFrame to support copy parameter (#16401)

🐞 Bug fixes

• add cluster_with_columns optimization toggle in python (#16446)
• Fix struct 'with_fields' schema for update dtypes (#16428)
• Fix error reading lists of CSV files that contain comments (#16426)
• make read_parquet() respect rechunk flag when using pyarrow (#16418)
• Improve read_excel dtype inference of "calamine" int/float results that include NaN (#16400)
• Update apply call in str_duration_ util. (#16412)

📖 Documentation

• Add interpolate_by entry to rst files. (#16422)
• Fix sphinx headers: 'Example' → 'Examples' (#16407)

Thank you to all our contributors for making this release possible!
@KDruzhkin, @alexander-beedie, @ankane, @cmdlineluser, @coastalwhite, @itamarst, @nameexhaustion, @ritchie46 and @stinodego