polars

Dataframes powered by a multithreaded, vectorized query engine, written in Rust

OTHER License

Downloads
9.7M
Stars
26.3K
Committers
213
View Code on GitHub Visit Website View on X
Ecosystems: Rust, Python

Bot releases are hidden (Show)

polars - Rust Polars 0.40.0

Published by ritchie46 5 months ago

πŸ’₯ Breaking changes

  • Remove incremental read based batched CSV reader (#16259)
  • separate rolling_*_by from rolling_*(..., by=...) in Rust (#16102)
  • Move CSV read options from CsvReader to CsvReadOptions (#16126)
  • Rename all 'Chunk's to RecordBatch (#16063)
  • prepare for join coalescing argument (#15418)
  • Rename to CsvParserOptions to CsvReaderOptions, use in CsvReader (#15919)
  • Add context trace to LazyFrame conversion errors (#15761)
  • Move schema resolving of file scan to IR phase (#15739)
  • Move schema resolving to IR phase. (#15714)
  • Rename LogicalPlan and builders to reflect their uses better (#15712)

πŸš€ Performance improvements

  • Use branchless uleb128 decoding for parquet (#16352)
  • Reduce error bubbling in parquet hybrid_rle (#16348)
  • use is_sorted in ewm_mean_by, deprecate check_sorted (#16335)
  • Optimize is_sorted for numeric data (#16333)
  • do not use pyo3-built (#16309)
  • Faster bitpacking for Parquet writer (#16278)
  • Avoid importing ctypes.util in CPU check script if possible (#16307)
  • Don't rechunk when converting DataFrame to numpy/ndarray (#16288)
  • use zeroed vec in ewm_mean_by for sorted fastpath (#16265)
  • use zeroable_vec in ewm_mean_by (#16166)
  • Improve cost of chunk_idx compute (#16154)
  • Don't rechunk by default in concat (#16128)
  • Ensure rechunk is parallel (#16127)
  • Don't traverse deep datasets that we repr as union in CSE (#16096)
  • Ensure better chunk sizes (#16071)
  • Don't rechunk in parallel collection (#15907)
  • Improve non-trivial list aggregations (#15888)
  • Ensure we hit specialized gather for binary/strings (#15886)
  • Limit the cache size for to_datetime (#15826)
  • skip initial null items and don't recompute slope in interpolate (#15819)
  • Fix quadratic in binview growable same source (#15734)

✨ Enhancements

  • Raise when joining on the same keys twice (#16329)
  • Don't require data to be sorted by by column in rolling_*_by operations (#16249)
  • Add struct.field expansion (regex, wildcard, columns) (#16320)
  • Faster bitpacking for Parquet writer (#16278)
  • Add struct.with_fields (#16305)
  • Handle implicit SQL string β†’ temporal conversion in the BETWEEN clause (#16279)
  • Add new index/range based selector cs.by_index, allow multiple indices for nth (#16217)
  • Show warning if expressions are very deep (#16233)
  • Native CSV file list reading (#16180)
  • Register memory mapped files and raise when written to (#16208)
  • Raise when encountering invalid supertype in functions during conversion (#16182)
  • Add SQL support for GROUP BY ALL syntax and fix several issues with aliased group keys (#16179)
  • Allow implicit string β†’ temporal conversion in SQL comparisons (#15958)
  • separate rolling_*_by from rolling_*(..., by=...) in Rust (#16102)
  • Add run-length encoding to Parquet writer (#16125)
  • add date pattern dd.mm.YYYY (#16045)
  • Add RLE to RLE_DICTIONARY encoder (#15959)
  • Support non-coalescing joins in default engine (#16036)
  • Move diagonal & horizontal concat schema resolving to IR phase (#16034)
  • raise more informative error messages in rolling_* aggregations instead of panicking (#15979)
  • Convert concat during IR conversion (#16016)
  • Improve dynamic supertypes (#16009)
  • Additional uint datatype support for the SQL interface (#15993)
  • Support Decimal read from IPC (#15965)
  • Add typed collection from par iterators (#15961)
  • Add by argument for Expr.top_k and Expr.bottom_k (#15468)
  • Add option to disable globbing in csv (#15930)
  • Add option to disable globbing in parquet (#15928)
  • Rename to CsvParserOptions to CsvReaderOptions, use in CsvReader (#15919)
  • Expressify dt.round (#15861)
  • Improve error messages in context stack (#15881)
  • Add dynamic literals to ensure schema correctness (#15832)
  • dt.truncate supports broadcasting lhs (#15768)
  • Expressify str.json_path_match (#15764)
  • Support decimal float parsing in CSV (#15774)
  • Add context trace to LazyFrame conversion errors (#15761)

🐞 Bug fixes

  • correct AExpr.to_field for bitwise and logical and/or (#16360)
  • cargo clippy for uleb128 safety comment (#16368)
  • Infer CSV schema as supertype of all files (#16349)
  • Address overflow combining u64 hashes in Debug builds (#16323)
  • Don't exclude explicitly named columns in group-by context' expr expansion (#16318)
  • Harden Series.reshape against invalid parameters (#16281)
  • Fix list.sum dtype for boolean (#16290)
  • Don't stackoverflow on all/any horizontal (#16287)
  • compilation error when both lazy and ipc features are enabled (#16284)
  • `rolling_*_by was throwing incorrect error when dataframe was sorted by contained multiple chunks (#16247)
  • Clippy Error for CPUID (#16241)
  • Reading CSV with low_memory gave no data (#16231)
  • Empty unique (#16214)
  • Fix empty drop nulls (#16213)
  • Fix get expression group-by state (#16189)
  • Fix rolling empty group OOB (#16186)
  • offset=-0i was being treated differently to offset=0i in rolling (#16184)
  • Fix panic on empty frame joins (#16181)
  • Fix streaming glob slice (#16174)
  • Fix CSV skip_rows_after_header for streaming (#16176)
  • Flush parquet at end of batches tick (#16073)
  • Check CSE name aliases for collisions. (#16149)
  • Don't override CSV reader encoding with lossy UTF-8 (#16151)
  • Add missing allow macros for windows (#16130)
  • Ensure hex and bitstring literals work inside SQL IN clauses (#16101)
  • Revert "Add RLE to RLE_DICTIONARY encoder" (#16113)
  • Respect user passed 'reader_schema' in 'scan_csv' (#16080)
  • Lazy csv + projection; respect null values arg (#16077)
  • Materialize dtypes when converting to arrow (#16074)
  • Fix casting decimal to decimal for high precision (#16049)
  • Fix printing max scale decimals (#16048)
  • Decimal supertype for dyn int (#16046)
  • Do not set sorted flag on lexical sorting (#16032)
  • properly handle nulls in DictionaryArray::iter_typed (#16013)
  • Fix CSE case where upper plan has no projection (#16011)
  • Crash/incorrect group_by/n_unique on categoricals created by (q)cut (#16006)
  • Ternary supertype dynamics (#15995)
  • Treat splitting by empty string as iterating over chars (#15922)
  • Fix PartialEq for DataType::Unknown (#15992)
  • Do not reverse null indices in descending arg_sort (#15974)
  • Finish adding typed_lit to help schema determination in SQL "extract" func (#15955)
  • do not panic when comparing against categorical with incompatible dtype (#15857)
  • Join validation for multiple keys (#15947)
  • Set default limit for String column display to 30 and fix edge cases (#15934)
  • typo in add_half_life takes ln(negative) (#15932)
  • Remove ffspec from parquet reader (#15927)
  • avoid WRITE+EXEC for CPUID check (#15912)
  • fix inconsistent decimal formatting (#15457)
  • Preserve NULLs for is_not_nan (#15889)
  • double projection check should only take the upstream projections into account (#15901)
  • Ensure we don't create invalid frames when combining unit lit + … (#15903)
  • Clear cached rename schema (#15902)
  • Fix OOB in struct lit/agg aggregation (#15891)
  • create (q)cut labels in fixed order (#15843)
  • Tag shrink_dtype as non-streaming (#15828)
  • drop-nulls edge case; remove drop-nulls special case (#15815)
  • ewm_mean_by was skipping initial nulls when it was already sorted by "by" column (#15812)
  • Consult cgroups to determine free memory (#15798)
  • raise if index count like 2i is used when performing rolling, group_by_dynamic, upsample, or other temporal operatios (#15751)
  • Don't deduplicate sort that has slice pushdown (#15784)
  • Fix incorrect is_between pushdown to scan_pyarrow_dataset (#15769)
  • Handle null index correctly for list take (#15737)
  • Preserve lexical ordering on concat (#15753)
  • Remove incorrect unsafe pointer cast for int -> enum (#15740)
  • pass series name to apply for cut/qcut (#15715)
  • count of null column shouldn't panic in agg context (#15710)

πŸ“– Documentation

  • Clarify arrow usage (#16152)
  • Solve inconsistency between code and comment (#16135)
  • add filter docstring examples to date and datetime (#15996)
  • update the link to R API docs (#15973)
  • Fix a typo in categorical section of the user guide (#15777)
  • Fix incorrect column name in LazyFrame.sort doc example (#15658)

πŸ“¦ Build system

  • Update Rust nightly toolchain version (#16222)
  • Don't import jemalloc (#15942)
  • Use default allocator for lts-cpu (#15941)
  • replace all macos-latest referrals with macos-13 (#15926)
  • pin mimalloc and macos-13 (#15925)
  • use jemalloc in lts-cpu (#15913)

πŸ› οΈ Other improvements

  • simplify interpolate code, add test for rolling_*_by with nulls (#16334)
  • Move expression expansion to conversion module (#16331)
  • Add polars-expr README (#16316)
  • Move physical expressions to new crate (#16306)
  • Use cls (not self) in classmethods (#16303)
  • conditionally print the CSEs (#16292)
  • Rename ChunkedArray.chunk_id to chunk_lengths (#16273)
  • Use Scalar instead of Series some aggregations (#16277)
  • Use CsvReadOptions in LazyCsvReader (#16283)
  • Do not hardcode bash path in Makefile (#16263)
  • Add IR::Reduce (not yet implemented) (#16216)
  • Remove incremental read based batched CSV reader (#16259)
  • move all describe, describe_tree and dot-viz code to IR instead of DslPlan (#16237)
  • move describe to IR instead of DSL (#16191)
  • Use Duration.is_zero instead of comparing Duration.duration_ns to 0 (#16195)
  • Remove unused code (#16175)
  • Don't override CSV reader encoding with lossy UTF-8 (#16151)
  • Move CSV read options from CsvReader to CsvReadOptions (#16126)
  • Bump sccache action (#16088)
  • Fix failures in test coverage workflow (#16083)
  • Rename all 'Chunk's to RecordBatch (#16063)
  • Use UnionArgs for DSL side (#16017)
  • Add some comments (#16008)
  • prepare for join coalescing argument (#15418)
  • Pin coverage job to MacOS 13 for now (#15918)
  • Reorganize from_iter and dispatch to collect_ca when possible (#15904)
  • More polars-io cleanup (#15885)
  • Improve type-coercion (#15879)
  • Move type coercion to IR conversion phase (#15868)
  • Reorganize polars_io::parquet module (#15860)
  • Reorganize polars_io::csv module (#15831)
  • Always expand horizontal_any/all (#15816)
  • Rename decimal_float to decimal_comma (#15817)
  • Move IO-related options structs to polars-io (#15806)
  • Split coverage calculation (#15780)
  • Update readme (#15787)
  • Move schema resolving of file scan to IR phase (#15739)
  • Factor out ensure_is_constant_duration (#15733)
  • Move schema resolving to IR phase. (#15714)
  • Rename LogicalPlan and builders to reflect their uses better (#15712)

Thank you to all our contributors for making this release possible!
@CanglongCl, @JulianCologne, @KDruzhkin, @MarcoGorelli, @NedJWestern, @NexVeridian, @NickCondron, @Robinsane, @ShivMunagala, @TobiasDummschat, @YichiZhang0613, @alexander-beedie, @avimallu, @bertiewooster, @brandon-b-miller, @c-peters, @coastalwhite, @dangotbanned, @datenzauberai, @deanm0000, @dependabot, @dependabot[bot], @eitsupi, @gasmith, @haocheng6, @ion-elgreco, @itamarst, @janpipek, @jr200, @jrycw, @jsarbach, @luke396, @marenwestermann, @max-muoto, @mbuhidar, @nameexhaustion, @orlp, @pydanny, @r-brink, @reswqa, @ritchie46, @stinodego, @thalassemia, @tharunsuresh-code, @twoertwein, @wence- and @wsyxbcl

polars - Python Polars 0.19.14

Published by ritchie46 11 months ago

πŸ† Highlights

  • Support Python 3.12 (#12094)
  • make 1D numpy to polars conversion zero-copy for numeric data (#12403)

⚠️ Deprecations

  • Rename DataFrame column index methods (#12542)
  • Rename Series.set_at_idx to scatter (#12540)
  • Deprecate Series.view (#12539)
  • Rename cumulative functions cumsum -> cum_sum and similar (#12513)
  • Rename take to gather (#12528)
  • Add dedicated horizontal aggregation methods to DataFrame (#12492)
  • Rename take_every to gather_every (#12531)
  • Deprecate Series.inner_dtype property (#12494)
  • Deprecate parse_int in favor of to_integer (#12464)
  • Deprecate DataType method is_not (#12458)
  • Deprecate Series methods is_boolean and is_utf8 (#12457)
  • Add DataType.is_integer and other dtype groups (#12200)

πŸš€ Performance improvements

  • speed up parquet download of streaming engine (#12544)
  • speed up cov/corr with SIMD + strength-reduction ~3x 0.19.13/ ~2x numpy (#12471)
  • apply predicates and statistics of parquet files in streaming mode (#12439)
  • use online algorithm for cov/corr ~2x (#12412)
  • make 1D numpy to polars conversion zero-copy for numeric data (#12403)

✨ Enhancements

  • Add dedicated horizontal aggregation methods to DataFrame (#12492)
  • support http scan_parquet (#12517)
  • Add support for UTF-8 BOM option in write_csv and sink_csv (#12253)
  • remove lexical (replace with atoi_simd, ryu, and itao). (#12512)
  • more changes for versioned plugins (#12504)
  • plugins add version and context (#12433)
  • Add DataType.is_integer and other dtype groups (#12200)
  • include i128 in more primitive functions (#12413)
  • write rolling functions as private expressions. (#12379)

🐞 Bug fixes

  • fix incorrect ternary agg states (#12538)
  • fix and improve ternary evaluation on groups (#12529)
  • saturating sub in debug msg (#12525)
  • fix panic when writing Decimal type to parquet (#12532)
  • pre-fefetch struct columns in async projection pd (#12514)
  • rechunk cross join output in streaming (#12511)
  • Ensure behaviour ofSeries comparison with timedelta matches that of other types (#12497)
  • fix as_list logical types (#12507)
  • fix streaming cross join on empty df (#12491)
  • dont overflow when calculating date range over very long periods (#12479)
  • Allow append/zip_with/extend on local categoricals (#12369)
  • Do not panic if time is invalid (#12466)
  • ensure explicit "return_dtype" is respected by map_dicts (#12436)
  • empty csv no-raise (#12434)
  • Fix scan_csv error type (#12355)
  • binary operations in aggregation context on literals (#12430)
  • raw HTML output alignment was incorrect for dtype in header (#12422)
  • update groups state after binary aggregation (#12415)
  • Remove extra \n when reading file-like object wi… (#12333)
  • Issue correct PolarsInefficientMapWarning for lshift/rshift operations (#12385)
  • revert ternary special broadcast, ensure broadcast is always to max height (#12395)
  • ensure first/last return null if empty (#12401)

πŸ› οΈ Other improvements

  • fix and improve ternary evaluation on groups (#12529)
  • Add polars-ds to list of community plugins (#12527)
  • Future-proof consortium standard test (#12524)
  • add schema test (#12523)
  • remove lexical (replace with atoi_simd, ryu, and itao). (#12512)
  • add test for previous commit (#12510)
  • Update polars-hash reference (#12505)
  • Add note on hash stability and mention polars-hash (#12496)
  • Support Python 3.12 (#12094)
  • Improved import polars timing test; now much more consistent/reliable (#12478)
  • Use .with_columns() in all .list namespace examples (#12475)
  • update rustc (#12468)
  • Fix docs trigger (#12449)
  • Update for new maturin release (#12437)
  • Remove 'experimental' tag for auto-structify setting (#12435)
  • make "DataFrame" and "Series" case more consistent across docs/comments/errors (#12428)
  • dprint/markdown link checker minor updates (#12409)
  • Use manylinux_2_17 for building x86-64 wheel (#12408)
  • Use manylinux 2.24 instead of 2.28 for compatibility reasons (#12397)
  • use with_columns in is_in example, and fix some bullet points not rendering (#12383)

Thank you to all our contributors for making this release possible!
@MarcoGorelli, @abstractqqq, @alexander-beedie, @c-peters, @cmdlineluser, @hirohira9119, @ion-elgreco, @jerome3o, @nameexhaustion, @reswqa, @ritchie46, @stinodego and @uchiiii

polars - Python Polars 0.17.5

Published by stinodego 12 months ago

πŸš€ Performance improvements

  • use online variance kernel for aggregation (#8306)

Thank you to all our contributors for making this release possible!
@ritchie46

polars - Python Polars 0.18.2

Published by ritchie46 over 1 year ago

πŸš€ Performance improvements

  • increase streaming groupby spill size from 256 to 10_000 (#9312)
  • perf(rust, python) Improve rolling min and max for nonulls (#9277)

✨ Enhancements

  • allow use of StringCache object as a function decorator (#9309)
  • allow use of Config object as a function decorator (#9307)
  • serde for 'to_physical' expr (#9294)

🐞 Bug fixes

  • fix rolling weighted mean (#9292)
  • fix overly-broad string matching in selectors (#9303)
  • fix when loading model data from upcoming pydantic 2.x release (#9296)

πŸ› οΈ Other improvements

  • fix extraneous indent in examples block (#9297)
  • Fix typo in Selectors documentation (#9295)

Thank you to all our contributors for making this release possible!
@alexander-beedie, @magarick, @ritchie46, @stinodego and @thomascamminady

polars - Python Polars 0.15.7

Published by ritchie46 almost 2 years ago

πŸš€ Performance improvements

  • impove performance reducing window functions with numeric output ~-14% (#5841)

✨ Enhancements

  • allow more pyarrow literals (#5842)
  • add sink_parquet (#5480)
  • release GIL when writing (#5830)
  • Support parsing more float string representations. (#5824)
  • implement mean aggregation for duration (#5807)
  • implement sensible boolean aggregates (#5806)

🐞 Bug fixes

  • correct invalid type in struct anyvalue access (#5844)
  • don't set fast_explode if null values in list (#5838)
  • duration formatting (#5837)
  • respect fetch in union (#5836)
  • keep f32 dtype in fill_null by int (#5834)
  • fix(python): fix delta issues (#5802)
  • err on epoch on time dtype (#5831)
  • fix panic in hmean (#5808)
  • asof join by logical groups (#5805)

πŸ› οΈ Other improvements

  • lazily import connectorx (#5835)

Thank you to all our contributors for making this release possible!
@chitralverma, @ghuls and @ritchie46

polars - Python Polars 0.14.31

Published by ritchie46 almost 2 years ago

πŸš€ Performance improvements

  • improve streaming primitve groupby (#5575)
  • vectorize integer vec-hash by using very simple, … (#5572)

✨ Enhancements

  • prefer streaming groupby if partitionable (#5580)

🐞 Bug fixes

  • fix ub due to invalid dtype on splitting dfs (#5579)

πŸ› οΈ Other improvements

  • Remove old Python changelog file (#5577)
  • namespace registration docs update (#5565)
  • Improve contributing guide (#5558)

Thank you to all our contributors for making this release possible!
@alexander-beedie, @ghuls, @ritchie46 and @stinodego

polars - Python Polars 0.14.15

Published by stinodego about 2 years ago

polars - Rust Polars 0.24.3

Published by stinodego about 2 years ago

polars - Rust Polars 0.24.0

Published by ritchie46 about 2 years ago

New rust polars release! πŸš€

This is the release of rust polars 0.24.0. This release comes with a lot of bug fixes, performance improvements and added functionality. The changes that stand out are larger than RAM memory mapping of IPC files and a new common-subplan-optimization that prunes duplicated sub-plan from the query plan and thereby potentially save a lot of duplicated work.

See more

Update to arrow2 0.14.0

See the 0.14.0 release for all upstream improvements.

New Contributors

Full Changelog: https://github.com/pola-rs/polars/compare/rust-polars-v0.23.0...rust-polars-v0.24.0

polars - Rust polars 0.23.0

Published by ritchie46 about 2 years ago

What's Changed

New Contributors

Full Changelog: https://github.com/pola-rs/polars/compare/rust-polars-v0.22.1...rust-polars-v0.23.0

polars - Rust polars 0.22.1

Published by ritchie46 over 2 years ago

What's Changed

New Contributors

Full Changelog: https://github.com/pola-rs/polars/compare/rust-polars-v0.21.1...rust-polars-v0.22.1

polars - Rust polars 0.21.1

Published by ritchie46 over 2 years ago

What's Changed

New Contributors

Full Changelog*: https://github.com/pola-rs/polars/compare/rust-polars-v0.20.0...rust-polars-v0.21.

polars - Rust polars 0.20.0

Published by ritchie46 over 2 years ago

New rust polars release! πŸš€

This release of 286 commits is here thanks to the contributions of: (in no specific order):

  • @moritzwilksch
  • @JakobGM
  • @illumination-k
  • @tamasfe
  • @ghuls
  • @alexander-beedie
  • @Maxyme
  • @universalmind303
  • @qiemem
  • @glennpierce
  • @nmandery
  • @ilsley
  • @marcvanheerden

did I forget your contribution, please ping me, I do this manually πŸ™ˆ

Most notable changes are:

  • Many bug fixes.
  • Many performance improvements.

features

  • Made representation of groups tuples more cache friendly #2431

  • Remove Seek requirement of readers

  • Add groupby_rolling as new entrance to expression API.

  • Improve CSV parsers stability and performance on several occasions

  • Horizontal aggregations are parallelized #2454

  • Reduce pivot code bloat and improve performance #2458

  • Struct data type added.

  • Extend methods that allow modification of the same memory if Arc::ref_count == 1

  • Avro readers and writers.

  • Improved rules of window expressions.

  • Support for us time unit.

  • Parquet use statistics in query optimizations.

  • Optimize projections in lazy computations. (Mostly useful when you deal with a large number of columns e.g. millions).

  • Improve performance and flexibility of melt operation @2799

  • new expressions

    • str.split
    • str.split_inclusive
    • arr.join
    • unique_stable
    • str.split_exact
    • count expression that does not require column names
    • arr.arg_min
    • arr.arg_max
    • arr.diff
    • arr.shift

Update to arrow2 0.10.0

See the 0.10.0 release for all upstream improvements.

Package Rankings
Top 0.54% on Pypi.org
Top 2.11% on Crates.io
Top 12.72% on Conda-forge.org
Top 18.03% on Npmjs.org
Related Projects
qsv

CSVs sliced, diced & analyzed.

11 Dec 2020 2,251
ohmyfpg

Oh My Fast Postgres!

26 Jun 2022 11
rust-python-example

Example of using Rust to Extend Python

11 Nov 2017 784
rpds

Rust persistent data structures

06 Jul 2017 1,228
pyxirr

Rust-powered collection of financial functions.

04 May 2021 167
polars-distance

Polars plugin for pairwise distance functions

04 Nov 2023 44
datafusion-sqlparser-rs

Extensible SQL Lexer and Parser for Rust

06 Feb 2018 2,732
lumber-jack

High performance & light weight alternative to Pandas with ML focused tooling. (Work in progress ...

15 Oct 2017 5
packed_simd

Portable Packed SIMD Vectors for Rust standard library

27 Jun 2018 592
orjson

Fast, correct Python JSON library supporting dataclasses, datetimes, and numpy

21 Nov 2018 6,140
pgrx

Build Postgres Extensions with Rust!

18 Dec 2019 3,223
sled

the champagne of beta embedded databases

11 Jan 2016 7,771
polars-st

Spatial extension for Polars DataFrames

20 Sep 2024 0
myval

Lightweight Apache Arrow data frame for Rust

14 Apr 2023 61
pyo3

Rust bindings for the Python interpreter

13 May 2017 12,132