3.0.0 (2022-04-25)

⚠ BREAKING CHANGES

ir: The following are breaking changes due to simplifying expression internals
- ibis.expr.datatypes.DataType.scalar_type and DataType.column_type factory
  methods have been removed, DataType.scalar and DataType.column class
  fields can be used to directly construct a corresponding expression instance
  (though prefer to use operation.to_expr())
- ibis.expr.types.ValueExpr._name and ValueExpr._dtype`` fields are not accassible anymore. While these were not supposed to used directly now ValueExpr.has_name(), ValueExpr.get_name()andValueExpr.type()` methods
  are the only way to retrieve the expression's name and datatype.
- ibis.expr.operations.Node.output_type is a property now not a method,
  decorate those methods with @property
- ibis.expr.operations.ValueOp subclasses must define output_shape and
  output_dtype properties from now on (note the datatype abbreviation dtype
  in the property name)
- ibis.expr.rules.cast(), scalar_like() and array_like() rules have been
  removed
api: Replace t["a"].distinct() with t[["a"]].distinct().
deps: The sqlalchemy lower bound is now 1.4
ir: Schema.names and Schema.types attributes now have tuple type rather than list
expr: Columns that were added or used in an aggregation or
mutation would be alphabetically sorted in compiled SQL outputs. This
was a vestige from when Python dicts didn't preserve insertion order.
Now columns will appear in the order in which they were passed to
aggregate or mutate
api: dt.float is now dt.float64; use dt.float32 for the previous behavior.
ir: Relation-based execute_node dispatch rules must now accept tuples of expressions.
ir: removed ibis.expr.lineage.{roots,find_nodes} functions
config: Use ibis.options.graphviz_repr = True to enable
hdfs: Use fsspec instead of HDFS from ibis
udf: Vectorized UDF coercion functions are no longer a public API.
The minimum supported Python version is now Python 3.8
config: register_option is no longer supported, please submit option requests upstream
backends: Read tables with pandas.read_hdf and use the pandas backend
The CSV backend is removed. Use Datafusion for CSV execution.
backends: Use the datafusion backend to read parquet files
Expr() -> Expr.pipe()
coercion functions previously in expr/schema.py are now in udf/vectorized.py
api: materialize is removed. Joins with overlapping columns now have suffixes.
kudu: use impala instead: https://kudu.apache.org/docs/kudu_impala_integration.html
Any code that was relying implicitly on string-y
behavior from UUID datatypes will need to add an explicit cast first.

Features

add repr_html for expressions to print as tables in ipython (cd6fa4e)
add duckdb backend (667f2d5)
allow construction of decimal literals (3d9e865)
api: add ibis.asc expression (efe177e), closes #1454
api: add has_operation API to the backend (4fab014)
api: implement type for SortExpr (ab19bd6)
clickhouse: implement string concat for clickhouse (1767205)
clickhouse: implement StrRight operation (67749a0)
clickhouse: implement table union (e0008d7)
clickhouse: implement trim, pad and string predicates (a5b7293)
datafusion: implement Count operation (4797a86)
datatypes: unbounded decimal type (f7e6f65)
date: add ibis.date(y,m,d) functionality (26892b6), closes #386
duckdb/postgres/mysql/pyspark: implement .sql on tables for mixing sql and expressions (00e8087)
duckdb: add functionality needed to pass integer to interval test (e2119e8)
duckdb: implement _get_schema_using_query (93cd730)
duckdb: implement now() function (6924f50)
duckdb: implement regexp replace and extract (18d16a7)
implement force argument in sqlalchemy backend base class (9df7f1b)
implement coalesce for the pyspark backend (8183efe)
implement semi/anti join for the pandas backend (cb36fc5)
implement semi/anti join for the pyspark backend (3e1ba9c)
implement the remaining clickhouse joins (b3aa1f0)
ir: rewrite and speed up expression repr (45ce9b2)
mysql: implement _get_schema_from_query (456cd44)
mysql: move string join impl up to alchemy for mysql (77a8eb9)
postgres: implement _get_schema_using_query (f2459eb)
pyspark: implement Distinct for pyspark (4306ad9)
pyspark: implement log base b for pyspark (527af3c)
pyspark: implement percent_rank and enable testing (c051617)
repr: add interval info to interval repr (df26231)
sqlalchemy: implement ilike (43996c0)
sqlite: implement date_truncate (3ce4f2a)
sqlite: implement ISO week of year (714ff7b)
sqlite: implement string join and concat (6f5f353)
support of arrays and tuples for clickhouse (db512a8)
ver: dynamic version identifiers (408f862)

Bug Fixes

added wheel to pyproject toml for venv users (b0b8e5c)
allow major version changes in CalVer dependencies (9c3fbe5)
annotable: allow optional arguments at any position (778995f), closes #3730
api: add ibis.map and .struct (327b342), closes #3118
api: map string multiplication with integer to repeat method (b205922)
api: thread suffixes parameter to individual join methods (31a9aff)
change TimestampType to Timestamp (e0750be)
clickhouse: disconnect from clickhouse when computing version (11cbf08)
clickhouse: use a context manager for execution (a471225)
combine windows during windowization (7fdd851)
conform epoch_seconds impls to expression return type (18a70f1)
context-adjustment: pass scope when calling adjust_context in pyspark backend (33aad7b), closes #3108
dask: fix asof joins for newer version of dask (50711cc)
dask: workaround dask bug (a0f3bd9)
deps: update dependency atpublic to v3 (3fe8f0d)
deps: update dependency datafusion to >=0.4,<0.6 (3fb2194)
deps: update dependency geoalchemy2 to >=0.6.3,<0.12 (dc3c361)
deps: update dependency graphviz to >=0.16,<0.21 (3014445)
duckdb: add casts to literals to fix binding errors (1977a55), closes #3629
duckdb: fix array column type discovery on leaf tables and add tests (15e5412)
duckdb: fix log with base b impl (4920097)
duckdb: support both 0.3.2 and 0.3.3 (a73ccce)
enforce the schema's column names in apply_to (b0f334d)
expose ops.IfNull for mysql backend (156c2bd)
expr: add more binary operators to char list and implement fallback (b88184c)
expr: fix formatting of table info using tabulate (b110636)
fix float vs real data type detection in sqlalchemy (24e6774)
fix list_schemas argument (69c1abf)
fix postgres udfs and reenable ci tests (7d480d2)
fix tablecolumn execution for filter following join (064595b)
format: remove some newlines from formatted expr repr (ed4fa78)
histogram: cross_join needs onclause=True (5d36a58), closes #622
ibis.expr.signature.Parameter is not pickleable (828fd54)
implement coalesce properly in the pandas backend (aca5312)
implement count on tables for pyspark (7fe5573), closes #2879
infer coalesce types when a non-null expression occurs after the first argument (c5f2906)
mutate: do not lift table column that results from mutate (ba4e5e5)
pandas: disable range windows with order by (e016664)
pandas: don't reassign the same column to silence SettingWithCopyWarning warning (75dc616)
pandas: implement percent_rank correctly (d8b83e7)
prevent unintentional cross joins in mutate + filter (83eef99)
pyspark: fix range windows (a6f2aa8)
regression in Selection.sort_by with resolved_keys (c7a69cd)
regression in sort_by with resolved_keys (63f1382), closes #3619
remove broken csv pre_execute (93b662a)
remove importorskip call for backend tests (2f0bcd8)
remove incorrect fix for pandas regression (339f544)
remove passing schema into register_parquet (bdcbb08)
repr: add ops.TimeAdd to repr binop lookup table (fd94275)
repr: allow ops.TableNode in fmt_value (6f57003)
reverse the predicate pushdown subsitution (f3cd358)
sort_index to satisfy pandas 1.4.x (6bac0fc)
sqlalchemy: ensure correlated subqueries FROM clauses are rendered (3175321)
sqlalchemy: use corresponding_column to prevent spurious cross joins (fdada21)
sqlalchemy: use replace selectables to prevent semi/anti join cross join (e8a1a71)
sql: retain column names for named ColumnExprs (f1b4b6e), closes #3754
sql: walk right join trees and substitute joins with right-side joins with views (0231592)
store schema on the pandas backend to allow correct inference (35070be)

Performance Improvements

datatypes: speed up str and hash (262d3d7)
fast path for simple column selection (d178498)
ir: global equality cache (13c2bb2)
ir: introduce CachedEqMixin to speed up equality checks (b633925)
repr: remove full tree repr from rule validator error message (65885ab)
speed up attribute access (89d1c05)
use assign instead of concat in projections when possible (985c242)

Miscellaneous Chores

deps: increase sqlalchemy lower bound to 1.4 (560854a)
drop support for Python 3.7 (0afd138)

Code Refactoring

api: make primitive types more cohesive (71da8f7)
api: remove distinct ColumnExpr API (3f48cb8)
api: remove materialize (24285c1)
backends: remove the hdf5 backend (ff34f3e)
backends: remove the parquet backend (b510473)
config: disable graphviz-repr-in-notebook by default (214ad4e)
config: remove old config code and port to pydantic (4bb96d1)
dt.UUID inherits from DataType, not String (2ba540d)
expr: preserve column ordering in aggregations/mutations (668be0f)
hdfs: replace HDFS with fsspec (cc6eddb)
ir: make Annotable immutable (1f2b3fa)
ir: make schema annotable (b980903)
ir: remove unused lineage roots and find_nodes functions (d630a77)
ir: simplify expressions by not storing dtype and name (e929f85)
kudu: remove support for use of kudu through kudu-python (36bd97f)
move coercion functions from schema.py to udf (58eea56), closes #3033
remove blanket call for Expr (3a71116), closes #2258
remove the csv backend (0e3e02e)
udf: make coerce functions in ibis.udf.vectorized private (9ba4392)

ibis - 2.1.1

Published by ibis-project-bot[bot] almost 3 years ago

2.1.1 (2022-01-12)

Bug Fixes

setup.py: set the correct version number for 2.1.0 (f3d267b)

ibis - 2.1.0

Published by ibis-project-bot[bot] almost 3 years ago

2.1.0 (2022-01-12)

Bug Fixes

consider all packages' entry points (b495cf6)
datatypes: infer bytes literal as binary #2915 (#3124) (887efbd)
deps: bump minimum dask version to 2021.10.0 (e6b5c09)
deps: constrain numpy to ensure wheels are used on windows (70c308b)
deps: update dependency clickhouse-driver to ^0.1 || ^0.2.0 (#3061) (a839d54)
deps: update dependency geoalchemy2 to >=0.6,<0.11 (4cede9d)
deps: update dependency pyarrow to v6 (#3092) (61e52b5)
don't force backends to override do_connect until 3.0.0 (4b46973)
execute materialized joins in the pandas and dask backends (#3086) (9ed937a)
literal: allow creating ibis literal with uuid (#3131) (b0f4f44)
restore the ability to have more than two option levels (#3151) (fb4a944)
sqlalchemy: fix correlated subquery compilation (43b9010)
sqlite: defer db connection until needed (#3127) (5467afa), closes #64

Features

allow column_of to take a column expression (dbc34bb)
ci: More readable workflow job titles (#3111) (d8fd7d9)
datafusion: initial implementation for Arrow Datafusion backend (3a67840), closes #2627
datafusion: initial implementation for Arrow Datafusion backend (75876d9), closes #2627
make dayofweek impls conform to pandas semantics (#3161) (9297828)