OpenLineage

Added

• Airflow integration uses new TaskInstance listener API for Airflow 2.3+ (#508) @mobuchowski
• Support for HiveTableRelation as input source in Spark integration (#683) @collado-mike
• Add HTTP and Kafka Client to openlineage-java lib (#480) @wslulciuc, @mobuchowski
• New SQL parser, used by Postgres, Snowflake, Great Expectations integrations (#644) @mobuchowski

Fixed

• GreatExpectations: Fixed bug when invoking GreatExpectations using v3 API (#683) @collado-mike

Added

• Python implements Transport interface - HTTP and Kafka transports are available (https://github.com/OpenLineage/OpenLineage/pull/530) @mobuchowski
• Add UnknownOperatorAttributeRunFacet and support in lineage backend (https://github.com/OpenLineage/OpenLineage/pull/547) @collado-mike
• Support Spark 3.2.1 (https://github.com/OpenLineage/OpenLineage/pull/607) @pawel-big-lebowski
• Add StorageDatasetFacet to spec (https://github.com/OpenLineage/OpenLineage/pull/620) @pawel-big-lebowski
• Airflow: custom extractors lookup uses only get_operator_classnames method (https://github.com/OpenLineage/OpenLineage/pull/656) @mobuchowski
• README.md created at OpenLineage/integrations for compatibility matrix (https://github.com/OpenLineage/OpenLineage/pull/663) @howardyoo

Fixed

• Dagster: handle updated PipelineRun in OpenLineage sensor unit test (https://github.com/OpenLineage/OpenLineage/pull/624) @dominiquetipton
• Delta improvements (https://github.com/OpenLineage/OpenLineage/pull/626) @collado-mike
• Fix SqlDwDatabricksVisitor for Spark2 (https://github.com/OpenLineage/OpenLineage/pull/630) @wjohnson
• Airflow: remove redundant logging from GE import (https://github.com/OpenLineage/OpenLineage/pull/657) @mobuchowski
• Fix Shebang issue in Spark's wait-for-it.sh (https://github.com/OpenLineage/OpenLineage/pull/658) @mobuchowski
• Update parent_run_id to be a uuid from the dag name and run_id (https://github.com/OpenLineage/OpenLineage/pull/664) @collado-mike
• Spark: fix time zone inconsistency in testSerializeRunEvent (https://github.com/OpenLineage/OpenLineage/pull/681) @sekikn

Added

• CI: add integration tests for Airflow's SnowflakeOperator and dbt-snowflake @mobuchowski
• Introduce DatasetVersion facet in spec @pawel-big-lebowski
• Airflow: add external query id facet @mobuchowski

Fixed

• Complete Fix of Snowflake Extractor get_hook() Bug @denimalpaca
• Update artwork @rossturk

Fixed

• Catch possible failures when emitting events and log them @mobuchowski

Added

• Extract source code of PythonOperator code similar to SQL facet @mobuchowski
• Add DatasetLifecycleStateDatasetFacet to spec @pawel-big-lebowski
• Airflow: extract source code from BashOperator @mobuchowski
• Add generic facet to collect environmental properties (EnvironmentFacet) @harishsune
• Update Airflow connection parsing to exclude AWS keys and extraneous Snowflake params @collado-mike
• OpenLineage sensor for OpenLineage-Dagster integration @dalinkim
• Java-client: make generator generate enums as well @pawel-big-lebowski

Fixed

• Airflow: increase import timeout in tests, fix exit from integration @mobuchowski
• Reduce logging level for import errors to info @rossturk
• Remove AWS secret keys and extraneous Snowflake parameters from connection uri @collado-mike
• Convert to LifecycleStateChangeDatasetFacet @pawel-big-lebowski

Added

• Proxy backend example using Kafka @wslulciuc
• Support Databricks Delta Catalog naming convention with DatabricksDeltaHandler @wjohnson
• Add javadoc as part of build task @mobuchowski
• Include TableStateChangeFacet in non V2 commands for Spark @mr-yusupov
• Support for SqlDWRelation on Databricks' Azure Synapse/SQL DW Connector @wjohnson
• Implement input visitors for v2 commands @pawel-big-lebowski
• Enabled SparkListenerJobStart events to trigger open lineage events @collado-mike

Fixed

• dbt: job namespaces for given dbt run match each other @mobuchowski
• Fix Breaking SnowflakeOperator Changes from OSS Airflow @denimalpaca

Added

• Support for dbt-spark adapter @mobuchowski
• New backend to proxy OpenLineage events to one or more event streams 🎉 @mandy-chessell @wslulciuc
• Add Spark extensibility API with support for custom Dataset and custom facet builders @collado-mike

Fixed

• airflow: fix import failures when dependencies for bigquery, dbt, great_expectations extractors are missing @lukaszlaszko
• Fixed openlineage-spark jar to correctly rename bundled dependencies @collado-mike

Added

• Separated tests between Spark 2 & 3 @pawel-big-lebowski
• Spark output metrics @OleksandrDvornik
• Databricks install README and init scripts @wjohnson
• Iceberg integration with unit tests @pawel-big-lebowski
• Kafka read and write support @OleksandrDvornik / @collado-mike
• Arbitrary parameters supported in HTTP URL construction @wjohnson
• Increased visitor coverage for Spark commands @mobuchowski / @pawel-big-lebowski

Fixed

• dbt: column descriptions are properly filled from metadata.json @mobuchowski
• dbt: allow parsing artifacts with version higher than officially supported @mobuchowski
• dbt: dbt build command is supported @mobuchowski
• dbt: fix crash when build command is used with seeds in dbt 1.0.0rc3 @mobuchowski
• spark: increase logical plan visitor coverage @mobuchowski
• spark: fix logical serialization recursion issue @OleksandrDvornik
• Use URL#getFile to fix build on Windows @mobuchowski

Fixed

• fix import in spark3 visitor @mobuchowski

Added

• Spark3 support @OleksandrDvornik / @collado-mike
• LineageBackend for Airflow 2 @mobuchowski
• Adding custom spark version facet to spark integration @OleksandrDvornik
• Adding dbt version facet @mobuchowski
• Added support for Redshift profile @AlessandroLollo

Fixed

• Sanitize JDBC URLs @OleksandrDvornik
• strip openlineage url in python client @OleksandrDvornik
• deploy spec if spec file changes @mobuchowski

Fixed

• Add dbt v3 manifest support @mobuchowski

Added

• Implement OpenLineageValidationAction for Great Expectations @collado-mike
• facet: add expectations assertions facet @mobuchowski

Fixed

• airflow: pendulum formatting fix, add tests @mobuchowski
• dbt: do not emit events if run_result file was not updated @mobuchowski

Fixed

• Default --project-dir argument to current directory in dbt-ol script @mobuchowski

Added

• Parse dbt command line arguments when invoking dbt-ol @mobuchowski. For example:

  $ dbt-ol run --project-dir path/to/dir
  

• Set UnknownFacet for spark (captures metadata about unvisited nodes from spark plan not yet supported) @OleksandrDvornik

Changed

• Remove model from dbt job name @mobuchowski
• Default dbt job namespace to output dataset namespace @mobuchowski
• Rename openlineage.spark.* to io.openlineage.spark.* @OleksandrDvornik

Fixed

• Remove instance references to extractors from DAG and avoid copying log property for serializability @collado-mike

OpenLineage is an open standard for lineage metadata collection designed to record metadata for a job in execution. The initial public release includes:

• An initial specification. The initial version 1-0-0 of the OpenLineage specification defines the core model and facets.
• Integrations that collect lineage metadata as OpenLineage events:
  • Apache Airflow with support for BigQuery, Great Expectations, Postgres, Redshift, Snowflake
  • Apache Spark
  • dbt
• Clients that send OpenLineage events to an HTTP backend. Both java and python are initially supported.