OpenLineage

Added

• Support custom transport types #1795 @nataliezeller1
  Adds a new interface, TransportBuilder, for creating custom transport types without having to modify core components of OpenLineage.
• Airflow: dbt Cloud integration #1418 @howardyoo
  Adds a new OpenLineage extractor for dbt Cloud that uses the dbt Cloud hook provided by Airflow to communicate with dbt Cloud via its API.
• Spark: support dataset name modification using regex #1796 @pawel-big-lebowski
  It is a common scenario to write Spark output datasets with a location path ending with /year=2023/month=04. The Spark parameter spark.openlineage.dataset.removePath.pattern introduced here allows for removing certain elements from a path with a regex pattern.

Fixed

• Spark: catch exception when trying to obtain details of non-existing table. #1798 @pawel-big-lebowski
  This mostly happens when getting table details on START event while the table is still not created.
• Spark: LogicalPlanSerializer #1792 @pawel-big-lebowski
  Changes LogicalPlanSerializer to make use of non-shaded Jackson classes in order to serialize LogicalPlans. Note: class names are no longer serialized.
• Flink: fix Flink CI #1801 @pawel-big-lebowski
  Specifies an older image version that succeeds on CI in order to fix the Flink integration.

Added

• SQL: parser improvements to support: copy into, create stage, pivot #1742 @pawel-big-lebowski
  Adds support for additional syntax available in sqlparser-rs.
• dbt: add support for snapshots #1787 @JDarDagran
  Adds support for this special kind of table representing type-2 Slowly Changing Dimensions.

Changed

• Spark: change custom column lineage visitors #1788 @pawel-big-lebowski
  Makes the CustomColumnLineageVisitor interface public to support custom column lineage.

Fixed

• Spark: fix null pointer in JobMetricsHolder #1786 @pawel-big-lebowski
  Adds a null check before running put to fix a NPE occurring in JobMetricsHolder
• SQL: fix query with table generator #1783 @pawel-big-lebowski
  Allows TableFactor::TableFunction to support queries containing table functions.
• SQL: fix rust code style bug #1785 @pawel-big-lebowski
  Fixes a minor style issue in visitor.rs.

Removed

• Airflow: Remove explicit pass from several extract_on_complete methods #1771 @JDarDagran
  Removes the code from three extractors.

Added

• Spark: properties facet #1717 by @tnazarew
  Adds a new facet to capture specified Spark properties.
• SQL: SQLParser supports alter, truncate and drop statements #1695 by @pawel-big-lebowski
  Adds support for the statements to the parser.
• Common/SQL: provide public interface for openlineage_sql package #1727 by @JDarDagran
  Provides a .pyi public interface file for providing typing hints.
• Java client: add configurable headers to HTTP transport #1718 by @tnazarew
  Adds custom header handling to HttpTransport and the Spark integration.
• Python client: create client from dictionary #1745 by @JDarDagran
  Adds a new from_dict method to the Python client to support creating it from a dictionary.

Changed

• Spark: remove URL parameters for JDBC namespaces #1708 by @tnazarew
  Makes the namespace value from an event conform to the naming convention specified in Naming.md.
• Make OPENLINEAGE_DISABLED case-insensitive #1705 by @jedcunningham
  Makes the environment variable for disabling OpenLineage in the Python client and Airflow integration case-insensitive.

Fixed

• Spark: fix missing BigQuery class in column lineage #1698 by @pawel-big-lebowski
  The Spark integration now checks if the BigQuery classes are available on the classpath before attempting to use them.
• DBT: throw UnsupportedDbtCommand when finding unsupported entry in args.which #1724 by @JDarDagran
  Adjusts the dbt-ol script to detect DBT commands in run_results.json only.

Removed

• Spark: remove unnecessary warnings for column lineage #1700 by @pawel-big-lebowski
  Removes the warnings about OneRowRelation and LocalRelation nodes.
• Spark: remove deprecated configs #1711 by @tnazarew
  Removes support for deprecated configs.

Added

• Clients: add DEBUG logging of events to transports #1633 by @mobuchowski
  Ensures that the DEBUG loglevel on properly configured loggers will always log events, regardless of the chosen transport.
• Spark: add CustomEnvironmentFacetBuilder class #1545 by New contributor @Anirudh181001
  Enables the capture of custom environment variables from Spark.
• Spark: introduce the new output visitors AlterTableAddPartitionCommandVisitor and AlterTableSetLocationCommandVisitor #1629 by New contributor @nataliezeller1
  Adds visitors for extracting table names from the Spark commands AlterTableAddPartitionCommand and AlterTableSetLocationCommand. The intended use case is a custom transport for the OpenMetadata lineage API.
• Spark: add column lineage for JDBC relations #1636 by @tnazarew
  Adds column lineage information to JDBC events with data extracted from query by the SQL parser.
• SQL: add Linux-aarch64 native library to Java SQL parser #1664 by @mobuchowski
  Adds a Linux-ARM version of the native library. The Java SQL parser interface had only Linux-x64 and MacOS universal binary variants previously.

Changed

• Airflow: get table database in Athena extractor #1631 by New contributor @rinzool
  Changes the extractor to get a table's database from the table.schema field or the operator default if the field is None.

Fixed

• dbt: add dbt seed to the list of dbt-ol events #1649 by New contributor @pohek321
  Ensures that dbt-ol test no longer fails when run against an event seed.
• Spark: make column lineage extraction in Spark support caching #1634 by @pawel-big-lebowski
  Collect column lineage from Spark logical plans that contain cached datasets.
• Spark: add support for a deprecated config #1586 by @tnazarew
  Maps the deprecated spark.openlineage.url to spark.openlineage.transport.url.
• Spark: add error message in case of null in url #1590 by @tnazarew
  Improves error logging in the case of undefined URLs.
• Spark: collect complete event for really quick Spark jobs #1650 by @pawel-big-lebowski
  Improves the collecting of OpenLineage events on SQL complete in the case of quick operations.
• Spark: fix input/outputs for one node LogicalRelation plans #1668 by @pawel-big-lebowski
  For simple queries like select col1, col2 from my_db.my_table that do not write output,
  the Spark plan contained just a single node, which was wrongly treated as both
  an input and output dataset.
• SQL: fix file existence check in build script for openlineage-sql-java #1613 by @sekikn
  Ensures that the build script works if the library is compiled solely for Linux.

Removed

• Airflow: remove JobIdMapping and update macros to better support Airflow version 2+ #1645 by @JDarDagran
  Updates macros to use OpenLineageAdapter's method to generate deterministic run UUIDs because using the JobIdMapping utility is incompatible with Airflow 2+.

Added

• Airflow: add new extractor for FTPFileTransmitOperator #1603 @sekikn
  Adds a new extractor for this Airflow operator serving legacy systems.

Changed

• Airflow: make extractors for async operators work #1601 @JDarDagran
  Sends a deterministic Run UUID for Airflow runs.

Fixed

• dbt: render actual profile only in profiles.yml #1599 @mobuchowski
  Adds an include_section argument for the Jinja render method to include only one profile if needed.
• dbt: make compiled_code optional #1595 @JDarDagran
  Makes compiled_code optional for manifest > v7.

Added

• Airflow: add new extractor for GCSToGCSOperator #1495 @sekikn
  Adds a new extractor for this operator.
• Flink: resolve topic names from regex, support 1.16.0 #1522 @pawel-big-lebowski
  Adds support for Flink 1.16.0 and makes the integration resolve topic names from Kafka topic patterns.
• Proxy: implement lineage event validator for client proxy #1469 @fm100
  Implements logic in the proxy (which is still in development) for validating and handling lineage events.

Changed

• CI: use ruff instead of flake8, isort, etc., for linting and formatting #1526 @mobuchowski
  Adopts the ruff package, which combines several linters and formatters into one fast binary.

Fixed

• Airflow: make the Trino catalog non-mandatory #1572 @JDarDagran
  Makes the Trino catalog optional in the Trino extractor.
• Common: add explicit SQL dependency #1532 @mobuchowski
  Addresses 0.19.2 breaking change to GE integration by including SQL dependency explicitly.
• DBT: adjust tqdm logging in dbt-ol #1549 @JDarDagran
  Adjusts tqdm to show the correct number of iterations and adds START events for parent runs.
• DBT: fix typo in log output #1493 @denimalpaca
  Fixes 'emittled' typo in log output.
• Great Expectations, Airflow: follow Snowflake dataset naming rules #1527 @mobuchowski
  Normalize Snowflake dataset and datasource naming rules among DBT/Airflow/GX; canonize old Snowflake account paths around making them all full size with account, region and cloud names.
• Java and Python Clients: Kafka does not initialize properties if they are empty; check and notify about Confluent-Kafka requirement #1556 @mobuchowski
  Fixes the failure to initialize KafkaTransport in the Java client and adds an exception if the required confluent-kafka module is missing from the Python client.
• Spark: add square brackets for list-based Spark configs #1507 @Varunvaruns9
  Adds a condition to treat configs with [] as lists. [] will be required for list-based configs starting with 0.21.0.
• Spark: fix several Spark/BigQuery-related issues #1557 @mobuchowski
  Fixes the assumption that a version is always a number; adds support for HadoopMapReduceWriteConfigUtil; makes the integration access BigQueryUtil and getTableId using reflection, which supports all BigQuery versions; makes logs provide the full serialized LogicalPlan on debug.
• SQL: only report partial failures `#1479 @mobuchowski
  Changes the parser so it reports partial failures instead of failing the whole extraction.

Added

• Airflow: add Trino extractor https://github.com/OpenLineage/OpenLineage/pull/1288 @sekikn
  Adds a Trino extractor to the Airflow integration.
• Airflow: add S3FileTransformOperator extractor https://github.com/OpenLineage/OpenLineage/pull/1450 @sekikn
  Adds an S3FileTransformOperator extractor to the Airflow integration.
• Airflow: add standardized run facet https://github.com/OpenLineage/OpenLineage/pull/1413 @JDarDagran
  Creates one standardized run facet for the Airflow integration.
• Airflow: add NominalTimeRunFacet and OwnershipJobFacet https://github.com/OpenLineage/OpenLineage/pull/1410 @JDarDagran
  Adds nominalEndTime and OwnershipJobFacet fields to the Airflow integration.
• dbt: add support for postgres datasources https://github.com/OpenLineage/OpenLineage/pull/1417 @julienledem
  Adds the previously unsupported postgres datasource type.
• Proxy: add client-side proxy (skeletal version) https://github.com/OpenLineage/OpenLineage/pull/1439 https://github.com/OpenLineage/OpenLineage/pull/1420 @fm100
  Implements a skeletal version of a client-side proxy.
• Proxy: add CI job to publish Docker image https://github.com/OpenLineage/OpenLineage/pull/1086 @wslulciuc
  Includes a script to build and tag the image plus jobs to verify the build on every CI run and publish to Docker Hub.
• SQL: add ExtractionErrorRunFacet https://github.com/OpenLineage/OpenLineage/pull/1442 @mobuchowski
  Adds a facet to the spec to reflect internal processing errors, especially failed or incomplete parsing of SQL jobs.
• SQL: add column-level lineage to SQL parser https://github.com/OpenLineage/OpenLineage/pull/1432 https://github.com/OpenLineage/OpenLineage/pull/1461 @mobuchowski @StarostaGit
  Adds support for extracting column-level lineage from SQL statements in the parser, including adjustments to Rust-Python and Rust-Java interfaces and the Airflow integration's SQL extractor to make use of the feature. Also includes more tests, removal of the old parser, and removal of the common-build cache in CI (which was breaking the parser).
• Spark: pass config parameters to the OL client https://github.com/OpenLineage/OpenLineage/pull/1383 @tnazarew
  Adds a mechanism for making new lineage consumers transparent to the integration, easing the process of setting up new types of consumers.

Fixed

• Airflow: fix collect_ignore, add flags to Pytest for cleaner output https://github.com/OpenLineage/OpenLineage/pull/1437 @JDarDagran
  Removes the extractors directory from the ignored list, improving unit testing.
• Spark & Java client: fix README typos @versaurabh
  Fixes typos in the SPDX license headers.

Added

• Airflow: support SQLExecuteQueryOperator #1379 @JDarDagran
  Changes the SQLExtractor and adds support for the dynamic assignment of extractors based on conn_type.
• Airflow: introduce a new extractor for SFTPOperator #1263 @sekikn
  Adds an extractor for tracing file transfers between local file systems.
• Airflow: add Sagemaker extractors #1136 @fhoda
  Creates extractors for SagemakeProcessingOperator and SagemakerTransformOperator.
• Airflow: add S3 extractor for Airflow operators #1166 @fhoda
  Creates an extractor for the S3CopyObject in the Airflow integration.
• Spec: add spec file for ExternalQueryRunFacet #1262 @howardyoo
  Adds a spec file to make this facet available for the Java client. Includes a README.
• Docs: add a TSC doc #1303 @merobi-hub
  Adds a document listing the members of the Technical Steering Committee.

Fixed

• Spark: improve Databricks to send better events #1330 @pawel-big-lebowski
  Filters unwanted events and provides a meaningful job name.
• Spark-Bigquery: fix a few of the common errors #1377 @mobuchowski
  Fixes a few of the common issues with the Spark-Bigquery integration and adds an integration test and configures CI.
• Python: validate eventTime field in Python client #1355 @pawel-big-lebowski
  Validates the eventTime of a RunEvent within the client library.
• Databricks: Handle Databricks Runtime 11.3 changes to DbFsUtils constructor #1351 @wjohnson
  Recaptures lost mount point information from the DatabricksEnvironmentFacetBuilder and environment-properties facet by looking at the number of parameters in the DbFsUtils constructor to determine the runtime version.

Added

• Spark: support latest Spark 3.3.1 https://github.com/OpenLineage/OpenLineage/pull/1183 @pawel-big-lebowski
  Adds support for the latest Spark 3.3.1 version.
• Spark: add Kinesis Transport and support config Kinesis in Spark integration https://github.com/OpenLineage/OpenLineage/pull/1200 @yogyang
  Adds support for sending to Kinesis from the Spark integration.
• Spark: Disable specified facets https://github.com/OpenLineage/OpenLineage/pull/1271 @pawel-big-lebowski
  Adds the ability to disable specified facets from generated OpenLineage events.
• Python: add facets implementation to Python client https://github.com/OpenLineage/OpenLineage/pull/1233 @pawel-big-lebowski
  Adds missing facets to the Python client.
• SQL: add Rust parser interface https://github.com/OpenLineage/OpenLineage/pull/1172 @StarostaGit @mobuchowski
  Implements a Java interface in the Rust SQL parser, including a build script, native library loading mechanism, CI support and build fixes.
• Proxy: add helm chart for the proxy backed https://github.com/OpenLineage/OpenLineage/pull/1068 @wslulciuc
  Adds a helm chart for deploying the proxy backend on Kubernetes.
• Spec: include possible facets usage in spec https://github.com/OpenLineage/OpenLineage/pull/1249 @pawel-big-lebowski
  Extends the facets definition with a list of available facets.
• Website: publish YML version of spec to website https://github.com/OpenLineage/OpenLineage/pull/1300 @rossturk
  Adds configuration necessary to make the OpenLineage website auto-generate openAPI docs when the spec is published there.
• Docs: update language on nominating new committers https://github.com/OpenLineage/OpenLineage/pull/1270 @rossturk
  Updates the governance language to reflect the new policy on nominating committers.

Changed

• Website: publish spec into new website repo location https://github.com/OpenLineage/OpenLineage/pull/1295 @rossturk
  Creates a new deploy key, adds it to CircleCI & GitHub, and makes the necessary changes to the release.sh script.
• Airflow: change how pip installs packages in tox environments https://github.com/OpenLineage/OpenLineage/pull/1302 @JDarDagran
  Uses deprecated resolver and constraints files provided by Airflow to avoid potential issues caused by pip's new resolver.

Fixed

• Airflow: fix README for running integration test https://github.com/OpenLineage/OpenLineage/pull/1238 @sekikn
  Updates the README for consistency with supported Airflow versions.
• Airflow: add task_instance argument to get_openlineage_facets_on_complete https://github.com/OpenLineage/OpenLineage/pull/1269 @JDarDagran
  Adds the task_instance argument to DefaultExtractor.
• Java client: fix up all artifactory paths https://github.com/OpenLineage/OpenLineage/pull/1290 @harels
  Not all artifactory paths were changed in the build CI script in a previous PR.
• Python client: fix Mypy errors and adjust to PEP 484 https://github.com/OpenLineage/OpenLineage/pull/1264 @JDarDagran
  Adds a --no-namespace-packages argument to the Mypy command and adjusts code to PEP 484.
• Website: release all specs since last_spec_commit_id, not just HEAD~1 https://github.com/OpenLineage/OpenLineage/pull/1298 @rossturk
  The script now ships all specs that have changed since .last_spec_commit_id.

Removed

• Deprecate HttpTransport.Builder in favor of HttpConfig https://github.com/OpenLineage/OpenLineage/pull/1287 @collado-mike
  Deprecates the Builder in favor of HttpConfig only and replaces the existing Builder implementation by delegating to the HttpConfig.

Added

• Airflow: add dag_run information to Airflow version run facet https://github.com/OpenLineage/OpenLineage/pull/1133 @fm100
  Adds the Airflow DAG run ID to the taskInfo facet, making this additional information available to the integration.
• Airflow: add LoggingMixin to extractors https://github.com/OpenLineage/OpenLineage/pull/1149 @JDarDagran
  Adds a LoggingMixin class to the custom extractor to make the output consistent with general Airflow and OpenLineage logging settings.
• Airflow: add default extractor https://github.com/OpenLineage/OpenLineage/pull/1162 @mobuchowski
  Adds a DefaultExtractor to support the default implementation of OpenLineage for external operators without the need for custom extractors.
• Airflow: add on_complete argument in DefaultExtractor https://github.com/OpenLineage/OpenLineage/pull/1188 @JDarDagran
  Adds support for running another method on extract_on_complete.
• SQL: reorganize the library into multiple packages https://github.com/OpenLineage/OpenLineage/pull/1167 @StarostaGit @mobuchowski
  Splits the SQL library into a Rust implementation and foreign language bindings, easing the process of adding language interfaces. Also contains a CI fix.

Changed

• Airflow: move get_connection_uri as extractor's classmethod https://github.com/OpenLineage/OpenLineage/pull/1169 @JDarDagran
  The get_connection_uri method allowed for too many params, resulting in unnecessarily long URIs. This changes the logic to whitelisting per extractor.
• Airflow: change get_openlineage_facets_on_start/complete behavior https://github.com/OpenLineage/OpenLineage/pull/1201 @JDarDagran
  Splits up the method for greater legibility and easier maintenance.

Fixed

• Airflow: always send SQL in SqlJobFacet as a string https://github.com/OpenLineage/OpenLineage/pull/1143 @mobuchowski
  Changes the data type of query from array to string to an fix error in the RedshiftSQLOperator.
• Airflow: include extra case when filtering URI query params https://github.com/OpenLineage/OpenLineage/pull/1144 @JDarDagran
  Includes the conn.EXTRA_KEY in the get_connection_uri method to avoid exposing secrets in URIs via the extra key.
• Airflow: enforce column casing in SQLCheckExtractors https://github.com/OpenLineage/OpenLineage/pull/1159 @denimalpaca
  Uses the parent extractor's _is_uppercase_names property to determine if the column should be upper cased in the SQLColumnCheckExtractor's _get_input_facets() method.
• Spark: prevent exception when no schema provided https://github.com/OpenLineage/OpenLineage/pull/1180 @pawel-big-lebowski
  Prevents evalution of column lineage when the schemFacet is null.
• Great Expectations: add V3 API compatibility https://github.com/OpenLineage/OpenLineage/pull/1194 @denimalpaca
  Fixes the Pandas datasource to make it V3 API-compatible.

Removed

• Airflow: remove support for Airflow 1.10 https://github.com/OpenLineage/OpenLineage/pull/1128 @mobuchowski
  Removes the code structures and tests enabling support for Airflow 1.10.

Added

• Airflow: improve development experience https://github.com/OpenLineage/OpenLineage/pull/1101 @JDarDagran
  Adds an interactive development environment to the Airflow integration and improves integration testing.
• Documentation: update issue templates for proposal & add new integration template https://github.com/OpenLineage/OpenLineage/pull/1116 @rossturk
  Adds a YAML issue template for new integrations and fixes a bug in the proposal template.
• Spark: add description for URL parameters in readme, change overwriteName to appName https://github.com/OpenLineage/OpenLineage/pull/1130 @tnazarew
  Adds more information about passing arguments with spark.openlineage.url and changes overwriteName to appName for clarity.

Changed

• Airflow: lazy load BigQuery client https://github.com/OpenLineage/OpenLineage/pull/1119 @mobuchowski
  Moves import of the BigQuery client from top level to local level to decrease DAG import time.

Fixed

• Airflow: fix UUID generation conflict for Airflow DAGs with same name https://github.com/OpenLineage/OpenLineage/pull/1056 @collado-mike
  Adds a namespace to the UUID calculation to avoid conflicts caused by DAGs having the same name in different namespaces in Airflow deployments.
• Spark/BigQuery: fix issue with spark-bigquery-connector >=0.25.0 https://github.com/OpenLineage/OpenLineage/pull/1111 @pawel-big-lebowski
  Makes the Spark integration compatible with the latest connector.
• Spark: fix column lineage https://github.com/OpenLineage/OpenLineage/pull/1069 @pawel-big-lebowski
  Fixes a null pointer exception error and an error when openlineage.timeout is not provided.
• Spark: set log level of Init OpenLineageContext to DEBUG https://github.com/OpenLineage/OpenLineage/pull/1064 @varuntestaz
  Prevents sensitive information from being logged unless debug mode is used.
• Java client: update version of SnakeYAML https://github.com/OpenLineage/OpenLineage/pull/1090 @TheSpeedding
  Bumps the SnakeYAML library version to include a key bug fix.
• dbt: remove requirement for OPENLINEAGE_URL to be set https://github.com/OpenLineage/OpenLineage/pull/1107 @mobuchowski
  Removes erroneous check for OPENLINEAGE_URL in the dbt integration.
• Python client: remove potentially cyclic import https://github.com/OpenLineage/OpenLineage/pull/1126 @mobuchowski
  Hides imports to remove potentially cyclic import.
• CI: build macos release package on medium resource class https://github.com/OpenLineage/OpenLineage/pull/1131 @mobuchowski
  Fixes failing build due to resource class being too large.

Fixed

• Fix Spark integration issues including error when no openlineage.timeout https://github.com/OpenLineage/OpenLineage/pull/1069 @pawel-big-lebowski
  OpenlineageSparkListener was failing when no openlineage.timeout was provided.

Added

• Support ABFSS and Hadoop Logical Relation in Column-level lineage https://github.com/OpenLineage/OpenLineage/pull/1008 @wjohnson
  Introduces an extractDatasetIdentifier that uses similar logic to InsertIntoHadoopFsRelationVisitor to pull out the path on the HDFS compliant file system; tested on ABFSS and DBFS (Databricks FileSystem) to prove that lineage could be extracted using non-SQL commands.
• Add Kusto relation visitor https://github.com/OpenLineage/OpenLineage/pull/939 @hmoazam
  Implements a KustoRelationVisitor to support lineage for Azure Kusto's Spark connector.
• Add ColumnLevelLineage facet doc https://github.com/OpenLineage/OpenLineage/pull/1020 @julienledem
  Adds documentation for the Column-level lineage facet.
• Include symlinks dataset facet https://github.com/OpenLineage/OpenLineage/pull/935 @pawel-big-lebowski
  Includes the recently introduced SymlinkDatasetFacet in generated OpenLineage events.
• Add support for dbt 1.3 beta's metadata changes https://github.com/OpenLineage/OpenLineage/pull/1051 @mobuchowski
  Makes projects that are composed of only SQL models work on 1.3 beta (dbt 1.3 renamed the compiled_sql field to compiled_code to support Python models). Does not provide support for dbt's Python models.
• Support Flink 1.15 https://github.com/OpenLineage/OpenLineage/pull/1009 @mzareba382
  Adds support for Flink 1.15.
• Add Redshift dialect to the SQL integration https://github.com/OpenLineage/OpenLineage/pull/1066 @mobuchowski
  Adds support for Redshift's SQL dialect in OpenLineage's SQL parser, including quirks such as the use of square brackets in JSON paths. (Note, this does not add support for all of Redshift's custom syntax.)

Changed

• Make the timeout configurable in the Spark integration https://github.com/OpenLineage/OpenLineage/pull/1050 @tnazarew
  Makes timeout configurable by the user. (In some cases, the time needed to send events was longer than 5 seconds, which exceeded the timeout value.)

Fixed

• Add a dialect parameter to Great Expectations SQL parser calls https://github.com/OpenLineage/OpenLineage/pull/1049 @collado-mike
  Specifies the dialect name from the SQL engine.
• Fix Delta 2.1.0 with Spark 3.3.0 https://github.com/OpenLineage/OpenLineage/pull/1065 @pawel-big-lebowski
  Allows delta support for Spark 3.3 and fixes potential issues. (The Openlineage integration for Spark 3.3 was turned on without delta support, as delta did not support Spark 3.3 at that time.)

Fixed

• Rename all parentRun occurrences to parent from Airflow integration https://github.com/OpenLineage/OpenLineage/pull/1037 @fm100
• Do not change task instance during on_running event https://github.com/OpenLineage/OpenLineage/pull/1028 @JDarDagran

Added

• Add BigQuery check support #960 @denimalpaca
• Add RUNNING EventType in spec and Python client #972 @mzareba382
• Use databases & schemas in SQL Extractors #974 @JDarDagran
• Implement Event forwarding feature via HTTP protocol #995 @howardyoo
• Introduce SymlinksDatasetFacet to spec #936 @pawel-big-lebowski
• Add Azure Cosmos Handler to Spark integration #983 @hmoazam
• Support OL Datasets in manual lineage inputs/outputs #1015 @conorbev
• Create ownership facets #996 @julienledem

Changed

• Use RUNNING EventType in Flink integration for currently running jobs #985 @mzareba382
• Convert task object into JSON encodable when creating Airflow version facet #1018 @fm100

Fixed

• Add support for custom SQL queries in v3 Great Expectations API #1025 @collado-mike

Added

• Add Spark 3.3.0 support #950 @pawel-big-lebowski
• Add Apache Flink integration #951 @mobuchowski
• Add ability to extend column level lineage mechanism #922 @pawel-big-lebowski
• Add ErrorMessageRunFacet #897 @mobuchowski
• Add SQLCheckExtractors #717 @denimalpaca
• Add RedshiftSQLExtractor & RedshiftDataExtractor #930 @JDarDagran
• Add dataset builder for AlterTableCommand #927 @tnazarew

Changed

• Limit Delta events #905 @pawel-big-lebowski
• Airflow integration: allow lineage metadata to flow through inlets and outlets #914 @fenil25

Fixed

• Limit size of serialized plan #917 @pawel-big-lebowski
• Fix noclassdef error #942 @pawel-big-lebowski

Added

• HTTP option to override timeout and properly close connections in openlineage-java lib. https://github.com/OpenLineage/OpenLineage/pull/909 @mobuchowski
• Dynamic mapped tasks support to Airflow integration https://github.com/OpenLineage/OpenLineage/pull/906 @JDarDagran
• SqlExtractor to Airflow integration https://github.com/OpenLineage/OpenLineage/pull/907 @JDarDagran
• PMD to Java and Spark builds in CI https://github.com/OpenLineage/OpenLineage/pull/898 @merobi-hub

Changed

• When testing extractors in the Airflow integration, set the extractor length assertion dynamic https://github.com/OpenLineage/OpenLineage/pull/882 @denimalpaca
• Render templates as start of integration tests for TaskListener in the Airflow integration https://github.com/OpenLineage/OpenLineage/pull/870 @mobuchowski

Fixed

• Dependencies bundled with openlineage-java lib. https://github.com/OpenLineage/OpenLineage/pull/855 @collado-mike
• PMD reported issues https://github.com/OpenLineage/OpenLineage/pull/891 @pawel-big-lebowski
• Spark casting error and session catalog support for iceberg in Spark integration https://github.com/OpenLineage/OpenLineage/pull/856 @wslulciuc

Added

• Add static code anlalysis tool mypy to run in CI for against all python modules (#802) @howardyoo
• Extend SaveIntoDataSourceCommandVisitor to extract schema from LocalRelaiton and LogicalRdd in spark integration (#794) @pawel-big-lebowski
• Add InMemoryRelationInputDatasetBuilder for InMemory datasets to Spark integration (#818) @pawel-big-lebowski
• Add copyright to source files (#755) @merobi-hub
• Add SnowflakeOperatorAsync extractor support to Airflow integration #869 @denimalpaca
• Add PMD analysis to proxy project (#889) @howardyoo

Changed

• Skip FunctionRegistry.class serialization in Spark integration (#828) @mobuchowski
• Install new rust-based SQL parser by default in Airflow integration (#835) @mobuchowski
• Improve overall pytest and integration tests for Airflow integration (#851, #858) @denimalpaca
• Reduce OL event payload size by excluding local data and including output node in start events (#881) @collado-mike
• Split spark integration into submodules (#834, #890) @tnazarew @mobuchowski

Fixed

• Conditionally import sqlalchemy lib for Great Expectations integration (#826) @pawel-big-lebowski
• Add check for missing class org.apache.spark.sql.catalyst.plans.logical.CreateV2Table in Spark integration (#866) @pawel-big-lebowski
• Fix static code analysis issues (#867, #874) @pawel-big-lebowski

Added

• Spark: Column-level lineage introduced for Spark integration (https://github.com/OpenLineage/OpenLineage/pull/698, https://github.com/OpenLineage/OpenLineage/pull/645) @pawel-big-lebowski
• Java: Spark to use Java client directly (https://github.com/OpenLineage/OpenLineage/pull/774) @mobuchowski
• Clients: Add OPENLINEAGE_DISABLED environment variable which overrides config to NoopTransport (https://github.com/OpenLineage/OpenLineage/pull/780) @mobuchowski

Fixed

• Set log to debug on unknown facet entry (https://github.com/OpenLineage/OpenLineage/pull/766) @wslulciuc
• Dagster: pin protobuf version to 3.20 as suggested by tests (https://github.com/OpenLineage/OpenLineage/pull/787) @mobuchowski
• Add SafeStrDict to skip failing attibutes (https://github.com/OpenLineage/OpenLineage/pull/798) @JDarDagran

Added

• openlineage-airflow now supports getting credentials from Airflows secrets backend (https://github.com/OpenLineage/OpenLineage/pull/723) @mobuchowski
• openlineage-spark now supports Azure Databricks Credential Passthrough (https://github.com/OpenLineage/OpenLineage/pull/595) @wjohnson
• openlineage-spark detects datasets wrapped by ExternalRDDs (https://github.com/OpenLineage/OpenLineage/pull/746) @collado-mike

Fixed

• PostgresOperator fails to retrieve host and conn during extraction (https://github.com/OpenLineage/OpenLineage/pull/705) @sekikn
• SQL parser accepts lists of sql statements (https://github.com/OpenLineage/OpenLineage/issues/734) @mobuchowski
• Missing schema when writing to Delta tables in Databricks (https://github.com/OpenLineage/OpenLineage/pull/748) @collado-mike