marquez

Added

• GA tag for website tracking @rossturk
• Basic CTE support in marquez-airflow @mobuchowski
• Airflow custom facets, bigquery statistics facets @mobuchowski
• Unit tests for class JobVersionDao @wslulciuc
• Sentry tracing support @julienledem
• OpenLineage facets support to API response models 🎉 @wslulciuc

Changed

• BigQueryRelationTransformer and deleted BigQueryNodeVisitor @collado-mike
• Bump postgres to 12.1.0 @wslulciuc
• Update spark job name to reflect spark application name and execution node @collado-mike
• Update marquez-airflow integration to use OpenLineage 🎉 @mobuchowski
• Migrate tests to junit 5 @mobuchowski
• Rewrite lineage IO sql queries to avoid job_versions_io_mapping_* tables @collado-mike
• Updated OpenLineage impl to only update dataset version on run completion @collado-mike

Changed

• Remove unused implementation of SQL parser in marquez-airflow @mobuchowski

Fixed

• Add inputs and outputs to lineage graph @henneberger
• Updated NodeId regex to support URIs with scheme and ports @collado-mike

Added

• Secret support for helm chart @KevinMellott91
• New seed cmd to populate marquez database with source, dataset, and job metadata allowing users to try out features of Marquez (data lineage, view job run history, etc) 🎉
• Docs on applying db migrations manually
• New LineageAPI to support data lineage queries 🎉 @henneberger
• Support for logging errors via sentry
• New Airflow example with Marquez 🎉

Changed

• Update OpenLinageDao to stop converting URI structures to contain underscores instead of colons and slashes @collado-mike
• Bump testcontainers dependency to v1.15.2 @ ShakirzyanovArsen
• Register output datasets for a run lazily @henneberger
• Refactor spark plan traversal to find input/output datasets from datasources @collado-mike
• Web UI project settings and default marquez port @phixMe
• Associate dataset inputs on run start @henneberger

Fixed

• Dataset description is not overwritten on update @henneberger
• Latest tags are returned from dataset @henneberger
• Airflow integration tests on forked PRs @mobuchowski
• Empty nominal end time support @henneberger
• Ensure valid dataset fields for OpenLineage @henneberger
• Ingress context templating for helm chart @KulykDmytro

Changed

• Use alpine image for marquez reducing image size by +50% @KevinMellott91
• Use alpine image for marquez-web reducing image size by +50% @KevinMellott91

Fixed

• Ensure marquez.DAG is (de)serializable

Added

• Modules: api, web, clients , chart, and integrations
• Working airflow example
• runs table indices for columns: created_at and current_run_state @phixMe
• New /lineage endpoint for OpenLineage support @henneberger
• New graphql endpoint @henneberger
• New spark integration @henneberger
• New API to list versions for a dataset

Changed

• Drop Source.type enum (now a string type)

Fixed

• Replace jdbi.getHandle() with jdbi.withHandle() to free DB connections from pool @henneberger
• Fix RunListener when registering outside of the MarquezContext builder @henneberger

Added

• Add indices to runs table for created_at and current_run_state columns @phixMe

Fixed

• Replace jdbi.getHandle() with jdbi.withHandle() to free DB connections from connection pool @henneberger

Added

• Add support for external ID on run creation @julienledem
• Throw RunAlreadyExistsException on run ID already exists
• Add BigQuery, Pulsar, and Oracle source types @sreev
• Add run ID support in job meta; the optional run ID will be used to link a newly created job version to an existing job run, while supporting updating the run state and avoiding having to create another run

Fixed

• Use postgres instead of db in marquez.dev.yml
• Allow multiple postgres containers in test suite @phixMe