OpenMetadata

Open Standard for Metadata. A Single place to Discover, Collaborate and Get your data right.

APACHE-2.0 License

Downloads
107.7K
Stars
4.2K
Committers
225
View Code on GitHub Visit Website View on X
Ecosystems: TypeScript
OpenMetadata - OpenMetadata 1.3.0-Release

Published by akash-jain-10 9 months ago

What’s New in OpenMetadata 1.3 Release

Lineage

  • Revamped the lineage UI for an intuitive and comprehensive view of data flow and transformations.
  • Organized nodes for better visibility with pagination support.
  • Improved the display of circular dependencies.
  • Nodes display the service icons, highlight dbt models, and show Data Quality results.
  • Lineage can be filtered to search by Ownership, Domain, Service, Service Type, Tier, and Classification Tags.
  • Supports search by Column and traces lineage even when the columns are renamed.
  • Enhanced user control with collapsible sub-graphs.
  • Supports editing the SQL queries for lineage edges from the UI.
  • Performance improvements for faster load of large graphs.

Data Observability Alerts

  • Data observability alerts have been distinguished from the other general-purpose notifications, making it easy to get to the crucial alerts quickly.
  • Sends alerts for schema changes and test case failures for the data assets that you follow.
  • The overall flow has been simplified to let you easily create alerts for schema changes in your data.
  • You can now get Data Quality alerts for specific Test Suites.
  • Users will be alerted for all the changes to the data assets that they own.

Incident Manager

  • Introduced Incidents Manager to improve the data quality resolution flow.
  • Incidents Manager summarizes all the test case results with information about the failure severity and resolution flow.
  • Supports assigning a resolution task to the users in OpenMetadata.
  • Tasks are created when a data quality test has been assigned to an Assignee or a Reviewer.
  • Resolved test failure also displays the comments posted on the resolution.
  • The Resolved Tab displays information on the Test case name, Execution date, Reason, Comments, and information on who Resolved the issue.

Knowledge Center ic-collate

  • Supports hierarchical pages to structure the articles.
  • You can easily associate knowledge articles with data assets.
  • The data assets page displays the related articles.
  • The block editor supports callouts to add notes, warnings, tables, and task lists.
  • Quicklinks are no longer separate pages; they redirect to external links.
  • Data assets can be associated with Quicklinks.
  • Added Search support for Knowledge articles to filter by Owner or Tags.
  • Supports preview for articles and Quicklinks.

Custom Metrics for Profiler

  • Supports custom metrics for the data profiler with custom SQL to keep track of your business metrics.
  • Custom metrics can be created at Table and Column levels.

Profiler and Data Quality

  • The Profiler has been improved to support sample data ingestion without computing other metrics.
  • Admins can configure the profiler to fetch up to 10,000 rows of sample data.
  • Sample data can be stored in S3 buckets.
  • Refined the default time range on the test case results page, adjusting it from the Last 3 days to the Last 30 days for a more encompassing view.

Connectors

  • New Google Cloud Storage for storage services. ic-collate
  • New Alation connector to migrate metadata into Collate. ic-collate
  • New Iceberg, SAS Viya, and Doris connectors.
  • Introduced the Spark Lineage Agent to extract metadata and end-to-end lineage from Spark jobs.
  • MSSQL and Oracle now support Stored Procedures.
  • We now exclude system indices from the Elasticsearch connector by default.
  • Added support for DB2 IBM I Series.
  • Pipeline services now get owner information.
  • Performance improvements for the Tableau Connector.
  • We now support metadata tag extraction from Databricks.
  • Supports the attribute Table Owner for metadata ingestion from Postgres.
  • We now extract table descriptions when ingesting metadata from Salesforce.

Glossary

  • Supports soft delete for the default glossaries in OpenMetadata.
  • Supports the creation of tasks to request tags or a description.
  • Only the Owner can edit the Glossary term.
  • Version history displays the Username instead of the User ID.

Localization

  • Now supports RTL UI for the Hebrew language.
  • New Dutch language translation.

Settings UI

  • The Settings page UI has been revamped.

Data Insights

  • Cost Analysis expanded to support BigQuery & Redshift. ic-collate
  • Improved the Data Insights Report sent via email.

Other Changes

  • Announcements can be notified over email, Slack, or Teams.
  • Alerts are sent to a user when they are mentioned in a task or activity feed.
  • We have improved the display of search results for column matches. When searching for columns, the matched results will be displayed and highlighted in the Preview pane.
  • Table Type filter has been added in the Advanced Search, so that users can exclude the temporary or staging tables from search.
  • Now it’s easy to filter the Data assets without a Owner.
  • Database and Schema were added to the Explore menu to enhance data discovery.
  • Custom properties are displayed on the right of the data asset details page.
  • We now display the Domain on the Users page.
  • Supports the sorting of data assets by popularity based on the number of followers and thumbs up as signals.
  • OpenMetadata can now handle metric history for ML models.
  • When configuring the Email settings, the Username and Password fields can be left blank.
  • We now support a test email button on the Email SMTP page.

New Contributors

Full Changelog: https://github.com/open-metadata/OpenMetadata/compare/1.2.5-release...1.3.0-release

OpenMetadata - OpenMetadata 1.3.0-Beta-Release

Published by akash-jain-10 9 months ago

What's Changed

New Contributors

Full Changelog: https://github.com/open-metadata/OpenMetadata/compare/1.2.5-release...1.3.0-beta-release

OpenMetadata - OpenMetadata 1.2.5-Release

Published by dhruvinmaniar123 9 months ago

What's Changed

Full Changelog: https://github.com/open-metadata/OpenMetadata/compare/1.2.4-release...1.2.5-release

OpenMetadata - OpenMetadata 1.2.4-Release

Published by akash-jain-10 10 months ago

What's Changed

Improvements

Bug Fixes

Full Changelog: https://github.com/open-metadata/OpenMetadata/compare/1.2.3-release...1.2.4-release

OpenMetadata - OpenMetadata 1.2.3-Release

Published by akash-jain-10 11 months ago

What's Changed

Full Changelog: https://github.com/open-metadata/OpenMetadata/compare/1.2.2-release...1.2.3-release

OpenMetadata - OpenMetadata 1.2.2-Release

Published by akash-jain-10 11 months ago

What's Changed

Full Changelog: https://github.com/open-metadata/OpenMetadata/compare/1.2.1-release...1.2.2-release

OpenMetadata - OpenMetadata 1.2.0-Release

Published by akash-jain-10 12 months ago

Domains and Data Products

  • Added support for Domains and Data Products.
  • Assets can be added to a Domain, and users can scope their discovery experience to one Domain.
  • Assets can also be added as Data Products in a Domain.

Search Index

  • Elasticsearch or Opensearch connectors can now bring in the search index metadata into OpenMetadata.
  • The connector will populate the index’s mapping, settings, and sample data.

Stored Procedures

  • Added support for Stored Procedures.
  • Snowflake, Redshift, and BigQuery connectors are updated to bring stored procedure metadata into OpenMetadata.
  • The metadata workflow will bring the Stored Procedures and parse their executions to extract lineage information.

Glossary Approval Workflow & Glossary Styling

  • Introduced a glossary approval workflow. An approval workflow is created if Reviewers are added to a glossary.
  • A task is added for reviewers to approve or reject the glossary term. The terms will show up in Draft status.
  • Only the reviewers can approve or reject the term.
  • Conversations are supported to discuss further about the terms.
  • If no reviewer is added, then the glossary terms are approved by default.
  • Introduced styling for glossary terms. Now you can add icons and color code the glossary terms for easy identification.
  • Color coding helps to visually differentiate and identify the data assets, when glossary terms are added to them.

OpenMetadata Browser Extension

  • Updated the Chrome browser extension for OpenMetadata with the new UI.
  • Added support for Databases, Database Schemas, Tables, Dashboards, Charts, Pipelines, and Topics.

Build Automation Applications

  • Added Applications into OpenMetadata, giving users a unique view of processes that can be scheduled and run in the platform.
  • Search Indexing and Data Insights Report have been converted into Applications.
  • UI displays all the available applications, which Admins can add or schedule.
  • We will continue to add new Applications in upcoming releases.

Lineage

  • Performance improvements made for lineage based on the new release of SQLfluff.
  • Added support for UPDATE … FROM Snowflake queries
  • Added column-level lineage support for SELECT * queries

Connectors

  • Greenplum connector is now supported.
  • Couchbase connector is now supported.
  • Azure Data Lake Storage is now supported. (Collate)

Customizable Landing Page

  • Admins can create Personas to group individuals in their company, such as Data Engineers, Data Stewards, or Data Scientists.
  • Admins can customize the landing page for each Persona with a set of supported widgets: Activity Feed, Announcements, Knowledge Center, etc.
  • We will add support for more widgets in upcoming releases.

Knowledge Center (Collate)

  • Backend APIs support creating, editing, and listing knowledge articles (with external links).
  • Knowledge articles and links can be associated with a Domain, Team, or an Entity.
  • UI support to build a Knowledge Center and expand the documentation of your company.

Cost Analysis Report (Collate)

  • The Usage Workflow will now also track how tables are Accessed and Updated.
  • This information will be used in the Data Insights workflow to show the evolution of your used and unused assets and compare them by size.
  • Support has been added for Snowflake, and we will continue to add more sources in upcoming releases.

What's Changed

New Contributors

Full Changelog: https://github.com/open-metadata/OpenMetadata/compare/1.1.7-release...1.2.0-release

Security Scan

We have attached the security scan results for the OpenMetadata Server Docker image. 0 vulnerabilities have been found.

OpenMetadata - OpenMetadata 1.2.0-beta-release

Published by akash-jain-10 about 1 year ago

What's Changed

New Contributors

Full Changelog: https://github.com/open-metadata/OpenMetadata/compare/1.1.7-release...1.2.0-beta-release

OpenMetadata - OpenMetadata 1.1.7-Release

Published by akash-jain-10 about 1 year ago

What's Changed

Full Changelog: https://github.com/open-metadata/OpenMetadata/compare/1.1.6-release...1.1.7-release

OpenMetadata - OpenMetadata 1.1.6-release

Published by akash-jain-10 about 1 year ago

What's Changed

Full Changelog: https://github.com/open-metadata/OpenMetadata/compare/1.1.5-release...1.1.6-release

OpenMetadata - OpenMetadata 1.1.5-Release

Published by dhruvinmaniar123 about 1 year ago

Full Changelog: https://github.com/open-metadata/OpenMetadata/compare/1.1.4-release...1.1.5-release

UI

  • Fixed signup page styling
  • Fixed OIDC Keyset retrieval

Ingestion

  • Fixed BigQuery multi-project ingestion
  • Upgrade Airflow base image to 2.6.3. This fixes Airflow's db migration.
  • Fixed table usage count
  • Add support for Tableau version 3.10
  • Add support for BigQuery datasets named with hyphens

Backend

  • Standardized timestamp format
  • Improved Data Insights error management to improve result accuracy
  • Improved API performance of Data Quality endpoints
OpenMetadata - OpenMetadata 1.1.4-Release

Published by pmbrull about 1 year ago

What's Changed

Full Changelog: https://github.com/open-metadata/OpenMetadata/compare/1.1.3-release...1.1.4-release

OpenMetadata - OpenMetadata 1.1.3-Release

Published by akash-jain-10 about 1 year ago

What's Changed

  • Name should be quoted before hashing in commit

Full Changelog: https://github.com/open-metadata/OpenMetadata/compare/1.1.2-release...1.1.3-release

OpenMetadata - OpenMetadata 1.1.2-Release

Published by akash-jain-10 about 1 year ago

What's New in Release 1.1.2

Data Quality

  • Added support for Postgres version 11.19.
  • Fixed MariaDB time column issues.

Connectors

  • Added JWT authentication support for Trino.
  • Fixed Snowflake connection test.
  • Fixed SageMaker ingestion.
  • Added external table support for BigQuery.

UI Improvements

  • Added Russian language support.
  • Supports Delete functionality for sample data.
  • Improved Schema page UX.
  • Table mentions now show Service, Schema and Database information.
  • Fixed the version history list.

Ingestion

  • Improved performance when ingesting table constraints.

Backend

  • Improved Glossary import validations.
  • Fixed Test Suite migrations and naming.
  • Fixed Classification migration.
  • Deprecated Flyway and using native migrations.
  • Improved Test Suite UI performance.

As always, please do review the upgrade docs to ensure a smooth transition to OpenMetadata 1.1.2.

What's Changed

New Contributors

Full Changelog: https://github.com/open-metadata/OpenMetadata/compare/1.1.1-release...1.1.2-release

OpenMetadata - OpenMetadata 1.1.1-release

Published by dhruvinmaniar123 about 1 year ago

What's Changed

New Contributors

Full Changelog: https://github.com/open-metadata/OpenMetadata/compare/1.1.0-release...1.1.1-release

OpenMetadata - OpenMetadata 1.1.0-release

Published by Anuj359 over 1 year ago

What's Changed

New Contributors

Full Changelog: https://github.com/open-metadata/OpenMetadata/compare/1.0.5-release...1.1.0-release

OpenMetadata - OpenMetadata 1.1.0-beta-release

Published by OnkarVO7 over 1 year ago

THIS IS BETA

Don't use this release to upgrade any existing environments. This is just to test out the new release features and migrations.

What's Changed

New Contributors

Full Changelog: https://github.com/open-metadata/OpenMetadata/compare/1.0.5-release...1.1.0-beta-release

OpenMetadata - OpenMetadata 1.0.5-release

Published by pmbrull over 1 year ago

What's Changed

Full Changelog: https://github.com/open-metadata/OpenMetadata/compare/1.0.4-release...1.0.5-release

OpenMetadata - OpenMetadata 1.0.4-release

Published by akash-jain-10 over 1 year ago

What's Changed

Full Changelog: https://github.com/open-metadata/OpenMetadata/compare/1.0.3-release...1.0.4-release

Package Rankings
Top 3.9% on Pypi.org
Badges
Extracted from project README
Release Stargazers of @open-metadata/OpenMetadata repo
Related Projects
multiwoven

🔥🔥🔥 Open Source Alternative to Hightouch, Census, and RudderStack. Leading Reverse ETL and Custom...

20 Oct 2023 636
crowd.dev

⚡️ The developer data platform to centralize community, product, and customer data

28 Jun 2022 3,292
customeros

CustomerOS is an open source platform that enables you to bring your customer back to the center ...

13 Sep 2022 80
featbit

A feature flags service written in .NET

28 Aug 2022 1,366
openverse

Openverse is a search engine for openly-licensed media. This monorepo includes all application code.

13 Apr 2021 205