Open Standard for Metadata. A Single place to Discover, Collaborate and Get your data right.
APACHE-2.0 License
Published by akash-jain-10 over 1 year ago
Full Changelog: https://github.com/open-metadata/OpenMetadata/compare/1.0.2-release...1.0.3-release
Published by Ashish8689 over 1 year ago
Full Changelog: https://github.com/open-metadata/OpenMetadata/compare/1.0.1-release...1.0.2-release
Published by pmbrull over 1 year ago
Bring parenthesis to Entity Name from https://github.com/open-metadata/OpenMetadata/pull/11580
Published by Ashish8689 over 1 year ago
Full Changelog: https://github.com/open-metadata/OpenMetadata/compare/1.0.0-release...1.0.1-release
Published by NiharDoshi99 over 1 year ago
Full Changelog: https://github.com/open-metadata/OpenMetadata/compare/0.13.3-release...1.0.0-release
Published by ulixius9 over 1 year ago
sqllineage
with openmetadata-sqllineage
by @nahuelverdugo in https://github.com/open-metadata/OpenMetadata/pull/9800
name
instead of displayName
#10136 by @ShaileshParmar11 in https://github.com/open-metadata/OpenMetadata/pull/10142
owner
by @TeddyCr in https://github.com/open-metadata/OpenMetadata/pull/10497
user
by @Sachin-chaurasiya in https://github.com/open-metadata/OpenMetadata/pull/10517
get_user_by_email
function by @TeddyCr in https://github.com/open-metadata/OpenMetadata/pull/10646
dataQuality/<specificity>
by @TeddyCr in https://github.com/open-metadata/OpenMetadata/pull/10970
Full Changelog: https://github.com/open-metadata/OpenMetadata/compare/0.13.3-release...1.0.0-beta-release
Published by ulixius9 over 1 year ago
This is a release from our WIP towards 1.0. We wanted to bring out some new features and fixes to the community (e.g., Glossary), for you to test things out and give feedback.
This release IS NOT INTENDED for any PROD environment or to be used to update any existing installation. Please, use an isolated run for these binaries. The docker quickstart would be the easiest way to handle this.
The OpenMetadata community will continue to patch things up and improve testing and stability for 1.0.
sqllineage
with openmetadata-sqllineage
by @nahuelverdugo in https://github.com/open-metadata/OpenMetadata/pull/9800
name
instead of displayName
#10136 by @ShaileshParmar11 in https://github.com/open-metadata/OpenMetadata/pull/10142
owner
by @TeddyCr in https://github.com/open-metadata/OpenMetadata/pull/10497
user
by @Sachin-chaurasiya in https://github.com/open-metadata/OpenMetadata/pull/10517
get_user_by_email
function by @TeddyCr in https://github.com/open-metadata/OpenMetadata/pull/10646
Full Changelog: https://github.com/open-metadata/OpenMetadata/compare/0.13.2-release...1.0.0-alpha-release
Published by Sachin-chaurasiya over 1 year ago
Full Changelog: https://github.com/open-metadata/OpenMetadata/compare/0.13.2-release...0.13.3-release
Published by aniketkatkar97 over 1 year ago
Full Changelog: https://github.com/open-metadata/OpenMetadata/compare/0.13.1-release...0.13.2-release
Published by pmbrull over 1 year ago
Full Changelog: https://github.com/open-metadata/OpenMetadata/compare/0.13.1-release...0.13.2-beta-release
Published by chirag-madlani almost 2 years ago
Full Changelog: https://github.com/open-metadata/OpenMetadata/compare/0.13.0-release...0.13.1-release
Published by ayush-shah almost 2 years ago
Data Insights, a game-changing feature, has been introduced that transforms the passive approach to data into a collaborative project toward improved data culture. Data Insights aims to provide a single-pane view of all the key metrics to best reflect the state of your data. OpenMetadata gathers all the metrics related to the metadata that you are extracting. The entities created the types of entities and the data evolution over a period of time. Based on these metrics, we provide analytics to assess the gathered data.
Admins can define the Key Performance Indicators (KPIs) and set goals within OpenMetadata to work towards better documentation, ownership, and tiering.
These goals are based on entities and driven to achieve targets within a specified time. For example, Admins can set goals to have at least 60% of the entities documented, owned and tiered by the end of Q1 2023.
The data insights dashboard provides a quick glance at aspects like data ownership, description coverage, data tiering, and so on. Teams can view a timeseries report to track progress and monitor the health of your data. In addition to the metrics on data, Admins can view the aggregated user activity and get insights into user engagement and user growth. Admins can check for Daily active users and know how the tool is being used.
The Data Insights Report is emailed weekly, so that teams can assess their performance relative to the KPIs set at an organizational level to improve data culture on an ongoing basis.
The lineage UI has been transformed to enhance user experience. Users can get a holistic view of an entity from the Lineage tab. When an entity is selected, the UI displays end-to-end lineage traceability for the table and column levels. Just search for an entity and expand the graph to unfold lineage. It’ll display the upstream and downstream for each node.
The Lineage Tab UI supports two-finger scrolling to zoom in or zoom out.
With the OpenMetadata UI, users can now create and deploy profiling workflows for the Datalake connector, which supports AWS S3 and GCS. In the next release, we’ll add the support to run tests as well as covering Azure ADLS.
Security
With the addition of the LDAP SSO in the current release, OpenMetadata supports nine SSOs, which includes Google, Azure, Okta, OneLogin, Auth0, Amazon Cognito, Keycloak, and custom OIDC. In the 0.12.1 release, support was added for basic authentication to sign up using a Username/Password.
OpenMetadata Roles and Policies treat Bots as a special user with access to all the APIs and entities, just like an Admin. Bots have been in use for ingestion to extract metadata, as well as for data profiler and so on. In the 0.13 release, we’ve created multiple bots to serve different scenarios. For example, Ingestion Bot, Lineage Bot, Data Quality and Profiler Bot.
Given the varying roles for specific bots, the policies and access control for bots has been redefined. Now, Bots can have their own policies. For example, the Ingestion Bot can create and update entities. The Profiler Bot can only update the profile of a table, and not have policies for any other entities or access to update table description, etc.
OpenMetadata already supports advanced search syntax. Since it is syntax-driven, it’s not easy to use for all except advanced users. In the current release, a Syntax Editor has been introduced for advanced search with And/Or conditions that help discover assets quickly. A huge thank you to Cristian Osiac from Bloomberg for helping with this feature.
In the 0.13 release, we have introduced four new connectors:
Domo, a cloud-based dashboard service. The Domo Business Cloud is a low-code data app platform that takes the power of BI to the next level by combining all your data and putting it to work across any business process or workflow. OpenMetadata supports Domo as a Database, Dashboard, as well as a Pipeline service.
Hacktoberfest has been a complete success at the OpenMetadata community, with three connectors being developed as part of the event:
AWS SageMaker, a fully managed machine learning service, where data scientists and developers can quickly and easily build and train machine learning models, and then directly deploy them into a production-ready hosted environment.
AWS Kinesis, a cloud-based messaging service that allows real-time processing of streaming large amounts of data per second.
AWS QuickSight, a cloud-scale business intelligence (BI) service that allows everyone in the organization to understand the data by asking questions in natural language, exploring through interactive dashboards, or automatically looking for patterns and outliers powered by machine learning.
Big thanks and congratulations to Michael Zhou for developing AWS QuickSight and to Tushar Mittal for adding both AWS SageMaker and AWS Kinesis.
Several improvements have been made to the ingestion framework. In the 0.12.1 release, we shipped the ability to add a custom service type. Users can now develop their own connector and ingest it as with any other supported service! If you’d like to learn more about that, you can check out the demo!
Major enhancements have been made to how data is extracted from Kafka and Redpanda Messaging services. Previously, OpenMetadata extracted all the Topics in the messaging queue and also connected to the Schema Registry to get the Schemas. These schemas were taken as one payload and published to OpenMetadata. We now parse Avro and Protobuf Schemas to extract the fields. Now, users can document each of these fields within a schema by adding descriptions and tags. Users can search based on the fields in the Schema of a Topic.
Soft deleted entities can be restored. Currently, only the ML Models are not supported.
Soft deleted teams can be restored. When restoring a soft deleted parent team, the child teams will not be restored by default.
Published by mohityadav766 almost 2 years ago
Full Changelog: https://github.com/open-metadata/OpenMetadata/compare/0.12.2-preview...0.12.3-release
Published by akash-jain-10 almost 2 years ago
1.3.5
to 2.0.0
by @Sachin-chaurasiya in https://github.com/open-metadata/OpenMetadata/pull/8277
py-cli-e2e-tests
by @ulixius9 in https://github.com/open-metadata/OpenMetadata/pull/8662
Full Changelog: https://github.com/open-metadata/OpenMetadata/compare/0.12.2-release...0.13.0-preview
Published by Vj-L about 2 years ago
Full Changelog: https://github.com/open-metadata/OpenMetadata/compare/0.12.2-preview...0.12.2-release
Published by akash-jain-10 about 2 years ago
TBD - Preview Release
Published by akash-jain-10 about 2 years ago
Full Changelog: https://github.com/open-metadata/OpenMetadata/compare/0.12.0-release...0.12.1-release
Published by Vj-L about 2 years ago
You can read the Release Blog here
or watch an awesome video showing the new features!
Prior releases supported a flat hierarchy of just Teams and Users. In 0.12, support has been added for the entire organizational hierarchy with Business Unit, Division, Department, and Groups. An organization from small to very large can now be modeled in OpenMetadata with this feature.
Access Control functionality has been revamped to support many use cases that were not possible before. Previously, a Role contained a single Policy, which consisted of simple Rules to Allow/Not Allow. The advanced rule configuration in the 0.12 release allows users to build more expressive rules using conditions.
OpenMetadata began support for Data Quality in the 0.10 release, and support was added for publishing Great Expectations results in the 0.11 release. Our goal with OpenMetadata is to define metadata standards for all things data and in this release, we are standardizing Tests and Data Quality metadata. Data Quality Tests can be expressed in JSON schema and now these tests can be added dynamically using the Test Definitions API. We have also added a custom SQL data quality test that allows you to write your data quality tests using SQL statements.
An interactive dashboard helps to visualize and explore the data from the Data Profiler. You can explore how your data is changing over time, and identify data drifts using this dashboard. You can also see how data quality is changing by looking at how tests are doing over time. What is even better is, that you can explore this at both the table level or drill down to each column level going back up to 60 days.
The UI supports the detailed exploration of data quality tests, and users can drill down for the details of the test results present in a time series fashion. Tests can be added easily from the Profiler tab in the UI, both at the Table and Column levels. The UI provides a one-glance update on the metrics with a summary of data quality at the Table and Column levels.
Informing users about upcoming changes to the data is a big challenge. In most organizations, a team sends an email well in advance about the change. But no one reads/tracks them and finally, when the change is done, many users are unprepared to handle it.
With Announcements, you can now inform your entire team of all the upcoming events and changes, such as deprecation, deletion, or schema changes. These announcements can be scheduled with a start date and an end date. All the users following your data are not only notified in Activity Feeds but a banner is also shown on the data asset details page for users to discover (or be reminded of) the announcement.
In 0.12, we’ve also streamlined the Notifications menu with two separate tabs for Tasks and Mentions, that’ll display only the recent notifications. You can always navigate to your User Profile page to view more activities.
Users can get timely updates about the metadata change events for all entities through APIs using webhooks. The webhook integration with Slack has been further improved in this release.
OpenMetadata also supports webhook integration to Microsoft Teams, just as it supports Slack. Users can choose to receive notifications for only the required entities by using event filters based on when an entity is created, updated, or deleted.
In the 0.11 release, a request to add or update descriptions for data assets could be converted to a Task. In the 0.12 release, Tasks can be created based on requests to create or update tags. Also, a glossary term approval workflow can be converted to a Task.
In 0.12, we have completely revamped how that secret is stored, accessed, and by whom; by introducing a Secrets Manager Interface to communicate with any Key Management Store. The KMS will mediate between any OpenMetadata internal requirement and sensitive information. That way, users can choose to use the underlying database as KMS, or any external system. The OpenMetadata community has already added support for AWS Key Management Service and AWS SSM.
New connectors are an essential part of every release in OpenMetadata. We are introducing four new connectors in this release:
We’ve enhanced the performance of workflows by having a separate workflow for Lineage and Usage. By using two workflows for computing specific pieces of information, we can effectively filter down the queries to extract lineage.
During table usage ingestion, the tables retrieved successfully will be cached, so that there is no need to repeat the same calls multiple times as many queries would be referencing the same tables.
Usage queries have been optimized.
A result limit has been added to Usage queries.
The OpenMetadata Settings dropdown menu has been transformed into a single, centralized Settings page for added convenience in viewing all the available options. The Global Settings comprises setting options for Team Members, Access based on Roles and Policies, Services, Data Quality, Collaboration, Custom Attributes, and Integrations for webhooks and bots. Admins can view or update settings for various services like Slack, MS Teams, Webhooks, etc from the Global Settings page.
The major UI UX improvements have been done around Roles and Policies and a Global Settings page. Quite a lot of tweaks have been made to the UI to improve the UX.
When creating a new user or when a user is registering for the first time, the dropdown menu for Teams now displays an option to ‘Show All’ teams. Previously, we supported the display of only the first 10 teams. An option has also been provided to search and filter.
UI improvements have been made on the Schema, Service, and Database details pages.
Manage Tab has been replaced with the manage button on the UI.
Full Changelog: https://github.com/open-metadata/OpenMetadata/compare/0.11.0-release...0.12.0-release
Published by akash-jain-10 about 2 years ago
Full Changelog: https://github.com/open-metadata/OpenMetadata/compare/0.11.4-release...0.11.5-release
Published by akash-jain-10 about 2 years ago
Full Changelog: https://github.com/open-metadata/OpenMetadata/compare/0.11.3-release...0.11.4-release