odd-platform

Changes

• Fix stack overflow error @damirabdul (#1294)
• Improve Sonar workflow jobs @DementevNikita (#1280)
• New FE test environment @Leshe4ka (#1277)
• React upgraded to v18 @Leshe4ka (#1274)
• Router improvements @Leshe4ka (#1255)
• Metrics bugfix + test enhancement @damirabdul (#1268)

Contributors

@DementevNikita, @Leshe4ka, @damirabdul and alina

The v0.11 release of the ODD Platform introduces the following enhancements:

• Metrics: With the implementation of Open Metrics Specification, ODD Platform introduces enhanced Metrics support. Users can ingest metrics for their data entities or dataset fields and observe last metric values, right within the platform.
• Search explanations: It is crucial to understand why a certain entity has appeared in search results. The ODD Platform can now display all user query entries in a specific entity in the Search API.
• New Dataset Structure UI: The Dataset Structure UI has been reworked and enhanced, to provide a better experience for users in the Data Discovery phase.

As usual, the v0.11 release includes bug fixes and maintenance work that further advance the performance, reliability, and usability of the ODD Platform.

Changes

• Fix LinkedUrlList for DataQualityTests @DementevNikita and @Leshe4ka (#1224)
• Search highlight @damirabdul (#1218)
• Remove unneeded waits for spinner when call "customElement" @shilgam (#1219)
• Azure integration @adyachenkoprov (#1216)

Contributors

@Leshe4ka, @DementevNikita, @adyachenkoprov, @damirabdul and @shilgam

Changes

• Additional links configuration @DementevNikita (#1217)
• Update integrations list @Vixtir (#1215)

Contributors

@DementevNikita and @Vixtir

Changes

• Add ODD IAM Logout handler @DementevNikita (#1214)
• Playwright | Enable the "github" reporter to get automatic failure annotations in GitHub actions @shilgam (#1213)
• Updated LinkedIn link @Leshe4ka (#1212)
• Updated readme @germanosin (#1210)
• Add alerts to activity @damirabdul (#1185)
• Fixed badges @germanosin (#1209)

Contributors

@DementevNikita, @Leshe4ka, @damirabdul, @germanosin and @shilgam

Changes

• Read groups claim from token @damirabdul (#1207)
• Support ODD IAM OAuth provider @DementevNikita (#1205)

Contributors

@DementevNikita and @damirabdul

Changes

• Remove javax annotations from Ingestion API @damirabdul (#1204)
• Remove is_deleted field from API models @damirabdul (#1192)
• Replace external library 'uuid' on the build-in function 'crypto.rand… @FlySpot (#1200)
• Fix concurrent modification error in tests and run tests in parallel (4 threads) @shilgam (#1193)
• Worflow triggering rules @shilgam (#1170)

Contributors

@FlySpot, @damirabdul and @shilgam

The ODD Platform v0.10 adds such features to the mix as:

• Preserving lineage state in query params: This allows users to easily share their lineage graph with other data practitioners, facilitating collaboration and ensuring that everyone has access to the most up-to-date and accurate data.
• Resending failed messages in Data Collaboration: The ODD Platform now attempts to resend messages in Data Collaboration providers (Slack) several times if an error occurs while sending.
• Housekeeping: The ODD Platform is now able to delete empty table partitions that are no longer of use, in order to increase its performance and maintain a clean underlying database.

Changes

• Fix bug with alert ingestion @DementevNikita (#1164)
• Paginate Alert's API for Data Entity @DementevNikita (#1157)
• FE packages updated @Leshe4ka (#1155)

Contributors

@DementevNikita and @Leshe4ka

One of the cornerstones of Data Observability is the ability to alert users about any data infrastructure issues. The v0.9 release of the ODD Platform introduces an enhanced Alert Management feature, which allows to stack alerts of the same type and automatically resolve existing open alerts based on the state of a data source. It also offers the ability to “switch” any entity into the maintenance mode, thus effectively blocking the ability to create new alerts.

The platform’s UI for Lineage has also been improved, which makes the data discovery process more convenient and efficient at the same time.

As usual, the v0.9 release includes bug fixes and maintenance work that further advance the performance, reliability, and usability of the ODD Platform.

We are happy to present ODD Platform v0.8.0!

"The v0.8 release of the ODD Platform is all about making data discovery and collaboration easier for our customers," says German Osin, Chief Product Owner of the ODD Platform. "With the latest iteration of RBAC, we give our customers more fine-grained control over who can access which data by granularly specifying their own data policies. The addition of OpenDataDiscovery profilers, which are powered by machine learning, makes it easy to enrich data with statistics and labels, while filtering data entities by DEG allows our customers to quickly find the data they need, collect stats on and in tables, label data source columns, and more."

"We are especially excited about our new Data Collaboration feature," continues Osin. "This allows our customers to start a conversation about data — be it tables, jobs, or anything else — in Open Data Discovery and continue it in Slack, with a single click of a button. In the future, we plan to store all user messages within the data entity so they can be indexed for quick search on the platform."

As usual, the v0.8 release includes bug fixes and maintenance work that further advance the performance, reliability, and usability of the ODD Platform.

The platform is built in close collaboration between Provectus and the open-source community. We invite the data community to contribute to the project, to help us realize our vision.

Thank you all for your contributions, we appreciate it!

Changes

• Fix FTS for external and internal tags and labels @DementevNikita (#1062)
• Introduce ODD Profiler support @Leshe4ka (#1060)
• Enhance odd-platform-enricher @DementevNikita (#1059)

Contributors

@DementevNikita and @Leshe4ka

Changes

• Markdown switched to light mode. @Leshe4ka (#1047)
• Filter by DEG's and tab showing DEGs in which an entity is @Leshe4ka (#1043)
• Creating metadata bugfix @Leshe4ka (#1041)
• Metadata bugfix @Leshe4ka (#1040)
• Metadata and description bugfix @Leshe4ka (#1039)
• Update integration list @Vixtir (#1034)
• Filter entities without penultimate versions in AlertLocator @DementevNikita (#1036)
• Update integrations list @Vixtir (#1030)
• Rename roles to titles @damirabdul (#1028)
• Owner name displayed firstly in toolbar. @Leshe4ka (#1027)

Contributors

@DementevNikita, @Leshe4ka, @Vixtir and @damirabdul

Changes

• Fix several bugs in Ingestion API @DementevNikita (#1025)
• Bugfix: avoid to creating new search by modifying search query @Leshe4ka (#1023)
• Introduce housekeeping for resolved alerts and outdated searches @DementevNikita (#1021)
• Add more verbose exception message to the resolveType operation in the Ingestion API getByDegOddrn endpoint @DementevNikita (#1018)
• Fix Ingestion getByOddrn endpoint @DementevNikita (#1017)
• Manually change identifier names in the codebase: snake_case -> camelCase @shilgam (#1015)
• RBAC Documentation @DementevNikita (#1012)

Contributors

@DementevNikita, @Leshe4ka, @damirabdul, @shilgam and dabdullin

ODD Platform v0.7 release includes several new and enhanced major features that make metadata easy to handle:

• Role-based Access Control: Having a solid governance in the data catalog is key to maintaining high quality metadata in organizations with multiple users. RBAC restricts the addition and update of data entity documentation to data owners only.
• Ingestion API: Unstable Ingestion API can lead to partial metadata ingestion, which causes poor quality metadata and negatively impacts the data discovery and data observability processes. We have reworked and enhanced ODD’s Ingestion API for greater stability and reliability, and more effective metadata ingestion.
• Various small enhancements have also been introduced to improve user experience:
• The dataset structure page has been significantly improved to provide better user experience when using large tables
• The starting page has been re-built from scratch, and now provides users with detailed statistics on their data
• Primary and sort keys (for wide column databases) are now part of the specification and structure page
• ARM64/M1 support has been enabled

As usual, this release includes bug fixes that further advance the performance and reliability of the ODD Platform.

Changes

• New design for my/upstream/downstream/popular entities in main overview page @nikitosinar (#899)
• Migrate Profile section to Redux toolkit #846 @NelyDavtyan (#859)
• Usage report block should be seen even if users haven't associated themselves with owners #909

Contributors

@NelyDavtyan and @nikitosinar

Changes

• Fix blocking repository update many query @damirabdul (#897)
• Tests overview ui fixes @nikitosinar (#889)
• Migrate DataSet structure section to Redux toolkit #847 @NelyDavtyan (#876)
• Fix wrong dataset field enums count @damirabdul (#893)
• Exclude non updatable fields from update many query @damirabdul (#892)
• Reduced lineage arrows length @Leshe4ka (#886)
• Virtual scrolling disallows from searching content on the structure page #335 @NelyDavtyan (#834)
• Long data source names are not being shown properly in the search filters' UI #813 @NelyDavtyan (#854)
• Sync owner form bug #819 @NelyDavtyan (#837)
• Remove puller support @DementevNikita (#882)
• Fix fetching all data entities in case no owners present in a 'My Object' tab @DementevNikita (#880)

Contributors

@DementevNikita, @Leshe4ka, @NelyDavtyan, @damirabdul and @nikitosinar

New Contributors

• @nikitosinar made their first contribution in https://github.com/opendatadiscovery/odd-platform/pull/889

Full Changelog: https://github.com/opendatadiscovery/odd-platform/compare/0.6.1...0.6.2

Fix a performance issue related to Ingestion API
@DementevNikita, @damirabdul

The ODD Platform v0.6 was enhanced to include such improvements as:

• SLA indicator: a new feature for business users that indicates the quality of datasets, helping them decide if they should use a specific dataset for analytics. The feature increases the trustworthiness of data, and accelerates and facilitates decision making.
• Activity feed: makes it easier to track all events associated with available data entities. Users can easily track and trace how the entities have been changed, and by whom.
• Data entity usage report: enables users to collect and handle statistics on completeness of their data entities. They can track what data entities are populated with tags, descriptions, and more.
• Session storage type selection: gives users the ability to select one of three storage types for session providers: In Memory, PostgreSQL, or Redis. The feature allows for a more flexible, scalable, and reliable development process.
• Alert notifications: a feature that allows sending alerts to Slack or Webhook as they are created or being resolved. The PostgreSQL replication mechanism is used to ensure the delivery of notifications.

The existing features for data classification and data structure visibility have been enhanced, and new abilities were added to the mix. Now, users of the platform can ingest tags and labels coming from Ingestion API, and more easily track such relational constraints as Primary key and Sort key labels.

The v0.6 release also includes UI enhancements and bug fixes that further advance the performance, usability, and reliability of the ODD Platform.

The ODD Platform is a work in progress.
We'd like to sincerely thank everyone who contributed in any way!

Changes

• Remove tags/labels which doesn't have any relations @damirabdul (#861)
• Fix SLA images loading for an ARM platform @DementevNikita (#860)
• Fix initial upsert in the statistics table @DementevNikita (#858)
• Rename severity statuses @damirabdul (#850)
• Usage report implementation @damirabdul (#839)

Contributors

@DementevNikita and @damirabdul