scalardb

Summary

This release has several bug fixes and vulnerability fixes.

Change logs

Bug fixes

• Avoid decrementing outstanding requests counter duplicately (#935)
• Fix CVE-2023-1428 and CVE-2023-32731 (#943)
• Fix CVE-2023-2976 (#954)

Summary

ScalarDB 3.10 includes many new features, improvements, bug fixes, vulnerability fixes, and documentation updates. Please see the following for a list of detailed changes.

Change logs

Enhancements

• Make transaction metadata nullable to support existing databases (#841)
• Add validation for consensus commit mutation operation (#873)
• Add an interface to add conditions in ScanAll (#889)
• Add relational scan for JDBC storage and consensus commit transaction (#900)
• Support condition for transactional operations (#899)
• Add relational scan for JDBC transactions (#925)
• Add integration tests for MariaDB to CI (#916)
• Add join() to DistributedTransactionManager (#944)
• Add table importing feature (#931)

Improvements

• Bump com.azure:azure-cosmos from 4.44.0 to 4.45.0 (#883)
• Add GHA workflow to automatically create pull requests to apply a merged PR changes (#884)
• Bump slackapi/slack-github-action from 1.23.0 to 1.24.0 (#886)
• Bump com.azure:azure-cosmos from 4.45.0 to 4.45.1 (#887)
• Remove untagged container images (#890)
• Bump com.oracle.database.jdbc:ojdbc8 from 21.9.0.0 to 21.10.0.0 (#892)
• Bump dropwizardMetricsVersion from 4.2.18 to 4.2.19 (#895)
• Bump info.picocli:picocli from 4.7.3 to 4.7.4 (#896)
• Handle PRs created by dependabot in auto-pr GHA workflow (#898)
• Bump com.azure:azure-cosmos from 4.45.1 to 4.46.0 (#904)
• Bump com.scalar-labs:scalar-admin from 2.1.0 to 2.1.1 (#910)
• Bump com.scalar-labs:scalar-admin from 2.1.1 to 2.1.2 (#918)
• Bump com.azure:azure-cosmos from 4.46.0 to 4.47.0 (#929)
• Revise Javadoc for exceptions (#930)
• Add integration tests for transaction with relational scan (#913)
• Add a constructor to UnsatisfiedConditionException (#917)
• Improve some toString expressions of Selection Operators (#920)

Bug fixes

• Bump scalar-labs/jre8 from 1.1.12 to 1.1.13 in /server (#881)
• Bump scalar-labs/jre8 from 1.1.12 to 1.1.13 in /schema-loader (#882)
• Use a proper PAT in the GHA workflow for automated PR creation (#885)
• Use the original Git commit author info not a response from GitHub API for the automated PR creation workflow (#891)
• Fix the wrong argument validations in Auto PR workflow (#906)
• Bump scalar-labs/jre8 from 1.1.13 to 1.1.14 in /schema-loader (#903)
• Bump scalar-labs/jre8 from 1.1.13 to 1.1.14 in /server (#902)
• Fix relational scan builder to keep ordering information and so on (#908)
• Bump org.xerial:sqlite-jdbc from 3.41.2.1 to 3.42.0.0 (#888)
• Run git cherry-pick --abort after it fails (#924)
• Records should not be rolled back in rollback() when the transaction state is marked as COMMITTED in 2PC (#909)
• Avoid decrementing outstanding requests counter duplicately (#935)
• Should call abort() when aborting transaction in transaction wrapper classes (#919)
• Fix CVE-2023-1428 and CVE-2023-32731 (#943)
• Fix CVE-2023-2976 (#954)
• Fix utility method to check transactional table metadata (#950)

Documentation

• Change HTML syntax to Markdown for images (master) (#880)
• Improve documents for Handle Exceptions (#897)
• Move Jekyll files for redirecting from the docs site hosted in this repository to the new docs site (#907)
• Revise configuration related documents (#905)
• Update the ScalarDB dependency version to 3.9.1 (#927)
• Add a redirect from the docs site hosted in this repository to the new docs site (#901)
• Revise document for handling exceptions (#932)

Summary

This release has several small improvements and vulnerability fixes.

Change logs

Improvements

• Bump com.scalar-labs:scalar-admin from 2.1.0 to 2.1.1 (#910)
• Improve some toString expressions of Selection Operators (#920)

Bug fixes

• Bump scalar-labs/jre8 from 1.1.12 to 1.1.13 in /server (#881)
• Bump scalar-labs/jre8 from 1.1.12 to 1.1.13 in /schema-loader (#882)
• Bump scalar-labs/jre8 from 1.1.13 to 1.1.14 in /schema-loader (#903)
• Bump scalar-labs/jre8 from 1.1.13 to 1.1.14 in /server (#902)
• Records should not be rolled back in rollback() when the transaction state is marked as COMMITTED in 2PC (#909)
• Should call abort() when aborting transaction in transaction wrapper classes (#919)
• Bump org.xerial:sqlite-jdbc from 3.41.2.1 to 3.42.0.0 (#888)

Documentation

• Change HTML syntax to Markdown for images (3.9) (#879)
• Change version in add-scalardb-to-your-build.md from 3.8.0 to 3.9.0 (#893)
• Revise configuration related documents (#905)

Summary

This release has several small improvements and vulnerability fixes.

Change logs

Improvements

• Bump com.scalar-labs:scalar-admin from 2.1.0 to 2.1.1 (#910)
• Improve some toString expressions of Selection Operators (#920)

Bug fixes

• Bump scalar-labs/jre8 from 1.1.12 to 1.1.13 in /server (#881)
• Bump scalar-labs/jre8 from 1.1.12 to 1.1.13 in /schema-loader (#882)
• Bump scalar-labs/jre8 from 1.1.13 to 1.1.14 in /schema-loader (#903)
• Bump scalar-labs/jre8 from 1.1.13 to 1.1.14 in /server (#902)
• Should call abort() when aborting transaction in transaction wrapper classes (#919)

Documentation

• Change HTML syntax to Markdown for images (3.8) (#878)

Summary

This release has several vulnerability fixes.

Change logs

Bug fixes

• Fix CVE-2023-0361 and CVE-2022-41723 (#834)
• Fix CVE-2022-41721 (#808)
• Fix CVE-2023-0286 (#821)

Summary

ScalarDB 3.9 includes many new features, improvements, bug fixes, vulnerability fixes, and documentation updates. Please see the following for a list of detailed changes.

ScalarDB provides a community edition and an enterprise edition. The community edition is available as open-source software that you can use under the Apache 2.0 License. The enterprise edition includes not only the features of the community edition but also many advanced features. Release notes for the enterprise edition are available under the “Enterprise edition” section. To use the features in the enterprise edition, you must have a license agreement with Scalar Inc.

Community edition

New features

• Added SQLite as a database that ScalarDB supports. (#762)

Improvements

• Added slf4j-simple to the sample application for getting started with ScalarDB. (#835)
• Updated the supported Java version when getting started to Java 8 or higher. (#840)
• Added the Decommissioning state to ScalarDB Server to achieve graceful shutdown. (#851)
• Unified the error-handling behavior of the admin API (e.g., an exception is thrown when dropping a namespace but the namespace has some tables). (#856)

Bug fixes

• Upgraded the integrated Java Runtime Environment (JRE) Docker image to 1.1.11 to fix security issues. CVE-2023-0286 (#821)
• Fixed issues where ScalarDB could not handle the write operation correctly when using the columns prefixed with before_ with Consensus Commit. (#844)

Documentation

• Updated the documentation for transaction APIs and exceptions, the Java API guide, and the two-phase commit transaction guide. (#804, #831)
• Updated the Cosmos DB name from Cosmos DB to Cosmos DB for NoSQL in the supported databases document. (#813)
• Updated the steps to create and configure a Cosmos DB for NoSQL account for ScalarDB in the getting-started guide. (#814)
• Updated the links for ScalarDB Server and Schema Loader in the backup and restore guide. (#810)
• Added how to repair stored procedures when using Cosmos DB for NoSQL in the backup and restore guide. (#825)

Enterprise edition

New features

ScalarDB Cluster

• Integrated the ScalarDB GraphQL and ScalarDB SQL interfaces with ScalarDB Cluster. As of ScalarDB 3.9, you no longer need to deploy dedicated components (i.e., containers) to provide GraphQL and SQL interfaces, respectively. In addition, you can more easily manage a ScalarDB Cluster.
• Added the direct-kubernetes client mode into the Client mode.
• Added the dedicated Schema Loader for ScalarDB Cluster.
• Added ScalarDB GraphQL integration to ScalarDB Cluster.

ScalarDB Analytics with PostgreSQL

• Added ScalarDB Analytics with PostgreSQL. As of ScalarDB 3.9, you can perform analytical queries on data in heterogeneous databases (i.e., RDBMS and NoSQL) managed by ScalarDB transparently by using a PostgreSQL foreign data wrapper (FDW). This feature eliminates the need for you to implement join or aggregation operations in applications, and enables you to easily perform routine or ad-hoc analysis by simply issuing SQL to PostgreSQL.

ScalarDB GraphQL

• Added two-phase commit (2PC) support for ScalarDB GraphQL. As of ScalarDB 3.9, you can more easily guarantee data consistency between GraphQL-based microservices. Adding support for 2PC not only reduces system development costs but also facilitates maintenance.
• Added two mechanisms to rebuild the GraphQL schema after a schema metadata change while the server is running.

ScalarDB SQL

• Added two-phase commit (2PC) support for Spring Data JDBC for ScalarDB. As of ScalarDB 3.9, you can more easily guarantee data consistency between Spring Framework–based microservices. Adding support for 2PC not only reduces system development costs but also facilitates maintenance.
• Enabled ScalarDbRepository#findAll() and ScalarDbRepository#count() in Spring Data JDBC for ScalarDB.
• Published JAR files for ScalarDB SQL CLI to the Maven repository.
• Added the default namespace name configuration scalar.db.sql.default_namespace_name. The value of this configuration is used when a namespace name is not specified in a SQL statement.

Improvements

ScalarDB Cluster

• Added logging to cluster nodes for recording detailed information.
• Unified scalar.db.cluster.node_port and scalar.db.cluster.node.port into scalar.db.cluster.node.port.
• Added an expiration time to the cluster node cache to wait for the current requests to finish before invalidating the cache.
• Added the Decommissioning state to gracefully shut down a cluster node.
• Revised to get the IP addresses of initial members when Kubernetes membership starts.
• Removed an unnecessary throws clause from the Membership#getMembersIpAddresses method.

ScalarDB GraphQL

• Revised to abort transactions that execute a CRUD or transaction operation (i.e., abort and commit) and throw an exception, except if transaction.commit() throws an UnknownTransactionStatusException error.

ScalarDB SQL

• Improved the ScalarDB SQL error mapping table and AbstractSqlJdbcStatement#execute error handling.
• Improved the Spring Data fragment repository class for ScalarDB.
• Updated com.scalar.db.sql.springdata.exception.ScalarDb*Exception to always have a transactionId field value.
• Revised the ScalarDB SQL API entirely.
• Added the Decommissioning state to ScalarDB SQL Server.
• Added a mechanism to expire the SqlSessionFactory cache and close the instance in ScalarDB JDBC.
• Updated the exception translator in Spring Data JDBC for ScalarDB.
• Updated the Metadata service interface to make it more intuitive.

Bug fixes

ScalarDB Cluster

• Upgraded gRPC Health Probe to 0.4.15 to fix security issues. CVE-2022-41721
• Upgraded the integrated JRE Docker image to 1.1.11 to fix security issues. CVE-2023-0286

ScalarDB GraphQL

• Upgraded the integrated JRE Docker image to 1.1.11 to fix security issues. CVE-2023-0286
• Upgraded the integrated JRE Docker image to 1.1.12 to fix security issues. CVE-2023-0361
• Upgraded the graphql-java version to 20.2 to fix security issues. CVE-2023-28867

ScalarDB SQL

• Upgraded gRPC Health Probe to 0.4.15 to fix security issues. CVE-2022-41721
• Upgraded the integrated JRE Docker image to 1.1.11 to fix security issues. CVE-2023-0286
• Fixed an incorrect error message when an error occurred while executing the SELECT operation.
• Upgraded the integrated JRE Docker image to 1.1.12 and gRPC Health Probe to 0.4.17 to fix security issues. CVE-2023-0361 CVE-2022-41723
• Revised to set the finished status of the failed transaction to avoid timeout error in the bidirectional streams flow in Server mode.

Documentation

ScalarDB GraphQL

• Added documentation for how to run ScalarDB GraphQL Server.
• Fixed some minor issues, such as typos and unnecessary statements.

ScalarDB SQL

• Updated the Spring Data JDBC for ScalarDB guide to mention users can use other retry libraries.
• Added a link to the Spring Data JDBC for ScalarDB guide to the README.
• Revised a heading for clarification when users are getting started with ScalarDB SQL.
• Revised the name of the "Spring Data integration with ScalarDB" feature to "Spring Data JDBC for ScalarDB".
• Updated the Javadoc files to match the changes in the release.

Summary

This release has a lot of enhancements, improvements, bug fixes, vulnerability fixes, and document improvements. Please see Change logs for the details.

Change logs

Enhancements

• Remove suspend() from TwoPhaseCommitTransactionManager (#715)
• Update release workflow to push container to ECR of Marketplace (#699)
• Add resume() to DistributedTransactionManager (#717)
• Add several methods to ActiveExpiringMap (#722)
• Allow to use placeholders in configuration values (#770)
• Add configurations for gRPC (#776)

Improvements

• Refactor server integration tests (#663)
• Use toUpperCase() for valueOf() of enum type (#666)
• Use ServiceLoader to load storage and transaction modules (#661)
• Improve error messages in OperationChecker (#667)
• Add logging for UnknownTransactionStatusException in Scalar DB Server (#669)
• Refactor JdbcAdminTest (#670)
• Refactor CosmosAdminTest (#671)
• Rename endpoint override config in Dynamo (#672)
• Enable EI_EXPOSE_REP and EI_EXPOSE_REP2 in SpotBugs (#668)
• Enable parallel/async commit by default (#676)
• Add more checks for conditions for Dynamo and Cosmos (#675)
• Refactor schema loader integration tests (#673)
• Delete setDriver() for BasicDataSource in JDBC (#682)
• Use DistributedTransactionAdmin for transaction tables in SchemaOperator (#684)
• Delete licenses (#686)
• Suppress exception thrown when deleting scalable target and scaling policy when using DynamoDB local. (#683)
• Upgrade Spotless Gradle Plugin (#694)
• Fix typo in admin integration test classes (#695)
• Move integration test base code to subproject (#697)
• Refactor ServiceLoader related classes (#698)
• Make the ConsensusCommitAdmin only return transaction table (#702)
• Add external Scalar DB property file support to getting-started application (#701)
• Refactor ConsensusCommitAdmin unit test (#704)
• In the Cosmos DB job of the CI, add a step to clean up gradle daemon logs file (#705)
• Disable include metadata tests if external server used in server integration tests (#711)
• Refactor consensus commit specific integration tests (#713)
• Refactor two-phase commit transaction integration tests (#712)
• Small integration test modifications for the resume feature (#724)
• Should throw IllegalStateException when transaction is not found and already exists (#718)
• Add transaction state management (#719)
• Separate active transaction management from AbstractDistributedTransactionManager and AbstractTwoPhaseCommitTransactionManager (#723)
• Fix dependency and spotbugs issues for >JDK 8 (#728)
• Fix two unit tests of OperationChecker (#731)
• Throw an exception when putting an empty value to a secondary index column in Dynamo DB (#729)
• Make JDK11+ use the same google java format version as JDK8 uses (#734)
• Make ParallelExecutor workers daemon thread (#733)
• Move vuln check to a separate daily workflow (#737)
• Conduct daily vulnerability checks on releases not only on main branch (#741)
• Should drop namespace only when no tables are in the namespace in Schema Loader (#740)
• Move common classes (#744)
• Throw TransactionNotFoundException when resuming a transaction but it's not found (#746)
• Post the vulnerability check result to Slack channel (#745)
• Deprecate getPartitionKey() and getClusteringKey() of Result (#750)
• Remove usages of getPartitionKey() and getClusteringKey() of Result from integration tests (#751)
• Validate contact points size in XxxxConfig to avoid "java.lang.ArrayIndexOutOfBoundsException: 0" (#752)
• Make integration test bases for transaction extendable (#754)
• Refactor server integration tests (#755)
• Small modifications for Javadoc and comments (#756)
• Should close admin in afterEach() in the repair integration tests (#767)
• build(fix): bump-up protobufVersion to 3.21.12 (latest) (#760)
• Refactor prepare logic (#771)
• Update Scheduled Vulnerability Check workflow (#749)
• Refactor validation logic (#775)
• Add transaction decorator (#781)
• Revisit commit and abort state logic (#783)
• Should synchronize in active transaction management (#785)
• Allow commit() and rollback() to be called in parallel in all coordinator/participant processes in 2PC transactions (#786)
• Load admins lazily in Schema Loader (#792)
• Rename Scalar DB to ScalarDB (#795)
• DynamoDB supports up to 100 actions per transaction (#796)
• Revert async commit default value (#797)

Bug fixes

• Should support --config= and -c= styles for configuration file option in Schema Loader (#690)
• Fix CVE-2021-3999, CVE-2022-1586 and CVE-2022-1587 (#700)
• Commit/rollback records should always be done for all records even on error (#680)
• Should use partition key and clustering key from result in recovery (#681)
• Fix typo in integration-test/archive.gradle (#706)
• Fix CVE-2022-27664 (#716)
• Should check the value range of BigIntColumn (#720)
• Fix CVE-2022-32149 (#727)
• Fix CVE-2022-42003 and CVE-2022-42004 (#726)
• Fix dependency errors for Maven projects (#735)
• Fix typos in exception messages (#714)
• Update protobuf and grpc to fix CVE-2022-3171 (#738)
• Fix CVE-2022-40151 and CVE-2022-40152 (#743)
• Upgrade docker image version to fix CVE-2021-46848 (#766)
• Fix CVE-2022-21363 and CVE-2021-2471 (#773)
• Fix CVE-2022-41946 (#774)
• fix: DynamoAdmin.namespaceExists to check full namespace (not prefix) (#782)
• Should care about ScanWithIndex in ScalarDbUtils.copyAndSetTargetToIfNot() (#793)
• Fix CVE-2022-42898 (#794)

Documentation

• Remove readthedocs (#664)
• Move Schema Loader document to docs directory (#665)
• Fix Javadoc warnings (#685)
• Update the Scalar DB supported database matrix with Scalar DB 3.7 (#703)
• Add description of limitations of data types (#721)
• Fix the broken link in the backup document (#725)
• Remove unnecessary key definition from API guide (#742)
• Add note for Put/Delete operation in the API guide (#747)
• Add ScalarDB Server's license description in the README (#753)
• docs: Clearer description on when to use online backups (#761)
• Add description how to build docker image of ScalarDB Server (#779)
• Update slides links (#789)

Bug fixes

• Use column name alias for projection expressions in the DynamoDB adapter (#264)

Enhancements

• Add JDBC adapter of Scalar DB (#133)
• Support table creation of Scalar DB on JDBC in schema-tool (#152)
• Enable dynamo backup (#159)
• Support secondary indexes on DynamoDB (#147)
• Set DynamoDB secondary index (#142)
• Set Cosmos secondary index (#141)
• Set Cassandra secondary index (#140)
• Support secondary indexes on Cosmos DB (#146)
• Extend TableMetadata for secondary indexes (#144)

Improvements

• Add integration tests for Get and Scan with a Cassandra secondary index (#139)
• Guard C* get and scan with a secondary index (#145)
• Check operations if they are given with correct keys (#136)

Bug fixes

• Remove prefix from indexes of DynamoDB (#161)
• Fix not to return unexpected results (#165)
• Fix to set RU for DynamoDB (#154)
• Fix snapshot scan bug (#151)
• Fix pip install failure in CI (#149)
• Fix Snapshot to handle deleteSet properly when a delete operation is mixed with other operations (#138)

Documentation

• Add more documentation including limitations in the Scalar DB docs (#162)
• Add a guide on the requirements in the underlining databases of Scalar DB (#156)
• Create a backup and restore guide (#148)

Bug fixes

• Use com.google.inject.Inject instead of javax.inject.Inject (#200)
• make attributes for secondary indexes (#189)
• Fix the bug where the error occurs when executing a Put operation without values in the JDBC adapter (#180)
• Fix the broken TLA+ link (#175)
• Wrong index name when a namespace prefix is not specified in the schema tool for DynamoDB (#173)
• Update pip URL (#171)

Bug fixes

• Fix unclosed scanner (#134)
• Fix inconsistent read bug (#135)

Enhancements

• Support DynamoDB auto-scaling in schema-tool (#116)
• Support partial update in Cosmos DB adapter (#118)
• Add namespace prefix (#123)
• Make isolation level and serializable strategy configurable with properties (#128)

Bug fixes

• FIx documentation bugs
• FIx Gradle bugs
• Fix schema-tool bugs

Enhancements

• Add DynamoDB support
• Update the getting started guide

Bug fixes

• Fix bugs in the schema tool

Enhancements

• Minor enhancements in the schema tool and the getting started document

Bug fixes

• Fix cast in Cosmos DB adapter (43cf19c6f3f4328492a06293052e7ee590953834)

Enhancements

• Add setters and getters for namespace and table name in DistributedStorage, DistributedTransaction, and DistributedTransactionManager classes.

Enhancements

• Add Cosmos DB support

Bug fixes

• Fix bugs in Serializable
• Fix bugs in Snapshot

Enhancements

• Introduce Serializable strategy
• Add a new Serializable strategy named Extra-Read
• Add TLA+ verification