ydb

Released on January 12, 2024. To update to version 2.8.0, select the Downloads section.

Features:

• Added configuration management commands for the cluster ydb admin config and ydb admin volatile-config.

• Added support for loading PostgreSQL-compatible data types by ydb import file csv|tsv|json command. Only for row-oriented tables.

• Added support for directory load from an S3-compatible storage in the ydb import s3 command. Currently only available on Linux and Mac OS.

• Added support for outputting the results of ydb table query execute, ydb yql and ydb scripting yql commands in the Apache Parquet format.

• In the ydb workload commands, the --executer option has been added, which allows to specify which type of queries to use.

• Added a column with median benchmark execution time in the statistics table of the ydb workload clickbench command.

• (Experimental) Added the generic request type to the ydb table query execute command, allowing to perform DDL and DML operations, return with arbitrarily-sized results and support for MVCC. The command uses an experimental API, compatibility is not guaranteed.

• (Experimental) In the ydb table query explain command, the --collect-diagnostics option has been added to collect query diagnostics and save it to a file. The command uses an experimental API, compatibility is not guaranteed.

Bug fixes:

• Fixed an error displaying tables in pretty format with Unicode characters.

• Fixed an error substituting the wrong primary key in the command ydb tools pg-convert.

Full Changelog: https://github.com/ydb-platform/ydb/compare/CLI_2.7.0...CLI_2.8.0

Released on October 23, 2023. To update to version 2.7.0, select the Downloads section.

Features:

• Added the ydb tools pg-convert command, which prepares a dump obtained by the pg_dump utility for loading into the YDB postgres-compatible layer.

• Added the ydb workload query load testing command, which loads the database with script execution queries in multiple threads.

• Added a command ydb scheme permissions list to list permissions.

• In the commands ydb table query execute, ydb table query explain, ydb yql, and ydb scripting yql, the --flame-graph option has been added, specifying the path to the file in which you need to save the visualization of query execution statistics.

• Special commands in the interactive query execution mode are now case-insensitive.

• Added validation for special commands and their parameters.

• Added table reading in the scenario with transactions in the command ydb workload transfer topic-to-table run.

• Added the --commit-messages option to the command ydb workload transfer topic-to-table run, specifying the number of messages in a single transaction.

• Added the options --only-table-in-tx and --only-topic-in-tx in the command ydb workload transfer topic-to-table run, specifying restrictions on the types of queries in a single transaction.

• Added new columns Select time and Upsert time in the statistics table in the command ydb workload transfer topic-to-table run.

Bug fixes:

• Fixed an error when loading an empty JSON list by commands: ydb table query execute, ydb scripting yql and ydb yql.

Full Changelog: https://github.com/ydb-platform/ydb/compare/CLI_2.6.0...CLI_2.7.0

Released on September 7, 2023. To update to version 2.6.0, select the Downloads section.

Features:

• Added --path option to ydb workload tpch run, which contains the path to the directory with tables created by the ydb workload tpch init command.

• Added ydb workload transfer topic-to-table run command, which loads the database with read requests from topics and write requests to the table.

• Added the option --consumer-prefix in the commands ydb workload topic init, ydb workload topic run read|full, specifying prefixes of consumer names.

• Added the --partition-ids option in the ydb topic read command, which specifies a comma-separated list of topic partition identifiers to read from.

• Added support for CSV and TSV parameter formats in YQL query execution commands.

• The interactive mode of query execution has been redesigned. Added new interactive mode specific commands: SET, EXPLAIN, EXPLAIN AST. Added saving history between CLI launches and auto-completion of YQL queries.

• Added the command ydb config info, which outputs the current connection parameters without connecting to the database.

• Added the command ydb workload kv run mixed, which loads the database with write and read requests.

• The --percentile option in the ydb workload topic run write|read|full commands can now take floating point values.

• The default values for the --seconds and --warmup options in the ydb workload topic run write|read|full commands have been increased to 60 seconds and 5 seconds, respectively.

• Changed the default value for the --supported-codecs option to RAW in the ydb topic create and ydb topic consumer add commands.

Bug fixes:

• Fixed string loss when loading with the ydb import file json command.

• Fixed ignored statistics during the warm-up of commands ydb workload topic run write|read|full.

• Fixed incomplete statistics output in the ydb scripting yql and ydb yql commands.

• Fixed incorrect output of progress bar in ydb tools dump and ydb tools restore commands.

• Fixed loading large files with the header in the ydb import file csv|tsv command.

• Fixed hanging of the ydb tools restore --import-data command.

• Fixed error Unknown value Rejected when executing the ydb operation list build index command.

Full Changelog: https://github.com/ydb-platform/ydb/compare/CLI_2.5.0...CLI_2.6.0

Bugfixes:

• Fix InFlightTxCount become negative during split/merge
• Allow compatibility between non-stable versions of different applications
• Various stability improvements (segfault, handle balancer and boot queue events in StateInit, data race when reporting issue counters, memory leak in TRopeArena)
• CPU overutilization in case of numerous topic consumers
• Fix import multiple converted csv columns
• Fix mem table merge when writing over committed deltas in tablets
• Check errors during read_table_scan and export
• Throttle datashard log messages

Features:

• Enable ANSI settings for generic query
• DML logging
• Request unit trailing header for ScanQuery/ScanYql sctipts
• Grpc KV api
• Change default metering mode of topics to request units
• Pqv0 support on server side
• Allow to maintain unavailable nodes in tablets checker
• Allow to reboot unavailable node
• Add Lazy Histogram for grpc request

Functionality:

• (Experimental) Implemented visibility of own changes. With this feature enabled you can read changed values from the current transaction, which has not been committed yet. This functionality also allows multiple modifying operations in one transaction on a table with secondary indexes. To enable this feature add enable_kqp_immediate_effects: true under table_service_config section into configuration file.
• (Experimental) Implemented read iterators. This feature allows to separate reads and computations. Read iterators allow datashards to increase read queries throughput. To enable this feature add enable_kqp_data_query_source_read: true under table_service_config section into configuration file.

Embedded UI:

• Navigation improvements:
  • Diagnostics and Development mode switches are moved to the left panel.
  • Every page has breadcrumbs.
  • Storage groups and nodes info are moved from left buttons to tabs on the database page.
• Query history and saved queries are moved to tabs over the text editor area in query editor.
• Info tab for scheme objects displays parameters using terms from CREATE or ALTER statements.
• Added column tables support.

Performance:

• For scan queries, you can now effectively search for individual rows using a primary key or secondary indexes. This can bring you a substantial performance gain in many cases. Similarly to regular queries, you need to explicitly specify its name in the query text using the VIEW keyword to use a secondary index.
• (Experimental) Added an option to give control of the system tablets of the database (SchemeShard, Coordinators, Mediators, SysViewProcessor) to its own Hive instead of the root Hive, and do so immediately upon creating a new database. Without this flag, the system tablets of the new database are created in the root Hive, which can negatively impact its load. Enabling this flag makes databases completely isolated in terms of load, that may be particularly relevant for installations, consisting from a roughly hundred or more databases. To enable this feature add alter_database_create_hive_first: true under feature_flags section into configuration file.

Bug fixes:

• Fixed a bug in the autoconfiguration of the actor system, resulting in all the load being placed on the system pool.
• Fixed a bug that caused full scanning when searching by prefix of the primary key using LIKE.
• Fixed bugs when interacting with datashard followers.
• Fixed bugs when working with memory in column tables.
• Fixed a bug in processing conditions for immediate transactions.
• Fixed a bug in the operation of iterator-based reads on datasharrd followers.
• Fixed a bug that caused cascading reinstallation of data delivery sessions to asynchronous indexes.
• Fixed bugs in the optimizer for scanning queries.
• Fixed a bug in the incorrect calculation of storage consumption by Hive after expanding the database.
• Fixed a bug that caused operations to hang on non-existent iterators.
• Fixed bugs when reading a range on a NOT NULL column.
• Fixed a bug in the replication of VDisks.
• Fixed a bug in the work of the run_interval option in TTL.

Full Changelog: https://github.com/ydb-platform/ydb/compare/23.1.26...23.2.12

Released on June 20, 2023. To update to version 2.5.0, select the Downloads section.

Features

• For the ydb import file command, a parameter --timeout has been added that specifies the time within which the operation should be performed on the server.
• Added a progress bar in commands ydb scheme rmdir --recursive and ydb import file.
• Added the command ydb workload kv run read-rows, which loads the database with requests to read rows using a new experimental API call ReadRows (implemented only in the main branch), which performs faster key reading than select.
• New parameters --warmup-time, --percentile, --topic have been added to the ydb workload topic, setting the test warm-up time, the percentile in the statistics output and the topic name, respectively.
• Added the ydb workload tpch command to run the TPC-H benchmark.
• Added the --ordered flag in the command ydb tools dump, which preserves the order by primary key in tables.

Performance

• The data loading speed in the ydb import file command has been increased by adding parallel loading. The number of threads is set by the new parameter --threads.
• A performance of the ydb import file json command has been increased by reducing the number of data copies.

Full Changelog: https://github.com/ydb-platform/ydb/compare/CLI_2.4.0...CLI_2.5.0

Released on May 24, 2023. To update to version 2.4.0, select the Downloads section.

Features

• Added the ability to upload multiple files in parallel with the command ydb import file.
• Added support for deleting column tables for the command ydb scheme rmdir --recursive.
• Improved stability of the command ydb workload topic.

Full Changelog: https://github.com/ydb-platform/ydb/compare/CLI_2.3.0...CLI_2.4.0

Released on May 5, 2023. To update to version 2.3.0, select the Downloads section.

What's new:

• Added interactive query execution mode. It can be launched using ydb yql command without arguments. This mode is experimental and is a subject to change.
• Added ydb table index rename command for atomic secondary index replacement or renaming.
• Added ydb workload topic command section that allows to run a workload of writes and reads to topics.
• Added --recursive option for ydb scheme rmdir command that allows to remove a directory recursively with all its content.
• Added topic and coordination node support for ydb scheme describe command.
• Added --commit option for ydb topic consumer command to commit offset for consumer.
• Added --columns option for ydb import file csv|tsv command to list column names in, instead of placing it into file header.
• Added --newline-delimited option for ydb import file csv|tsv command that confirms that there is no newline characters inside records which allows to read from several sections of a file simultaneously.

Bug fixes:

• Fixed a bug that caused executing the ydb import file command to consume too much memory and CPU.

Full Changelog: https://github.com/ydb-platform/ydb/compare/CLI_2.2.0...CLI_2.3.0

Released on March 3, 2023. To update to version 2.2.0, select the Downloads section.

What's new:

• Fixed the error that didn't allow specifying supported compression algorithms when adding a topic consumer.
• Added support for streaming YQL scripts and queries based on parameters transferred via stdin.
• YQL query parameter values can now be transferred from a file.
• Password input requests are now output to stderr instead of stdout.
• You can now save the root CA certificate path in a profile.
• Added a global parameter named --profile-file to use the specified file as storage for profile settings.
• Added a new type of load testing: ydb workload clickbench.

Full Changelog: https://github.com/ydb-platform/ydb/compare/CLI_2.1.1...CLI_2.2.0

Released on December 30, 2022. To update to version 2.1.1, select the Downloads section.

Improvements:

• Added support for the --stats parameter of the ydb scheme describe command for column-oriented tables.
• Added support for Parquet files to enable their import with the ydb import command.
• Added support for additional logging and retries for the ydb import command.

Full Changelog: https://github.com/ydb-platform/ydb/compare/CLI_2.1.0...CLI_2.1.1

Released on November 18, 2022. To update to version 2.1.0, select the Downloads section.

What's new:

• You can now create a profile non-interactively.
• Added the ydb config profile update and ydb config profile replace commands to update and replace profiles, respectively.
• Added the -1 parameter for the ydb scheme ls command to enable output of a single object per row.
• You can now save the IAM service URL in a profile.
• Added support for username and password-based authentication without specifying the password.
• Added support for AWS profiles in the ydb export s3 command.
• You can now create profiles using stdin. For example, you can pass the YC CLI yc ydb database get information command output to the ydb config profile create command input.

Bug fixes:

• Fixed the error when request results were output in JSON-array format incorrectly if they included multiple server responses.
• Fixed the error that disabled profile updates so that an incorrect profile was used.

Full Changelog: https://github.com/ydb-platform/ydb/commits/CLI_2.1.0

Functionality:

• Added initial table scan when creating a CDC changefeed. Now, you can export all the data existing at the time of changefeed creation.
• Added atomic index replacement. Now, you can atomically replace one pre-defined index with another. This operation is absolutely transparent for your application. Indexes are replaced seamlessly, with no downtime.
• Added the audit log: Event stream including data about all the operations on YDB objects.

Performance:

• Improved formats of data exchanged between query stages. As a result, we accelerated SELECTs by 10% on parameterized queries and by up to 30% on write operations.
• Added autoconfiguring for the actor system pools based on the workload against them. This improves performance through more effective CPU sharing.
• Optimized the predicate logic: Processing of parameterized OR or IN constraints is automatically delegated to DataShard.
• (Experimental) For scan queries, you can now effectively search for individual rows using a primary key or secondary indexes. This can bring you a substantial gain in performance in many cases. Similarly to regular queries, to use a secondary index, you need to explicitly specify its name in the query text using the VIEW keyword.
• The query's computational graph is now cached at query runtime, reducing the CPU resources needed to build the graph.

Bug fixes:

• Fixed bugs in the distributed data warehouse implementation. We strongly recommend all our users to upgrade to the latest version.
• Fixed the error that occurred on building an index on NOT NULL columns.
• Fixed statistics calculation with MVCC enabled.
• Fixed errors with backups.
• Fixed the race condition that occurred at splitting and deleting a table with SDC.

Full Changelog: https://github.com/ydb-platform/ydb/compare/22.5.10...23.1.26

What's new:

• Added changefeed configuration settings to transfer additional info about changes to topic.
• Added support of rename for tables with TTL setting enabled.
• Added retention period setting for cdc-log.

BugFixes & Improvements:

• Fixed: Error inserting 0 rows via BulkUpsert.
• Fixed: Error when importing Date/DateTime columns from CSV.
• Fixed: Error importing data from CSV with line breaks in values.
• Fixed: Error importing data from CSV with empty values.
• Improved Query Processing performance (WorkerActor was replaced with SessionActor).
• DataShard compaction now starts immediately after split or merge operations.

Full Changelog: https://github.com/ydb-platform/ydb/compare/22.4.44...22.5.10

• Minor bugfix and improvements
• Fixed security issue (CVE-2022-28228)

Full Changelog: https://github.com/ydb-platform/ydb/compare/22.4.31...22.4.44

YDB Topics and Change Data Capture (CDC):

• A new Topic API is introduced in YDB. YDB topic is an entity for storing unstructured messages and delivering them to multiple subscribers.
• A new TopicAPI is supported in YDB CLI and SDK. The API enables topic management operations and streaming message publish/subscribe.
• Added support of table data changes capture into a topic using CDC change feeds.

SDK:

• YDB Topics support added in YDB Golang SDK.
• Official database/sql driver to work with YDB is now supported for Golang.

Embedded UI:

• CDC change feeds and Secondary Indexes are displayed in Database scheme hierarchy as separate objects.
• Visualization of query explain plan was completely redesigned.
• Storage groups with problems are now more visible.
• Various optimizations based on UX assessment

Query Processing:

• Added Query Processor 2.0 — a brand new version of execution engine for OLTP-queries with significant improvements comparing to previous one.
• Improved performance: up to 10% better read throughout, up to 60% better write throughput.
• A feature for NOT NULL constraint for Primary Key has been added. Can be set up at table creation stage only.
• Added support of online secondary index rename without stopping the service.
• Improved query explain representation now includes graph for physical operators.

Core:

• Read-only transactions now use a consistent snapshot that doesn't conflict with concurrent writes.
• Added BulkUpsert support for tables with async secondary indexes.
• Added TTL support for tables with async secondary indexes.
• Added option to compress data during export to S3.
• Added initial version of audit log for DDL statements.
• Added support of authentication with static credentials.
• Added system tables for query performance diagnostics.

Full Changelog: https://github.com/ydb-platform/ydb/compare/22.2.47...22.4.31