clickhouse-backup

IMPROVEMENTS

• Added 'allow_empty_backups' and 'api.create_integration_tables' options
• Wait for clickhouse in server mode (fix #169)
• Added Disk Mapping feature (fix #162)

BUG FIXES

• Fixed 'ftp' remote storage (#164)

It is the last release of v0.x.x

IMPROVEMENTS

• Added 'create_remote' and 'restore_remote' commands
• Changed update config behavior in API mode

BUG FIXES

• fix #154
• fix #165

IMPROVEMENTS

• Support for new versions of ClickHouse (#155)
• Support of Atomic Database Engine (#140, #141, #126)
• Support of multi disk ClickHouse configurations (#51)
• Ability to upload and download specific tables from backup
• Added partitions backup on remote storage (#83)
• Added support for backup/upload/download schema only (#138)
• Added new backup format select it by compression_format: none option

BROKEN CHANGES

• Changed backup format
• Incremental backup on remote storage and 'flashback' feature is not supported now, but will supported in future versions

IMPROVEMENTS

• Added CLICKHOUSE_AUTO_CLEAN_SHADOW option for cleaning shadow folder before backup. Enabled by default.
• Added CLICKHOUSE_SYNC_REPLICATED_TABLES option for sync replicated tables before backup. Enabled by default.
• Improved statuses of operations in server mode

BUG FIXES

• Fixed bug with semaphores in server mode

IMPROVEMENTS

• Added TLS support for clickhouse and api (#133)

BUG FIXES

• Fixed ignoring 'config' flag in server mode
• Fixed 'restore' command in server mode

IMPROVEMENTS
Added integration with clickhouse cli. Now backups can be managed via clickhouse-client
How it works:

1. Run clickhouse-backup api

clickhouse-backup server

2. Create backup_actions and backup_list tables

CREATE TABLE system.backup_actions (command String, start DateTime, finish DateTime, status String, error String) ENGINE=URL('http://127.0.0.1:7171/backup/actions', JSONEachRow)
CREATE TABLE system.backup_list (name String, created DateTime, size Int64, location String) ENGINE=URL('http://127.0.0.1:7171/backup/list', JSONEachRow)

3. Manage backup via clickhouse queries

INSERT INTO system.backup_actions (command) VALUES ('create backup1')
INSERT INTO system.backup_actions (command) VALUES ('upload backup1')
SELECT * FROM system.backup_list
┌─name───────────┬─────────────created─┬──size─┬─location─┐
│ backup1        │ 2020-12-01 08:43:47 │     0 │ local    │
│ backup1.tar.gz │ 2020-12-01 08:43:54 │ 74249 │ remote   │
└────────────────┴─────────────────────┴───────┴──────────┘
SELECT * FROM system.backup_actions
┌─command────────┬───────────────start─┬──────────────finish─┬─status──┬─error─┐
│ create backup1 │ 2020-12-01 09:01:02 │ 2020-12-01 09:01:02 │ success │       │
│ upload backup1 │ 2020-12-01 09:01:05 │ 2020-12-01 09:01:05 │ success │       │
└────────────────┴─────────────────────┴─────────────────────┴─────────┴───────┘

IMPROVEMENTS

• Added 'Flashback' restore feature, which allows restoring only changed parts. This can be useful for replicated tables. (thanks @nikk0)
• Added '--rm' and '--drop' flags to delete tables before restoring. (thanks @max-wittig)
• Added ability to choose AWS S3 storage class. (thanks @andrii29)

BUG FIXES

• Tables containing the "-" character in the name could not be restored. This has been fixed. (thanks @LH-Datarocks)
• Fixed error "open /var/lib/clickhouse-server/store/shadow: no such file or directory" on backup creation for ClickHouse after 20.8. (thanks @everythings-gonna-be-alright and @hohaftf)

IMPROVEMENTS

• Added REST API Server (thanks @nmcclain)
• Added ClickHouse integration via REST API. Now you can manage backups from ClickHouse.
• Added FTP support (thanks @jawr)
• Added Azure Blob Storage support (thanks @atykhyy)

IMPROVEMENTS

• Added support of Tencent COS cloud object storage provider (thanks @kasnet)
• Added CLICKHOUSE_TIMEOUT and CLICKHOUSE_FREEZE_BY_PART options. It's useful when your tables have too many parts and default freezing doesn't work.
• Added 'rpm' and 'deb' packages to release page (thanks @Felixoid)

BUG FIXES

• Fixed connection issue when ClickHouse password contains special characters (https://github.com/AlexAkulov/clickhouse-backup/issues/69)

BUG FIXES

• Restoring some tables might not work correctly. That has been fixed (thanks @yuzhichang)

IMPROVEMENTS

• Added S3_DISABLE_CERT_VERIFICATION option (thanks @martenlindblad)
• Removed default value for GCS_CREDENTIALS_FILE option, it led to issue when running on GCE (thanks @nikon72ru)
• Now the view tables will be restored at the end

IMPROVEMENTS

• Added support of Google Cloud Storage (thanks @przemekd)
• Now tables will be processed in an orderly manner (thanks @yuzhichang)
• Now config file location may be defined via $CLICKHOUSE_BACKUP_CONFIG

BROKEN CHANGES

• 'backups_to_keep_s3' option in config file was renamed into 'backups_to_keep_remote'
• 'disable_progress_bar', 'backups_to_keep_local', 'backups_to_keep_remote' settings were moved to 'general' section in config file
• 'restore-schema' and 'restore-data' were united into one command. You can restore the schema and data separately, as before, using the '--schema' and '--data' flags (thanks @dcastanier)

IMPROVEMENTS

• Support of ClickHouse v19.15 was added
• AWS Server-Side Encryption support was added (thanks @pelletier)
• Auto create 'backup' folder on download (thanks @dcastanier)

IMPROVEMENTS

• Now restrictions defined by flag 'tables' or CLICKHOUSE_SKIP_TABLES will be also applied for metadata (https://github.com/AlexAkulov/clickhouse-backup/issues/43)
• S3 part was rewritten on non-blocking pipes. It allowed to execute reading from disk, compression and uploading simultaneously.
• Library mholt/archiver was updated to version with multi-threading 'gzip' support and now 'gzip' has been selected as compression format by default (https://github.com/AlexAkulov/clickhouse-backup/issues/14)

Comparison table of upload 31GiB data with v0.4.0:

+------+----------------+----------------+
|      |     v0.4.0     |     v0.4.1     |
+------+----------------+----------------+
| tar  | Time: 8m14s    | Time: 7m50s    |
|      | Size: 30.8GB   | Size: 30.8GB   |
+------+----------------+----------------+
| lz4  | Time: 22m50s   | Time: 15m15s   |
|      | Size: 24GB     | Size: 24.5GB   |
+------+----------------+----------------+
| gzip | Time: 13m21s   | Time: 6m54s    |
|      | Size: 21GB     | Size: 24.2GB   |
+------+----------------+----------------+

IMPROVEMENTS

• Upload and download were rewritten on the official AWS library. This allows to support more s3 cloud providers. In particular Yandex.Cloud now works correctly (https://github.com/AlexAkulov/clickhouse-backup/issues/30)
• Support for tables containing special and national characters in name was added (https://github.com/AlexAkulov/clickhouse-backup/issues/28)
• Part size has been increased to 100MiB by default
• App will retry to upload and download data 30 times in case of network errors (https://github.com/AlexAkulov/clickhouse-backup/issues/36)

DEPREСATIONS

• The 'tree' strategy and the 'dry-run' flag, which were marked as deprecated in v0.3.2, are removed

BUG FIXES

• Error while creating backups of tables different from MergeTree has been fixed (https://github.com/AlexAkulov/clickhouse-backup/issues/44)

IMPROVEMENTS

• Check that table is created before 'restore-data' was added
• Support for comma separated list for '--tables' flag was added

IMPROVEMENTS

• Support of the newest versions of ClickHouse was added (above 19.4.4.33)

IMPROVEMENTS

• The 'list' command can print latest and penultimate of backup name. Now you can use the output of the 'list' command as parameters for another command.
  For example, create new backup and upload diff:

  clickhouse-backup create
  clickhouse-backup upload --diff-from=$(clickhouse-backup list local penult) $(clickhouse-backup list local latest)
  

• The 'delete' command has been added. Now you can remove specific backups from local and s3.

  clickhouse-backup delete local <backup_name>
  clickhouse-backup delete s3 $(clickhouse-backup list s3 penult)
  

Added AWS IAM support

BUG FIXES

• Bug because of which the newest backups are deleted instead of the oldest ones has been fixed

BUG FIXES

• lz4 compression might cause error "lz4: invalid source or destination buffer too short". It was caused by bug in old version of github.com/pierrec/lz4. It was fixed. Please, check your backups on S3.

DEPREСATIONS

• 'dry-run' flag and 'tree' strategy were marked as deprecated

IMPROVEMENTS

• Performance of uploading and downloading was improved. See comparison table of uploading of 30GB below

+-----+---------------------+---------------------+
|     |        v0.3.1       |       v0.3.2        |
+-----+---------------------+---------------------+
| tar | Time: 45m:58s       | Time: 14m:34s       |
|     | Disk Usage: 70%     | Disk Usage: 58%     |
|     | Network Util: 100Mb | Network Util: 300Mb |
+-----+---------------------+---------------------+
| lz4 | Time: 28m:47s       | Time: 26m:35s       |
|     | Disk Usage: 30%     | Disk Usage: 26%     |
|     | Network Util: 120Mb | Network Util: 130Mb |
+-----+---------------------+---------------------+

• Now progress bar shows counts in bytes