v1.1.0

IMPROVEMENTS

• Added concurrency settings for upload and download, which allow loading table data in parallel for each table and each disk for multi-disk storages
• Up golang version to 1.17
• Updated go libraries dependencies to actual version (exclude azure)
• Add Clickhouse 21.8 to test matrix
• Now S3_PART_SIZE not restrict upload size, partSize calculate depends on MAX_FILE_SIZE
• improve logging for delete operation
• Added S3_DEBUG option to allow debug S3 connection
• Decrease number of SQL queries to system.* during backup commands
• Added options for RBAC and CONFIGs backup, look to clickhouse-backup help create and clickhouse-backup help restore for details
• Add S3_CONCURRENCY option to speedup backup upload to S3
• Add SFTP_CONCURRENCY option to speedup backup upload to SFTP
• Add AZBLOB_USE_MANAGED_IDENTITY support for ManagedIdentity for azure remote storage, thanks https://github.com/roman-vynar
• Add clickhouse-operator kubernetes manifest which run clickhouse-backup in server mode on each clickhouse pod in kubernetes cluster
• Add detailed description and restrictions for incremental backups.
• Add GCS_DEBUG option
• Add CLICKHOUSE_DEBUG option to allow low-level debug for clickhouse-go

BUG FIXES

• fix #266 properly restore legacy backup format
• fix #244 add read_timeout, write_timeout to client-side timeout for clickhouse-go
• fix #255 restrict connection pooling to 1 in clickhouse-go
• fix #256 remote_storage: none, was broke compression
• fix #266 legacy backups from version prior 1.0 can't restore without allow_empty_backup: true
• fix #223 backup only database metadata for proxy integrated database engines like MySQL, PostgreSQL
• fix GCS global buffer wrong usage during UPLOAD_CONCURRENCY > 1
• Remove unused SKIP_SYNC_REPLICA_TIMEOUTS option

BUG FIXES

• Fixed silent cancel uploading when table has more than 4k files (fix #203, #163. Thanks mastertheknife)
• Fixed download error for zstd and brotli compression formats
• Fixed bug when old-format backups hadn't cleared

IMPROVEMENTS

• Added diff backups
• Added retries to restore operation for resolve complex tables dependencies (Thanks @Slach)
• Added SFTP remote storage (Thanks @combin)
• Now databases will be restored with the same engines (Thanks @Slach)
• Added create_remote and restore_remote commands
• Changed of compression format list. Added zstd, brotli and disabled bzip2, sz, xz

BUG FIXES

• Fixed empty backup list when S3_PATH and AZBLOB_PATH is root
• Fixed azblob container issue (Thanks @atykhyy)

IMPROVEMENTS

• Added 'allow_empty_backups' and 'api.create_integration_tables' options
• Wait for clickhouse in server mode (fix #169)
• Added Disk Mapping feature (fix #162)

BUG FIXES

• Fixed 'ftp' remote storage (#164)

It is the last release of v0.x.x

IMPROVEMENTS

• Added 'create_remote' and 'restore_remote' commands
• Changed update config behavior in API mode

BUG FIXES

• fix #154
• fix #165

IMPROVEMENTS

• Support for new versions of ClickHouse (#155)
• Support of Atomic Database Engine (#140, #141, #126)
• Support of multi disk ClickHouse configurations (#51)
• Ability to upload and download specific tables from backup
• Added partitions backup on remote storage (#83)
• Added support for backup/upload/download schema only (#138)
• Added new backup format select it by compression_format: none option

BROKEN CHANGES

• Changed backup format
• Incremental backup on remote storage and 'flashback' feature is not supported now, but will supported in future versions

IMPROVEMENTS

• Added CLICKHOUSE_AUTO_CLEAN_SHADOW option for cleaning shadow folder before backup. Enabled by default.
• Added CLICKHOUSE_SYNC_REPLICATED_TABLES option for sync replicated tables before backup. Enabled by default.
• Improved statuses of operations in server mode

BUG FIXES

• Fixed bug with semaphores in server mode

IMPROVEMENTS

• Added TLS support for clickhouse and api (#133)

BUG FIXES

• Fixed ignoring 'config' flag in server mode
• Fixed 'restore' command in server mode

IMPROVEMENTS
Added integration with clickhouse cli. Now backups can be managed via clickhouse-client
How it works:

1. Run clickhouse-backup api

clickhouse-backup server

2. Create backup_actions and backup_list tables

CREATE TABLE system.backup_actions (command String, start DateTime, finish DateTime, status String, error String) ENGINE=URL('http://127.0.0.1:7171/backup/actions', JSONEachRow)
CREATE TABLE system.backup_list (name String, created DateTime, size Int64, location String) ENGINE=URL('http://127.0.0.1:7171/backup/list', JSONEachRow)

3. Manage backup via clickhouse queries

INSERT INTO system.backup_actions (command) VALUES ('create backup1')
INSERT INTO system.backup_actions (command) VALUES ('upload backup1')
SELECT * FROM system.backup_list
┌─name───────────┬─────────────created─┬──size─┬─location─┐
│ backup1        │ 2020-12-01 08:43:47 │     0 │ local    │
│ backup1.tar.gz │ 2020-12-01 08:43:54 │ 74249 │ remote   │
└────────────────┴─────────────────────┴───────┴──────────┘
SELECT * FROM system.backup_actions
┌─command────────┬───────────────start─┬──────────────finish─┬─status──┬─error─┐
│ create backup1 │ 2020-12-01 09:01:02 │ 2020-12-01 09:01:02 │ success │       │
│ upload backup1 │ 2020-12-01 09:01:05 │ 2020-12-01 09:01:05 │ success │       │
└────────────────┴─────────────────────┴─────────────────────┴─────────┴───────┘

IMPROVEMENTS

• Added 'Flashback' restore feature, which allows restoring only changed parts. This can be useful for replicated tables. (thanks @nikk0)
• Added '--rm' and '--drop' flags to delete tables before restoring. (thanks @max-wittig)
• Added ability to choose AWS S3 storage class. (thanks @andrii29)

BUG FIXES

• Tables containing the "-" character in the name could not be restored. This has been fixed. (thanks @LH-Datarocks)
• Fixed error "open /var/lib/clickhouse-server/store/shadow: no such file or directory" on backup creation for ClickHouse after 20.8. (thanks @everythings-gonna-be-alright and @hohaftf)

IMPROVEMENTS

• Added REST API Server (thanks @nmcclain)
• Added ClickHouse integration via REST API. Now you can manage backups from ClickHouse.
• Added FTP support (thanks @jawr)
• Added Azure Blob Storage support (thanks @atykhyy)

IMPROVEMENTS

• Added support of Tencent COS cloud object storage provider (thanks @kasnet)
• Added CLICKHOUSE_TIMEOUT and CLICKHOUSE_FREEZE_BY_PART options. It's useful when your tables have too many parts and default freezing doesn't work.
• Added 'rpm' and 'deb' packages to release page (thanks @Felixoid)

BUG FIXES

• Fixed connection issue when ClickHouse password contains special characters (https://github.com/AlexAkulov/clickhouse-backup/issues/69)

BUG FIXES

• Restoring some tables might not work correctly. That has been fixed (thanks @yuzhichang)

IMPROVEMENTS

• Added S3_DISABLE_CERT_VERIFICATION option (thanks @martenlindblad)
• Removed default value for GCS_CREDENTIALS_FILE option, it led to issue when running on GCE (thanks @nikon72ru)
• Now the view tables will be restored at the end

IMPROVEMENTS

• Added support of Google Cloud Storage (thanks @przemekd)
• Now tables will be processed in an orderly manner (thanks @yuzhichang)
• Now config file location may be defined via $CLICKHOUSE_BACKUP_CONFIG

BROKEN CHANGES

• 'backups_to_keep_s3' option in config file was renamed into 'backups_to_keep_remote'
• 'disable_progress_bar', 'backups_to_keep_local', 'backups_to_keep_remote' settings were moved to 'general' section in config file
• 'restore-schema' and 'restore-data' were united into one command. You can restore the schema and data separately, as before, using the '--schema' and '--data' flags (thanks @dcastanier)

IMPROVEMENTS

• Support of ClickHouse v19.15 was added
• AWS Server-Side Encryption support was added (thanks @pelletier)
• Auto create 'backup' folder on download (thanks @dcastanier)

IMPROVEMENTS

• Now restrictions defined by flag 'tables' or CLICKHOUSE_SKIP_TABLES will be also applied for metadata (https://github.com/AlexAkulov/clickhouse-backup/issues/43)
• S3 part was rewritten on non-blocking pipes. It allowed to execute reading from disk, compression and uploading simultaneously.
• Library mholt/archiver was updated to version with multi-threading 'gzip' support and now 'gzip' has been selected as compression format by default (https://github.com/AlexAkulov/clickhouse-backup/issues/14)

Comparison table of upload 31GiB data with v0.4.0:

+------+----------------+----------------+
|      |     v0.4.0     |     v0.4.1     |
+------+----------------+----------------+
| tar  | Time: 8m14s    | Time: 7m50s    |
|      | Size: 30.8GB   | Size: 30.8GB   |
+------+----------------+----------------+
| lz4  | Time: 22m50s   | Time: 15m15s   |
|      | Size: 24GB     | Size: 24.5GB   |
+------+----------------+----------------+
| gzip | Time: 13m21s   | Time: 6m54s    |
|      | Size: 21GB     | Size: 24.2GB   |
+------+----------------+----------------+

IMPROVEMENTS

• Upload and download were rewritten on the official AWS library. This allows to support more s3 cloud providers. In particular Yandex.Cloud now works correctly (https://github.com/AlexAkulov/clickhouse-backup/issues/30)
• Support for tables containing special and national characters in name was added (https://github.com/AlexAkulov/clickhouse-backup/issues/28)
• Part size has been increased to 100MiB by default
• App will retry to upload and download data 30 times in case of network errors (https://github.com/AlexAkulov/clickhouse-backup/issues/36)

DEPREСATIONS

• The 'tree' strategy and the 'dry-run' flag, which were marked as deprecated in v0.3.2, are removed

BUG FIXES

• Error while creating backups of tables different from MergeTree has been fixed (https://github.com/AlexAkulov/clickhouse-backup/issues/44)

IMPROVEMENTS

• Check that table is created before 'restore-data' was added
• Support for comma separated list for '--tables' flag was added

IMPROVEMENTS

• Support of the newest versions of ClickHouse was added (above 19.4.4.33)

IMPROVEMENTS

• The 'list' command can print latest and penultimate of backup name. Now you can use the output of the 'list' command as parameters for another command.
  For example, create new backup and upload diff:

  clickhouse-backup create
  clickhouse-backup upload --diff-from=$(clickhouse-backup list local penult) $(clickhouse-backup list local latest)
  

• The 'delete' command has been added. Now you can remove specific backups from local and s3.

  clickhouse-backup delete local <backup_name>
  clickhouse-backup delete s3 $(clickhouse-backup list s3 penult)