wal-g

This is a technical pre-release aimed at mitigating several CVEs in dependencies (#1675, #1715). These vulenrabilities have no known ways to be exploited against WAL-G. But these days we should keep CVE scanners calm.
Also this pre-release gives access to several features that didn't make it into 3.0.0 because were not tested enough (#1662, #1646, #1652).
As usual, handful of bugs fixed.

Thanks to everyone who contributed!

What's Changed

• Support DigitalOcean Spaces (NoSuchKey during ListFolder) by @turt2live in https://github.com/wal-g/wal-g/pull/1646
• Skip archive_command check on standby by @x4m in https://github.com/wal-g/wal-g/pull/1662
• fixes for ao check config by @debebantur in https://github.com/wal-g/wal-g/pull/1664
• Fix for GP incremental backups by @debebantur in https://github.com/wal-g/wal-g/pull/1666
• Add unit tests for internal.databases.greenplum.buildFetchCommand by @denisSurkov in https://github.com/wal-g/wal-g/pull/1661
• Catchup-send and catchup-receive commands by @x4m in https://github.com/wal-g/wal-g/pull/1652
• Collecting statistics about read/written bytes by @debebantur in https://github.com/wal-g/wal-g/pull/1670
• Add Docs for undocumented debug env variables by @sebastiangrill in https://github.com/wal-g/wal-g/pull/1640
• sanikadze:doc_gp_gp_pitr by @Sanikadze in https://github.com/wal-g/wal-g/pull/1672
• Implement lockFile for windows by @passcod in https://github.com/wal-g/wal-g/pull/1673
• upgrade golang.org/x/net to resolve CVE-2023-39325 by @devinturner in https://github.com/wal-g/wal-g/pull/1675
• Fix WALG_STATSD_EXTRA_TAGS by @nick-kentik in https://github.com/wal-g/wal-g/pull/1676
• Fix gp tests by @debebantur in https://github.com/wal-g/wal-g/pull/1677
• add path to recovery.conf for greenplum by @avkashin in https://github.com/wal-g/wal-g/pull/1679
• Clarified dev installing instruction on Ubuntu for go 1.16+ by @homper in https://github.com/wal-g/wal-g/pull/1681
• added IsTablespaceMapExists test by @NiripsaKakVsegda in https://github.com/wal-g/wal-g/pull/1680
• split mongodb restore on stages under flag by @Apelsin234 in https://github.com/wal-g/wal-g/pull/1685
• add support to redis acl user by @NotAmigo in https://github.com/wal-g/wal-g/pull/1682
• Add glob pattern support in storagetools by @ReshetnikovPavel in https://github.com/wal-g/wal-g/pull/1684
• add PGSSLCERT, PGSSLKEY, PGSSLROOTCERT support by @NiripsaKakVsegda in https://github.com/wal-g/wal-g/pull/1687
• WALG-1627. MySQL fix file double Close() on file by @ostinru in https://github.com/wal-g/wal-g/pull/1628
• add create upload input unit tests by @danilkaz in https://github.com/wal-g/wal-g/pull/1690
• add tests for SetUserData in postgres/generic_meta_interactor by @SixBlueBlack in https://github.com/wal-g/wal-g/pull/1697
• Fix lll lint violation, update golangci version by @serprex in https://github.com/wal-g/wal-g/pull/1686
• Exclude standby.signal by @munakoiso in https://github.com/wal-g/wal-g/pull/1702
• WALG-1665. User proper DSNs in MySQL documentation & validate DSN format it earlier by @ostinru in https://github.com/wal-g/wal-g/pull/1692
• unit test for internal/backup.go/FetchMetadata by @ValeriaIM in https://github.com/wal-g/wal-g/pull/1695
• Bump golangci/golangci-lint-action from 4.0.0 to 5.3.0 by @dependabot in https://github.com/wal-g/wal-g/pull/1703
• Save isIncremented attribute by @debebantur in https://github.com/wal-g/wal-g/pull/1704
• Add charset converter to convert non utf-8 char by @anisurrahman75 in https://github.com/wal-g/wal-g/pull/1705
• Bump golangci/golangci-lint-action from 5.3.0 to 6.0.1 by @dependabot in https://github.com/wal-g/wal-g/pull/1706
• WALG-1683. Save CPU arch / OS to mysql sentinel. by @ostinru in https://github.com/wal-g/wal-g/pull/1693
• added without metadata to gp by @debebantur in https://github.com/wal-g/wal-g/pull/1708
• Tests for isPagedFile by @Themplarer in https://github.com/wal-g/wal-g/pull/1710
• Added validation for S3 credentials by @Themplarer in https://github.com/wal-g/wal-g/pull/1711
• fixed hex parsing in timeline by @debebantur in https://github.com/wal-g/wal-g/pull/1712
• Added retention period for s3 by @debebantur in https://github.com/wal-g/wal-g/pull/1713
• Add few unit test on azure ConfigureStorage by @Loraan in https://github.com/wal-g/wal-g/pull/1714
• Add Uploader interface mock, test on UploadSentinel and make mock command by @Loraan in https://github.com/wal-g/wal-g/pull/1716
• Update modules in order to resolve CVEs. by @montaguethomas in https://github.com/wal-g/wal-g/pull/1715

New Contributors

• @turt2live made their first contribution in https://github.com/wal-g/wal-g/pull/1646
• @sebastiangrill made their first contribution in https://github.com/wal-g/wal-g/pull/1640
• @Sanikadze made their first contribution in https://github.com/wal-g/wal-g/pull/1672
• @passcod made their first contribution in https://github.com/wal-g/wal-g/pull/1673
• @devinturner made their first contribution in https://github.com/wal-g/wal-g/pull/1675
• @avkashin made their first contribution in https://github.com/wal-g/wal-g/pull/1679
• @homper made their first contribution in https://github.com/wal-g/wal-g/pull/1681
• @NiripsaKakVsegda made their first contribution in https://github.com/wal-g/wal-g/pull/1680
• @NotAmigo made their first contribution in https://github.com/wal-g/wal-g/pull/1682
• @danilkaz made their first contribution in https://github.com/wal-g/wal-g/pull/1690
• @SixBlueBlack made their first contribution in https://github.com/wal-g/wal-g/pull/1697
• @ValeriaIM made their first contribution in https://github.com/wal-g/wal-g/pull/1695
• @anisurrahman75 made their first contribution in https://github.com/wal-g/wal-g/pull/1705
• @Themplarer made their first contribution in https://github.com/wal-g/wal-g/pull/1710
• @Loraan made their first contribution in https://github.com/wal-g/wal-g/pull/1714
• @montaguethomas made their first contribution in https://github.com/wal-g/wal-g/pull/1715

Full Changelog: https://github.com/wal-g/wal-g/compare/v3.0.0...v3.0.1

This is a pre-release to test the aws iam roles config setting.

Release notes:
This release mainly consists of various bug fixes and improvements.

Common:

• Add arm(aarch64) build pipeline for ubuntu20.04 binaries (#1304)

PostgreSQL:

• Add warning output on backup-fetch retries (#1325)
• Remove redundant check for --without-files-metadata flag (#1321)
• PostgreSQL 15 Support (#1319)
• Fix of deleting WAL-E backups (#1292)

MySQL/MariaDB:

• Fix a regression in MariaDB backup-push handler (#1313)
• Binlog cache fixes (#1281)

Greenplum:

• Optimize delete handler (#1300, #1302)
• Optimize storage listing queries during the backup push (#1312)
• Use AO files metadata instead of backup references (#1308)
• Refactor AO/AOCS storage metadata queries (#1280, #1278)

Storages:

• Azure authentication using the default credential chain (#1282)
• Update aws-sdk-go to v1.44.7 (#1262)
• Add check read/write to storage tools (#1275)

Release notes:

This release contains a huge amount of new features, bug fixes, and performance improvements. It is worth noting that v2.0 Postgres delta backups are not backward-compatible with the pre-2.0 versions. It means that the v2.0 delta backup can't be restored with the v1.1 or earlier versions. Also, this release introduces the initial support for Greenplum physical backups.

WAL-G for MySQL now supports the parallel backup-push/backup-fetch and has an optimized binlog upload mechanism. WAL-G for SQLServer now supports the backup encryption and compression as well as the external storage backup import/export.

Common:

• Update lz4, brotli #1165, #1179
• Multiple libsodium fixes #1087, #1227
• Bash/zsh shell completions #1250, #1252
• Add a 'flags' subcommand #1033
• Disable CPU Limits in turbo mode #1121
• Metrics forwarding via statsd protocol #1240

SQLServer:

• Add SQLServerDBConcurrency setting to limit concurrent database operations #1067
• Add LATEST keyword handling #1110
• Add norecovery option for backup log command #1115
• Add SQLServer compression / encryption #1161
• Backup import and export to the external storage #1184

Postgres:

• Add CopyComposer for full backups optimization using remote copy API #1012
• Add streamed JSON sentinel serialization to decrease memory usage #1103
• Extract heavy files and tarfilesets from the backup sentinel #1114
• Introduce WALG_TAR_DISABLE_FSYNC to skip fsync during backup-fetch #1144
• Add reduced memory usage backup mode #1101
• Improvements and fixes for remote backup #1175
• Add pgBackRest backups restore support (beta) #1139, #1202
• Add 'wal-restore' command #977
• Add 'delete garbage' command #1217, #1218
• Return custom error code when WAL file does not exist #1195
• Add WALG_ALIVE_CHECK_INTERVAL to fail backup on Postgres process failure #1260
• Reformat LSN output #1259

MySQL:

• Add wal-g binlog-analyze #1085
• Check binlog GTIDs before pushing binlogs to S3 #1190
• Add --check-gtids to wal-g binlog-push #1085
• Add --until-binlog-last-modified-time option to wal-g-mysql binlog-replay #1154
• Improve MySQL restore performance #1131
• Allow backups on cascade replicas #1168
• Fix mysql backup of a lagging replica #1223
• Make mysql bilong-push status observable to 3rd party tools #1229

Greenplum:

• Initial physical backups support for GPDB 6.19.2+. API might change in the future versions.

Storages:

• Add get/put/rm/cat to storage tools #1069, #1164
• S3: Add download by range #1083
• S3: Add connection reset retries #1137
• S3: Add WALG_S3_MAX_RETRIES setting #1148
• S3: Fix S3_LOG_LEVEL setting #1158
• S3: AWS authentication fixes #1209
• S3: Yandex Cloud session token auth support #1213
• SSH: Ensure trailing delimiter in the storage prefix #1130
• Azure: Switch to the recommended azblob SDK #1135, #1171
• Azure: Fix ListFolder never returning error #1208
• Swift: Ignore files that were not found during ListFolder #1129

This is a pre-release to test S3 authentication with mechanisms other than key/secret.

This pre-release introduces the Greenplum physical backups support.

This pre-release introduces the S3_LOG_LEVEL setting to help with debugging the S3 storage interaction issues.

This is a pre-release intended to test the s3 connection reset retries fix (https://github.com/wal-g/wal-g/pull/1137).

Some interesting features/fixes:

• Include libsodium in the GitHub release builds. Exit with failure if libsodium is requested but WAL-G was not compiled with libsodium support #1062 CVE-2021-38599
• UserData is now required to be a valid JSON #1049
• backup-list now sorts backups by the astronomical time of the completition #907

Postgres:

• .history files overwrite prevention fix #1052
• wal-verify can now be run from a standby #1014
• wal-verify now ignores the permanent backups #1014, #1002

MongoDB:

• Add support for MongoDB 5.0 #1041

SQLServer:

• Fix MSSQL driver import #1038

MySQL:

• Add --turbo flag to disable limiters #1027
• Fix --detailed backup-list bug #1026
• Add --until flag to binlog-push #1005

Redis:

• Add 6.2 support & redis-cli bug workaround #1022

Storages

• S3: customer SSE key support #1042 (https://github.com/wal-g/storages/pull/48)
• Azure: support AZURE_STORAGE_SAS_TOKEN without the leading '?' #1056
• Azure: disable the unnecessary syslog writes #1058

Release notes:

This time we decided to bump the number to 1.0 to switch to a common versioning pattern.
WAL-G 1.0 is compatible with prior versions (v0.2.0+). This might change with later releases.
Also, WAL-G 1.0 can restore WAL-E backups and WALs.
Good news for non-PostgreSQL databases: WAL-G is now production-ready for MS SQL, MySQL databases, meanwhile the API can be changed in future releases. MongoDB and Redis support is in the beta stage.

Some interesting features/fixes:

• #780 PostgreSQL remote backups
• #905 Improve compatibility with PostgreSQL 13, introduced WALG_PREFETCH_DIR option
• #824 New option was added to wal-verify, now it requires an argument timeline|integrity
• #903 It's now not allowed to create non-permanent delta backups from permanent backups
• #842 Fix compatibility with S3 implementations that do not support ListObjectsV2
• #950 It's now again possible to tell WAL-G to rename .ready files to .done directly
• #898, #962 WAL prefetch is now slightly faster
• #858 Catchup-list command to list delta backups taken specifically for catchup
• #847 Support for Yandex Cloud KMS
• #817, #833 Multipart retries for GCS storage
• #873 Support for AzureUSGovernmentCloud, AzureChinaCloud, AzureGermanCloud
• #899 Add metadata for WAL files
• #913, #917 New option to delete single backup for PostgreSQL
• Add Redis backups. Now we have Redis support with backup-fetch, backup-push, backup-list and delete commands. For full info see Redis.md
• Add MySQL catchup mode: allow replay binlogs while uploading new ones
• Add SQLServer single database restore with renaming
• Huge amount of bug fixes, tests and improvements for Redis/MongoDB/SQLServer/etc

Thanks!

This is QA pre-release per request #970
We will cut a release when the QA cycle will be finished.

This is QA pre-release to test changes in background WAL upload

This is QA pre-release intended to test libsodium settings.
#776

Notable changes in this release include:

1. Fixes for S3 and GCP storages #656 #756.
2. Add wal-show command to get information about wal storage folder.
3. Add wal-verify command. It checks the integrity of WAL history starting from the oldest backup available in current timeline history.
4. Add wal-receive command. You can use WAL-G as a replica running on another host to ensure RPO=0. This is beta functionality: API may change in the future.
5. Add reverse delta unpack for backup-fetch (--reverse-unpack flag).
6. Add redundant archives skipping for backup-fetch (--skip-redundant-tars flag, designed to work in pair with reverse delta unpack).
7. Add page checksum verification for backup-push (--verify flag).

You can find more about some of these new features in Daniil Zakhlystov's post

This is QA prerelease to test new wal-receive feature.

This is QA pre-release to cope with AWS S3 SDK bug. See https://github.com/wal-g/storages/pull/22, #656 and #727 for details.

This is a QA pre-release to test refined logic of parallel WAL upload.
See #644 for details.

Notable changes in this release include:

• Support of Libsodium encryption
• Retryer for AWS storage
• Go modules
• Add catchup commands. It allows cut replication lag on a PostgreSQL cluster.
  Simple scenario:

1. Get current LSN of a replica with high replication lag.
2. Create an incremental backup with catchup-push since the replica LSN.
3. Apply it to the replica with catchup-fetch command.

Thanks for all your contributions, bug reports, and feature requests!

Draft prerelease to test #593

In this pre-release:

1. Fix prefetch for LZO archives. Bug #472 was greatly reducing the parallelism of WAL-E archive restoration.
2. Support SCRAM-SHA-256 authentication (actually, just PGX upgrade)
3. Fix support of 9.4 (we tried to check system identifier in a way that is not compatible with 9.4)