wal-g

This is a technical pre-release aimed at mitigating several CVEs in dependencies (#1675, #1715). These vulenrabilities have no known ways to be exploited against WAL-G. But these days we should keep CVE scanners calm.
Also this pre-release gives access to several features that didn't make it into 3.0.0 because were not tested enough (#1662, #1646, #1652).
As usual, handful of bugs fixed.

Thanks to everyone who contributed!

What's Changed

• Support DigitalOcean Spaces (NoSuchKey during ListFolder) by @turt2live in https://github.com/wal-g/wal-g/pull/1646
• Skip archive_command check on standby by @x4m in https://github.com/wal-g/wal-g/pull/1662
• fixes for ao check config by @debebantur in https://github.com/wal-g/wal-g/pull/1664
• Fix for GP incremental backups by @debebantur in https://github.com/wal-g/wal-g/pull/1666
• Add unit tests for internal.databases.greenplum.buildFetchCommand by @denisSurkov in https://github.com/wal-g/wal-g/pull/1661
• Catchup-send and catchup-receive commands by @x4m in https://github.com/wal-g/wal-g/pull/1652
• Collecting statistics about read/written bytes by @debebantur in https://github.com/wal-g/wal-g/pull/1670
• Add Docs for undocumented debug env variables by @sebastiangrill in https://github.com/wal-g/wal-g/pull/1640
• sanikadze:doc_gp_gp_pitr by @Sanikadze in https://github.com/wal-g/wal-g/pull/1672
• Implement lockFile for windows by @passcod in https://github.com/wal-g/wal-g/pull/1673
• upgrade golang.org/x/net to resolve CVE-2023-39325 by @devinturner in https://github.com/wal-g/wal-g/pull/1675
• Fix WALG_STATSD_EXTRA_TAGS by @nick-kentik in https://github.com/wal-g/wal-g/pull/1676
• Fix gp tests by @debebantur in https://github.com/wal-g/wal-g/pull/1677
• add path to recovery.conf for greenplum by @avkashin in https://github.com/wal-g/wal-g/pull/1679
• Clarified dev installing instruction on Ubuntu for go 1.16+ by @homper in https://github.com/wal-g/wal-g/pull/1681
• added IsTablespaceMapExists test by @NiripsaKakVsegda in https://github.com/wal-g/wal-g/pull/1680
• split mongodb restore on stages under flag by @Apelsin234 in https://github.com/wal-g/wal-g/pull/1685
• add support to redis acl user by @NotAmigo in https://github.com/wal-g/wal-g/pull/1682
• Add glob pattern support in storagetools by @ReshetnikovPavel in https://github.com/wal-g/wal-g/pull/1684
• add PGSSLCERT, PGSSLKEY, PGSSLROOTCERT support by @NiripsaKakVsegda in https://github.com/wal-g/wal-g/pull/1687
• WALG-1627. MySQL fix file double Close() on file by @ostinru in https://github.com/wal-g/wal-g/pull/1628
• add create upload input unit tests by @danilkaz in https://github.com/wal-g/wal-g/pull/1690
• add tests for SetUserData in postgres/generic_meta_interactor by @SixBlueBlack in https://github.com/wal-g/wal-g/pull/1697
• Fix lll lint violation, update golangci version by @serprex in https://github.com/wal-g/wal-g/pull/1686
• Exclude standby.signal by @munakoiso in https://github.com/wal-g/wal-g/pull/1702
• WALG-1665. User proper DSNs in MySQL documentation & validate DSN format it earlier by @ostinru in https://github.com/wal-g/wal-g/pull/1692
• unit test for internal/backup.go/FetchMetadata by @ValeriaIM in https://github.com/wal-g/wal-g/pull/1695
• Bump golangci/golangci-lint-action from 4.0.0 to 5.3.0 by @dependabot in https://github.com/wal-g/wal-g/pull/1703
• Save isIncremented attribute by @debebantur in https://github.com/wal-g/wal-g/pull/1704
• Add charset converter to convert non utf-8 char by @anisurrahman75 in https://github.com/wal-g/wal-g/pull/1705
• Bump golangci/golangci-lint-action from 5.3.0 to 6.0.1 by @dependabot in https://github.com/wal-g/wal-g/pull/1706
• WALG-1683. Save CPU arch / OS to mysql sentinel. by @ostinru in https://github.com/wal-g/wal-g/pull/1693
• added without metadata to gp by @debebantur in https://github.com/wal-g/wal-g/pull/1708
• Tests for isPagedFile by @Themplarer in https://github.com/wal-g/wal-g/pull/1710
• Added validation for S3 credentials by @Themplarer in https://github.com/wal-g/wal-g/pull/1711
• fixed hex parsing in timeline by @debebantur in https://github.com/wal-g/wal-g/pull/1712
• Added retention period for s3 by @debebantur in https://github.com/wal-g/wal-g/pull/1713
• Add few unit test on azure ConfigureStorage by @Loraan in https://github.com/wal-g/wal-g/pull/1714
• Add Uploader interface mock, test on UploadSentinel and make mock command by @Loraan in https://github.com/wal-g/wal-g/pull/1716
• Update modules in order to resolve CVEs. by @montaguethomas in https://github.com/wal-g/wal-g/pull/1715

New Contributors

• @turt2live made their first contribution in https://github.com/wal-g/wal-g/pull/1646
• @sebastiangrill made their first contribution in https://github.com/wal-g/wal-g/pull/1640
• @Sanikadze made their first contribution in https://github.com/wal-g/wal-g/pull/1672
• @passcod made their first contribution in https://github.com/wal-g/wal-g/pull/1673
• @devinturner made their first contribution in https://github.com/wal-g/wal-g/pull/1675
• @avkashin made their first contribution in https://github.com/wal-g/wal-g/pull/1679
• @homper made their first contribution in https://github.com/wal-g/wal-g/pull/1681
• @NiripsaKakVsegda made their first contribution in https://github.com/wal-g/wal-g/pull/1680
• @NotAmigo made their first contribution in https://github.com/wal-g/wal-g/pull/1682
• @danilkaz made their first contribution in https://github.com/wal-g/wal-g/pull/1690
• @SixBlueBlack made their first contribution in https://github.com/wal-g/wal-g/pull/1697
• @ValeriaIM made their first contribution in https://github.com/wal-g/wal-g/pull/1695
• @anisurrahman75 made their first contribution in https://github.com/wal-g/wal-g/pull/1705
• @Themplarer made their first contribution in https://github.com/wal-g/wal-g/pull/1710
• @Loraan made their first contribution in https://github.com/wal-g/wal-g/pull/1714
• @montaguethomas made their first contribution in https://github.com/wal-g/wal-g/pull/1715

Full Changelog: https://github.com/wal-g/wal-g/compare/v3.0.0...v3.0.1

Hello everyone!

This is a much-anticipated release. In the future, we will strive to avoid accumulating such a large number of changes. We will code frequently, ship frequently, and release frequently!

Noteworthy Changes

• Every database engine now benefits from failover storage. This technique can save the day if one of the storage systems goes down. However, do not forget to monitor them anyway.
• For Postgres, we have a WAL archiving daemon available. Its primary purpose is to improve the efficiency of parallel archiving. Additionally, WAL-G supports the partial restoration of a cluster. If you have large, unnecessary tables, you can exclude them from the restoration process, the Postgres will skip over the WAL records for these tables.
• Greenplum's PITR (Point-in-Time Recovery) is undergoing its maturation process. Partial restore is supported in a similar way to Postgres. We now have tools to verify the integrity of AO\AOCS files in storage without actually restoring backups. This technique is necessary to ensure developers can sleep well, as Greenplum backups are relatively new, and extra precautions need to be taken to ensure they are safe and reliable.
• MongoDB backups can now benefit from binary backups, which significantly reduce the impact of backups on a running workload.
• MySQL now supports incremental backups, also known as delta backups. However, this feature is still in beta mode and the API may change in the future.
• Also, we now have beta support for etcd. Changes necessary for real PITR have not yet been merged into etcd, and can be found at https://github.com/etcd-io/etcd/pull/17233.

Detailed changes log

• Correct the --without-files-metadata docs by @usernamedt in https://github.com/wal-g/wal-g/pull/1327
• expose zstd compression by @heinerstilz in https://github.com/wal-g/wal-g/pull/1326
• [WIP] Multithread downloading directory by @hariton27sy in https://github.com/wal-g/wal-g/pull/1297
• Add WALG_GP_AOSEG_SIZE_THRESHOLD setting by @usernamedt in https://github.com/wal-g/wal-g/pull/1329
• Respect WALG_NETWORK_RATE_LIMIT globally by @usernamedt in https://github.com/wal-g/wal-g/pull/1330
• Restrict rate limiting only to backup-push/backup-fetch commands by @usernamedt in https://github.com/wal-g/wal-g/pull/1331
• Initial mvp mongodb binary backup by @KhurtinDN in https://github.com/wal-g/wal-g/pull/1285
• Add delta backups for Greenplum by @usernamedt in https://github.com/wal-g/wal-g/pull/1333
• Fix sorting in detailed mongodb backup list by @KhurtinDN in https://github.com/wal-g/wal-g/pull/1334
• Add @VGoshev and @KhurtinDN as codeowners of MongoDB part of Wal-G by @VGoshev in https://github.com/wal-g/wal-g/pull/1335
• Add tests, docs and minor fixes for Greenplum delta backups by @usernamedt in https://github.com/wal-g/wal-g/pull/1337
• Fix mongodb binary backup during mongod do checkpoint by @KhurtinDN in https://github.com/wal-g/wal-g/pull/1342
• Add rate limiter for mongodb by @KhurtinDN in https://github.com/wal-g/wal-g/pull/1343
• Add data catalog size for PostgreSQL/Greenplum backups by @usernamedt in https://github.com/wal-g/wal-g/pull/1344
• Add minor improvements for Greenplum by @usernamedt in https://github.com/wal-g/wal-g/pull/1345
• postgresq.md add cmake installation by @rogaliiik in https://github.com/wal-g/wal-g/pull/1348
• Support ARM64 builds on MacOS by @usernamedt in https://github.com/wal-g/wal-g/pull/1349
• Implement delete target backup for Greenplum by @usernamedt in https://github.com/wal-g/wal-g/pull/1350
• Add delete garbage command for Greenplum by @usernamedt in https://github.com/wal-g/wal-g/pull/1351
• Add limiters to Greenplum segment backup-push/backup-fetch by @usernamedt in https://github.com/wal-g/wal-g/pull/1352
• Fix failing docs builds by @usernamedt in https://github.com/wal-g/wal-g/pull/1354
• Fix distro name for ubuntu-20.04 in arm release by @williancolognesitrimble in https://github.com/wal-g/wal-g/pull/1355
• Bump golangci/golangci-lint-action from 3.2.0 to 3.3.0 by @dependabot in https://github.com/wal-g/wal-g/pull/1361
• Add minimal mongod config path option by @KhurtinDN in https://github.com/wal-g/wal-g/pull/1360
• Fix closing mongodb backup cursor by @KhurtinDN in https://github.com/wal-g/wal-g/pull/1359
• Replace mongodb binary backup to tar archive format by @KhurtinDN in https://github.com/wal-g/wal-g/pull/1357
• Fix binary backup of mongodb 4.2 by @KhurtinDN in https://github.com/wal-g/wal-g/pull/1364
• Add daemon mode for wal-push command by @rogaliiik in https://github.com/wal-g/wal-g/pull/1353
• Change error to warning when creating symlink, that already exists by @IncubusRK in https://github.com/wal-g/wal-g/pull/1320
• [Greenplum] do not fail backup on increment reading error by @usernamedt in https://github.com/wal-g/wal-g/pull/1365
• Only dump relevant env variables with LOG_LEVEL=DEVEL by @LeGEC in https://github.com/wal-g/wal-g/pull/1367
• Do not print secrets in debug logs by @LeGEC in https://github.com/wal-g/wal-g/pull/1368
• Bump golangci/golangci-lint-action from 3.3.0 to 3.3.1 by @dependabot in https://github.com/wal-g/wal-g/pull/1380
• enable using aws iam roles by @Qwiz in https://github.com/wal-g/wal-g/pull/1377
• notify_d watchdog for daemon service by @rogaliiik in https://github.com/wal-g/wal-g/pull/1371
• Use pure go implementation for zstd by @LeGEC in https://github.com/wal-g/wal-g/pull/1376
• Restore debug logging improvements by @usernamedt in https://github.com/wal-g/wal-g/pull/1389
• postgresql: fix data path mismatch on local backups by @LeGEC in https://github.com/wal-g/wal-g/pull/1383
• Fix Greenplum deleted AO segment file error handling by @usernamedt in https://github.com/wal-g/wal-g/pull/1391
• Implement max file size for mysql by @bularond in https://github.com/wal-g/wal-g/pull/1374
• Fix non-string config settings handling by @usernamedt in https://github.com/wal-g/wal-g/pull/1393
• Makefile: change compilation and linking of libbrotli to optional (and off by default) by @LeGEC in https://github.com/wal-g/wal-g/pull/1379
• add unit tests for the WriteBackupListDetails function by @tewboat in https://github.com/wal-g/wal-g/pull/1395
• Add recovery_target_timeline setting to Greenplum restore config by @usernamedt in https://github.com/wal-g/wal-g/pull/1398
• Add detailed json backup-list for Greenplum by @usernamedt in https://github.com/wal-g/wal-g/pull/1399
• Add restore-point-list command for gp by @rogaliiik in https://github.com/wal-g/wal-g/pull/1397
• Update golangci-lint version to 1.50.1 by @serprex in https://github.com/wal-g/wal-g/pull/1381
• Add restore-point-ts flag for Greenplum backup-fetch by @usernamedt in https://github.com/wal-g/wal-g/pull/1403
• Greenplum: populate pg_hba.conf on master host during restore by @usernamedt in https://github.com/wal-g/wal-g/pull/1404
• enable linters for cmd/redis internal/databases/redis and pkg/storages by @teewar2 in https://github.com/wal-g/wal-g/pull/1407
• Fix wal-g-mongo mongod restore procedure by @KhurtinDN in https://github.com/wal-g/wal-g/pull/1414
• Add mongo-rs-members parameter to binary mongo restore by @KhurtinDN in https://github.com/wal-g/wal-g/pull/1418
• [MongoDB] add opportunity restore mongodb backup with + 1 version by @Apelsin234 in https://github.com/wal-g/wal-g/pull/1417
• Add mongo-rs-name and mongo-rs-members parameters to binary mongo restore by @KhurtinDN in https://github.com/wal-g/wal-g/pull/1419
• add unit test for greenplum/backup_fetch_handler by @teewar2 in https://github.com/wal-g/wal-g/pull/1406
• fix bug with mixed up arguments by @Apelsin234 in https://github.com/wal-g/wal-g/pull/1421
• Postgres requires data dir to be 0700 or 0750. by @Aevin1387 in https://github.com/wal-g/wal-g/pull/1420
• Add support mongodb 6.0 by @KhurtinDN in https://github.com/wal-g/wal-g/pull/1416
• Fix connect to mongodb during restore procedure by @KhurtinDN in https://github.com/wal-g/wal-g/pull/1428
• Bump golangci/golangci-lint-action from 3.3.1 to 3.4.0 by @dependabot in https://github.com/wal-g/wal-g/pull/1426
• MDB-21194: [MongoDB] fix bug with purge garbage and change logic for retains by @Apelsin234 in https://github.com/wal-g/wal-g/pull/1429
• Implementation of the new binlog-server command by @Fizic in https://github.com/wal-g/wal-g/pull/1408
• Bump github.com/prometheus/client_golang from 1.7.1 to 1.11.1 in /internal/tools by @dependabot in https://github.com/wal-g/wal-g/pull/1433
• Bump golang.org/x/text from 0.3.6 to 0.3.8 in /internal/tools by @dependabot in https://github.com/wal-g/wal-g/pull/1437
• Implement retry download backup by @bularond in https://github.com/wal-g/wal-g/pull/1394
• adding push metrics as a defer in the upload method to make sure metrics are pushed even on failure by @francoispqt in https://github.com/wal-g/wal-g/pull/1423
• Revert #1423 by @usernamedt in https://github.com/wal-g/wal-g/pull/1442
• Implement database composer and partial fetch for Postgres by @Catmoonlight in https://github.com/wal-g/wal-g/pull/1434
• [MongoDB] fix retryable error from code to name by @Apelsin234 in https://github.com/wal-g/wal-g/pull/1444
• Bump actions/setup-go from 3 to 4 by @dependabot in https://github.com/wal-g/wal-g/pull/1446
• Adaptive timeout for mongodb restore by @Apelsin234 in https://github.com/wal-g/wal-g/pull/1449
• [Greenplum] Store metadata about backup restore points by @usernamedt in https://github.com/wal-g/wal-g/pull/1448
• Add Postgres partial restore by database names by @Catmoonlight in https://github.com/wal-g/wal-g/pull/1447
• Add tests postgres sentinel dto is incremental by @AlexUran in https://github.com/wal-g/wal-g/pull/1455
• Add tests for NewDtoSerializer by @denisSurkov in https://github.com/wal-g/wal-g/pull/1460
• feat: removing binlogs after using them by @Fizic in https://github.com/wal-g/wal-g/pull/1457
• Update golangci-lint in .github/workflows to 1.52.2 by @serprex in https://github.com/wal-g/wal-g/pull/1464
• support flag latest for backup and oplog reply by @Apelsin234 in https://github.com/wal-g/wal-g/pull/1463
• add tests to LSN Regex by @IvanSibirtsev in https://github.com/wal-g/wal-g/pull/1462
• Refactor storage abstractions by @usernamedt in https://github.com/wal-g/wal-g/pull/1467
• use ProtonMail/go-crypto/openpgp instead of golang.org/x/crypto/openpgp by @IvanSibirtsev in https://github.com/wal-g/wal-g/pull/1468
• fix: flag in mysql binlog-server by @Fizic in https://github.com/wal-g/wal-g/pull/1475
• [Postgres] Add failover storages for wal-push and wal-fetch by @usernamedt in https://github.com/wal-g/wal-g/pull/1466
• Add prefetch prefix for wal-prefetch cmd by @denisSurkov in https://github.com/wal-g/wal-g/pull/1471
• Make wal-g-mysql binlog-server work with new MySQL by @ostinru in https://github.com/wal-g/wal-g/pull/1474
• add GetLogsDstSettings and ConfigureCompressor tests by @amagoosebitch in https://github.com/wal-g/wal-g/pull/1479
• Correct mistakes and inaccuracies I found in the documentation by @rdjjke in https://github.com/wal-g/wal-g/pull/1478
• Add backup utility tests by @Cockamamie in https://github.com/wal-g/wal-g/pull/1481
• Implement st transfer command by @rdjjke in https://github.com/wal-g/wal-g/pull/1480
• Bump golangci/golangci-lint-action from 3.4.0 to 3.5.0 by @dependabot in https://github.com/wal-g/wal-g/pull/1484
• Fix panics in st transfer command when --overwrite flag is set by @rdjjke in https://github.com/wal-g/wal-g/pull/1485
• MongoDB binary restore: flag to not perform host reconfiguration by @ElectricR in https://github.com/wal-g/wal-g/pull/1488
• Enable database alive check for Postgresql by default by @usernamedt in https://github.com/wal-g/wal-g/pull/1486
• continue inactive PR #1453 (Tables partial restore for Postgresql) by @nikifkon in https://github.com/wal-g/wal-g/pull/1490
• export daemon message type & constants by @sergey-arefev in https://github.com/wal-g/wal-g/pull/1494
• Fix mariadb tests by @usernamedt in https://github.com/wal-g/wal-g/pull/1495
• Update Golang to 1.20 version by @anikin-aa in https://github.com/wal-g/wal-g/pull/1492
• Bump golangci/golangci-lint-action from 3.5.0 to 3.6.0 by @dependabot in https://github.com/wal-g/wal-g/pull/1496
• [Postgres] Fix races in alive storage checker by @usernamedt in https://github.com/wal-g/wal-g/pull/1498
• wal-fetch command in daemon mode by @sergey-arefev in https://github.com/wal-g/wal-g/pull/1499
• walg-daemon-client by @sergey-arefev in https://github.com/wal-g/wal-g/pull/1500
• fix action order in test jobs: checkout -> setup-go by @sergey-arefev in https://github.com/wal-g/wal-g/pull/1501
• Add strict requirements for default tablespace relfilenode parsing by @usernamedt in https://github.com/wal-g/wal-g/pull/1503
• Add partial restore for Greenplum by @nikifkon in https://github.com/wal-g/wal-g/pull/1489
• fix daemon client build ldflags values by @sergey-arefev in https://github.com/wal-g/wal-g/pull/1504
• [PG] wal-g daemon \ client: ArchiveNonExistence by @sergey-arefev in https://github.com/wal-g/wal-g/pull/1505
• Support failover storages in backup-push and backups in st transfer by @rdjjke in https://github.com/wal-g/wal-g/pull/1502
• fix SdNotify: send in goroutine by @sergey-arefev in https://github.com/wal-g/wal-g/pull/1508
• Add Greenplum AO/AOCS files deduplication age limit by @usernamedt in https://github.com/wal-g/wal-g/pull/1507
• Add WALG_ prefix to all the s3 related envs by @anikin-aa in https://github.com/wal-g/wal-g/pull/1512
• chore: remove refs to deprecated io/ioutil by @testwill in https://github.com/wal-g/wal-g/pull/1515
• Fix panic in the multistorage folder_reader.go by @usernamedt in https://github.com/wal-g/wal-g/pull/1519
• Fix typo in wal-g storage tool cat handler by @reshke in https://github.com/wal-g/wal-g/pull/1520
• Fix deadlock in GpTarBallComposer between AddHeader and addFileWorker by @mialinx in https://github.com/wal-g/wal-g/pull/1524
• Pin the CI golang version to 1.20.5 by @usernamedt in https://github.com/wal-g/wal-g/pull/1525
• Fix WAL-G prefetch leaving zombie processes in daemon mode by @usernamedt in https://github.com/wal-g/wal-g/pull/1526
• Dead lock fix by @noname0443 in https://github.com/wal-g/wal-g/pull/1530
• MariaDB Test Fix by @noname0443 in https://github.com/wal-g/wal-g/pull/1533
• Support failover storages in backup-fetch by @rdjjke in https://github.com/wal-g/wal-g/pull/1514
• support flag latest for backup-show by @Apelsin234 in https://github.com/wal-g/wal-g/pull/1529
• Now supported: additional headers for s3 requests by @debebantur in https://github.com/wal-g/wal-g/pull/1535
• Fix bug with failover storages in backup-fetch by @rdjjke in https://github.com/wal-g/wal-g/pull/1537
• MongoDB binary restore: Added flag for particular replica set member ids by @ElectricR in https://github.com/wal-g/wal-g/pull/1532
• Bump gopkg.in/yaml.v3 from 3.0.0-20210107192922-496545a6307b to 3.0.0 in /internal/tools by @dependabot in https://github.com/wal-g/wal-g/pull/1541
• Correct typo in PG doc by @Quentinchampenois in https://github.com/wal-g/wal-g/pull/1543
• Enhancement: Read file content from STDIN for wal-g st put. by @reshke in https://github.com/wal-g/wal-g/pull/1539
• WAL-G storage tool copy command by @reshke in https://github.com/wal-g/wal-g/pull/1531
• Don't fail commands if unable to write storage statuses cache file by @rdjjke in https://github.com/wal-g/wal-g/pull/1544
• [PG] Optimize docker-compose & dockerfiles for postgresql docker tests by @sergey-arefev in https://github.com/wal-g/wal-g/pull/1510
• Print storage names in backup-list by @rdjjke in https://github.com/wal-g/wal-g/pull/1538
• Feature/envelope pgp by @Alviner in https://github.com/wal-g/wal-g/pull/1547
• Fix daemon client size regression & add test by @usernamedt in https://github.com/wal-g/wal-g/pull/1552
• fix: use yc endpoint and optional file creds with envelope gpg by @Alviner in https://github.com/wal-g/wal-g/pull/1553
• Pass Context around Folder upload functions, add PG WAL push timeout by @ne2pit in https://github.com/wal-g/wal-g/pull/1546
• PG Integration tests and linter local dev improvements by @ne2pit in https://github.com/wal-g/wal-g/pull/1551
• Cancel long mongodb uploads by context by @rdjjke in https://github.com/wal-g/wal-g/pull/1548
• Allow to specify storage names in PG commands by @rdjjke in https://github.com/wal-g/wal-g/pull/1550
• fix: read exactly bytes by @Alviner in https://github.com/wal-g/wal-g/pull/1555
• add myself to reviewer for mongo and redis by @Apelsin234 in https://github.com/wal-g/wal-g/pull/1558
• Fix target storage name specifying in PG commands by @rdjjke in https://github.com/wal-g/wal-g/pull/1557
• fix: b64 encoded envelope key length by @Alviner in https://github.com/wal-g/wal-g/pull/1556
• Add gpg key id to the envelope encryption header by @Alviner in https://github.com/wal-g/wal-g/pull/1559
• Feature/use external key identifier by @Alviner in https://github.com/wal-g/wal-g/pull/1561
• upgrade github.com/yandex-cloud/go-sdk to fix CVE-2020-26160 by @zlcnju in https://github.com/wal-g/wal-g/pull/1563
• Bump actions/checkout from 3 to 4 by @dependabot in https://github.com/wal-g/wal-g/pull/1554
• Bump golangci/golangci-lint-action from 3.6.0 to 3.7.0 by @dependabot in https://github.com/wal-g/wal-g/pull/1534
• Don't require user $HOME dir to exist by @rdjjke in https://github.com/wal-g/wal-g/pull/1565
• Support failover storages in backup deletion by @rdjjke in https://github.com/wal-g/wal-g/pull/1560
• Improve storage fail detection by putting the object by @ne2pit in https://github.com/wal-g/wal-g/pull/1549
• Fix golangci-lint found issues on master by @usernamedt in https://github.com/wal-g/wal-g/pull/1567
• Don't trim slash at the beginning of ssh folder path by @rdjjke in https://github.com/wal-g/wal-g/pull/1568
• Establish ssh connections lazily by @rdjjke in https://github.com/wal-g/wal-g/pull/1569
• Add tests for GenericMetaFetcher.Fetch by @olafeonk in https://github.com/wal-g/wal-g/pull/1572
• Log storage names by @rdjjke in https://github.com/wal-g/wal-g/pull/1570
• Fix deleting backups from failover storages in PG by @rdjjke in https://github.com/wal-g/wal-g/pull/1573
• Don't throw error in SSH folder if deleting object doesn't exist by @rdjjke in https://github.com/wal-g/wal-g/pull/1574
• Respect upload context in bguploader by @usernamedt in https://github.com/wal-g/wal-g/pull/1575
• Introduce a setting to turn off storage alive checks by @rdjjke in https://github.com/wal-g/wal-g/pull/1579
• Add MongoDB 7.0 support. Remove MongoDB 4.2 support. Fix dropIndexes operation replay. by @dapqa in https://github.com/wal-g/wal-g/pull/1583
• Recheck storages and don't wait cache to expire if they all are dead by @rdjjke in https://github.com/wal-g/wal-g/pull/1582
• MySQL incremental backup by @ostinru in https://github.com/wal-g/wal-g/pull/1571
• Fixed UUID filtering for nested applyOps commands by @dapqa in https://github.com/wal-g/wal-g/pull/1596
• Reuse SSH connections opened once when daemon started by @rdjjke in https://github.com/wal-g/wal-g/pull/1586
• Chec AO/AOCS table EOF by @debebantur in https://github.com/wal-g/wal-g/pull/1594
• S3 throttling test by @debebantur in https://github.com/wal-g/wal-g/pull/1591
• Backup selector tests by @KosovGrigorii in https://github.com/wal-g/wal-g/pull/1603
• Initial ETCD + WAL-G implementation by @KosovGrigorii in https://github.com/wal-g/wal-g/pull/1605
• Fix delete garbage in gp by @debebantur in https://github.com/wal-g/wal-g/pull/1609
• check that delete garbage picks correct oldest backup by @KosovGrigorii in https://github.com/wal-g/wal-g/pull/1607
• Retry fetch file by @debebantur in https://github.com/wal-g/wal-g/pull/1608
• Detect alive storages by stats by @rdjjke in https://github.com/wal-g/wal-g/pull/1602
• Fix closing SSH storage + log conection errors by @rdjjke in https://github.com/wal-g/wal-g/pull/1617
• fixed sequential tests by @debebantur in https://github.com/wal-g/wal-g/pull/1618
• Partial restore wildcard by @debebantur in https://github.com/wal-g/wal-g/pull/1613
• Etcd walg wal push wal fetch by @KosovGrigorii in https://github.com/wal-g/wal-g/pull/1620
• Add namespace to greenplum partial-restore by @debebantur in https://github.com/wal-g/wal-g/pull/1624
• Bump actions/cache from 3 to 4 by @dependabot in https://github.com/wal-g/wal-g/pull/1630
• Bump actions/setup-go from 4 to 5 by @dependabot in https://github.com/wal-g/wal-g/pull/1606
• propagate error from ParseLSN function by @sivukhin in https://github.com/wal-g/wal-g/pull/1612
• MongoDB binary restore: Add resetup of shardIdentity field for mongod… by @ElectricR in https://github.com/wal-g/wal-g/pull/1622
• Add backup selection to ao/aocs check by @debebantur in https://github.com/wal-g/wal-g/pull/1631
• fixes for ao/aocs length check by @debebantur in https://github.com/wal-g/wal-g/pull/1633
• STORAGES doc using wrong Azure variable name by @fgendc in https://github.com/wal-g/wal-g/pull/1626
• Add --after option support for PostgreSQL delete retain command by @victor-tsykanov in https://github.com/wal-g/wal-g/pull/1424
• Add tests for partitionStrings by @ReshetnikovPavel in https://github.com/wal-g/wal-g/pull/1598
• MongoDB binary restore: use most recent to given oplog from oplog arc… by @ElectricR in https://github.com/wal-g/wal-g/pull/1638
• MongoDB binary restore: If shardIdentity document does not exist, ass… by @ElectricR in https://github.com/wal-g/wal-g/pull/1641
• Bump golangci/golangci-lint-action from 3.7.0 to 4.0.0 by @dependabot in https://github.com/wal-g/wal-g/pull/1643
• Pinned minio version in throttling test by @debebantur in https://github.com/wal-g/wal-g/pull/1645
• Avoid resorting to pg_resetwal in catchup test by @x4m in https://github.com/wal-g/wal-g/pull/1642
• GP, prevent multiple backups in the same time by @debebantur in https://github.com/wal-g/wal-g/pull/1634
• Fix MariaDB and GP tests by @ostinru in https://github.com/wal-g/wal-g/pull/1647
• Allow extra tags to be added to statsd metrics by @nick-kentik in https://github.com/wal-g/wal-g/pull/1649
• Fixed file walking in aochecks by @debebantur in https://github.com/wal-g/wal-g/pull/1654
• Added s3 response code metrics by @debebantur in https://github.com/wal-g/wal-g/pull/1653
• Bump softprops/action-gh-release from 1 to 2 by @dependabot in https://github.com/wal-g/wal-g/pull/1655
• Added config settings for handling S3 throttling by @debebantur in https://github.com/wal-g/wal-g/pull/1656
• Bump google.golang.org/protobuf from 1.26.0-rc.1 to 1.33.0 in /internal/tools by @dependabot in https://github.com/wal-g/wal-g/pull/1658
• Changed s3_response_code metric to gauge by @debebantur in https://github.com/wal-g/wal-g/pull/1660

New Contributors

• @heinerstilz made their first contribution in https://github.com/wal-g/wal-g/pull/1326
• @KhurtinDN made their first contribution in https://github.com/wal-g/wal-g/pull/1285
• @VGoshev made their first contribution in https://github.com/wal-g/wal-g/pull/1335
• @rogaliiik made their first contribution in https://github.com/wal-g/wal-g/pull/1348
• @williancolognesitrimble made their first contribution in https://github.com/wal-g/wal-g/pull/1355
• @IncubusRK made their first contribution in https://github.com/wal-g/wal-g/pull/1320
• @Qwiz made their first contribution in https://github.com/wal-g/wal-g/pull/1377
• @bularond made their first contribution in https://github.com/wal-g/wal-g/pull/1374
• @tewboat made their first contribution in https://github.com/wal-g/wal-g/pull/1395
• @teewar2 made their first contribution in https://github.com/wal-g/wal-g/pull/1407
• @Apelsin234 made their first contribution in https://github.com/wal-g/wal-g/pull/1417
• @Fizic made their first contribution in https://github.com/wal-g/wal-g/pull/1408
• @francoispqt made their first contribution in https://github.com/wal-g/wal-g/pull/1423
• @Catmoonlight made their first contribution in https://github.com/wal-g/wal-g/pull/1434
• @AlexUran made their first contribution in https://github.com/wal-g/wal-g/pull/1455
• @denisSurkov made their first contribution in https://github.com/wal-g/wal-g/pull/1460
• @IvanSibirtsev made their first contribution in https://github.com/wal-g/wal-g/pull/1462
• @amagoosebitch made their first contribution in https://github.com/wal-g/wal-g/pull/1479
• @rdjjke made their first contribution in https://github.com/wal-g/wal-g/pull/1478
• @Cockamamie made their first contribution in https://github.com/wal-g/wal-g/pull/1481
• @ElectricR made their first contribution in https://github.com/wal-g/wal-g/pull/1488
• @nikifkon made their first contribution in https://github.com/wal-g/wal-g/pull/1490
• @sergey-arefev made their first contribution in https://github.com/wal-g/wal-g/pull/1494
• @anikin-aa made their first contribution in https://github.com/wal-g/wal-g/pull/1492
• @testwill made their first contribution in https://github.com/wal-g/wal-g/pull/1515
• @noname0443 made their first contribution in https://github.com/wal-g/wal-g/pull/1530
• @debebantur made their first contribution in https://github.com/wal-g/wal-g/pull/1535
• @Quentinchampenois made their first contribution in https://github.com/wal-g/wal-g/pull/1543
• @Alviner made their first contribution in https://github.com/wal-g/wal-g/pull/1547
• @ne2pit made their first contribution in https://github.com/wal-g/wal-g/pull/1546
• @zlcnju made their first contribution in https://github.com/wal-g/wal-g/pull/1563
• @olafeonk made their first contribution in https://github.com/wal-g/wal-g/pull/1572
• @dapqa made their first contribution in https://github.com/wal-g/wal-g/pull/1583
• @KosovGrigorii made their first contribution in https://github.com/wal-g/wal-g/pull/1603
• @sivukhin made their first contribution in https://github.com/wal-g/wal-g/pull/1612
• @fgendc made their first contribution in https://github.com/wal-g/wal-g/pull/1626
• @victor-tsykanov made their first contribution in https://github.com/wal-g/wal-g/pull/1424
• @ReshetnikovPavel made their first contribution in https://github.com/wal-g/wal-g/pull/1598
• @nick-kentik made their first contribution in https://github.com/wal-g/wal-g/pull/1649

Full Changelog: https://github.com/wal-g/wal-g/compare/v2.0.1...v3.0.0

This is a pre-release to test the aws iam roles config setting.

Release notes:
This release mainly consists of various bug fixes and improvements.

Common:

• Add arm(aarch64) build pipeline for ubuntu20.04 binaries (#1304)

PostgreSQL:

• Add warning output on backup-fetch retries (#1325)
• Remove redundant check for --without-files-metadata flag (#1321)
• PostgreSQL 15 Support (#1319)
• Fix of deleting WAL-E backups (#1292)

MySQL/MariaDB:

• Fix a regression in MariaDB backup-push handler (#1313)
• Binlog cache fixes (#1281)

Greenplum:

• Optimize delete handler (#1300, #1302)
• Optimize storage listing queries during the backup push (#1312)
• Use AO files metadata instead of backup references (#1308)
• Refactor AO/AOCS storage metadata queries (#1280, #1278)

Storages:

• Azure authentication using the default credential chain (#1282)
• Update aws-sdk-go to v1.44.7 (#1262)
• Add check read/write to storage tools (#1275)

Release notes:

This release contains a huge amount of new features, bug fixes, and performance improvements. It is worth noting that v2.0 Postgres delta backups are not backward-compatible with the pre-2.0 versions. It means that the v2.0 delta backup can't be restored with the v1.1 or earlier versions. Also, this release introduces the initial support for Greenplum physical backups.

WAL-G for MySQL now supports the parallel backup-push/backup-fetch and has an optimized binlog upload mechanism. WAL-G for SQLServer now supports the backup encryption and compression as well as the external storage backup import/export.

Common:

• Update lz4, brotli #1165, #1179
• Multiple libsodium fixes #1087, #1227
• Bash/zsh shell completions #1250, #1252
• Add a 'flags' subcommand #1033
• Disable CPU Limits in turbo mode #1121
• Metrics forwarding via statsd protocol #1240

SQLServer:

• Add SQLServerDBConcurrency setting to limit concurrent database operations #1067
• Add LATEST keyword handling #1110
• Add norecovery option for backup log command #1115
• Add SQLServer compression / encryption #1161
• Backup import and export to the external storage #1184

Postgres:

• Add CopyComposer for full backups optimization using remote copy API #1012
• Add streamed JSON sentinel serialization to decrease memory usage #1103
• Extract heavy files and tarfilesets from the backup sentinel #1114
• Introduce WALG_TAR_DISABLE_FSYNC to skip fsync during backup-fetch #1144
• Add reduced memory usage backup mode #1101
• Improvements and fixes for remote backup #1175
• Add pgBackRest backups restore support (beta) #1139, #1202
• Add 'wal-restore' command #977
• Add 'delete garbage' command #1217, #1218
• Return custom error code when WAL file does not exist #1195
• Add WALG_ALIVE_CHECK_INTERVAL to fail backup on Postgres process failure #1260
• Reformat LSN output #1259

MySQL:

• Add wal-g binlog-analyze #1085
• Check binlog GTIDs before pushing binlogs to S3 #1190
• Add --check-gtids to wal-g binlog-push #1085
• Add --until-binlog-last-modified-time option to wal-g-mysql binlog-replay #1154
• Improve MySQL restore performance #1131
• Allow backups on cascade replicas #1168
• Fix mysql backup of a lagging replica #1223
• Make mysql bilong-push status observable to 3rd party tools #1229

Greenplum:

• Initial physical backups support for GPDB 6.19.2+. API might change in the future versions.

Storages:

• Add get/put/rm/cat to storage tools #1069, #1164
• S3: Add download by range #1083
• S3: Add connection reset retries #1137
• S3: Add WALG_S3_MAX_RETRIES setting #1148
• S3: Fix S3_LOG_LEVEL setting #1158
• S3: AWS authentication fixes #1209
• S3: Yandex Cloud session token auth support #1213
• SSH: Ensure trailing delimiter in the storage prefix #1130
• Azure: Switch to the recommended azblob SDK #1135, #1171
• Azure: Fix ListFolder never returning error #1208
• Swift: Ignore files that were not found during ListFolder #1129

This is a pre-release to test S3 authentication with mechanisms other than key/secret.

This pre-release introduces the Greenplum physical backups support.

This pre-release introduces the S3_LOG_LEVEL setting to help with debugging the S3 storage interaction issues.

This is a pre-release intended to test the s3 connection reset retries fix (https://github.com/wal-g/wal-g/pull/1137).

Some interesting features/fixes:

• Include libsodium in the GitHub release builds. Exit with failure if libsodium is requested but WAL-G was not compiled with libsodium support #1062 CVE-2021-38599
• UserData is now required to be a valid JSON #1049
• backup-list now sorts backups by the astronomical time of the completition #907

Postgres:

• .history files overwrite prevention fix #1052
• wal-verify can now be run from a standby #1014
• wal-verify now ignores the permanent backups #1014, #1002

MongoDB:

• Add support for MongoDB 5.0 #1041

SQLServer:

• Fix MSSQL driver import #1038

MySQL:

• Add --turbo flag to disable limiters #1027
• Fix --detailed backup-list bug #1026
• Add --until flag to binlog-push #1005

Redis:

• Add 6.2 support & redis-cli bug workaround #1022

Storages

• S3: customer SSE key support #1042 (https://github.com/wal-g/storages/pull/48)
• Azure: support AZURE_STORAGE_SAS_TOKEN without the leading '?' #1056
• Azure: disable the unnecessary syslog writes #1058

Release notes:

This time we decided to bump the number to 1.0 to switch to a common versioning pattern.
WAL-G 1.0 is compatible with prior versions (v0.2.0+). This might change with later releases.
Also, WAL-G 1.0 can restore WAL-E backups and WALs.
Good news for non-PostgreSQL databases: WAL-G is now production-ready for MS SQL, MySQL databases, meanwhile the API can be changed in future releases. MongoDB and Redis support is in the beta stage.

Some interesting features/fixes:

• #780 PostgreSQL remote backups
• #905 Improve compatibility with PostgreSQL 13, introduced WALG_PREFETCH_DIR option
• #824 New option was added to wal-verify, now it requires an argument timeline|integrity
• #903 It's now not allowed to create non-permanent delta backups from permanent backups
• #842 Fix compatibility with S3 implementations that do not support ListObjectsV2
• #950 It's now again possible to tell WAL-G to rename .ready files to .done directly
• #898, #962 WAL prefetch is now slightly faster
• #858 Catchup-list command to list delta backups taken specifically for catchup
• #847 Support for Yandex Cloud KMS
• #817, #833 Multipart retries for GCS storage
• #873 Support for AzureUSGovernmentCloud, AzureChinaCloud, AzureGermanCloud
• #899 Add metadata for WAL files
• #913, #917 New option to delete single backup for PostgreSQL
• Add Redis backups. Now we have Redis support with backup-fetch, backup-push, backup-list and delete commands. For full info see Redis.md
• Add MySQL catchup mode: allow replay binlogs while uploading new ones
• Add SQLServer single database restore with renaming
• Huge amount of bug fixes, tests and improvements for Redis/MongoDB/SQLServer/etc

Thanks!

This is QA pre-release per request #970
We will cut a release when the QA cycle will be finished.

This is QA pre-release to test changes in background WAL upload

This is QA pre-release intended to test libsodium settings.
#776

Notable changes in this release include:

1. Fixes for S3 and GCP storages #656 #756.
2. Add wal-show command to get information about wal storage folder.
3. Add wal-verify command. It checks the integrity of WAL history starting from the oldest backup available in current timeline history.
4. Add wal-receive command. You can use WAL-G as a replica running on another host to ensure RPO=0. This is beta functionality: API may change in the future.
5. Add reverse delta unpack for backup-fetch (--reverse-unpack flag).
6. Add redundant archives skipping for backup-fetch (--skip-redundant-tars flag, designed to work in pair with reverse delta unpack).
7. Add page checksum verification for backup-push (--verify flag).

You can find more about some of these new features in Daniil Zakhlystov's post

This is QA prerelease to test new wal-receive feature.

This is QA pre-release to cope with AWS S3 SDK bug. See https://github.com/wal-g/storages/pull/22, #656 and #727 for details.

This is a QA pre-release to test refined logic of parallel WAL upload.
See #644 for details.

Notable changes in this release include:

• Support of Libsodium encryption
• Retryer for AWS storage
• Go modules
• Add catchup commands. It allows cut replication lag on a PostgreSQL cluster.
  Simple scenario:

1. Get current LSN of a replica with high replication lag.
2. Create an incremental backup with catchup-push since the replica LSN.
3. Apply it to the replica with catchup-fetch command.

Thanks for all your contributions, bug reports, and feature requests!

Draft prerelease to test #593