wal-g

Features:

• Preview of tablespace support
• delete everything - when you do not need an archive anymore

Bugs fixed:

• Major race condition in case of highly concurrent wal uploads fixed #417
• Fix --version #381

In this pre-release, we fixed a bug of incorrect pipeline closing on network error. For GCS it could manifest in haning wal-g backup-push in case of a transient network fault (#266)

In this prerelease:

1. Fix compatibility with WAL-E : previous pre-release did not contain LZO-enabled binary. Now there is one binary again, but this binary is compatible with WAL-E.
2. Fix configuration of SWIFT storage from config-file

This pre-release fixes numerous bugs:

1. Crash on deleting old backups #319
2. Escaping of new lines in WALG_PGP_KEY #320
3. Fixed many different problems with config file usage
4. Fixed home directory detection problem with getent #216

Also, there are some features:

1. Add aws/kms + minio/sio symmetric client-side encryption #315
2. Avoid mutating interactions with archive_status #127

We have significantly improved our CI. Now it is faster with more tests.

Thanks to everyone involved!

Notable changes in this release include:

• Support of brotli compression (additionally to lz4 and lzma).
• WAL-delta backups (a.k.a. fast block-level incremental backup). This feature enables scanning of WAL during archivation, the gathered information is used to make delta backup much faster.
• PGP encryption without external GPG.
• Prefault of database pages during recovery. When standby is requesting new segments of WAL we know which blocks need to be put into page cache and try to prefetch them to make WAL replay faster.
• Support for S3, GCS, SWIFT, FS, and Azure storages.

Many thanks to those who contributed by pull requests, bug reports, issue discussions, and feature requests!

This pre-release fixes several bugs:

1. Parallel wal-push could be stuck in deadlock under several circumstances. #222
2. delete could remove first WAL segment needed to the oldest backup done by WAL-E #230 #232
3. delete could remove WALs incorrectly in GCP bucket #230 #233

Many thanks for reporting these bugs @brouberol @CyberDem0n @kel00s and others.

This pre-release intended for verification of fix for #222 : a deadlock in parallel WAL archiving in case of high concurrency uploads.

This pre-release fixes bug that could manifest with high WALG_UPLOAD_DISK_CONCURRENCY and network-constrained backup-pushes (as opposition to disk-constrained).
Due to the race condition in finishing uploads last tar file could be left unfinished.
For details see #198

This release fixes bug that could manifest with high WALG_UPLOAD_DISK_CONCURRENCY and network-constrained backup-pushes (as opposition to disk-constrained).
Due to the race condition in finishing uploads last tar file could be left unfinished.
For details see #198

This pre-release contains new settings for reading PGP keys (see #173 ).
This new API may change in the release version, but most probably it will not.

Thanks to @Savichev-Igor for his work and discussion about the feature.

This pre-release fixes:

1. Bug in delta backups (introduced in v0.2.3, see #177 #185 )
2. Bug in WAL-E restoration (introduced in v 0.2.3 too, see #172 #184 )

Thanks to @tam-i13 , @cjhubert, @DeathBorn and others who reported these problems.

This pre-release fixes WAL-E backup-fetch.
In previous pre-releases, during the introduction of Storage Folder abstraction, we misplaced check of dedicated pg_control tar. This check was performed before we knew that backup is done via WAL-E and has no separate pg_control tar.

This prerelease allows using GCS with default service account credentials.
Also, fixed one more bug in the retry logic of backup-fetch.

In this prerelease was fixed issue of broken retries during backup-fetch.

This is an experimental version with a major refactoring of the storage logic.
This refactoring allowed to include support of Google Cloud Storage and file system for storage platform.

lz4 codec is updated to fix lz4 issues of v0.2.0.

Also, @Savichev-Igor restructured files structure and build logic to make it easier to dive in for future contributors.

This is a pre-release with new cool features:

1. WAL-delta technology speed up delta backup dramatically
2. Page prefault speed up WAL replay by warming page cache of file system
3. Brotli encoder is now supported

Severe WAL delete error fixed.
Due to incorrect WAL window computation, WAL retention window could be much narrower than oldest backups. LATEST backup would still recovery correctly (including PITR), but older backups could miss necessary WALs for recovery.
The issue was introduced in v0.1.13 (2018-10-27) with commit https://github.com/wal-g/wal-g/commit/8791dfed2210f30145c3dc9f692b1e4a80b997c8 , version v0.1.12 is not affected.

In future releases delete tests will be covered by integration e2e test, this release includes only unit tests.

Also, this release includes some minor error formatting enhancements.

This pre-release fixes problem of LZO archives padded with zeroes (see comments in #22 ) and ensures WALs are kept correctly even in case of storing in the same folder after pg_upgrade(fix #121, but do not do this anyway).

Fixes:

1. Zstd is temporarily disabled due to corruption reports. We have reproduction and working on fixing the issue. The issue arises on delta backups with a base larger than few Tb on highly concurrent uploads. More info can be found here and here.
2. Fixed WAL prefetch. WALs close to the end of the current timeline were downloaded without parallelism.

Features:

1. Backup-fetch now retries failed tars with decreasing concurrency to mitigate issues of S3 implementations with throttling on the server side.
2. WAL-G now has end-to-end integration tests in docker.

News:

1. Considering Brotli as alternative codec with good CPU vs compression ratio tradeoff.
2. Forked branch REL0_1_STABLE to separate development and support.

Stay tuned.

Bug fixes:

1. Better compatibility with WAL-E backups and archive (see #100 )
2. More safety features and error prevention (#102 #109 )

Features:

1. Zstd and Lzma compression (#99 )
2. Disk and network throttling (#101 )

Thanks to everyone, who reported, fixed, suspected, diagnosed and mitigated bugs, crafted new features and pushed WAL-G further.