duplicacy

• Upgrade storj.io/uplink to v1.12.1 to fix an 64 bit alignment issue: https://github.com/gilbertchen/duplicacy/commit/e99dfea048e0e42092835e9e57ee56dfbdb6184b

• Save storj credentials in keychain/keyring after a storj storage has been successfully created: https://github.com/gilbertchen/duplicacy/commit/7bfc0e7d5114c448ae7c90c6cda06faccdf78eab

• Removed a false check that caused an initialization error with zstd-enabled storages: https://github.com/gilbertchen/duplicacy/pull/653 (thanks @gorbak25)
• Allow two copy-compatible storages to have different compression levels: https://github.com/gilbertchen/duplicacy/commit/4e9d2c4cca7b4ebd394323fa154918da9bf2e939
• Skip identical entries when listing chunks: https://github.com/gilbertchen/duplicacy/commit/d92b1734f476d7f67f176d7f853c7013baffb301
• Upgraded Go version to 1.19.13

• Added a Samba backend: https://github.com/gilbertchen/duplicacy/commit/3a81c1065add9ec885c6fe88126446308b563e5f
• Implemented zstd compression: https://github.com/gilbertchen/duplicacy/commit/53b0f3f7b6b998e74d79b98ea53b60eb90a26aad
• Added support for custom OneDrive credentials: https://github.com/gilbertchen/duplicacy/pull/632 (thanks to @sevimo123)
• Added support for SharePoint document libraries to the ODB backend: https://github.com/gilbertchen/duplicacy/pull/633 (thanks to @sevimo123)
• Fixed a connection leak in the Dropbox backend on some http errors: https://github.com/gilbertchen/go-dropbox/pull/5 (thanks to @northnose)
• Fixed bugs that caused the B2 and Google Drive backends to fail to download metadata chunks that have been marked as fossils: https://github.com/gilbertchen/duplicacy/commit/ff207ba5bfe89a4ccdfa229bd225f5cf4850e35a https://github.com/gilbertchen/duplicacy/commit/1f9ad0e35c3fb24118d746612d4fad33626228be
• Fixed a crash when some backends return with an empty entry path: https://github.com/gilbertchen/duplicacy/commit/cdf8f5a8575fa50e22cad126a00140e38618b9fc
• Fixed a crash caused by concurrent access to a map during chunk verification: https://github.com/gilbertchen/duplicacy/pull/649 (thanks to @northnose)
• Fixed a chunk leak when listing files in a revision: https://github.com/gilbertchen/duplicacy/commit/9be475f876c0f4eab9cea7fa0b356d927d1628f9

• Updated github.com/minio/highwayhash to 1.02 to fix incorrect hashes for Erasure Coding on arm64 [1]: https://github.com/gilbertchen/duplicacy/commit/6a7a2c80480aefba59c690eaf32ad742ec687667
• Add -rewrite to the check command to fix corrupted chunks with Erasure Coding enabled:
• Fixed a bug that didn't preserve the version bit when copying snapshots in the old V2 format [2]: https://github.com/gilbertchen/duplicacy/commit/58f0d2be5a4356307dfecba56f1fd0ea982f6a92
• Use the updated access token after a token refresh in the dropbox backend: https://github.com/gilbertchen/go-dropbox/commit/60ebcdbdd837d267393cab74f5ba3523e23e3160 (by @stuckj)
  https://github.com/gilbertchen/duplicacy/commit/bc2d762e41c5ce11be01bffe32b39edc4d5b2eaa
• Fixed a chunk leak with check -files: https://github.com/gilbertchen/duplicacy/commit/b8c7594dbf422878d167411ae635ceeb890034d7
• Fixed 2 bugs that are results of handling zero-byte files incorrectly: https://github.com/gilbertchen/duplicacy/commit/3472206bcf861a28dfec61410b5970a7700fb4d6

Notes:
[1] The highwayhash is only used to calculate the checksum of each shard in Erasure Coding and not used anywhere else. So you are affected by this bug only if you enable Erasure Coding on arm64 machines. This bug is present in both 3.0.1 and 2.7.2 versions of the macOS m1 builds, and only in 3.0.1 of the linux arm64 builds.

In this release both the correct and incorrect versions of the highwayhash will be calculated so there is no need to fix existing storages, but you can use the newly added -rewrite option to rewrite affected chunks using the correct version of the hash if desired.

[2] If you are using 3.0.1 to copy snapshots created with 2.7.2 or earlier, this bug will affect you. In 3.0.1, the version bit (0 for V2 snapshots and 1 for V3 snapshots) was the only flag used to determine the format of a snapshot. However, in this release, the first byte of the snapshot file is also used to distinguish between V2 and V3 formats. As a result, there is no need to rewrite V2 snapshots that were already copied with the incorrect version bit.

• Completely rewrite the backup engine to achieve near-constant memory usage regardless of backup size (https://github.com/gilbertchen/duplicacy/pull/625)
• New storage backend for Storj (https://github.com/gilbertchen/duplicacy/commit/6009f64b66d844ad8db023e9bf6bd3b38a8ebeca)
• Use long-lived refresh token for the Dropbox backend (https://github.com/gilbertchen/duplicacy/commit/cde660ee9f528324a3f07305c06cf0951ac0184c)
• Update the Swift storage to V2 (https://github.com/gilbertchen/duplicacy/pull/626)
• Add a global option -print-memory-usage to print memory usage during execution
• Find the storage path in shared folders first when connecting to Google Drive (https://github.com/gilbertchen/duplicacy/commit/e43e848d47176144049b93938f09820c61effc72)
• Support GCD impersonation via modified service account file (https://github.com/gilbertchen/duplicacy/pull/612)
• Built with Go 1.16.15 with Go modules enabled

• Completely rewrite the backup engine to achieve near-constant memory usage regardless of backup size (https://github.com/gilbertchen/duplicacy/pull/625)
• New storage backend for Storj (https://github.com/gilbertchen/duplicacy/commit/6009f64b66d844ad8db023e9bf6bd3b38a8ebeca)
• Use long-lived refresh token for the Dropbox backend (https://github.com/gilbertchen/duplicacy/commit/cde660ee9f528324a3f07305c06cf0951ac0184c)
• Update the Swift storage to V2 (https://github.com/gilbertchen/duplicacy/pull/626)
• Add a global option -print-memory-usage to print memory usage during execution
• Find the storage path in shared folders first when connecting to Google Drive (https://github.com/gilbertchen/duplicacy/commit/e43e848d47176144049b93938f09820c61effc72)
• Support GCD impersonation via modified service account file (https://github.com/gilbertchen/duplicacy/pull/612)
• Built with Go 1.19 with Go modules enabled

• Make -vss work with Big Sur: https://github.com/gilbertchen/duplicacy/commit/7c36311aa973c4189a68c4eae4191fa8bf9417e9
• Skip chunks already verified in previous runs when running check -chunks: https://github.com/gilbertchen/duplicacy/commit/d7c1903d5a5b06716265a3ea24bd135764a0813d
• Fixed reading/writing extended attributes on Linux by switching to github.com/pkg/xattr: https://github.com/gilbertchen/duplicacy/commit/b392302c0680ca4de569630031c43bab21f51d82
• Fixed a bug that caused a fresh restore of large files to fail without the -overwrite option: https://github.com/gilbertchen/duplicacy/commit/bec3a0edcd6329ed57d287eec996858a3621f567
• Validate that a repository id can only contain letters, numbers, dashes, and underscores: https://github.com/gilbertchen/duplicacy/commit/5eed6c65f688aa5dda1c45143837f4f0f886fff4
• Fixed a crash in the Dropbox backend caused by a nil Seeker: https://github.com/gilbertchen/go-dropbox/commit/2233fa1dd846b3a3e8060b6c1ea12883deb9d288

This release is mainly to fix a crash caused by unaligned 64-bit integers on 32-bit OS. If you're running Duplicacy on 64-bit OS there is no need to upgrade.

Changes:

• Fixed a 64-bit integer alignment problem on 32-bit OS: https://github.com/gilbertchen/duplicacy/commit/0da55f95ab9605f6053c7c67949419d99d377aa0
• Allow chunks subdirectories in the local disk storage to be symlinks : https://github.com/gilbertchen/duplicacy/commit/7e372edd68d59785f0d2bffe66c198717bcbc50e https://github.com/gilbertchen/duplicacy/commit/2f407d6af9f15c9fa862a60aaac8037ecaa7ba76

• Added support for erasure coding to detect and recover from corrupt chunks: https://github.com/gilbertchen/duplicacy/pull/606
• Added a -persist option to check and restore to continue despite errors: https://github.com/gilbertchen/duplicacy/pull/595 (by @twlee79)
• Added exclude_by_attribute preference to exclude files based on os-specific extended attributes: https://github.com/gilbertchen/duplicacy/pull/498 (by @plasticrake)
• Fix and improvements for the OneDrive backend: https://github.com/gilbertchen/duplicacy/commit/eecbb8fa9904ccec424a4cd46299ca278e04f11e
• Improvements for the WebDAV backend: https://github.com/gilbertchen/duplicacy/pull/549 (by @Jos635), https://github.com/gilbertchen/duplicacy/commit/e3e3e97046ee17299a973082e1cb0843fc0d3020
• Fixed a bug that broke the Swift backend: https://github.com/gilbertchen/duplicacy/pull/594 (by @alecuyer)
• Added a -download-threads option to the copy command for specifying the number of downloading threads: https://github.com/gilbertchen/duplicacy/commit/947006411b5f5306d55f40516e977f3c100b986c
• Show progress log messages during copy: https://github.com/gilbertchen/duplicacy/commit/947006411b5f5306d55f40516e977f3c100b986c
• Show progress log messages when verifying chunks (check -chunks): https://github.com/gilbertchen/duplicacy/commit/d0b3b5dc2ed068e5c5bdf6978e9fa8516fb34a57

This release is to fix a bug where upload errors may be ignored by the sftp backend resulting in incomplete chunk files.

If you're backing up to an sftp storage, we suggest you upgrade to this version and run a check command with the -chunks option to find out if any chunk is affected by this bug.

This release is to fix a bug that causes a check command to fail if there are empty temporary chunk files in the disk or sftp storage. These files may be created by a failed or ongoing chunk upload so they are usually harmless and should not throw an error.

New Features:

• Add a new backend for StorageMadeEasy's File Fabric storage: https://github.com/gilbertchen/duplicacy/commit/ed8b4393bec35eb1e4410535944685501d5523cb
• Add a -key-passphrase option to pass in passphrase for RSA private key: https://github.com/gilbertchen/duplicacy/commit/089e19f8e6474aa7202bce3e49ec6bcb9f66ece4
• Use multiple threads to list the chunks directory for Google Drive: https://github.com/gilbertchen/duplicacy/commit/153f6a2d206e79b970bfe0fcc995eaad7a5c5025
• Add a variation of the SFTP backend to support older ciphers and key exchange algorithms for compatibility: https://github.com/gilbertchen/duplicacy/commit/1adcf5689048887cc05c0a600235249e606423d1
• Error out in the check command if there are 0-size chunks: https://github.com/gilbertchen/duplicacy/commit/fe854d469dbd31b4165b3062bb727e5aa1e3be9d

Bug fixes:

• Fix a crash when a username is not specified in the WebDAV storage url: https://github.com/gilbertchen/duplicacy/commit/1da7e2b53667b9ecab5e75fe65de45a1a5088591
• Fixed a bug in Dropbox backend that can cause 0-byte files to be uploaded after retrying on an error: https://github.com/gilbertchen/duplicacy/commit/5e28dc491184e4bd8d4dd33426b2ad1ea5612442
• Retry on "unexpected EOF" errors for the webdav backend: https://github.com/gilbertchen/duplicacy/commit/f2f07a120d81927cdf5d9b960be8373b05cfdfc0
• Clear the loaded content after a snapshot has been verified to reduce memory usage: https://github.com/gilbertchen/duplicacy/commit/5d459990777cc955c5de0b263cae0aef7dfb467f
• Ignore 0-byte chunks passed in by the chunk reader: https://github.com/gilbertchen/duplicacy/commit/09e3cdfebf0a8913c2742bcbde6d1beee3fc7e0d

• Fixed a bug that caused check -chunks to store every chunk in the local cache .duplicacy/cache: https://github.com/gilbertchen/duplicacy/commit/6ca8b8dff0f6216c66c82ceb5ef1b883616b7d47
• Fixed a bug that the token file path for OneDrive Business isn't saved to KeyChain/Keyring: https://github.com/gilbertchen/duplicacy/commit/9c3122b814b88de3ec66211ab15b0c8f7d49c065

• Fixed a bug in ssh login with encrypted private key: https://github.com/gilbertchen/duplicacy/commit/835af1133481111523838c774f446c0173d99bca

• Add support for Microsoft OneDrive for Business: https://github.com/gilbertchen/duplicacy/commit/d26ffe2cffba2064c3c76b0cb9a93001b2892190
• Add support for Google Shared Drive: https://github.com/gilbertchen/duplicacy/pull/579 by @rsanger
• Ignore a "operation not supported" error when backing up to a Samba share that doesn't support the sync operation: https://github.com/gilbertchen/duplicacy/pull/590 by @fbarthez
• Add a -chunks option to the check command for verifying chunks: https://github.com/gilbertchen/duplicacy/commit/22d6f3abfc61c12d6c2d518f5bdd7e0f346c5a10
• Allow a ssh private key file to be used together with a signed certificate trusted by the server: https://github.com/gilbertchen/duplicacy/pull/547 by @philband
• Fail the backup if the repository can't be accessed or there are no files in the repository: https://github.com/gilbertchen/duplicacy/commit/810303ce25bd444d5ca1e840519abb5ae0dcaa3b
• Access Google Drive via service accounts: https://github.com/gilbertchen/duplicacy/commit/7208adbce20421d0801fe11beb404953fb76ed2d
• Add a global -suppress option to suppress logs with given ids: https://github.com/gilbertchen/duplicacy/commit/749db78a1f47666b48f5bdfc10b82d6103353d87
• Call ReadAll() on the http body to allow the http keepalive connection to be reused in the dropbox backend: https://github.com/gilbertchen/duplicacy/pull/593 by @freaksdotcom
• Update github.com/aws/aws-sdk-go and google.golang.org/api to the latest: https://github.com/gilbertchen/duplicacy/commit/eebcece9e038532eebee37bceadbb61de3b5d973

This is an emergency release for fixing a bug in the copy command when the destination storage is RSA-encrypted: https://github.com/gilbertchen/duplicacy/commit/6699e2f44046e5c861f11efe01785a474db4815d

You're affected by this bug if you're

1. running 2.3.0 and copying from a source storage of any type (unencrypted, RSA-encrypted, and non-RSA-encrypted) to an RSA-encrypted destination storage
2. running 2.4.0 and copying from a source storage (unencrypted or non-RSA-encrypted) to an RSA-encrypted destination storage

You'll need to clean up the storage and start from fresh; otherwise either all chunks are not encrypted by RSA (case 2), or all chunks are encrypted by RSA (case 1) causing all commands (backup, list, check, and prune) to require the private key to work.

• Reduced the memory usage of the copy command by releasing the list of chunk hashes after processing each snapshot: https://github.com/gilbertchen/duplicacy/commit/d43fe1a282e4c7e46ab2b3eb2b3849c9e55652d1
• Fix the retry logic in the sftp backend: https://github.com/gilbertchen/duplicacy/commit/e888b6d7e5609ba13cfc665c9c34def86eba9834
• Support free downloading from B2 with a custom URL: https://github.com/gilbertchen/duplicacy/pull/589
• Fixed a bug that caused all copied chunks to be RSA encrypted: https://github.com/gilbertchen/duplicacy/commit/cc88abd547fa6ee1b82fa6a4eeabca9844484098
• Fixed handling of repository ids with spaces in the b2 backend: https://github.com/gilbertchen/duplicacy/commit/a6ce64e715cd0e9b3340effe7a06e38aa2428fdd

Features:

• RSA encryption (https://forum.duplicacy.com/t/new-feature-rsa-encryption/2662)
• Allow a custom location for the filters file (https://github.com/gilbertchen/duplicacy/commit/a99f059b52a9d45b245a15b36582667a3ab20652)

Bug fixes:

• For the check command, -tabular should imply -all just like -stats (https://github.com/gilbertchen/duplicacy/commit/165152493c02affc3dad0c3fe334ff09dfedf57e)
• Make -vss work on Mac OS Catalina (https://github.com/gilbertchen/duplicacy/pull/578 by @gboudreau)
• Update pkg/sftp to 1.10.1 to fix a deadlock in the sftp backend (https://github.com/gilbertchen/duplicacy/commit/9abfbe1ee06660ac320bf7bc767c1380458081b1, see discussion at https://forum.duplicacy.com/t/deadlock-during-backup/2835)
• If a symlink is a directory, match it against the patterns as a directory (https://github.com/gilbertchen/duplicacy/commit/b32c3b2cd52ee99f17c2252c938b8e5d932a6586)
• Fixed a path not found error on Windows when the path to a symlink is too long (https://github.com/gilbertchen/duplicacy/commit/ca7d927840d0764b6e83fb728531ecd1cfb0a154)

Improvements:

• Throw an error, instead of a warning, if pre/post script fails (https://github.com/gilbertchen/duplicacy/commit/694494ea54aa5284ebc9804a4dce18e31768d713)
• Increase the number of retries for the b2 backend (https://github.com/gilbertchen/duplicacy/commit/e02041f4ed7d5027cfab2c56403c633d1dcad083)

• Fix a B2 401 error when using an application key by providing the bucket name to b2_list_buckets (https://github.com/gilbertchen/duplicacy/pull/570 by @philband)