duplicacy

• Better handling of B2 authorization failures (https://github.com/gilbertchen/duplicacy/commit/045be3905b38341ecb2cc6db66bcd8d8f25bb852)
• Fixed a bug that caused 'check -files' to download the same chunk multiple times if shared by multiple small files (https://github.com/gilbertchen/duplicacy/commit/4da7f7b6f900aadc45e440da9786bd92005b043d)
• Update github.com/gilbertchen/go.dbus to fix a double close bug when accessing keyring on linux (https://github.com/gilbertchen/duplicacy/commit/41668d4bbd3652de45f63ae81479be4ef9ce549c)
• Don't compare hashes of empty files in the diff command (https://github.com/gilbertchen/duplicacy/commit/9d4ac34f4b64f17ce2ea415f9b0a13f386ee491c)
• Retry on broken pipe errors in the Azure backend (https://github.com/gilbertchen/duplicacy/commit/6efcd37c5cf86c8699cefa2abc224832282faba0)

This release is mostly a bug fix version for 2.2.0 which has a bug that can't restore individual files due to parent directories not being created correctly: https://github.com/gilbertchen/duplicacy/commit/47c4c25d8b3a99bc73e3004b2dabfe6356749957

Other changes:

• Licensing change: the CLI license is now per-computer: https://github.com/gilbertchen/duplicacy/commit/37781f954084fc3f768ce45a2ed3680886eedd64
• Add a TRACE log message when skipping an unchanged file during restore: https://github.com/gilbertchen/duplicacy/commit/6e7d45caacdee182430a430c4b74607b16e0a941

• Allow the filters file to include other filters files (https://github.com/gilbertchen/duplicacy/pull/514)
• On Windows add the \?\ prefix to add paths to support UNC paths in the storage URL (https://github.com/gilbertchen/duplicacy/commit/53548a895f0ba9e9ade8b8dbb21f15fde4c164ef)
• Support ssh private key files encrypted by passphrases (https://github.com/gilbertchen/duplicacy/commit/8aa67c816254fb4f3592c0a3dc3de1ecd12a9424)
• Add a Sync call before closing a file when uploading a file to local storage (https://github.com/gilbertchen/duplicacy/commit/bb652d0a8c7e89e9f0d4cb8acef7fdc453e21a6b)
• Ignore false malware warnings from Google Drive that prevent some chunks to be downloaded (https://github.com/gilbertchen/duplicacy/pull/447)
• Retry on EOF errors in the SFTP backend (https://github.com/gilbertchen/duplicacy/pull/489)
• Replace special characters in environment variable name with underscores (https://github.com/gilbertchen/duplicacy/pull/495)
• Fixed a webdav compatibility issue with rclone (https://github.com/gilbertchen/duplicacy/commit/2b56d576c7ac192d8fefa512b91840f85c79f61e)
• Set the content length for upload in the webdav backend (https://github.com/gilbertchen/duplicacy/commit/d16273fe2bd1e10c770122e9dda2bc085b9e69aa)
• Fixed a bug where a wrong variable is used as the number of threads causing incorrect rate limits (https://github.com/gilbertchen/duplicacy/commit/43a5ffe0116b753180b76adb182e0fd9df5a722b)
• Fixed a bug where filenames starting with i or e are mistakenly interpreted as regex (https://github.com/gilbertchen/duplicacy/commit/abcb4d75c186051d57b15cdb895c9081d2ef24b0)
• Fix a memory issue that causes check -tabular uses too much memory with hundreds of revisions (https://github.com/gilbertchen/duplicacy/commit/4b69c1162e442ae62c961344a037830c30ac24e1)
• Add an additional lookup for a chunk not in the known chunk list (https://github.com/gilbertchen/duplicacy/commit/1da151f9d954e034f72252934c95e7bc707abe76)
• Fixed a MoveFile bug in Wasabi when the storage is at the root of a bucket (https://github.com/gilbertchen/duplicacy/commit/a6fe3d785e8bbb07fd9ab1499b282ed159139595)
• Retry on 408 errors from Google Drive (https://github.com/gilbertchen/duplicacy/pull/529)
• The cat command doesn't need to load the entire file into memory (https://github.com/gilbertchen/duplicacy/commit/458687d543a03b6e6a74d2a6a20b5b8bc81062d8)
• Rework the Backblaze B2 backend (https://github.com/gilbertchen/duplicacy/commit/57a408a577d472d17f7018071bf4887f0b43a891)
  • All APIs include UploadFile are done via the call() function
  • New retry mechanism limiting the maximum backoff each time to 1 minute
  • Add an env var DUPLICACY_B2_RETRIES to specify the number of retries
  • Handle special/unicode characters in repositor ids
  • Allow a directory in a bucket to be used as the storage destination

• Fixed a bug in calculating the padding size during chunk encryption: https://github.com/gilbertchen/duplicacy/commit/21b3d9e57f087d0d7913b0bd444557d4a5208929
• Print the number of files if available in the snapshot file before downloading the file list: https://github.com/gilbertchen/duplicacy/commit/244b797a1c9c1e8ae2d48dd7bb464fe76ba613fc
• Don't list snapshots whose tags don't match the given one when the -tag is provided: https://github.com/gilbertchen/duplicacy/commit/073292018cf42361d05e43d9499fa0c3b387d3b8
• Show more statistics in the check command (for the new web-based GUI): https://github.com/gilbertchen/duplicacy/commit/15f15aa2ca5b035a9e5b15331548d2c997b12d04
• In some backends the benchmark command may incorrectly list the chunks directory when looking for previous temporary files: https://github.com/gilbertchen/duplicacy/commit/d8e13d8d85936e88481f9ced6ca2e6e6e3f6d1d0
• Optimizing restore to avoid reading newly created sparse file: https://github.com/gilbertchen/duplicacy/commit/bfb4b44c0ab9afbbd33cd641c2dab49c3518b751
• Align snapshot times to the beginning of days when calculating the time differences so that prune operations running on the same day will prune the same set of old snapshots: https://github.com/gilbertchen/duplicacy/commit/22a0b222db56c8bbbe0b250e1f2b5fa751b5aaee
• Make B2 backend work with application keys (based on https://github.com/gilbertchen/duplicacy/pull/475 by @bekriebel): https://github.com/gilbertchen/duplicacy/commit/674d35e5cae603b641a048609ff48335a36ebb36
• Restore UID and GID of symlinks: https://github.com/gilbertchen/duplicacy/commit/a7d2a941bec536b54f89c8c1656598e3edf16fa8
• Fixed a divide by zero bug when the repository has only zero-byte files: https://github.com/gilbertchen/duplicacy/commit/39d71a325606698abcf060630622f3ec31590faa
• Do not update the Windows keyring file if the password remains unchanged: https://github.com/gilbertchen/duplicacy/commit/9d10cc77fcacb16d0910999ddaa39c94cd5faf72
• Continue to check other snapshots when one snapshot has missing chunks: https://github.com/gilbertchen/duplicacy/commit/e8b8922754056f74b605c8deef0b834ab1c9d391
• Record deleted snapshots in the fossil collection and if any deleted snapshot still exists then nuke the fossil collection: https://github.com/gilbertchen/duplicacy/commit/93cc632021a9281ae8dec6225c66e7c43f6607fa
• Add Git commit numbers to version info: https://github.com/gilbertchen/duplicacy/commit/48cc5eaedb400dffb888e4eae797aeb70b62a44a
• Removed a redundant call to manager.chunkOperator.Resurrect (which can cause a crash): https://github.com/gilbertchen/duplicacy/commit/f304b64b3f607a61bdda9e3ece74fe6b022fb6ec
• Remove extra newline in the PRUNE_NEWSNAPSHOT log message: https://github.com/gilbertchen/duplicacy/commit/8ae7d2a97d4bf44359f806d8cb21546cba806275
• Fix crashes on 32 bit machines caused by misaligned 64 bit integers: https://github.com/gilbertchen/duplicacy/commit/fce42348617039134300101a54af4990aa7d6c3c
• Fix "Failed to fossilize chunk" errors in wasabi backend: https://github.com/gilbertchen/duplicacy/pull/459 (by @jtackaberry)
• Add a -storage option to the benchmark comman: https://github.com/gilbertchen/duplicacy/commit/89769f39067370f0b3500ec462df9341953737e3

• Fixed a bug causing certain new snapshots to be not counted when deciding which fossils can be deleted (https://github.com/gilbertchen/duplicacy/commit/72dfaa8b6b21fd9fbdefe06c67ae6c3a153231f2)
• Added a benchmark command to test disk and transfer performance (#449)
• Support multi-threaded pruning (#441)
• Fixed restoration of basic UNIX file permissions (#417)
• Added macOS APFS snapshot support (#415)
• Fixed a crashing bug when showing the history of excluded files (https://github.com/gilbertchen/duplicacy/commit/0e585e4be4e4e6a61ab2e1ebf591105c3bb060d0)
• Add unreferenced fossils to the fossil collection instead of deleting them immediately (https://github.com/gilbertchen/duplicacy/commit/e03cd2a8800a91425284cd70335fb11c4d7a9a18)
• Added an -enum-only option to the backup command to enumerate the repository only (https://github.com/gilbertchen/duplicacy/commit/aadd2aa390355b21a2d986da15457c211900b941)
• Added a -repository option to the init and add command to specify an alternate repository path (https://github.com/gilbertchen/duplicacy/commit/72239a31c42dd6e01521be19f45d13ed9caf635f)
• Implemented the WebDAV backend (#394)
• Added a -nobackup-file option to the set command to skip directories containing the specified file (#392)
• Add an environment variable DUPLICACY_DECRYPT_WITH_HMACSHA256 to force compatibility with Vertical Backup (https://github.com/gilbertchen/duplicacy/commit/b1c1b47983dce6ee3606bc2dbb81b321ec793ef2)
• Skipped chunks should not be counted when calculating downloading percentage during restore (https://github.com/gilbertchen/duplicacy/commit/23a2d91608e3c28652732e59c0f3d8eb1fa07c31)
• Added a global option -commend to allow Duplicacy processed to be identified by arguments (#391)
• Follow symlinks that point to UNC paths on Windows (https://github.com/gilbertchen/duplicacy/commit/b99f4bffec6ad499628991bcdd2e8998e8a810ec)
• Added a -vss-timeout option to set VSS creation timeout (https://github.com/gilbertchen/duplicacy/commit/be2856ebbd0a90092a775a34b36ff52340e4b379)
• Reduced memory consumption for prune operation (#329)
• Added a new Wasabi storage backend largely based on S3 but optimized to reduce storage cost for deleted objects (#322)
• Print git commit number in version string (https://github.com/gilbertchen/duplicacy/commit/48cc5eaedb400dffb888e4eae797aeb70b62a44a)
• Record deleted snapshots in the fossil collection and if any deleted snapshot still exist nuke the fossil collection (https://github.com/gilbertchen/duplicacy/commit/93cc632021a9281ae8dec6225c66e7c43f6607fa)
• Continue to check other snapshots when one snapshot has missing chunks (https://github.com/gilbertchen/duplicacy/commit/e8b8922754056f74b605c8deef0b834ab1c9d391)

• Retry on temporary network errors in the Azure backend
• Added an OpenStack Swift backend based on github.com/ncw/swift
• Fixed a bug when both tag and the retention policy are specified for the prune command
• Fixed bugs in restoring extended attributes
• Unload the extended attributes from last snapshot in order to save memory
• Limited derivation keys to 64 bytes since snapshot file paths used as keys may be longer
• Fixed a bug that caused -hash to have no effect
• Correctly handle spaces in file paths for the B2 backend
• Improved the Hubic backend to retry on various errors
• Fixed a bug that caused -hash to have no effect
• Don't download a fossil directly; turn it back to a chunk and download the chunk instead
• Add a -storage-name option to the init command to specify the storage name
• Add the global -profile option to enable profiling via http
• Allow the -bit-identical option to the add command to copy the config file as it is
• Disable caching when restoring files in SnapshotManager
• Removed aes128-cbc from the supported ciphers by HiDrive
• Refresh expired tokens unconditionally on authorization errors for Hubic and OneDrive
• Fixed a bug that prevents the file specifying the chunk nesting levels from being loaded and parsed
• Fix the GCD directory creating bug; only save directories in the id cache
• Remove existing config and save a local copy when changing the storage password
• Create the storage folder on gcd storage if it doesn't exist

• Optimize the copy command to skip chunks existing on the destination storage
• Add the -dry-run option to the backup command
• Include storage name when looking up passwords for non-default storages
• Fix prune bug when last snapshot is removed for issue
• Add regex matching to include/exclude filters
• Add the -tabular option to the check command to show tabular usage and deduplication statistics
• Improve the backoff algorithm for the Google Drive storage
• Use b2_download_file_by_name rather than b2_list_file_names to check file existence
• Retry downloads with corrupted content up to three times
• Use random salt and make the number of iterations configurable for storage key derivation
• Add an -ignore-owner option to skip setting uid/gid on restored files
• Unify the chunk nesting level to 1 for all storages
• Fix a bug in splitting the existing file that caused all chunks to be redownloaded
• Add a -bit-identical option to the add command to make a bit-identical copy of the config file
• Increase the timeout for shadow copy creation on Windows
• Various changes to improve password management

• Fixed OneDrive 503 errors by sending GET requests with a nil body
• Fixed symbolic link handling on Windows
• The copy command now skips chunks already on destination
• Fixed a bug in setting the upload/download rate limit for the copy command
• Fixed a bug in setting the number of threads for the copy command
• Update aws/aws-sdk-go to version 1.10.41
• Don't save passwords/credentials to keyring if they are retrieved from environment/preference
• Don't ask for ssh password if a ssh key file is available
• Fixed a bug in retrieving passwords from gnome-keyring
• In GCD backend each thread should have its own backoff value
• Fixed a bug in storage passwords in preferences for non-default storages

• Updated Azure storage backend to support retrying on temporary errors
• The restore command now preserves empty directories
• Fixed a chunk not found error caused by Windows drives with data deduplication on
• Fixed a bug that caused truncated files not to be restored correctly
• Added a flat:// storage backend that can take a flat chunks directory on local or networked drives
• Added a samba:// storage backend that is basically a local drive backend but with caching enabled (for networked drives)

• Fixed incorrect progress statistics during backup and restore
• Increased HTTPS timeouts for overloaded Hubic servers
• Force in-place restore mode with a non-default preference path
• Added new S3 storage backends to support minio and other s3-compatible providers
• Fixed a bug that caused a size -1 error
• The history command can now skip revisions in which the specified file does exist
• Fixed a bug in the fixed-size chunking algorithm
• Improve OneDrive backend by retrying on various errors
• Full sparse file support

• Retry on authentication errors for Google Drive
• Fixed inaccurate restore statistics that mistakenly take existing files into account
• Enhance the B2 storage backend to retry on 3xx status codes
• Fixed a bug that caused backup to fail due to incorrect file size
• Fixed incorrect progress report during restore
• Verify ids of snapshot chunks before uploading them

• Fixed a bug in disk or SFTP storage backends that caused a directory creation error when multiple threads are used
• Fixed a bug that caused the copy operation to crash
• Added a -pref-dir option to the init command to specify an alternate location for the .duplicacy preference directory (contributed by @ech1965)
• Rewrote the S3 storage backend using the official Amazon S3 client library

• Allow a root drive to be selected as the storage on Windows
• Fixed a divide by zero error when verbose logging is turned on
• Fixed a bug that caused restoration of two adjacent files to crash

• The -threads option is now supported by all storage backends, and for both backup and restore
• A Google Cloud Storage backend based on the official Google client library
• Fixed a bug the caused incorrect hash or invalid index errors when restoring zero-size files

• Improved exponential backoff for S3 storage

• Built with go 1.8.1
• Fixed a bug in the OneDrive storage caused by server side changes

• Optimized memory handling of file attributes
• Fixed a bug that prevented top-level folders from being restored on Windows
• Fixed a bug that caused restoration to fail due to an error on changing timestamps of symbolic links

• Fixed a bug that caused backup to error out when there are invalid Unicode characters in filenames
• Handle Windows deduplication reparse points as if they were regular files

• Once in a while skip the chunk cache in order to keep the connection alive
• Fixed a bug in SFTP storage where a connection error is not immediately reported.