azure-storage-azcopy

Documentation changes

1. all was historically an available status option for jobs show but is now documented.

Bug Fixes

1. Fixed a hard crash when persisting ACLs from remote filesystems on Windows.
2. Fixed a hard crash when deleting folders containing a % in the name from Azure Files.
3. Fixed a bug which made Managed Disks data access authentication mode unusable with auto login.

Version 10.16.0

New features

1. Added time-based flag for remove to include files modified before/after certain date/time.
2. Added --output-level flag which allows users to set output verbosity.
3. Added --preserve-posix-properties flag that allows user to persist the results of statx(2)/stat(2) syscall on upload.
4. Implemented setprops command that allows users to set specific properties of Blobs, BlobFS, and Files.
5. Implemented multi-auth for managed disks (SAS+OAuth) when the managed disk export account requests it.

Bug fixes

1. Fixed issue 1506: Added input watcher to resolve issue since job could not be resumed.
2. Fixed issue 1794: Moved log-level to root.go so log-level arguments do not get ignored.
3. Fixed issue 1824: Avoid creating .azcopy under HOME if plan/log location is specified elsewhere.
4. Fixed isue 1830, issue 1412, and issue 873: Improved error message for when AzCopy cannot determine if source is directory.
5. Fixed issue 1777: Fixed job list to handle respective output-type correctly.
6. Fixed win64 alignment issue.

Version 10.15.0

New features

1. Added support for OAuth forwarding when performing Blob -> Blob copy.
2. Allow users to dynamically change the bandwidth cap via messages through the STDIN.
3. GCS -> Blob is now GA.
4. Enable MinIO(S3) logs in DEBUG mode.
5. Upgraded Go version to 1.17.9.

Bug fixes

1. Resolved alignment of atomicSuccessfulBytesInActiveFiles.
2. Fixed issue where last-write-time was still getting persisted even when --preserve-smb-info is false.
3. Fixed issue where concurrency was always AUTO for Azure Files despite explicit override.
4. Removed outdated load command following the deprecation of the cflsload package.

Version 10.14.1

Bug fixes

1. Fixed issue #1625 where a panic occurs during sync scanning.
2. Fixed remove issue when account has versioning enabled.

Version 10.14.0

New features

1. Feature to permanently delete soft-deleted
   snapshots/versions of the blobs has been added (preview). --permanent-delete=none/snapshots/version/snapshotsandversions.
2. Feature to preserve properties and ACLs when copying to Azure file share root directory.
3. Pin all APIs to use the default service version 2020-04-08 and let users decide the service version via
   AZCOPY_DEFAULT_SERVICE_API_VERSION environment variable. Previously, few APIs were not respecting the AZCOPY_DEFAULT_SERVICE_API_VERSION environment variable.

Bug fixes

1. Fixed issue in which AzCopy failed to copy to classic blob container with preserve blob access tier.
2. Fixed issue 1630 : AzCopy created extra empty
   directories at destination while performing S2S transfer from one ADLS Gen2 account to another ADLS Gen2 account.
3. Changed the way AzCopy was using to obtain and set ACLs to ensure accuracy.
4. Clarify error message for azcopy sync when source or destination cannot be detected.
5. Report error when client provided key(CPK) encryption is applied to DFS endpoint.
6. Fixed issue 1596 : AzCopy failed to transfer files
   (with '/.' in their path) from AWS S3 to Azure blob storage.
7. Fixed issue 1474 : AzCopy panicked when trying to re-create an already open plan file.
8. Improved handling of Auth error against single file.
9. Fixed issue 1640 : Recursive copy from GCS bucket to Azure container failed
   with FileIgnored error when using --exclude-path.
10. Fixed issue 1655 : AzCopy panicked when using --include-before flag.
11. Fixed issue 1609 : blockid converted to lower case in AzCopy logs.
12. Fixed issue 1643, issue 1661 : Updated Golang version to 1.16.10 to fix security vulnerabilities in Golang packages.

Version 10.13.0

New features

1. Added Arc VM support for authorization via managed identity.
2. Widen managed disk scenario to all md- accounts instead of just md-impexp- accounts.
3. The concurrency is now set to AUTO for Azure Files by default to avoid throttling.
4. Decrease the number of create directory calls for Azure Files to avoid throttling.
5. Added the from-to flag for sync.

Bug fixes

1. Fixed the memory usage issue with generating the list of skipped/failed transfers in JSON output.
2. Fixed ADLS Gen2 ACL copying where intermediate folders were missed.
3. Fixed the S3 to Blob scenario using the login command.
4. Fixed dry-run for dfs endpoints.
5. Fixed incorrect percentage-done shown while resuming job.
6. Fixed login issues on the ARM platforms.
7. Fixed incorrect progress status for the sync command.
8. Fixed concurrency map access problem for folder creation tracker.
9. Fixed resuming with a public source.

Version 10.12.2

Bug fixes

1. Fix deleting blobs that are of a different type than the specified copy
2. Fix --delete-destination on Windows download

Version 10.12.1

Bug fixes

1. Fixed the problem of always receiving overwrite prompt on azure files folders.

Version 10.12.0

New features

1. Added support for include and exclude regex flags, which allow pattern matching on the entire paths.
2. Added dry run mode for copy, remove, and sync. This feature allows the user to visualize the changes before committing them.
3. For SMB aware locations, preserve-smb-info flag is now true by default.
4. Improved how folder lmts are obtained to allow time-based filters for folders.
5. Added support for ACL copying between HNS enabled accounts. The preserve-smb-permissions flag is now deprecated and has been renamed to preserve-permissions.

Bug fixes

1. Allow from-to to be set for the remove command.
2. Fixed the problem where resume command did not honor AZCOPY_DEFAULT_SERVICE_API_VERSION.
3. Fixed the new version check.
4. Fixed sync issue on Windows where paths are case-insensitive.
5. Added prompt for invalid characters when importing from S3.
6. Fixed bug where public S3 buckets cannot be listed.
7. Sanitize SAS tokens in JSON output for skipped and failed transfers.
8. Improved folder property preservation across resumes.

Version 10.11.0

New features

1. Improved performance for copying small blobs (with size less than 256MiB) with Put Blob from URL.
2. Added mirror mode support in sync operation via mirror-mode flag. The new mode disables last-modified-time based comparisons and overwrites the conflicting files and blobs at the destination if this flag is set to true.
3. Added flag disable-auto-decoding to avoid automatic decoding of URL-encoded illegal characters when uploading from Windows. These illegal characters could have encoded as a result of downloading them onto Windows which does not support them.
4. Support custom mime type mapping via environment variable AZCOPY_CONTENT_TYPE_MAP.
5. Output message on the CLI when AzCopy detects a proxy for each domain.
6. Interpret DFS endpoints as Blob endpoint automatically when performing service-to-service copy.

Bug fixes

1. Tolerate enumeration errors for Azure Files and not fail the entire job when a directory is deleted/modified during scanning.
2. Log skipped transfers to the scanning log.
3. Fixed pipe upload by adding missing fields such as Metadata, Blob Index Tags, Client Provided Key, Blob Access Tier, etc.
4. Fixed issue of clean up for the benchmark command.

Version 10.10.0

New features

1. Support sync for Local/Blob <-> Azure File.
2. Download to temporary file path (.azDownload-[jobID]-[name]) before renaming to the original path.
3. Support CPK by name and CPK by value.
4. Offer knob to disable application logging (Syslog/Windows Event Log).
5. Trust zonal DNS suffix for OAuth by default.
6. Added include-directory-stub flag for the copy command, to allow copying of blobs with metadata of hdi_isfolder:true.
7. Display more fields for the list command, please refer to the help message for example.
8. Provide environment variable to set request try timeout, to allow faster retries.

Bug fixes

1. Improve job progress updating mechanism to improve scalability for larger jobs.
2. Time limit the container creation step, to avoid hanging and improve UX.
3. Set SMB info/permission again after file upload and copy, to fully preserve the integrity of the permission string and last-write-time.
4. Fixed module import problem for V10.

Version 10.9.0

New features

1. Added preview support for importing from GCP Storage to Azure Block Blobs.
2. Added scanning logs which have low output by default but can become verbose if desired to help in debugging.
3. Support preservation of tags when copying blobs.
4. Added last modified time info to the list command.

Bug fixes

1. Removed unexpected conflict prompt for file share folders with special characters in the name, such as ";".

Version 10.8.0

New features

1. Added option to disable parallel blob listing
2. Added support for uploading large files upto 4TiB. Please refer the public documentation for more information
3. Added support for include-beforeflag. Refer this for more information

Bug fixes

1. Fixed issue #1246 of security vulnerability in x/text package
2. Fixed issue share snapshot->share copy with SMB permissions

Version 10.7.0

New features

1. Added support for auto-login when performing data commands(copy/sync/list/make/remove). Please refer to our documentation for more info.
2. Added blob-tags flag for setting blob index tags when performing copy command. Please note that we support setting blob tags only when tags are explicitly specified. Refer to the public documentations to know more.

Bug fixes

1. Fixed issue #1139 to preserve content-type in service-to-service transfer.
2. Fixed issue to allow snapshot restoring.
3. Fixed issue with setting content-type of an empty file when performing copy command.

Improvements

1. Added support for setting tier directly at the time of upload API call instead of performing a separate set tier API call.

Version 10.6.1

Bug fixes

1. Fix issue #971 with scanning directories on a public container
2. Fix issue with piping where source and destinations were reversed
3. Allow piping to use OAuth login
4. Fix issue where transfers with overwrite flag set to IfSourceNewer would work incorrectly
5. Fix issue #1139, incorrect content type in BlobStorage
6. Issue #1192, intermittent panic when AzCopy job is abort
7. Fix issue with auto-detected content types for 0 length files

New features

1. azcopy sync now supports the persistence of ACLs between supported resources (Windows and Azure Files) using the --persist-smb-permissions flag.
2. azcopy sync now supports the persistence of SMB property info between supported resources (Windows and Azure Files) using the --persist-smb-info flag. The information that can be preserved is Created Time, Last Write Time and Attributes (e.g. Read Only).
3. Added support for uploading higher block & blob size (Please note that copy operations are not yet supported.)
   • For service version 2019-12-12 or higher, the block size can now be less than or equal to 4000 MiB. The maximum size of a block blob therefore can be 190.7 TiB (4000 MiB X 50,000 blocks)
4. Added support for Blob Versioning
   • Added list-of-versions flag (specifies a file where each version id is listed on a separate line) to download/delete versions of a blob from Azure Storage.
   • Download/Delete a version of blob by directly specifying its version id in the source blob URL.

Bug fixes

1. Logging input command at ERROR level.

New features

• Allow more accurate values for job status in jobs commands, e.g. completed with failed or skipped transfers.

Bug fixes

• Fixed issue with removing blobs with hdi_isfolder=true metadata when the list-of-files flag is used.
• Manually unfurl symbolic links to fix long file path issue on UNC locations.

New features

1. Improved scanning performance for most cases by adding support for parallel local and Blob enumeration.
2. Added download support for the benchmark command.
3. A new way to quickly copy only files changed after a certain date/time. The copy command now accepts
   the parameter --include-after. It takes an ISO 8601-formatted date, and will copy only those files that were
   changed on or after the given date/time. When processing large numbers of files, this is faster than sync or
   --overwrite=IfSourceNewer. But it does require the user to specify the date to be used. E.g. 2020-08-19T15:04:00Z
   for a UTC time, 2020-08-19T15:04 for a time in the local timezone of the machine running Azcopy,
   or 2020-08-19 for midnight (00:00), also in the local timezone.
4. When detecting content type for common static website files, use the commonly correct values instead of looking them up in the registry.
5. Allow the remove command to delete blob directory stubs which have metadata hdi_isfolder=true.
6. The S3 to Blob feature now has GA support.
7. Added support for load command on Linux based on Microsoft Avere's CLFSLoad extension.
8. Each job now logs its start time precisely in the log file, using ISO 8601 format. This is useful if you want to
   use that start date as the --include-after parameter to a later job on the same directory. Look for "ISO 8601 START TIME"
   in the log.
9. Stop treating zero-item job as failure, to improve the user experience.
10. Improved the naming of files being generated in benchmark command, by reversing the digits.
    Doing so allows the names to not be an alphabetic series, which used to negatively impact the performance on the service side.
11. Azcopy can now detect when setting a blob tier would be impossible. If azcopy cannot check the destination account type, a new transfer failure status will be set: TierAvailabilityCheckFailure

Bug fixes

1. Fixed the persistence of last-write-time (as part of SMB info when uploading) for Azure Files. It was using the creation time erroneously.
2. Fixed the SAS timestamp parsing issue.
3. Transfers to the File Service with a read-only SAS were failing because we try listing properties for the parent directories.
   The user experience is improved by ignoring this benign error and try creating parent directories directly.
4. Fixed issue with mixed SAS and AD authentication in the sync command.
5. Fixed file creation error on Linux when decompression is turned on.
6. Fixed issue on Windows for files with extended charset such as [%00 - %19, %0A-%0F, %1A-%1F].
7. Enabled recovering from unexpectedEOF error.
8. Fixed issue in which attribute filters does not work if source path contains an asterisk in it.
9. Fixed issue of unexpected upload destination when uploading a whole drive in Windows (e.g. "D:").

Bug fixes

1. Fixed bug where AzCopy errored if a filename ended with slash character. (E.g. backslash at end of a Linux filename.)

Version 10.3.4

New features

1. Fixed feature parity issue by adding support for "ifSourceNewer" option on the overwrite flag. It serves as a replacement of the '\XO' flag in V8.

Bug fixes

1. Fixed jobs clean command on Windows which was previously crashing when the with-status flag was used.