Changelog

v2.2.2 - 13 Sep 2023

Bugfixes

• Fixed cp and pipe to not omit some of the metadata flags. (#657)

v2.2.1 - 23 Aug 2023

Bugfixes

• Fixed incorrect s5cmd version output (#650)

v2.2.0 - 21 Aug 2023

Features

• Added pipe command. (#182)
• Added presign command. (#634) @zemul
• Added file types to select queries with more range of options to set during the query. (#494)
• Added --include flag to cp, rm, and sync commands. (#516)
• Added --show-progress flag to cp command. (#51)
• Added --content-disposition flag to cp command. (#569)
• Added --show-fullpath flag to ls command. (#596)
• Added --metadata flag to cp and pipe commands to set arbitrary metadata for the objects. (#537)

Improvements

• Implemented concurrent multipart download support for cat command. (#245)
• Upgraded minimum required Go version to 1.19. (#583)
• ListObjectsV2 S3 API is enabled for Google Cloud Storage. (#617)
• Added installation instructions for FreeBSD. (#573) @ehaupt
• Added ppc64le support. (#552) @mgiessing

Bugfixes

• Fixed a bug that causes sync command with whitespaced flag value to fail. (#541) ataberkgrl
• Fixed a bug introduced with external sort support in sync command which prevents sync to an empty destination with --delete option. (#576)
• Fixed a bug in sync command, which previously caused the command to continue running even if an error was received from the destination bucket. (#564)
• Fixed a bug that causes local files to be lost if downloads fail. (#479)
• Fixed a bug where cp command could not upload a non-regular file to remote destination. (#618)
• Fixed a crash where a file or a remote object is removed or renamed after it is listed to be operated on. (#620)

Changelog

Breaking changes

• Adjacent slashes in key are no longer removed when uploading to remote. Before s5cmd cp file.txt s3://bucket/a//b///c/ would copy to s3://bucket/a/b/c/file.txt but now to s3://bucket/a//b///c/file.txt.(#459)
• --endpoint-url will not accept URLs without scheme such as example.com. Instead, it will give an error and ask for an url with a scheme; either http://example.com or https://example.com (#496).

Features

• Added --content-type and --content-encoding flags to cp command. (#264)
• Added --profile flag to allow users to specify a named profile. (#353)
• Added --credentials-file flag to allow users to specify path for the AWS credentials file instead of using the default location.
• Added --all-versions flag to ls, rm, du and select subcommands to apply operation on(/over) all versions of the objects. (#475)
• Added --version-id flag to cat, cp/mv, rm, du and select subcommands to apply operation on(/over) a specific versions of the object. (#475)
• Added bucket-version command to configure bucket versioning. Bucket name alone returns the bucket versioning status of the bucket. Bucket versioning can be configured with set flag. (#475)
• Added --raw flag to cat and select subcommands. It disables the wildcard operations. (#475)
• Added bench.py script under new benchmark folder to compare performances of two different builds of s5cmd. (#471)

Improvements

• Disable AWS SDK logger if log level is not trace. (#460)

• Allow adjacent slashes to be used as keys when uploading to remote. (#459)

• Debian packages are provided on releases page (#380)

• Upgraded minimum required Go version to 1.17.

• The sync command uses external sort instead of internal sort. This change
  reduces RAM usage from ~10 GB to ~1.5 GB for sync operation of a directory containing
  1,000,000 files at a cost of speed (20% slower for 1,000,000 objects). For smaller
  directories (~50,000 files) there is no significant change in speed. (#483)

• Improve auto-completion support of s5cmd for zsh and bash, start supporting pwsh and stop the support for fish. Now s5cmd can complete bucket names, s3 keys in a bucket and the local files. However, install-completion flag no longer installs the completion script to *rc files instead it merely gives instructions to install autocompletion and provides the autocompletion script (#500).

Bugfixes

• Fixed a bug where (--stat) prints unnecessarily when used with help and version commands (#452)
• Changed cp error message to be more precise. "given object not found" error message now will also include absolute path of the file. (#463)
• Fixed a bug where some part of the destination path is removed by cp and sync subcommands (#360)
• Fixed a bug where proxy is not being used when --no-verify-ssl flag is used. (#445)
• Fixed unknown url format error when object key also includes s3:// e.g. s5cmd ls s3://foo/bar/s3://baz (#449)
• Fixed a bug where the local file created for the download operation was not deleted if the download fails in Windows. (#348)

Changelog

• c02fdb6 Add a packaging badge (#439)
• 914e701 Add benchmark script to compare two different builds of s5cmd (#471)
• 36d74f6 Add installation instructions for conda (#468)
• 6e3fc4f Change cp error not being descriptive (#463)
• 38b0fdf Make it possible to override content-type and content-encoding from CLI arguments (#467)
• d5dde3c Remove stat flag unnecessary prints (#455)
• eea87b1 Update changelog for v2.0.0 (#458)
• 8220438 changelog: update Improvements sections (#473)
• 2a6c7cc ci: use Go 1.19 for builds, remove 1.16 support (#481)
• 84c3696 command/app: force endpoint-url to have scheme (#496)
• 5e867c9 command: add extensive autocompletion support (#500)
• 13aa68a command: fix target file is created despite the download failure (#477)
• 7c90f8a command: retry upload on s3.NoSuchUpload (#470)
• 5b64469 disable aws sdk logger if log level is not "trace" (#460)
• 711ddf0 e2e/app_test: fix failed test due to new cleanup (#504)
• d949fe0 e2e/util_test: enable tests to work using external endpoint (#506)
• 37e9a68 e2e/util_test: remove unnecessary test directory cleanup (#512)
• d50ef44 e2e: refactor cleanup behavior using t.Cleanup (#502)
• c67760e goreleaser: add nfpm configuration for Debian packages (#437)
• 88a4641 log, parallel: fix help flag panicking (#509)
• 90b7418 main: use signal.NotifyContext to handle signals (#510)
• 3e08061 readme: fix ci badge URL (#546)
• e9c4d7e readme: we dont support inline comments since v2 (#486)
• c01c5a2 storage/s3: Fix proxy with no-verify-ssl flag (#457)
• d21a491 storage/s3: add SlowDown as retryable (#507)
• 6b87fb4 storage/url: Allow usage of adjacent slashes (#459)
• 22592e5 storage/url: add nil check for URL.filterRegex in URL.Match (#511)
• 83ce8bc storage/url: allow wildcards to match new line characters (#505)
• 6727b11 storage/url: fixes the bug where some part of destination path is removed by cp (#456)
• 717bf5b storage: add profile flag (#453)
• 73f4abb storage: remove unused notImplemented type (#476)
• a8a226f update readme: add a section about numworkers/concurrency (#465)
• deb16c9 vendor: update urfave/cli to v2.11.2 (#495)
• 123b1c7 vendor: upgrade gofakes3 to v0.0.11 (#472)

Changelog

Breaking changes

• Dropped inline comment feature for run command. Previously s5cmd supported a command with an inline comment like ls s3://bucket/object.gz # inline comment. (#309)
• Changed homebrew installation command on macOS. Users can install s5cmd via brew install peak/tap/s5cmd. (#356)
• Print usage errors to stderr instead of stdout and do not show help text on usage error. (#399)
• Working directory of the Docker image changed to /aws from /. (#412)

Features

• Added sync command to synchronize two given buckets, prefixes, or objects. (#3)
• Added AWS Single Sign-On (SSO) profiles support. (#385)
• Added --force-glacier-transfer flag to select command. (#346)
• Added --ignore-glacier-warnings flag to cp, mv and select commands. (#346)
• Added --request-payer flag to include x-amz-request-payer in header while sending GET, POST and HEAD requests. (#297) @Kirill888
• Added --use-list-objects-v1 flag to force using S3 ListObjects API instead of ListObjectsV2 API. (#405) @greenpau
• Added trace log level(--log=trace) which enables SDK debug logs.(#363)

Improvements

• Upgraded minimum required Go version to 1.16.

Bugfixes

• Fixed a bug about precedence of region detection, which auto region detection would always override region defined in environment or profile. (#325)
• Fixed a bug where errors did not result a non-zero exit code. (#304)
• Print error if the commands file of run command is not accessible. (#410)
• Updated region detection call to use current session's address resolving method (#314)
• Fixed a bug where lines with large tokens fail in run command. sync was failing when it finds multiple files to remove. (#435, #436)
• Print usage error if given log level(--log) is not valid. (#430)
• Fixed a bug where (--stat) is ignored when log level is error. (#359)

Changelog

Breaking changes

• Dropped inline comment feature for run command. Previously s5cmd supported a command with an inline comment like ls s3://bucket/object.gz # inline comment. (#309)
• Changed homebrew installation command on macOS. Users can install s5cmd via brew install peak/tap/s5cmd. (#356)
• Print usage errors to stderr instead of stdout and do not show help text on usage error. (#399)
• Working directory of the Docker image changed to /aws from /. (#412)

Features

• Added sync command to synchronize two given buckets, prefixes, or objects. (#3)
• Added AWS Single Sign-On (SSO) profiles support. (#385)
• Added --force-glacier-transfer flag to select command. (#346)
• Added --ignore-glacier-warnings flag to cp, mv and select commands. (#346)
• Added --request-payer flag to include x-amz-request-payer in header while sending GET, POST and HEAD requests. (#297) @Kirill888
• Added --use-list-objects-v1 flag to force using S3 ListObjects API instead of ListObjectsV2 API. (#405) @greenpau

Improvements

• Upgraded minimum required Go version to 1.16.

Bugfixes

• Fixed a bug about precedence of region detection, which auto region detection would always override region defined in environment or profile. (#325)
• Fixed a bug where errors did not result a non-zero exit code. (#304)
• Print error if the commands file of run command is not accessible. (#410)
• Updated region detection call to use current session's address resolving method (#314)
• Fixed a bug where lines with large tokens fail in run command. sync was failing when it finds multiple files to remove. (#435, #436)

Changelog

Breaking changes

• Dropped inline comment feature for run command. Previously s5cmd supported a command with an inline comment like ls s3://bucket/object.gz # inline comment. (#309)
• Changed homebrew installation command on macOS. Users can install s5cmd via brew install peak/tap/s5cmd. (#356)
• Print usage errors to stderr instead of stdout and do not show help text on usage error. (#399)

Features

• Added sync command to synchronize two given buckets, prefixes, or objects. (#3)
• Added AWS Single Sign-On (SSO) profiles support. (#385)
• Added --force-glacier-transfer flag to select command. (#346)
• Added --ignore-glacier-warnings flag to cp, mv and select commands. (#346)
• Added --request-payer flag to include x-amz-request-payer in header while sending GET, POST and HEAD requests. (#297) @Kirill888
• Added --use-list-objects-v1 flag to force using S3 ListObjects API instead of ListObjectsV2 API. (#405) @greenpau

Improvements

• Upgraded minimum required Go version to 1.16.

Bugfixes

• Fixed a bug about precedence of region detection, which auto region detection would always override region defined in environment or profile. (#325)
• Fixed a bug where errors did not result a non-zero exit code. (#304)
• Print error if the commands file of run command is not accessible. (#410)
• Updated region detection call to use current session's address resolving method (#314)

Changelog

ed0556d Brew formula update for s5cmd version v1.3.0
e81b978 add cache-control and expires options (#318)
fb03bce all: enable -race flag for tests (#336)
5a611da command/cp&rm: add --raw flag (#315)
fbd99c5 command/cp: add force-glacier-transfer flag (#312)
b4c91bd command/cp: allow overriding bucket region (#270)
d80fc02 command/expand: fix data race for concurrent writes (#330)
68f797b command/mb: fix incorrect usages in help output (#316)
d5e5ea3 command/mv: fix option usage (#338)
babc8d5 command/run: fix concurrent writes to the flags list (#335)
604f03a command: add --exclude flag (#334)
9defce6 command: append object error to multierror to return non-zero exit code error (#328)
5495dd4 command: implement 'remove bucket' feature (#310)
c2a9d31 command: implement 'select' command (#300)
02f63b3 makefile: fix check-fmt target (#327)
89baee6 readme: add 'windows' to the list of supported operating systems (#323)
c73528f readme: add steps for installation using macports (#311)
4130a7e storage/fs: change order of precedence for URL expansion (#322)
5ba7ee0 storage/s3: do not retry on expired token (#337)

With this release, Windows is supported.

Breaking changes

• Dropped storage class short codes display from default behaviour of ls operation. Instead, use -s flag with ls to see full names of the storage classes when listing objects.

Features

• Added Server-side Encryption (SSE) support for mv/cp operations. It uses customer master keys (CMKs) managed by AWS Key Management Service. (#18)
• Added an -s flag to show full form of storage class when listing objects. (#165)
• Add access control lists (ACLs) support to enable managing access to buckets and objects. (#26)

Bugfixes

• Fixed infinite repetition issue on mv/cp operations which would occur if the destination matched the source wildcard. (#168)
• Fixed windows filepath issue, where backslashes should be treated as the path delimiter. (#178)
• All tests pass on windows, by converting and treating file paths to UNIX filepath format.
• Fixed a transfer issue where the object path contains particular regex metacharacters. (#111) @brendan-matroid
• Correctly parse object paths that contain whitespaces in run-mode. (#111) @brendan-matroid

Improvements

• Retry when connection closed by S3 unexpectedly. (#189) @eminugurkenar

This is a major release with many breaking changes.

Breaking changes

• Dropped get command. Users could get the same effect with s5cmd cp <src> ..
• Dropped nested command support.
• Dropped ! command. It was used to execute shell commands and was used in
  conjunction with nested commands.
• s5cmd -f and s5cmd -f - usage has changed to s5cmd run. run command
  accepts a file. If not provided, it'll listen for commands from stdin.
• Exit code for errors was 127. It is 1 now.
• Dropped exit command. It was used to change the shell exit code and usually
  a part of the nested command usage.
• Dropped local->local copy and move support. (#118)
• All error messages are sent to stderr now.
• -version flag is changed to version command.
• Dropped batch-rm command. It was not listed in the help output. Now that we
  support variadic arguments, users can remove multiple objects by providing wildcards or multiple arguments to s5cmd rm command. (#106)
• Virtual host style bucket name resolving is enabled by default for S3 and GCS. If you provide a custom endpoint via --endpoint-url flag (other than GCS and S3 transfer acceleration), s5cmd will fall back to the path-style. (#92)
• Listing a non-existent object will return exit code 1, instead of 0. (#23)
• -ds, -dw, -us and -uw global flags are no longer available. Multipart concurrency and part size flags are now part of the cp/mv command. New replacement flags are --concurrency | -c and --part-size | -p. (#110)
• s5cmd cp command follows symbolic links by default (only when uploading to s3 from local filesystem). Use --no-follow-symlinks flag to disable this feature. (#17)
• Dropped -parents flag from copy command. Copy behaviour has changed to preserve the directory hierarchy as a default. Optional -flatten flag is added to flatten directory structure. (#107)
• Dropped -vv verbosity flag. --log flag is introduced.

Features

• Added mb command to make buckets. (#25)
• Added --json flag for JSON logging. (#22)
• Added S3 transfer acceleration support. (#40)
• Added Google Cloud Storage support. (#81)
• Added cat command to print remote object contents to stdout (#20)

Bugfixes

• Correctly set Content-Type of a file on upload operations. (#33)
• Fixed a bug where workers are unable to consume job if there are too many
  outstanding wildcard expansion requests. (#12, #58)

Improvements

• Pre-compiled binaries are provided on releases page. (#21)
• AWS Go SDK is updated to support IAM role for service accounts. (#32)
• For copy/move operations, s5cmd now creates destination directory if missing.
• Increase the soft limit of open files to 1000 and exits immediately when it encounters too many open files error. (#52)

Add Go modules support