azure-storage-fuse

Bug Fixes:

#658 : Fixed segfault in streaming when file is 0 byte in length
#660 : Added mp4 to list of supported content types
#665 : SYNC and DIRECT open flags are not supported. New cli param added to supress them in open file call

New Cli Options:

--ignore-open-flags : Supress unsupported flags in open file call.

Bug Fixes:

#638 : Mount blob using Managed Identity or service principal failed in Azure China Cloud

• Gen2 accounts were not showing correct ACL permissions if there are additional service-principals in ACL
• Fix for using MSI Secret from environment variable

Feature addition

• Read Streaming support added for 'read-only' mounts. Instead of downloading entire file only part of it will be downloaded on demand.
• Background download support. Instead of downloading entire file in open call, if configured, open will start download in background and read/write call will wait untill download completes. Thus making 'open' call non-blocking.
• Attribute cache and content cache can be invalidated using 'fsync' comand on file.
• 'fsyncdir' will invalidate attribute-cache of all its children

New Cli Options:

--background-download : Set this true if you want file download to run in the background on open.
--invalidate-on-sync : Set this to true if you want the particular file or directory content and attribute cache to be invalidated on 'fsync' or 'fsyncdir'. Refer README for more details.
--streaming : Enable streaming. Works only for 'read-only' mount. Add '-o ro' in mount cli and set this flag to 'true' to enable streaming.
--stream-cache-mb : Limit total amount of data being cached in memory with streaming.
--max-blocks-per-file : Maximum number of blocks to be cached in memory for read streaming.
--block-size-mb=16 : Size (in MB) of a block to be downloaded during streaming. Refer README for more details.

Bug Fixes:

#604 : Linux file list commands is only showing 5000 files in a folder
#609 : Expose max retry count and retry time out settings
#612 : Segmentation fault when running blobfuse on a pod on AKS version 1.19.11
#613 : blobfuse mount with invalid credentials still returns succeed

New Cli Options:

--max-retry : Maximum retry count if the failure codes are retryable. Default count is 26.
--max-retry-interval-in-seconds : Maximum number of seconds between 2 retries, retry interval is exponentially increased but it can never exceed this value. Default naximum interval is 60 seconds.
--basic-remount-check : Try checking for a remount by reading /etc/mtab instead of calling the syscall setmntent (#612)
--pre-mount-validate : Skip cURL version check and validate storage connection before mount. (#613)

Changes:

• Features:

  • Blobfuse will block initial on mount list API calls to storage if the duration is configured while mounting, to save billing money.
  • Blobfuse can be configured to mount through Proxy Server
  • Blobfuse will upload only newly created files or modified files to storage, Note: changes to metadata only will not propagate unless there are changes to file contents. It can be configured to turn off this flag.
  • Added Azure CLI script for generating SasToken
  • Content-type update in blob, based on file extension

• Bug Fixes:

  • Corrected available cache calculation for non sudo users
  • #540: MSI and ADLS now working properly with gen-1 and gen-2 accounts
  • #548 :Fixed concurrency issues affecting MSI performance
  • #552 : Made Disk thresholds configurable, check README for CLI options
  • #578 : Reduced the libCurl minimum version post-fork authentication
  • #586: fixed the erroring out of chmod and rename with SAS token on ADLS account
  • #584: Evict read only files as well from cache
  • #590: Added Proxy support with custom client certificate file
  • #593 : Resolved race conflict issue when multiple threads read the same blob

• New Cli Options :

  • --upload-modified-only=true . Default is true so unless there are writes to the file it will not be reuploaded to Storage.
  • --set-content-type=true. Set the 'content-type' property in blob while uploading. This will be based on the file extension.
  • --cancel-list-on-mount-seconds=0. After mount block the list call for give number of seconds. After mounting libfuse automatically fires a list call to list all files and directories which results into unintended calls to container.
  • --cache-poll-timeout-msec=1. Caching polling and eviction check time interval (in milliseconds). Default is 1 milisecond.
  • --high-disk-threshold-in-seconds. The upper limit percentage of disk usage. Disk cache cleanup will be triggered when it reaches this percentage.
  • --low-disk-threshold-in-seconds. The lower limit percentage for used disk space below which GC will not clear disk cache.
  • --ca-cert-file=. Provide client certificate file for proxy environment
  • --httpsproxy= < proxy >. HTTPS proxy through which connection to storage is allowed.
  • --httpproxy= < proxy >. HTTP proxy through which connection to storage is allowed.

This release has the following fixes and enhancements

1. Parameter file_cache_timeout_in_seconds will be honored based on the time the file was downloaded to the file system as opposed to the original modification date in blob.
2. All retries follow exponential backoff to prevent throttling due to exceeded RPS.
3. Blob names will allow '&', '%', '+' and "="
4. Sticky bits will be allowed in ADLS Gen 2 permission list , this is a fix for the error "Unexpected number of permissions"
5. Blobfuse 1.3.6 is backward compatible with Curl versions < 7.54, this addresses the handshake issue faced by CentOS and RHEL versions < 8

1. ADLS Gen 2 enhanced to non-public blob endpoints.
2. SAS Token refresh without remount feature implemented.
3. File names with '#' now encoded and supported.
4. Bug fix: Rename operation in ADLS Gen 2
5. Added build support for Debian 10

Blobfuse 1.3.4 has the following bug fixes only

1. Fix for FileNotFound error when the nested blob directory lexicography overrode the parent directory alone lexicography
2. RPM script modified to compile with RHEL 7.8

This release has the following bug fixes and perf improvements

1. Perf improvements with file listing when attribute cache is turned on , mount param for the same --use-attr-cache=true
2. Perf improvements with MSI authentication
3. Delete bug fix for ADLS
4. Memory error aka Input/Output error bug fix

This release has the following features and bug fixes

1. Compliant with RHEL 8.2 and Ubuntu 18.04 LTS updated version**
2. Command line parameter for symlinks switched to --no-symlinks=true, the previous one--no_symlinks will also accepted.
3. Bug fix for 'ls' on listdir throwing Input/output error for million plus records.
   ** Note: Ubuntu 18.04 latest upgrade has an updated fuse version so previous versions of blobfuse < 1.3.2 will not mount anymore on the latest kernel upgraded Ubuntu 18.04

1. Added ADLS Gen2 Storage features. Use --use-adls=true for ADLS storage accounts
2. Added Symlink support: Symlinks is on by default. It can be turned off to gain performance. Optional commandline param to turn it off : --no_symlinks=true
3. Re-added support for filenames named with '' aka backslashes
4. Added command-line options to disallow the reuse of the tmp directory and block non-empty tmp-path : --empty-dir-check=true.
5. Added support for RHEL 8.1
6. Added options to make cache configurable in Megabytes. Optional commandline param --cache-size-mb=1000 will specify a cache size of 1 GB
7. Added options to specify concurrency. Optional commandline param --max-concurrency=14 will reset the default concurrency of 12.
8. Released for Ubuntu 20.04

Bug fix:
Fixed occasional directory not accessible errors for directories with large size files.
NOTE: Public packages are currently published only for Ubuntu18.04 and Ubuntu16.04

New Features:
MSI Custom endpoint support.
Dynamic logger injection - refer to Readme logging section for more detail.
Bug fixes
Fixed the Frequent file not found error for large volumes and multiinstance scenarios.
Error message rephrasing
Mounting restriction added for non-empty tmp_path.

NOTE: Public packages are currently published only for Ubuntu18.04 and Ubuntu16.04

New Feature:

• attribute cache mount option
  Details: --use-attr-cache=true|false : Enables attributes of a blob being cached. False by default.
• support for Debian

Improvements:

• Cache will clear itself once reaching a high threshold and will reduce the cache to below the low threshold. Timeout cache still exists, so if the timeout is met and the threshold is not met, the cache will still clear itself

Fixes:

• seg fault issue with parsing responses (#249)

This release improves file listing algorithms in readdir function:

• It improves performance as it uses more efficient comparison of vectors
• It avoids ENOMEM 12 issue since readdir no longer creates new vectors

This is the first GA / non-preview release of blobfuse. The only changes since v1.0.1-RC-Preview are:

• Fixed a bug regarding SAS tokens; blob-service SAS tokens will now work with blobfuse, as long as the permissions are wide enough.
• Added a small stress testing framework, and the results of the initial run of stress tests. Going forward, we will use this to monitor improvements in performance, and to guard against perf regressions.

This release improves logging statements for debugging purposes, and fixes a few bugs:

• Fix for being able to compile on Ubuntu 17 & Debian 9
• Fixed some slow file handle leaks that can cause blobfuse to start failing if run for a long time at high concurrency.

This release includes all features planned for the 1.0.0 release, which will be released after additional testing and bug fixes. New features since 0.4 include:

• Improved retry mechanism for faulty / slow connections, throttling, etc
• Improved timeout mechanism for faulty / slow connections
• Added full support for truncate (previously, only truncating to 0 was supported.)

The v0.4.0 release includes:

• SAS token support for auth, as well as Shared Key
• Logging to syslog, at configurable levels
• New strategy for representing directories as blobs
• Framework for white-box C++ tests
• Various bug fixes and small changes

The v0.3.1 release addresses the read cache issue described in #120

The v0.3.0 release includes:

• Better support for last-modified-time on files
• Support for non-public clouds
• Support for larger files (up to max block blob size)
• Correcting ownership values returned from get_attr
• A fix for a potential data-loss / data-corruption bug
• Other misc. bug fixes