seaweedfs

• Shell:
  • volume.tier.move adds capability to use wildcard to specify collections #2221
  • collection.list calculates size and file count for a collection #2224
  • Add volume.deleteEmpty to purge empty volumes
• Volume
  • Deletion can skip volume size checking #2225

• Filer
  • Fix deletion of large directories in postgres and sqlite.
  • LevelDB and RocksDB add bloom filter for faster existence checking #2211
  • Optimization: improve random range query read speed for large files
• Shell
  • Adds fs.rm to remove files or directories #2208
  • Adds fs.mkdir to make directories
  • Adds -retry option for volume.fix.replication #2191
  • Adds -retry option for volumeServer.evacuate #2191
  • s3.bucket.delete avoid waiting for large object metadata
  • volume.check.disk skips errors and check next volumes #2129
• Volume
  • Add webp extension for resizing #2217
• Master
  • Ensure unique ids for snowflake sequencer #2213
• Filer.copy
  • Fail early if assign request failed #2216
• weed fuse
  • Fix error handling #2209
• Minor
  • weed -h subcommand list is not sorted, makes discovery difficult #2214
  • weed scaffold -config= should output to stdout #2212

• S3
  • Fixed an important bug where files larger than 4MB will fail the upload. #2195 The bug was introduced since 2.51 (June 6th, 2021).
• Shell
  • volume.check.disk adds retries in case the volumes are just moved #2194
  • volume.fsck deletes a volume if the volume has only orphaned data #2190
• Volume
  • Change all writes to fsync during graceful stopping. #2193

• Volume Server
  • default readMode to proxy.
• Minor
  • fixes related to 64 bit alignment on 32 bit OS.
  • Shell: volume.fsck fix purging orphaned chunks #2188

• Volume
  • Fix bug with in-memory index in large_volume version. #2162 This bug caused a few successfully uploaded chunks could not be read back. Upgrade is strongly recommended.
  • Reduce memory consumption for in-memory index in environments wtih high concurrent writes
  • Add readMode.redirect=proxy mode to read non local volumes. #2168
  • Address unaligned 64-bit atomic operation on 32-bit systems #2177
• Filer
  • Better range query for mp4 video #2156
  • Cassandra store supports TokenAwareHostPolicy #2179
• Daemonized Fuse
  • fix weed fuse parameters parsing #2178
• Mount
  • Recursively rename directory locally #2169
• S3
  • Authorization: StreamingSigned mode enforces access control #2180 This bug allowed any user with read access to write to any buckets in StreamingSigned mode. Upgrade is strongly recommended.
  • Add metadata with multipart upload #2173 #2174

• Filer
  • slow metadata topic read may lose some change events #2117. This affects weed filer.sync over slow network. This is also useful for consistent metadata with weed mount, and affects gRPC subscribe metadata changes.
• Shell
  • volume.fsck -findMissingChunksInFiler checks missing chunks in filer
• weed fuse
  • Fix: Loosing environment variables at StartProcess #2160

This is a release to resolve the binary build issue caused by travis-ci migration.

• FUSE
  • Daemonized weed fuse #2141
• Master
  • Avoid repeated leader redirection #2146
• Add process metrics of weed itself #2147

• FUSE mount
  • Fix a deletion bug #2138 introduced in 2.53. This would cause the file chunks are not actually deleted, which can be fixed by running volume.fix in weed shell.
  • Skip flushing if file is deleted. This enables WinFsp to mount SeaweedFS on windows via sshfs #2110
• Master
  • fix stats when a collection is deleted

• Filer
  • when deleting a bucket, remove all bucket metadata for mysql and postgres filer stores. (mysql2, postgres2, and sqlite stores are not affected.) #2118
• Minor
  • Return correct exitcode when wheed upload fails #2115

• FUSE mount
  • Support macfuse 4.x on MacOS
  • Add current and parent directoreis . and .. #2105
  • Avoid file handle locking when setattr
  • Invalidate FUSE cached entries #2108
• Filer
  • Add path-specific option to enforce read only
  • Parallelize file data uploading to volume servers. (Performance Improvement)
• Filer Store
  • Fix error on Mongodb E11000 duplicate key error collection: seaweedfs.filemeta index: directory_1_name_1 dup key
• Improvements
  • use bytes.Buffer to reduce memory allocations and garbage collections (Performance Improvement)
  • keep alive for streaming connections #2096
  • optionally disable current upload limit

• Fuse Mount
  • Add fuse subcommand to use weed with mount #2085 #2099
• Filer Store
  • Support sqlite as filer meta store
• Filer.bacup
  • backup to local directory optionally is incremental #2084
  • escape colon from path on windows #2084
• S3
  • Object write cache control and expires #2079
  • Object get with If-Match: bogus ETag #2080
  • Fix saving metadata during put-object #2092
• Minor
  • recreate grpc connections if too many errors #2098
  • weed server fix free space parameters

• FUSE Mount
  • Support multiple filers! #2015 #1531
• Filer
  • Receate grpc connections if having transport error #2070
  • Waits in case master lost connection temporarily
  • bootstrap from peer filer #1861
• Master
  • avoid possible nil when node is disconnected from its parent #2073
• S3
  • Adds errors if requests are signed but no authentication is setup #2075
  • Deprecate filer.options.buckets_fsync. Use path-specific configuration instead.
• Java Client 1.6.6
  • Fix replication setting not passed to filer
  • Throw exception when assign fails

• Mount
  • Fix bugs related to renaming files and directories #2068 #2064
  • Fix reading file when file is still being written #2065
• weed filer.copy
  • Adds option to skip copying files if file sizes are the same #2067

• Volume
  • ahead of time volume assignment #2049
  • add retry to assign volumes #2056
• FUSE Mount
  • write to local temporary file before uploading
• Filer
  • delete specific tags #2041
  • fix wrong log error #2051
• Shell
  • optional parallel copy ec shards #2048
• S3
  • return 404 if bucket does not exist #2039
• Java Client
  • Add exists() function
  • make SeaweedInputStream throw FileNotFoundException

• FUSE Mount
  • Fix regression on data inconsistencies between multiple mounts #2038
  • Handle random reads more memory efficient #2037
• S3
  • authorization uses bucket wild cards #2030
• Minor
  • Volume server: -minFreeSpace option supports specific free space size #2025
  • Do not compress on brotlii archives #2031, rar files #2032

• FUSE mount
  • delay new file creation until file is closed, unless file is opened exclusively
• Volume Server
  • Erasure Code: transient errors may cause thundering herd effect #2012
  • Do not add new volumes when below minFreeSpacePercent #2017
• Filer
  • Mysql/Postgres may have some leftover entries in database when deleting folders #2006
• Minor
  • filer directory listing adds namePatternExclude #2023
  • filer directory listing ensures case sensitive when checking name patterns
  • weed shell lock shows which server holds the lock #1983
  • weed filer.copy includes empty folders #2016
  • volume server report errors properly #2012

• FUSE mount
  • Much more efficient memory usage.
• Bugs
  • volumeServer.evacuate uniform movement #1990
  • volume.check.disk skipping readonly volume loop #2002
  • volume.tier.move: avoid data loss when destination volume server already has the volume #2001
  • avoid loop with volume assign alternative request #1996
  • weed shell return to the terminal correct #1995
  • filer pload to a directory without "/" suffix #1988

• Add weed gateway for easier submitting blobs
• FUSE Mount
  • Release view cache to avoid OOM
• Filer
  • Avoid stuck uploader #1980
• Add a basic FTP server in Java #1968

• Mount
  • Add Read Only mode #1961
• Filer
  • Fix possible nil chunk when uploading #1966

• Filer
  • Add customizable upsertQuery for sql file stores. #1946 #1948
  • Add concurrent upload size limit to avoid OOM
  • Ensure creating tables #1957
• Java client 1.6.4
  • ByteBuffer.clear supports higher JDK versions
• Volume
  • Add concurrent upload size limit to avoid OOM