seaweedfs

• Master
  • Admin scripts change to sleep between each iterations. Previous wrong implementation starts the clock at the beginning of previous iteration, instead of at the end of last iteration. This can cause tight loop, which can cause actioning based on stale cluster metadata and lose some volumes. Related to #2476
• Shell
  • volume.list supports different level of details
  • lock can show the long running job that is holding the lock.
• S3
  • Supports Audit log #2498
  • Handle Response-Conent-Disposition 7bf14452f7882c803c7c86dcb7b81be65bf2934e
  • volume.fsck skips purging on EC volumes 94e5c0a4544d970fec33f4d7f41b0944a57612f2
• Mount
  • Fix renaming operation with the correct old parent path. #2500

• Filer Store
  • Redis3 adds support for redis sentinel
  • Mongodb listing directory handles failures #2488
• Filer
  • Adds retry when reading from volume servers #2486
• Docker
  • Removed "crontab" module in docker entry point
• S3
  • copy object API needs to espcae special characters #2482
• Volume Server
  • fix loading old volume format #2487
  • Erasure Coding: failed to load decoded volumes #2489
• Gateway to Remote Object Storage
  • avoid creating the same bucket with a different randomized names #2492
• Shell
  • volume.check.disk was using the wrong source and target locations #2268
  • volume.fix.replication fix misplaced volumes #2416
• Java Client
  • Writes add retries
  • Sync Java client version with SeaweedFS to 2.81 also.

• Helm charts update resource API versions to Kubernetes 1.22 #2464
• Gateway to remote object store
  • Adds retries when writing to remote storage
• Filer
  • Redis2 supports Redis Sentinel #2469
• Master
  • Adds default s3.clean.uploads -timeAgo=24h to default operations.

• S3
  • Avoid forbidden for HeadBucketHandler if owners are specified #2434
• Filer
  • fix reading from local data center and remote data centers
• Volume
  • prevent nil response #2452

• Docker
  • docker entrypoint working directory changed to /data
• Filer
  • Randomize same-dc servers and other-dc servers separately, if there are multiple replicas.
• Filer Store
  • Fix wrong SQL template in mysql
• Shell
  • Volume configure replication by collection pattern #2438
• S3
  • force overwrite s3-is-admin to avoid possible access control attack #2433

• Filer
  • Add Automatic Peer Discovery and -peers option is removed.
  • Paginate large directories to delete file chunks
  • Randomized chunk reads if replicated
• S3
  • Fix ListObject if more than 10000 objects.
  • Skip permission checking for creating bucket if the bucket already exists
  • Adjust permission to READ for HEAD bucket operation #2417
• Shell
  • Filer address is optional now.
  • Add command cluster.ps to list filers.
• Docker
  • Build correctly with ARM #2418

• Filer Store
  • Fix pagination bug on redis2 and redis_cluster2. This caused issue when running registry with SeaweedFS S3.
• S3
  • fix DeleteBucketLifecycleConfiguration #2397
  • Add CORS support
• Cloud Tier
  • Remove tagging since not all S3 vendors support this.
• Erasure Coding
  • By default skip erasure coding if less than recommended minimum of 4 nodes.

This release is mostly fixing minor issues. There are no known major issues.

• S3
  • Fix object tagging #2389
• Volume Server
  • Stream out process during long running volume copying process, to avoid possible timeout. #2386 #2396

• Fix an S3 upload bug introduced in 2.72 #2383. The bug may corrupt files less than 512 bytes. The fix is in #2387.
• Fix an S3 upload bug introduced in 2.71 where files more than 8GB will get truncated to 8GB.

• Filer, FUSE Mount, and Hadoop compatible file system 1.6.9
  • More efficient file read on large files. Previous implementation was O(N*N) with N as number of chunks. This new algorithm is closer to O(N) win practice.
• FUSE mount
  • Rename operation now is synchronized with filer. Previous implementation was sort of hacking and could not handle deep directory renaming, and need to rely on metadata updates from filer, which has delays and may cause race condition.
• Volume
  • Fix binary searching by timestamp, which caused volume.balance error #2364
• S3
  • Use response-content-disposition to overwrite default content-disposition #2371
  • Save content-disposition as extended properties #2371
  • Save content-type during multipart upload and normal object upload #2376 #2383
  • Add Bucket ACL handler #2380
  • Add empty bucket life cycle handler and empty bucket policy handler, empty object acl handler
  • Continuously listen to configuration file updates #2342
• Bugs
  • S3 : nil pointer due to race condition #2370

• Filer Store
  • Add Redis3 to store 10+ millions to billions of files under one directory.
  • Add docker image for RocksDB as filer store
• Filer / S3
  • Fix nil error when assigning file id #2370
• Volume Server
  • Add gRPC API to read all files in one volume.
• Shell
  • volume.fix.replication can fix multiple volumes #1846
• filer.copy tries both assigning volume and uploading data #2351
• Parameter changes:
  • s3: default to allow empty folder for better performance
  • master Etcd sequencer is removed
  • TiKV filer store is removed.

There are no major bugs reported recently. This release is mostly bug fixing on metadata subscription for accurate statistics.

• Cloud Monitoring
  • Adds statistics for directory count and file count.
  • Automatically remove empty folders in buckets, to be more compatible with S3 object store.

Minor:

• Default to 10000 + port for grpc port. This was the original behavior, but was changed in 2.69, which needs to specify grpc port if using a non-default http port.

• Experimental Cloud Monitoring
• Gateway to Remote Object Storage
  • Added weed filer.remote.gateway command for write back local object store changes.
• S3
  • Avoid overwriting object when put object ACL/LegalHold/Retension/LockConfiguration. These operations send empty put requests, which previously were treated as empty file uploads. The empty files will overwrite existing files. So upgrade is strongly recommended, especially when using third party tools.
• Filer
  • Redis store needs to clean up sub folders left by batch deletion.
  • Avoid xattr printed out as HTTP headers #2336

• Cloud Drive cache and sync cloud data to local SeaweedFS cluster

  • remote.cache supports replicated cache.

• Supports custom gRPC ports

• Java Client and Hadoop Compatible File System

  • Release 1.6.8, which needs to go together with current SeaweedFS 2.68 or later.

• Improvements

  • Better IPv6 support.
  • FUSE Mount
    • Add bloom filter to improve performance #2321
    • CompactionTableSizeMultiplier of leveldb use default value. #2325
  • Erasure Coding
    • Clean up if failed in the middle

• S3
  • Fix wrong checking during object copy.
• filer.remote.sync
  • Automatically detect the primary remote storage
  • Randomize bucket name in remote storage to avoid bucket name conflicts.

• Cloud Drive cache and sync cloud data to local SeaweedFS cluster
  • Adds Gateway to Remote Object Storage to automatically create or delete buckets on remote object storage.
  • Adds support for Storj.
  • Adds support for Filebase
  • Adds support for HDFS (WIP)
• Volume Server
  • Moving files also copies modification time, to ensure TTL volumes can work as expected.
• Filer Store
  • Supports TiKV
• S3 API
  • CopyObject return http Status 400 Bad Request for non-existing source #2306

Improvements

• Detect IPv6 adddresses #2310
• Grafana page improves on S3 cost analysis #2290

• Cloud Drive cache and sync cloud data to local SeaweedFS cluster
  • Add support for Azure
  • Add support for BackBlaze
  • Add support for Aliyun
  • Add support for Tencent
  • Add support for Wasabi
  • Distributed parallel downloading.
• Mount
  • Invalidate mount meta cache #2289

• Cloud Drive cache and sync cloud data to local SeaweedFS cluster
  • Support Google Cloud Storage, in addition to existing AWS S3.
• Built with Go 1.17
• Mount
  • Increase concurrency with pooled goroutines #2263
  • Fix a mounting issue #2275
• Filer, S3, Volume servers
  • Read write with streaming gzip, #2274 showed 10% overall performance improvement and 90% memory allocation reduced for small compressible files

Big New Feature!

• Cloud Drive https://github.com/chrislusf/seaweedfs/wiki/Cloud-Drive-Architecture
  It can cache S3 cloud storage data and metadata to local.
  Effectively this is a local cache with unlimited size.
  Only S3 is implemented for now.
  It works with all existing FUSE/Hadoop/S3/WebDAV/HTTP protocols.
  The local changes can be write back to cloud, to work with existing cloud ecosystems.

New Feature:

• Add weed master.follower for scalable volume id lookup.

Improvements:

• Filer
  • Stream read from volume servers to reduce memory usage.
• Shell
  • volume copying: clean up stale volume data files #2250
  • volume.tier.move makes up changes if volume move failed.
  • volume.tier.move, volume.balance, etc fix empty cookie reading problem.
  • Parallelized volume.tier.move
• Volume Server
  • Move volume lookup operations to gRPC.
• S3
  • Add liveness or readiness probe for s3 api handler on /status path. #2245

Bug Fixes:

• Filer
  • gRpc connection error on filer when no volume left #2243
• Volume Server
  • [Version 2.62] Volume server crash when i access file from public url #2247

New Features:

• Adds weed autocomplete for auto completion
• Volume
  • Add concurrentDownloadLimitMB to limit concurrent file download size, to avoid possible memory growth

Minor Improvements:

• Shell
  • volume.tier.move distribute volumes better to avoid moving all volumes to one destination
  • volume.tier.move skip not found error on deletion
  • volume.fix.replication avoid retries even when there is no error #2235
  • volume.balance balance read only volumes first
• filer.backup, filer.meta.backup, filer.meta.tail, filer.sync
  • Add TLS grpc support