seaweedfs

You can now setup a production-ready SeaweedFS S3 on AWS with cloudformation to enjoy the fastest S3 API, and support SeaweedFS!

• Filer
  • Add YDB as filer store #2996
  • Fix pagination with Etcd filer store #3016
  • filer.sync pass attribute for mount #3012
• S3
  • Fix an edge case for multipart upload #3011
  • Skip unix socket mode on windows #3013
• Other small fixes
  • fix filer helm pvc configuration #3020
  • docker compose use -ip.bind=0.0.0.0

You can now setup a production-ready SeaweedFS S3 on AWS with cloudformation to enjoy the fastest S3 API, and support SeaweedFS!

Welcome everyone to take a short ongoing survey with 5 questions! This should help us grow in the right direction!
https://us14.list-manage.com/survey?u=3c1e6dfe1d207b42913bf9518&id=cb0fb93858

• Filer
  • Add an experimental filerGroup option to separate different-purposed filers into their own namespaces
• Shell
  • volume.check.disk add param volumeId to check specific volumn
  • volume.tier.upload progress starts negative #2992
  • volume.tier.upload skip if already copied to remote #2993
• Volume
  • volume: sync to disk before copying volume files #2976
• S3
  • Avoid empty listMultipartUploads response
  • s3 backend support customizing storage class
• Fix ipv6 #2983

This release starts to trim out uncommonly used libraries with large size when compiled, including: "elastic, gocdk, sqlite, hdfs".
If needed, please compile them directly with the Makefile.

You can now setup SeaweedFS S3 on AWS with cloudformation

Welcome everyone to take a short ongoing survey with 5 questions! This should help us grow in the right direction!
https://us14.list-manage.com/survey?u=3c1e6dfe1d207b42913bf9518&id=cb0fb93858

• Shell
  • vacuum volume by collection and volume id #2936
  • volume.list add option to show only read only volumes, by collection name pattern, or by volumeId #2940
• Filer
  • Fix http response error code when reading has error #2960
• Volume
  • Fix bug when deleting volume or unmount volume if a volume server has multiple directories
• Minor
  • Helm Charts: filer using leveldb2 may lose data #2935
• S3
  • handle implicit user name #2957

Welcome everyone to take a short ongoing survey with 5 questions! This should help us grow in the right direction!
https://us14.list-manage.com/survey?u=3c1e6dfe1d207b42913bf9518&id=cb0fb93858

You can now setup SeaweedFS S3 on AWS with cloudformation

• S3
  • check object name and uploadID when processing multi upload #2901
• Shell
  • s3.clean.uploads use jwt token #2900
  • cluster_check fix nil #2905
  • cluster_check add network latency and clock skew checking
• FUSE Mount
  • set file write time correctly to work with rsync #2867
• Master
  • Avoid repeated logs for full volumes.
• Filer
  • Fix starting error on windows. #2930

• Filer and Mount
  • Track and read from Erasure Coding encoded shards
• Filer
  • Improved UI for creating directory, deletion, renaming, upload progress.
• FUSE Mount
  • Fix nil during stress test
• CSI Driver
  • Fix leftover unix socket file

• Mount
  • Adjust quota
  • Fix a map concurrent access bug. #2866
• Shell
  • volume.fsck check specific bucket
  • add cluster.check to check network health
• Erasure Coding
  • encode small chunk return error maybe have some bug #2827
• S3
  • Fix test multiple object delete key limit #2840
  • Fix test multipart upload incorrect etag #2841
  • Fix test delete nonempty bucket #2083

Welcome everyone to take a short ongoing survey with 5 questions! This should help us grow in the right direction!
https://us14.list-manage.com/survey?u=3c1e6dfe1d207b42913bf9518&id=cb0fb93858

• Gateway to Remote Storage
  • Set to correct file size when used together with weed mount
  • Support Contabo remote storage
• S3
  • Multipart upload verifies uploaded parts
• Volume Server
  • Fail fast if too many concurrent requests, to avoid dead lock due to replication.
• Master
  • Always clear previous logs to avoid server is promotable #2804
  • Remote deleted peers of raft server. #2804

Welcome everyone to take a short survey with 5 questions! This should help us grow in the right direction!
https://us14.list-manage.com/survey?u=3c1e6dfe1d207b42913bf9518&id=cb0fb93858

• Master, Filer, S3, IAM also bind to localhost, in addition to -ip.bind. This should avoid exposing to all network interfaces while make most common use cases easy.
• Added a "-config_dir" option to customize *.toml configuration file directory #2753
• CSI Driver v1.0.8
  • Support volume capacity.
  • Support option to create volumes with specific disk type. https://github.com/seaweedfs/seaweedfs-csi-driver/issues/50
  • Releases for amd64, 386, armv7, and arm64.
• Volume Server
  • Add mTLS support for both master and volume http server #2760
• Filer
  • Add option skipCheckParentDir to reduce one metadata lookup.

• Security
  • Default ip bind address to be the same as "-ip" instead of all network interfaces. #1937
• S3
  • Use unix domain socket to connect to local filer.
• Mount
  • Add back support for random read and write. Limit memory usage and disk usage with lots of random read and write.
  • Chmod for mounted folder.
• Bugs
  • #2730, #2740, #2741, #2747

This release supports quota for FUSE mount! When mounted to a specific collection, the usage can be easily collected by checking the collection's volumes. So mount quota can be efficiently supported.

• CSI Driver
  • Support quota https://github.com/seaweedfs/seaweedfs-csi-driver/issues/48
• Mount
  • Support quota when mounted to a collection.
  • Fix directory pagination when using midnight commander
  • Fix error with multiple filers #2719
• filer, s3, volume server more efficient managed memory allocation.

This is mostly a bug fix version related to the new "weed mount" implementation.

• Mount
  • Add back -filer.path support
  • Mark directory uncached. Release file handles.
  • Correctly fsync files that are opened and renamed.
• filer.backup
  • Backup encrypted chunks.

• FUSE Mount Rewrite
  • Faster stream read. Confirmed in this discussion. https://github.com/chrislusf/seaweedfs/discussions/2711
  • Better OS caching.
  • Faster directory listing on linux.
  • Better POSIX compliant.
  • Less memory usage and allocation.
• Filer
  • Support upload to a specific offset. #2694
• Server
  • Fix starting with "-aim" option. #2696

• Master
  • Make heartbeat interval and election timeout of masters configurable #2661
• Mount
  • Disable cache by default to reduce disk space usage.
• Filer
  • Add redis_lua to reduce network trips to redis servers using Lua script. #2676
  • Add http API to read file metadata. #2685
  • Add http API to move a file. #2690
  • Fixed a directory listing bug that may affect Cassandra, Redis, Mongodb, Elastic, Etcd. 7bc67399e4660df55550eaa3328c1e0882c1ae21
• Volume Server
  • Add heathz check to avoid draining pod with last replicas #2680
  • Sync with disk first during the shutdown process a129bda7d98bdbd8ca41c3aeb94739582686a6f1
• S3
  • Fixed a bug from v2.89 where multipart upload are not found. #2665

• Shell
  • volume.check.disk fix wrong logic to compare #2626
  • volume.balance adds delay during tight loop #2637
  • volume.tier.move avoid double counting #2637
• Filer
  • Fix OOM when reading empty files in some cases #2641
  • Fix HTTP header preconditions 7f0c7930
• Filer Sync
  • Fix replicating partially updated file 9405eaef
• S3
  • Added new bucket handlers for more compatibility with AWS S3 Protocol #2635
  • Add error when copy in file, not directory #2632
  • Fix nil in multipart upload #2636
• Mount
  • Add a beta version of weed mount2 #2668 This caused some compilation issue due to underlying package changes. So windows and freebsd versions are not building.
• Volume
  • Fix a problem os.WriteFile() does not fsync.

What's Changed

• Master prom metrics by @kmlebedev in https://github.com/chrislusf/seaweedfs/pull/2613
• async volumeTierMove by @divanikus in https://github.com/chrislusf/seaweedfs/pull/2618

Full Changelog: https://github.com/chrislusf/seaweedfs/compare/2.87...2.88

• S3
  • Bucket Quota Management !!!
• Mount
  • Better support for large file random writes. Use swap file instead of all buffered in memory. Reduce memory consumption. Avoids IO corruption during Sqlite vaccum (https://github.com/chrislusf/seaweedfs/issues/2609).
• Filer
  • Deduplicate metadata events during retries. #2542
  • Cassandra filer store adds connection timeout option. #2597
• Master
  • Adds metrics port for Prometheus. #2602

This release made weed mount more POSIX compliant and working under concurrent read write scenarios.

• Mount
  • Address renaming issues found in #2561
  • Follow POSIX to not allow renaming to non-empty folder.
  • Follow POSIX to check name is too long.
  • Follow POSIX to adjust ctime if file is truncated or metadata is updated.
  • Follow POSIX to adjust ctime if file metadata is updated.
  • Follow POSIX to adjust source file ctime when creating hard link.
  • Follow POSIX to fail deletion if directory is not empty.
  • Follow POSIX to check permissions during deletion.
• S3
  • Fix admin permission #2583
  • Fail fast when "X-Amz-Copy-Source" is a folder #2593
• Shell
  • volume.check.disk avoid memory leaking.
• Filer
  • Add ttimeout setting for Cassandra store #2597
• Commands
  • weed fix takes path as parameter #2563
• Minor Bugs
  • weed filer.copy report error if volume creation fails #2579

• S3
  • Use path prefix to configure access to directories in S3 bucket #2551
• Mount
  • Skip special characters in file names #2559
  • Change entry name after renaming #2561
• Volume
  • volume.fsck skips fixing read only volumes #2562

• Filer and S3
  • Secure Filer http endpoints with JWT and use them in S3 Client #2543
  • Avoid duplicated metadata messages #2550
  • Dynamically adjust connection timeout for slow donwloads. #2541
  • Filer API support fsync option #2528
  • Filer list entries operation use context to break jobs #2539
• Mount adds more tweaks to improve read write speed:
  • Remove limits of parallel requests.
  • Avoid wrong pattern detection due to lock waiting.
  • Disable FUSE write back cache to avoid duplicated writes.
  • Fix:weed mount clean up function was called twice.
• Java Client
  • Sync to version 2.84 to avoid duplicated metadata messages. #2550

Happy Christmas!

This holiday edition has major read and write performance improvements for weed mount. Please let me know if you noticed any difference for your use case.

• FUSE Mount
  • Improve random read performance by only fetching the range instead of the whole chunk.
  • Improve stream read performance by only caching the first chunk to disk.
  • Improve random write performance with temporary chunked file.
  • Improve stream write performance with parallel uploading and without writing to a temporary file.
• Master
  • Revert adding default admin scripts. #2525
  • assign fileId according to DataNode with empty DataCenter and Rack #2527
• S3
  • Audit log default to async mode #2503
  • Bind to a specific IP address #2516
  • Pass through s3 presigned headers #2502
• Filer
  • allow upload file to specific dataNode #2530
  • add "datacenter, rack and datanode" for path specific configuration #2533
• Gateway to Remote Object Store
  • Skip writing temporary uploaded multipart files to remote storage #2509