seaweedfs

• S3
  • Fix weed s3 startup error #3607

• Mount
  • Fix corrupted mount reads after changes on file handle #3536
• Volume
  • cloud tiering fix reading s3 reading, consume all read response body #3584
• Shell
  • volume.tier.move add option to change replication. #3583
• Filer
  • Fix rocksdb store incorrectly report not found #3555
  • Fix arangodb transactions and bucket deleting #3586 #3587
• filer.sync adds options to adjust concurrency.
• filer.backup and filer.sync include headers during backup and sync #3532
• S3
  • Fix configuring IAM for the same user
  • Close response body when S3 copying objects
• Misc fixes on data race conditions

• Volume Server
  • Optimize compaction commit step #3388
  • Fix bug on delete count #3499
• Master
  • Remove old raft servers if they do not answer ping requests #3398
  • Send out deleted volume ids before dropping a volume server #3530
• Shell
  • Add speed limit parameter when moving volumes #3478
  • Stop long running jobs if lock is lost
• Gateway To Remote Object Storage
  • Parallelize filer.remote.sync uploading to remote object storage
• Filer Async Backup
  • use S3 upload manager for uploading smaller sized files #3531
• S3
  • Handle preflight CORS requests #3481 #3496
  • Fix for explicitly created directory #3457 #3498
• Mount
  • Implement lseek syscall in FUSE #3491
• Filer UI
  • Fix reloading page before all files are uploaded #3534

File content uploading now is more robust. There are two steps, assigning a file id from master and uploading to volume server. Previously the two steps are usually retried independently. However, it is possible that after step 1, the volume server will change the volume to read only, especially during volume operations, e.g., rebalance, erasure coding, etc. In this release, the two steps are retried together. This should help for filer.copy, mount, webdav, etc.

Another fix is Erasure Coding for volume server, when a volume server has multiple directories and a shared index directory. This combination caused some left over files, which caused other volume manage problems.

• Filer
  • Retry read requests if not found in one replica
  • Fix if-modified-since check doesn't fire on "exact" #3467
  • Fix file uploading for /etc #3476
• S3
  • Fix regression on HEAD directory operation #3457
• filer.backup
  • backup small files if the file is saved in filer (saveToFilerLimit > 0) #3468
• Mount
  • Implement copy_file_range syscall #3475
  • Fix buffer leaks in file gaps. #3472
  • Fix hanging read goroutines in chunk cache #3473
  • Avoid race conditions with current filer address #3474
• Volume
  • Fix metrics reporting on volume file count.
  • Fix EC encoding/decoding problem when multiple dir locations and shared index directory are used together.
• Minor
  • filer.sync offset initialization #3450 #3451
  • S3 fix metrics reporting for deleted collections #3453

• Revert fsync change in 3.21 that is causing write performance problem. #3443

• S3

  • Fix s3 pagination #3436 #3166
  • Fix head object

• Mount

  • Adjust stats reporting for disk utilization du command #3424
  • Avoid memory alignment problem on 32 bit CPU #3427

• WebDAV

  • Fix nil exception #3440

• Master

  • Add randomness when vacuuming #3396
  • Avoid infinite loop with raft when starting admin scripts #3421

• Docker

  • Fix "file not found" for docker entrypoint #3430

You can setup a production-ready SeaweedFS S3 on AWS with cloudformation to enjoy the fastest S3 API, and support SeaweedFS!

Besides many small fixes, the 2 major improvements are:

• The FUSE mount is getting more stable. The long-over-due fix for truncate operation enables SQLite to run safely.
• The filer.sync can synchronize multiple file changes in parallel. Before the fix, all changes can only be applied one-by-one in the target filer cluster.

Change log:

• FUSE Mount
  • Fix truncate operation caused data write failure when new file is smaller in size #2609 #3384
  • Fix link count for hard links #3386
  • Adjust df stats reporting when usage is close to the limit #3407
  • Fix wrong logic for detecting random read pattern
• filer.sync
  • Parallelize synchronization from 1 to 128 copying goroutines.
  • Fix synchronization logic in active-active mode #3328
  • Fix when excluded path is empty
• Filer
  • Filer prefer volume servers in the same data center #3405
  • Add download speed limit support #3408
• Helm Charts
  • Add support for initContainers #3399
  • Misc monitoring improvements #3406
• Filer Store
  • Allow postgresql to use standard environment variables for connection #3413
• Shell
  • filer.meta.load add quiet mode to reduce verbose output #3414

You can setup a production-ready SeaweedFS S3 on AWS with cloudformation to enjoy the fastest S3 API, and support SeaweedFS!

This is the first release under https://github.com/seaweedfs/seaweedfs
The docker images will be migrated later.

• Helm Charts
  • Add PVC storage option for master and filer data and logs. #3367
• Filer.sync
  • Optionally exclude directories to sync #3372
• Mount
  • Fix symlink during git status caused by symlink size reporting #3351
• Volume
  • delete disk_size metrics when collection deleted #3341
• S3
  • Buckets are not created and deleted correctly on the filer with the same signature when they are created and deleted #3374
  • IAM: PutUserAction should completely replace identity actions #3378
• Master
  • Add healthy indicator for raft status #3385

You can now setup a production-ready SeaweedFS S3 on AWS with cloudformation to enjoy the fastest S3 API, and support SeaweedFS!

The 3.17 has a bug that adds a 150ms sleep during each file read operation. It is deleted and here is the same release notes from 3.17.

This release solved 2 major problems:

1. When a master is down, the masters will elect a new leader and the volume servers will report itself to the new leader. This process will take many seconds. During this time, volume id lookup can fail. This is solved in #3350 by reusing previous maps.
2. weed mount fixed wrong logic about hard links.

Here is the list of changes:

• Volume Server
  • Faster loading leveldb volumes. #3337
  • volume server can evacuate to a specific target server #3298
• S3
  • Avoid duplicated actions in IAM PutUserPolicy #3340
• Filer
  • Fallback to previous volume id map to ensure smooth master leader switching #3350
  • Use fallback if volume url is not found #3318
  • leveldb3 filer store adds instant bucket deletion
• Mount
  • Better support for hard links.
  • Fix fix concurrent map read and map write #3344
• Shell
  • volume.fix.replication can change replication by collection #3346

You can now setup a production-ready SeaweedFS S3 on AWS with cloudformation to enjoy the fastest S3 API, and support SeaweedFS!

• Volume Server
  • Loading volumes in parallel to the number of CPU. #3303 #3309
• S3
  • Check whether there is a duplicate accessKey when modifying iam. #3305 #3308
  • File upload should return ETag instead of Etag #3314
• Filer
  • Fix error duplicated local subscription detected #3320
• Mount
  • Fix fatal error: sync: Unlock of unlocked RWMutex #3306
  • Better detecting sequential read and write mode.
  • fix bug during busy writes #3315

You can now setup a production-ready SeaweedFS S3 on AWS with cloudformation to enjoy the fastest S3 API, and support SeaweedFS!

• Master
  • Fix filtering by volume id during vacuum.
• S3
  • Fix pagination ListObjectsV2 under bucket root. #3166
  • Fix IAM CreatePolicy error #3289
• Mount
  • Random read also try to use the local cache first
  • File handle release memory
• Minor
  • Mount adjust reporting on disk usage.
  • Filer store for TiKV support TLS #3286

If you do not want to build the binary yourself, and do not want to wait for the release, you can just download the binaries here, which are automatically built on every commit to the master branch.

The file names are formatted as name-yyyymmdd-HHMM-..., and the build time is in UTC time. Usually there is a 4 minutes delay between the commit and the built. The built log is https://github.com/chrislusf/seaweedfs/actions

You can now setup a production-ready SeaweedFS S3 on AWS with cloudformation to enjoy the fastest S3 API, and support SeaweedFS!

• Filer
  • Fix misc connection edge conditions with master, reducing volume xx not found errors.
• S3
  • Fix pagination. #3263
• Filer Store
  • Add back TiKV support. #3211
• Filer Sync
  • Fix reading timestamp
• WebDAV
  • Fixes the issue where files created via WebDAV show as 0-bytes size when read via fuse. #3250
• Minor
  • weed filer improve command help, add supported filer store list #3235
  • volumeServer.evacuate fix logic add option targetServer #3255
  • handle errors for GenerateDirUuid method #3238

You can now setup a production-ready SeaweedFS S3 on AWS with cloudformation to enjoy the fastest S3 API, and support SeaweedFS!

• Filer
  • Add additional file id lookup when master disconnects.
  • Reconnect to master with updated TLS keys #3226
• Java
  • Fix data uploading when encryption is enabled. #3211
• Filer Store
  • Add back TiKV support #3233
• Master
  • Reduce connections created during vacuum #3218
• FUSE Mount
  • Stats report total size and used size as physical space instead of logical size.

You can now setup a production-ready SeaweedFS S3 on AWS with cloudformation to enjoy the fastest S3 API, and support SeaweedFS!

This is an emergency release since the fix for #3177 is deemed critical. The bug happens for range requests on large files.

• S3
  • Add circuit breaker for each bucket if over concurrent request count limit or size limit #3189
  • Add bucket-specific metrics #3190
  • Return 204 for AbortMultipartUploadHandler #3198
• Volume Server
  • Reduce upfront memory consumption for low-density volumes
  • Add timeout if concurrent upload is over the volume sever limit #3081
• Add weed update to automatically update to the latest version #3184
• Mount
  • Fix a subtle bug causing git checkout to fail.
• Filer Sync
  • Add metrics for the offset of the filer.sync and last send timestamp of the filer subscription #3182
• Filer/Mount
  • Fix invalid chunk data when failed to read manifests on large files. #3195
• Filer UI
  • Provide filer.ui.deleteDir options for server command. #3180
• Remote Object Store Gateway
  • Remove support for HDFS.

You can now setup a production-ready SeaweedFS S3 on AWS with cloudformation to enjoy the fastest S3 API, and support SeaweedFS!

This is an emergency release since the fix for #3177 is deemed critical. The bug happens for range requests on large files.

• Volume Server
  • For range requests on large files, avoid writing too much data. #3177
• Filer Sync
  • Can start separate sync processes to sync different folders, with offset for each folder #3179

You can now setup a production-ready SeaweedFS S3 on AWS with cloudformation to enjoy the fastest S3 API, and support SeaweedFS!

• Remote Object Store Gateway
  • Fix compatibility problem with BackBlaze by removing tagging.
• Mount
  • Increase sequence read speed #3074
  • add disableXAttr in mount option #3140
• S3
  • Detect TTL based on fs.configure setting #3075
  • Fix remove parent directory in NextCloud #3130
  • Fix S3 test: invalid range for ranged request #3154
• Filer Sync
  • Add option to sync from a timestamp #3155
• Filer UI
  • force reload filer page when upload/delete/rename/create directory #3165
• Minor
  • Filer fix customizable local socket file name #3147

You can now setup a production-ready SeaweedFS S3 on AWS with cloudformation to enjoy the fastest S3 API, and support SeaweedFS!

• Volume Server
  • Stream reading large needles, reusing allocated memories. Hopefully this can avoid or reduce latency spikes due to memory GC.
• Mount
  • file handle locks entry object better
• Shell
  • volume.fix.replication fix retry logic #3136
• Filer Sync
  • Fix missing source server uploaded files to target when target down #3132
• Minor
  • fix error reporting for "Need to a hdd disk type" #3128

You can now setup a production-ready SeaweedFS S3 on AWS with cloudformation to enjoy the fastest S3 API, and support SeaweedFS!

• Filer
  • New filers with embedded filer stores can bootstrap correctly from its peers #3089
• S3
  • Fake directory related fix to be compatible with NextCloud #3086
  • Keep authentication enabled in case identities are set to empty #3084
• Volume Server
  • Fix possible nil during compaction #3105
• Mount
  • Fix partial ReadDir results #3102

You can now setup a production-ready SeaweedFS S3 on AWS with cloudformation to enjoy the fastest S3 API, and support SeaweedFS!

This fixed a bug introduced since 2.93, where files larger than 10000*chunkSize will have wrong content during read time. The data storage is fine but the read operation has bug. The default chunks size for 2MB for weed mount and 4MB for weed filer. So files larger than 20GB or 40GB will need this fix.

• Filer
  • Use fsync option from fs.configure if not overwritten #3079
  • Reading files larger than 10000 * chunkSize will have error result.
• Shell
  • volume.fix.replication avoid deleting volume replica if collection does not match #3056
• Volume
  • Avoid two volume servers starting from the same directory #3059

You can now setup a production-ready SeaweedFS S3 on AWS with cloudformation to enjoy the fastest S3 API, and support SeaweedFS!

• S3
  • Add metadata and tagging in s3 api CopyObject #3047
• Shell
  • volume.tier.upload fix atomic add int64 #3038
• Filer
  • YDB filer store improvements #3043