seaweedfs

• Filer: fix filer read error (https://github.com/chrislusf/seaweedfs/issues/1179)

This has a fix for a compaction bug introduced in 1.48

• volume: fix compaction
• weed fix: generate .idx files correctly

• Tiered Storage
  • Support reverting cloud tier to local tier.
• Erasure Coding
  • Support reverting EC volumes to normal volumes.
• Filer
  • Support extended attributes.
  • Add route by latency option for Redis store.
  • Returns 404 if not found.
  • Optional default recursive deletion.
  • Add fs.meta.cat to see file meta data.
  • Fix fs.meta.save concurrency problem.
• Volume
  • Lower readonly volume memory usage.
  • Support deletions on read only volumes.
  • Avoid core dump when creating volumes and disk is full.

• FUSE mount
  • Fix nil directory problem
• Volume Server
  • Fix missing leading dot in extension

• Feature
  • beta version of transparent cloud integration
    • support tiering to S3
  • Filer
    • support memsql
    • add "mode" parameter when uploading
  • Hadoop Compatible File System
    • release 1.2.4
• Performance
  • mount
    • add option "-dirListCacheLimit" to speed up large directory long-format listing ("ls -l")
  • Filer
    • async directory deletion, separate deletion for metadata and data
  • shell
    • faster filer.meta.save in parallel
• Bugs
  • WebDav
    • fix writing file (#1114 )
• Tweaks
  • docker-compose set to correct master IP option.
  • master submit support "dataCenter" parameter
• Experts
  • Filer deletion add option skipChunkDeletion
  • Replication will check healthy volume servers first (https://github.com/chrislusf/seaweedfs/commit/3727d2488fac473bc49af8239c15f440e9552adb)
  • Graceful volume server shutdown (https://github.com/chrislusf/seaweedfs/commit/a9dbd52e063263cf418bb30805f90e6a9f69fd1a)
  • fuse: check mount point available before do runmount(https://github.com/chrislusf/seaweedfs/commit/0b822a6fb546e642daaf81947a2f943f396e255b)
  • gRPC change directory listing to streaming

• shell
  • fs.meta.save missed directory data with more than 1024 entries (#1130)
  • ec.balance did not balance correctly (#1111)
  • add current filer directory default value
• master
  • add option to store file sequence to etcd
  • assign volume can optionally specify the number of volumes to grow
• volume
  • ec reading error if file size is greater than a limit (#1106)
  • use read lock to avoid hang during heartbeat (https://github.com/chrislusf/seaweedfs/pull/1107)
• filer
  • parallelize weed filer.copy for single file
  • support tikv as filer store

• Filer
  • follow the redirected master #1070
  • Recursive deletion optionally ignore any errors #1062
  • avoid hard-coded upload timeout
• Hadoop Compatible File System
  • fix jar issues on Cloudera HDFS CDH
• Volume
  • in rare cases when volume is created but no superblock #1079
  • skip readonly volume checking when destroy #1063
  • avoid nil needle map #1061
• Experimental
  • grpc API for server side json select pushdown
• increase thread safety by avoiding rand

• Released 1.1.6 Hadoop clients, verified working with HDFS/Spark/HBase.
• master
  • master followers can also lookup and redirect, improves master scalability
  • clients are redirected to new leader, instead of looping through all masters.
• filer
  • support Etcd as filer store, avoiding any SPOF.
  • fix redis configuration typo for database. #1031
• volume
  • fail the volume deletion if compaction is in progress, #1035
  • protect against nil needle map, #1037
  • truncating the temp files when compacting, avoid wrong .dat file after compaction.
• mount
  • add umask option, related to #978
• shell
  • added an option to set the output file name
  • added an option to see verbose output, default to non-verbose mode
• weed backup
  • add ttl and replication parameter

• master:
  • ensure only one exclusive vacuum process
• volume
  • vacuum related optimizations and bug fixes
  • fix volume count metrics bug #1013
  • write requests also checks matching cookies #1014
• shell
  • add fs.mv
• filer
  • set file name when uploading to a directory (#1005)
  • default filer.copy to chunk at 32MB
  • detect a filer path is a file or a directory #1004
• s3
  • add location for multipart upload #1002

• filer
  • leveldb2: fix directory listing https://github.com/chrislusf/seaweedfs/issues/1001

• filer
  • default to leveldb2 store if filer store is not configured
  • add metrics for go memory, gc stats
  • avoid concurrent modification to result slice, fix #972
• volume
  • fix bug with 8000GB version if using in memory index
  • default file ttl to volume ttl, related to #992

• metrics
  • publish metrics to Prometheus Gateway
  • shared Grafana dashboard https://grafana.com/dashboards/10423

• erasure coding: supports blob deletion
• metrics: support Prometheus gateway
• filer
  • fix issue https://github.com/chrislusf/seaweedfs/issues/986 if you are using filer in 1.34, 1.35, please upgrade ASAP.
  • fix Cassandra creating file via FUSE mount

• Support Rack-Aware 10.4 Erasure Coding for warm storage. See https://github.com/chrislusf/seaweedfs/wiki/Erasure-Coding-for-warm-storage
• Security: support JWT for volume server read access

• NOTICE: For filer with a large leveldb store, initial upgrading will trigger overdue background compactions, to optimize the leveldb files. This can cause high CPU and disk usage. Just let it run.
• filer
  • add leveldb2 for faster local leveldb
  • optimize leveldb parameters.
• master
  • configurable JWT expires_after_seconds
• shell
  • add volume.balance command
• volume
  • add option to throttle background volume compaction
• add weedfuse for fstab

• Added WebDAV
• volume
  • optionally slow down compaction (b335f81a4fb51e57b0da37b2b662e823369459f6)
• filer
  • listen on specified ip address (dabc9c9a891b7b563ae17dbb4030ddecdde73483)
  • remove customizable grpc port (21fe5bde7cf0e619b4d9f2b2a15f8541988a646b)
• server
  • add option to start S3 API gateway

• UI
  • fix volume server metrics (#935)
• shell
  • add fs.cat to see file content

• shell
  • add fs.meta.save, fs.meta.load
  • add fs.meta.notify, removing command "weed filer.export"
  • add volume.copy, volume.move, volume.mount, volume.unmount, volume.delete
• filer
  • rename -port.public to -port.readonly
  • add locking for in memory filer store
  • filer.copy add local compression level
  • adjust mysql database DDL
• volume
  • reset leader if error happens
  • support http status NotModified 304

• weed filer
  • optimized leveldb memory usage
  • removed btree and boltdb options
  • added leveldbMedium and leveldbLarge options
• docker
  • added cronjob mode to run "volume.fix.replication" every 5 hours
• important fix
  • close gRPC connections if failure happens

• weed volume
  • add builds for large disk. volume size can be up to 8000GB.
• weed s3
  • multipart upload goes to the right bucket
  • fix aws s3 ls lists directories but not files (fix #911)
• weed replicate
  • replicate atomic renaming to other systems
  • add GoCDK PubSub support
• weed filer
  • mysql/postgres use ReadCommitted transaction isolation level
  • increase default mysql/postgres dir or name length to 65535 (just instruction changes)
  • fix directory cache invalidation error (fix #910)
• weed shell
  • add fs.cd, fs.pwd to change to a directory and print current directory
• weed filer.copy
  • set default chunk size to 32MB
  • parallelize weed filer.copy
  • client side gzip compression
  • keep original file owner id and group id
• weed upload
  • set default chunk size to 32MB