seaweedfs

SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, cross-DC active-active replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding.

APACHE-2.0 License

Downloads
315
Stars
21.4K
Committers
336

Bot releases are hidden (Show)

seaweedfs -

Published by chrislusf over 3 years ago

  • Master
    • Supports snowflake file key generation. #1945
  • Filer
    • MySQL store ensures name column is case sensitive #1931
  • Java Client
    • Support running on windows
  • Minor
    • Master avoid possible nil when nodes are disconnected #1928
    • Mount releases resources when Forget() is called.
    • Shell: volume.check.disk adds optional slow mode #1930
seaweedfs -

Published by chrislusf over 3 years ago

  • Shell
    • Add volume.check.disk to fix inconsistency for replicated volumes #1923
  • S3
    • CopyObject endpoint support copying object to itself #1922
    • dir with slash key does not exist #1917
seaweedfs -

Published by chrislusf over 3 years ago

  • FUSE Mount
    • regression: filer.path option results in empty mount in 2.32 #1906
  • S3
    • count correction for the number of files in the dir #1904
  • Minor
    • remove fasthttp package
seaweedfs -

Published by chrislusf over 3 years ago

  • FUSE Mount
    • Resolve memory leak by adding BatchForget in fuse library. The leak was introduced in version 2.23.
  • Shell
    • Fix a problem with volumeServer.evacuate and volume.balance #1534
    • add s3.clean.uploads to clean up stale multipart uploads #1855
  • Minor
    • S3 escape special characters in file names #1884
    • Filer leveldb, rocksdb metadata store automatically create metadata folers #1901
    • Filer mysql2, postgres2 trigger actions on bucket creation and deletion #1877
    • -ip.bind default to empty
seaweedfs -

Published by chrislusf over 3 years ago

Fixing a problem that .vif file for each volume is not generating. It can cause failures when moving volumes during rebalancing or fix replication. #1878

seaweedfs - 2.30 (has bug #1878)

Published by chrislusf over 3 years ago

  • Filer
    • Adds weed filer.meta.backup command to backup metadata to a different filer store.
  • Cloud Tier
    • Upload to non-AWS S3 gateways
  • S3
    • ListBuckets show accessible buckets #1837
    • MultipleObjectDelete also deletes folders #1857
  • Volume Server
    • Fix EC shard count logic #1863
    • Add experimental TCP support
  • In general
    • Support IPv6
    • Misc fixes for nil objects
seaweedfs -

Published by chrislusf over 3 years ago

  • Add filer.backup command
    • Avoids external message queue. Supports S3, GCP, Azure, B2, local disk, etc.
    • Adds incremental backup mode for daily new or updated files.
  • filer.tier.move
    • avoid repeated move for replicated volumes #1792
  • S3
    • return http status code 200 for empty files #1830 #1831
  • Java Client
    • ByteBuffer.clear supports higher JDK versions #1836
  • command line options
seaweedfs -

Published by chrislusf over 3 years ago

This is a minor release. Adding back weed scaffold -config=xxx

  • Tiered Storage
    • Support custom tags for tiers.
    • Add volume.tier.move to move data from one tier to the other.
  • Minor fixes
    • filer: avoid encryption and compression at the same time #1828
    • volume: avoid fixed vacuum timeout for large volumes
seaweedfs -

Published by chrislusf over 3 years ago

  • Support command line arguments in a configuration file #1823
  • Filer
    • Fix postgres2 mysql2 avoid repeatedly creating tables
    • Fix fs.configure should try to read from entry content #1792
  • Volume
    • Heartbeat should fail fast #1820
  • Minor
    • filer: do not print password on error #1809
    • webdav: add replication setting #1817
seaweedfs -

Published by chrislusf over 3 years ago

S3

  • Fix a critical bug with S3 in 2.24, 2.25 releases #1806 Critical - S3 Delete Multiple Objects, delete the bucket
  • Tiering
  • Java
  • Kubernetes
    • Updated Helm charts #1789 #1790
  • FUSE mount
    • Fix with correct filer path #1799
    • Ensure proper release #1803
  • Minor
    • Fail fast if idx files are missing #1796
    • Replication has a few changes with RabbitMQ #1801
    • Add http max idle connections per host #1802
    • Performance increase with fasthttp client
    • Mount remove unnecsary flush on release
seaweedfs - 2.24 (has bug #1806)

Published by chrislusf over 3 years ago

  • WebDAV
    • Can start together with filer or server as weed filer -webdav or weed server -webdav.
  • S3
    • DeleteMultipleObjects efficiently clean up leftover empty folders #1772
    • Return 204 when deleting nonexistent files #1776
  • Replication
    • RabbitMQ adds delayed retries with dead letter exchange #1773
  • Filer
    • Fix regression with ElasticSearch filer store #1774
  • Java
    • Adds official Java client API with examples, with user friendly SeaweedInputStream and SeaweedOutputStream.
  • Hadoop compatible file system
    • Adds optional fs.seaweed.volume.server.access to access volume servers via their publicUrl settings, or use filer as a proxy.
    • Enabled flush() function.
seaweedfs -

Published by chrislusf over 3 years ago

  • Filer

    • Fix filer store leveldb and hbase miss files when listing large directories more than 1024 entries #1768
    • Fix filer store etcd listing bugs #1767
  • FUSE Mount

    • Invalidate kernel cache properly #1752
    • Add volumeServerAccess modes for publicUrl and filerProxy
    • Report error when flush()
  • S3

    • Optimized DeleteMultipleObjectsHandler
  • Replicate

    • Add local and local_incremental to backup to local file systems #1759 #1761 #1762
  • Shell

    • volume.fix.replication adds collectionPattern with wildcard characters #1758
  • Minor

    • Fix concurrent map reads
    • FUSE Mount on FreeBSD skip daemon timeout
    • Avoid possible nil reader #1754
seaweedfs -

Published by chrislusf over 3 years ago

  • Filer Store
    • Add mysql2, postgres2 for dedicated tables for each bucket. This ensures instant bucket deletion even for large buckets.
  • Filer
    • Support appending to a file with "op=append" URL parameter.
    • Fix http status 206 setting #1753
  • Fuse Mount
    • Use DIRECT_IO on linux to avoid page cache and ensure write-once-read-many with multiple weed mount instances. #1752
    • Limit background requests and congestion threshold
  • Filer meta
    • Watch meta data changes in JSON format by weed filer.meta.tail.
  • Filer Sync
    • weed filer.sync can optionally use filer as a proxy to access volume servers, avoid network setup to expose all volume servers.
  • S3
    • Fix for listing objects if more than 1000 in the folder
seaweedfs -

Published by chrislusf almost 4 years ago

  • Filer
    • Small values are saved to filer store only, saving one network hop to volume server.
    • Fix bug to ensure seamless meta data replication under stress loads.
    • Add leveldb3 to have dedicated filer store for each bucket, for instant large bucket deletion.
    • Avoid http status 206 when error happens for range requests.
  • Mount
    • Fix a data read error due to a race condition when write and then read back quickly.
  • Filer meta tail
    • Add feature to submit metadata updates to ElasticSearch.
  • S3
    • Bug fix: #1741 key count during list objects
  • Minor fixes
    • Fix viper concurrent modification when starting servers.
    • Filer store: nil entry error if not found
seaweedfs -

Published by chrislusf almost 4 years ago

  • Mount
    • Default to 128 concurrent writers to avoid OOM
  • Filer
    • RocksDB adds native support for TTL
    • Upgrade redis module
  • Cloud Tiering
    • Fix s3 upload "use of closed network connection" #1734
  • Add weed filer.cat command
    • Read files directly from volume servers, without proxying through filer
  • Server
    • Add option to disable master
seaweedfs -

Published by chrislusf almost 4 years ago

New

  • Filer
    • Add RockDB as filer store

Bug fixes

  • S3
    • Fix prefix listing #1722
    • Respect Content-Type on PUT #1720
  • Shell
    • Support shell.toml to switch cluster easily #1664
  • Mount
    • Fix weed mount with TLS #1714
  • Master
    • Fix nil snapshot #1718
seaweedfs -

Published by chrislusf almost 4 years ago

seaweedfs - 2.17 Christmas Version!

Published by chrislusf almost 4 years ago

  • Filer
    • Redis, Cassandra can support super large directories
    • Add HBase for filer store
    • Make path-specific stores portable by trimming out common prefixes
    • Add -defaultStoreDir so that filer.toml can be skipped #1659
  • S3
    • Fix a bug causing loop with empty folders #1701
    • Fix bucket-specific Admin permissions
seaweedfs -

Published by chrislusf almost 4 years ago

  • Volume Server
    • Avoid repeated loading the same volume. #1682
  • Filer
    • Support path-specific filer store. (EXPERIMENTAL in 2.16, there are incompatible changes in the git tip)The meta data for one branch of the directory tree can be stored on a separate filer store, for better efficiency and scalability.
    • Fix Mysql/Postgres error handling for non-english languages.
    • Return http status 499 for client closed connection. #1684
  • Minor
    • Expose master UI when http APIs are disabled. #1650
    • Detect IP address better. #1671
    • Prevent mistakenly delete empty collection. #1677
    • filer.copy errors a little bit more obviously. #1678
seaweedfs -

Published by chrislusf almost 4 years ago

  • Shell
    • fix fs.configure for path-specific storage configuration, e.g., Replication, TTL, Collection.
    • add s3.configure to update the s3 identities without restart
  • FUSE mount
    • Avoid memory leaking
  • S3
    • move "delete-directory-if-empty" operation to read time
    • Fix listing with sub folders #1656
    • Avoid cross-collection file rename
  • Filer
    • Paginate through large folders
    • Change to /etc/seaweedfs/filer.conf on filer for path-specific configuration.
Package Rankings
Top 31.22% on Formulae.brew.sh
Top 1.73% on Proxy.golang.org
Top 7.32% on Alpine-edge
Top 37.52% on Pypi.org
Badges
Extracted from project README
Slack Twitter Build Status GoDoc Wiki Docker Pulls SeaweedFS on Maven Central Artifact Hub nodion piknik keepsec Docker Pulls Stargazers over time