juicefs

JuiceFS is a distributed POSIX file system built on top of Redis and S3.

APACHE-2.0 License

Stars
9.9K
Committers
130
juicefs - v1.0.0-beta2

Published by SandyXSD over 2 years ago

JuiceFS v1.0.0-beta2 is the second beta release for v1.0. It has 150+ commits from 16 contributors, thanks to @SandyXSD @zhijian-pro @yuhr123 @xiaogaozi @davies @sanwan @AIXjing @Suave @tangyoupeng @zwwhdls @201341 @zhexuany @chnliyong @liufuyang @rayw000 @fredchen2022 !

New

  • Supports BadgerDB (an embeddable key-value database) as metadata engine (#1340)
  • Supports WebDAV protocol to access JuiceFS files (#1444)
  • Supports read-only clients connecting to a Redis Sentinel-controlled replica (#1433)
  • Added version control of metadata and clients (#1469)
  • Added categories and long descriptions for all commands (#1488, #1493)
  • Added a new option no-bgjob for service commands to disable background jobs like clean-up, backup, etc. (#1472)
  • Added metrics for number of rawstaging blocks (#1341)
  • Added cross-platform compile script (#1374)

Changed

  • cmd: help command is removed; use --help/-h flag to get help information (#1488)
  • cmd: print usage if not enough args (#1491)
  • cmd/format: only try to create bucket when it really doesn't exist (#1289)
  • cmd/format: prevend reusing object storage when formating a volume (#1420, #1449)
  • cmd/destroy: show information of active sessions (#1377)
  • cmd/config: add sanity check of new values (#1479)
  • cmd/umount: ignore error from umount (#1425)
  • meta: add progress bar for CompactAll (#1317)
  • meta: hide password in meta-url (#1333, #1361)
  • meta: use META_PASSWORD env and omit unnecessary characters (#1388)
  • meta: limit the number when scanning sessions/files (#1397)
  • meta: limit the number of clients running background cleanup jobs (#1393)
  • meta: continue dump when encountering non-fatal errors (#1462)
  • meta/sql: increase max idle connections to CPU*2 for SQL engine (#1443)
  • object: set ContentType when putting object (#1468)
  • object: support skip https certificate verify (#1453)
  • object/s3: format support pvc link (#1382)
  • object/qingstor: support private cloud and replace the sdk repository (#1303)
  • vfs: don't do backup for read-only clients (#1435)
  • vfs: add BackupMeta in config (#1460)
  • utils: log now contains the exact line of caller, and is colorized in terminal (#1404, #1318, #1312)
  • utils: simplify the usage of progress bar (#1325)
  • utils: add SleepWithJitter to reduce collision of background jobs (#1412)
  • SDK: upgrade Hadoop common to 3.1.4 (#1411)
  • SDK: java sdk push gateway support multiple volumes (#1492)
  • Other: update chart move it to a standalone repository (#1281, #1336, #1348)
  • Improves documentation and coverage of tests

Bugfix

  • cmd: fix buffer-size in gc and fsck (#1316)
  • cmd/bench: convert PATH to absolute path (#1305)
  • meta: return EROFS as soon as possible (#1477)
  • meta/redis: fix leaked inodes in Redis (#1353)
  • meta/tkv: fix divide by zero error when dumping meta (#1369)
  • meta/tikv: fix scan of tikv, limiting the upperbound (#1455)
  • meta/memkv: fix scanKeys, returning a sorted list (#1381)
  • meta/sql: delete warning message for empty directory (#1442)
  • meta/sql: fix return value of mustInsert (#1429)
  • vfs: fixed deadlock when truncate a released file handle. (#1383)
  • vfs/trash: fix access to trash dir (#1356)
  • vfs/backup: skip dir objects when scanning meta backups (#1370)
  • vfs/backup: fix incorrect inode number when using subdir (#1385)
  • utils: fix the contention between progress bar and logger (#1436)
  • Windows: fix rename fails because the chunk file is still open (#1315)
  • Windows: fix mkdir on windows platform (#1327)
  • SDK: hadoop: fix umask apply (#1338, #1394)
  • SDK: hadoop: fix libjfs.so load bug (#1458)
  • other: fix legend of "Operations" panel in Grafana template (#1321)
juicefs - v1.0.0-beta1

Published by davies almost 3 years ago

JuiceFS v1.0.0-beta1 is the first beta release for v1.0, arrived three months after v0.17. It has 300+ commits from 22 contributors, thanks to @SandyXSD @davies @xiaogaozi @yuhr123 @zhijian-pro @sanwan @zwwhdls @tangyoupeng @Suave @chiyutianyi @201341 @suzaku @reusee @tisonkun @chenjie4255 @dragonly @nature1995 @fredchen2022 @Shoothzj @nsampre @supermario1990 @sjp00556 !

JuiceFS v1.0.0-beta1 is released under the Apache License 2.0.

New

  • Backups the whole metadata as a compressed JSON file into object storage in every hour by default, so we can get most of the data back in case of losing the entire meta database. (#975)
  • Added trash bin: removed or overwritten file will be moved to trash bin, will be deleted after configured days. (#1066)
  • Added config command to update configuration of an existing volume (#1137)
  • Added destroy command to clean up all data & metadata of a volume (#1164)
  • Added a option to limit the concurrent deletes (#917)
  • Added a option to register prometheus metrics api to consul (#910)
  • Added a option to S3 gateway to keep the etag information of uploaded objects (#1154)
  • Added --check-all and --check-new to verify data integrity (#1208)
  • sync command supports anonymous access to S3 (#1228)

Changed

  • meta: don't return EINVAL when encountering unknown flags (#862)
  • cmd/bench: remove default benchmark PATH (#864)
  • hadoop: sort list res to compatible with hdfs (#889)
  • expose metrics in S3 gateway (#897)
  • cleanup broken mountpoint (#912)
  • cmd/info: add '--recursive' flag to get summary of a tree (#935)
  • cache: force sync upload if local disk is too full for writeback (#943)
  • Speed up metadata dump&load for redis (#954)
  • fsck: list broken files (#958)
  • write to json file by stream (#970)
  • speedup listing on file (#622)
  • meta: unify transaction backoff for redis/tkv (#999)
  • speed up metadata dumping and loading for Tikv V2 (#998)
  • check AK and SK for gateway (#1001)
  • add a option to provide customized bucket endpoint (#1008)
  • change frsize to 4096 (#1016)
  • Speed up dump for sql engine (#1006)
  • format support provide only bucket name (#1022)
  • meta/tkv: fix clean stale session (#1041)
  • add namespace and label to existing metrics (#1056)
  • release memory after dumping meta (#1093)
  • meta: unify counters for all metadata engines (#1119)
  • Keep SGID for directory (#1133)
  • Speedup hadoop concat by delete src concurrently (#1163)
  • change default cache-size to 100GB (#1169)
  • metrics: expose only JuiceFS metrics to prometheus (#1185)
  • cleanup reference of unused slice in gc command (#1249)
  • Adjust log level for xorm and TiKV when they are in use (#1229)
  • hadoop: Users in supergroup are superusers (#1202)
  • Check permission with multiple groups (#1205)
  • Ret the region of the object storage compatible with the s3 protocol from the env (#1171)
  • Added prefix to metrics in gateway (#1189)
  • improve s3 url parsing (#1234)

Bugfix

  • go-fuse: return ENOSYS if xattr is disabled (#863)
  • Fix WebDAV backend (#899)
  • meta: always update cache after got newest attributes (#902)
  • sql.go: fix delete file range error (#904)
  • meta: update parent when rename (#905)
  • sqlmeta: fix delete chunk sql error (#931)
  • Fix the bug that file credential authentication fails when using GCS (#933)
  • hadoop: compatible with kitedata in sqoop parquet use case (#941)
  • Fix redis-sentinel addresses parsing (#953)
  • utils: fix cond and its test (#983)
  • workaround for List with marker (#984)
  • sync: ignore broken symlink (#1015)
  • meta/sql: fix delete condition (#1018)
  • metadata: should not skip error if open a file not exist (#1035)
  • Fix minio with sync (#1055)
  • meta: remove xattr only when nlink <= 0 (#1078)
  • meta/sql: fix parentDst nlink in rename (#1082)
  • Fix upload bandwidth limit (#1086)
  • Fix lua script handling big inode number (#1095)
  • meta: fix lookup of corrupt entries (#1098)
  • Fix potential metadata corrupt in Redis caused by gc (#1110)
  • Fix fmtKey to sliceKey (#1143)
  • Fix mkdir/rename/delete/rmr with trailing slash (#1149)
  • Fix gateway GetObjectInfo http code 500 to 404 (#1158)
  • meta: fix nlink of parentDst in rename (#1170)
  • Sync: distinguish DeleteSrc from DeleteDSt (#1197)
  • Fix subdir in S3 gateway (#1201)
juicefs - v0.17.5

Published by chnliyong almost 3 years ago

JuiceFS v0.17.5 is a patch release for v0.17, which has the following changes:

  • Sync command crashes for broken symlinks (#1028).
  • Fixed the returned code when open a non-existed file (#1035).
  • Fixed the leaked key after cleanup stale sessions in TiKV (#1041).
  • Fixed sync from/to some prefix of MinIO (#1055).
  • Fixed the problem that extended attributes of hard links are removed unexpectedly (#1078).
  • Fixed the wrong used space in df (introduced in v0.17.2) (#1096).
  • Fixed a bug in gc command that could corrupt file in Redis (#1110).

Thanks to @davies @SandyXSD @chiyutianyi for contributions!

juicefs - v0.17.2

Published by chnliyong almost 3 years ago

Changelog

JuiceFS v0.17.2 is the second patch release for v0.17, which has the following changes:

  • Fixed potential data leak for SQL engine (#931 #1018).
  • Fixed a bug in CopyFileRange() (77a3a6252d).
  • Fixed authentication for GCS in container (#933).
  • Don't fill all the disk space in writeback mode(#943).
  • Fixed parsing address for Redis Sentinel (#953).
  • Added a workaround to sync with object storages that are not compatible with S3 (#984).
  • Increase backoff for Redis/TiKV to avoid potential failure under high contention (#999).
  • Added a check for AK/SK for S3 gateway (#1001).
  • Added an option to provide a customized endpoint for object storage (#1008).
  • Chnaged frsize to 4096 to be compatible with local filesystem (#1016).

Thanks to @SandyXSD @davies @chiyutianyi @zhijian-pro @nsampre for contributions!

juicefs - v0.17.1

Published by chnliyong almost 3 years ago

JuiceFS v0.17.1 is a patch release for v0.17, which has the following changes:

  • Return ENOSYS if attr is disabled (#863).
  • Ignore unknown flags in setattr() (#862).
  • Sort files when list a directory in Java SDK to be compatible with HDFS (#889).
  • Upgrade nats-server to v2.2+ to address CVE-2021-3127 (#893).
  • Enable metrics for S3 gateway (#897).
  • Fixed WebDAV backend (#899).
  • Refresh cache once new attributes found (#902).
  • Upgrade dgrijalva/jwt-go to golang-jwt/jwt to address CVE-2020-26160 (#903)
  • Fixed parent of renamed files/dirs (#905).
  • Fixed chunk deletions for SQL engine (#904)
  • Upgrade gjson to fix CVE-2021-42836 (#912).

Thanks to contributions from @SandyXSD @tangyoupeng @davies @zhijian-pro @chiyutianyi !

juicefs - v0.17.0

Published by davies about 3 years ago

JuiceFS v0.17 arrived one month after 0.16, with 80+ commits from 9 contributors (@SandyXSD, @davies, @xiaogaozi, @yuhr123, @Suave @tangyoupeng @201341 @zwwhdls @allwefantasy), thanks to them!

This release improved the performance when JuiceFS is used for temporary data by using an in-memory meta engine (memkv) and delayed uploading. For example, we can use JuiceFS as the shuffle and spill disk without worrying about running out of space.

Linux Test Project was used to verify the compatibility of JuiceFS, please checkout the current results here.

This release introduced metadata cache for Java SDK and S3 gateway (similar to metadata cache in kernel), which could be turned to improve performance significantly.

New

  • Added a option to delay upload in writeback mode (#736 #743), which is useful for temporary data.
  • Added an in-memory meta engine (memkv://) for temporary data (#751 #779).
  • Support CREATE and REPLACE flag in setxattr (#770).
  • Added metrics for in-memory cache (#776).
  • Support rename flags (NoReplace and Exchange) (#787).
  • New colorful result for bench command (#810).
  • Added entry and attributes cache for Java SDK and S3 gateway (#835).

Changed

  • Default logging directory in macOS was changed to user's home directory (#744).
  • Limit the number of retry for listing to 3 to avoid infinite loop (#745).
  • Show total size of valid objects in gc command (#746).
  • Disable SHA256 checksum for S3 and other compatible object store to reduce CPU usage (#754).
  • Hide operations on internal files from access log (#766).
  • Require Go 1.15 to build JuiceFS and build the release with Go 1.16 (#771).
  • Inherit gid of parent when SGID is set (#772).
  • Keep SGID when file is non-group-executable (#773).
  • Allow remove broken dir/file (#784).
  • Retry transactions for TxnLockNotFound from TiKV (#789).
  • Cleanup current session during umount (#796).
  • Reduce memory allocation for OBS and Webdav backend (#800).
  • Support escaped access key for KS3 (#830).
  • Support lookup . and .. (#842).
  • No warning if compaction fail with missing objects(#844).
  • Increase available inodes based on current usage (#851).
  • Allow update access key and secret key with default compress algorithm (#855).

Bugfix

  • Fixed a leak in SQL engine (#728).
  • Fixed a bug that may crash client (#729).
  • Fixed valid bytes of progress bar for gc command (#746).
  • Fixed warmup with a long list of files (#752).
  • Fixed supporting of secured Redis connection (regression in v0.16) (#758).
  • Fixed data corruption in SQL and TiKV engine when some slices are skipped during compaction (#759).
  • Fixed metrics for read bytes in Hadoop SDK (#761).
  • Fixed multipart upload in S3 gateway (#765).
  • Fixed POSIX locks on interweaved regions (#769).
  • Fixed latency metrics for libRADOS (#793).
  • Fixed concat in Java SDK and multipart upload (#817).
  • Fixed nlink of parent when rename directories (#839).
  • Fixed transaction for read-only mount (#844).
juicefs - v0.16.2

Published by davies about 3 years ago

JuiceFS v0.16.2 is patched version for v0.16, is recommended to upgrade.

Bugfix

  • Retries LIST three times to avoid infinite loop (#745).
  • Fixed valid bytes of progress bar for gc command (#746).
  • Fixed warmup with a long list of files (#752).
  • Fixed supporting of secured Redis connection (regression in v0.16) (#758).
  • Fixed data corruption in SQL and TiKV engine when some slices are skipped during compaction (#759).
  • Fixed metrics for read bytes in Hadoop SDK (#761).
juicefs - v0.16.1

Published by chnliyong about 3 years ago

JuiceFS v0.16.1 arrived one month after 0.15.2, with 80+ commits from 11 contributors (@davies, @Sandy4999 , @xiaogaozi @tangyoupeng @zhijian-pro @chnliyong @Suave @themaxdavitt @xuhui-lu @201341 @zwwhdls ), thanks to them!

The biggest feature is supporting TiKV as the meta engine, which is a distributed transactional key-value database. With TiKV, JuiceFS can store trillions of files and exabytes of data, please .

BREAKING CHANGE

The meaning of password in Redis Sentinel URL is changed from Sentinel password to Redis server password, please update the password in the URL if you use sentinel and their password are different.

New

  • Supports TiKV as meta engine and object store (#610 #629 #631 #636 #645 #633 #663 #666 #675 #704).
  • Added limit for upload/download bandwidth (#611).
  • Added a virtual file (.config) to show current configurations (#652).
  • Added a subcommand stats to watch performance metrics in realtime (#702 #721).
  • Added progress bar for fsck, gc, load and dump (#683 #684 #724)
  • Disabled updatedb for JuiceFS (#727).

Changed

  • Speedup listing on file store (#593).
  • Upgrade KS3 SDK to 1.0.12 (#638).
  • Update mtime in fallocate (#602).
  • Improved performance for writing into file store (#621).
  • Changed the password in Redis URL to Redis Server (#620)
  • Support mixed read/write in Go/Java SDK (#647).
  • Enable debug agent for sync command (#659).
  • Improved stability for random write workload (#664).
  • Avoid some memory copy in block cache (#668).
  • disable fsync in writeback mode to be closer to local file system (#696).
  • profile: show final result when interval is set to 0 (#710).

Bugfix

  • Fixed stats with MySQL engine (#590).
  • Fixed case insensitivity with MySQL engine(#591).
  • Fixed atime of file in Java SDK(#597).
  • Fixed a bug that block write when memory cache is enabled (#667).
  • Fixed fd leak in block cache (#672).
  • Fixed stale result for restarted transaction (#678)
  • Fixed a bug under mixed write and truncate (#677).
  • Fixed race condition under mixed read/write (#681).
  • Fixed compatibility with Redis clones.
  • Fixed data usage in read-only mode (#698).
  • Fixed key leak in Redis (#694).
  • Fixed a bug about collate with MySQL 5.6 (#697).
  • Fixed a bug that may cause crash when writeback_cache is used (#705).
  • Fixed pid of updated POSIX lock (#708).
  • Added a workaround for a data loss bug in Huawei OBS SDK (#720).
  • Fixed the metrics of uploaded bytes (#726).
  • Fixed a leak of chunk and sustained inode in SQL engine (#728).
  • Fixed a bug that may crash client (#729).
juicefs - v0.15.2

Published by chnliyong over 3 years ago

JuiceFS v0.15.2 arrived 1 month after v0.14.2, with more than 60+ changes from 8 contributors (@davies, @Sandy4999 , @xiaogaozi, @yuhr123, @Suave, @zzcclp , @tangyoupeng, @chnliyong, @yuhr123), thanks to them.

This release introduced new tool to backup and restore metadata, which can also be used to migrate metadata between different meta engines, check Backup and Restore Metadata for details.

This release also improved the performance significantly for read/write heavy workload by utilizing page cache in kernel.

This release is backward-compatible with previous releases, should be safe to upgrade.

New

  • Added command dump and load to backup and restore metadata (#510, #521, #529, #535, #551).
  • Added an option (--read-only) to mount as read-only (#520).
  • Support command auto-completion (#530, #534).
  • Run benchmark in parallel (-p N) (#545).
  • Support PostgreSQL as meta engine (#542).
  • Added an option (--subdir) to mount a sub-directory (#550).
  • Support WebDAV as an object store (#573).
  • Allow enable writeback cache in kernel by -o writeback_cache (#576).
  • Added a option to redirect the logging into a file in background mode (#575).

Changed

  • Changed the batch size of LIST request to 1000 (some object storage may fail with 400 for larger limit) (1385587).
  • Exclude log4j from Java SDK to avoid potential conflict (#501).
  • Exit when unknown option found (d6a39f11db).
  • Report type of meta engine and storage together with usage (#504).
  • Changed the default configuration of Java SDK and S3 gateway to be consistent with juicefs mount (#517).
  • Keep page cache in kernel when files opened without changes (#528, #537).
  • Change REDIS-URL to META-URL in docs (#552).

Bugfix

  • Fixed the memory leak in B2 client (#500).
  • Handle BucketAlreadyExists error for all object storage (#561).
  • Fixed a bug with SQLite and PostgreSQL engine when high bit of lock owner is 1 (#588)
  • Fixed updating stats for MySQL engine (#590).
  • Fixed case sensitivity for MySQL engine (#591).
  • Fixed potential leak for files overwritten by rename (#594, #495)
juicefs - v0.14.2

Published by chnliyong over 3 years ago

JuiceFS v0.14.2 received 30+ contributions from @davies @xiaogaozi @tangyoupeng @Sandy4999 @chnliyong @yuhr123 @xyb @meilihao @frankxieke , thanks to them!

New

  • Added quota for space and inodes for whole volume (#495).
  • Lists all the client session in juicefs status (#491).
  • Cleanup any leaked inodes in Redis with juicefs gc (#494).
  • Supports sticky bits in Java SDK (#475).
  • Added juicefs.umask for Java SDK (#462).
  • Empty the Hadoop trash automatically (#456).

Changed

  • Returns ENOSPC rather than IOError when Redis runs out of memory (#479).

Bugfix

  • Allow superuser to change file mode in Java SDK (#467).
  • Allow owner to change grouping of file (#470).
juicefs - v0.13.1

Published by chnliyong over 3 years ago

JuiceFS v0.13.1 is a bugfix release for v0.13. We have created first release branch for 0.13, which will only have bugfix in future patch releases.

New

  • Support flock for SQL engine (#422).
  • Support posix lock for SQL engine (#425).
  • Global user/gropu id mapping for Java SDK (#439, #447)
  • Added benchmark results for different meta engine (#445).

Bugfix

  • Fixed transaction conflict for SQL engine(#448).
  • Fixed build on macos 11.4 (#452).
  • Fixed parsing redis versions (1c945d746be376831e706761a6a566d236123be3).
  • Ignore deleted file during sync (6e06c0ebd6a2e906235a80807418d18f5ea8a84a).
  • Fixed permission check in Lua script (used by Java SDK and S3 Gateway) (#430).
  • Fixed juicefs sync in distributed mode (#424)
juicefs - v0.13.0

Published by chnliyong over 3 years ago

JuiceFS v0.13 arrived 1 month after v0.12.1, with more than 80 changes from 9 contributors (@davies, @Sandy4999 , @xiaogaozi, @yuhr123, @polyrabbit, @suzaku, @tangyoupeng, @angristan, @chnliyong), thanks to them.

The biggest feature in v0.13 is using SQL database as meta engine, SQLite, MySQL and TiDB are supported right now, we will add others later. Using SQL database will be slower than using Redis, but they have better persistency and scalability than Redis, is better in the cases that data safety and number of files are more important than performance, for example, backup.

New

  • Support SQL database (SQLite, MySQL and TiDB) as meta engine (#375).
  • Added profile to analyze access log (#344).
  • Added status to show the setting and status (#368).
  • Added warmup to build cache for files/directory (#409).
  • Build Java SDK for Windows (#362).
  • Use multiple buckets as object store (#349).
  • Collect metrics for Java SDK (#327).
  • Added virtual file /.stats to show the internal metrics (#314).
  • Allow build a minimized binary without S3 gateway and other object storages (#324).

Changed

  • Enable checksum for Swift (a16b106808aa1).
  • Added more buckets for object latency distribution (#321).
  • Added a option (--no-agent) to disable debug agent (#328).
  • Added internal details at the end of benchmark (#352).
  • More metrics for block cache(#387).
  • Speed up path resolution for Java SDK and S3 gateway using Lua script (#394).
  • Restart the Redis transaction after some known failures (#397).
  • Remove the limit on the number of cached blocks (#401).

Bugfix

  • Fixed a bug in SetAttr to refresh new written data (429ce80100).
  • Fixed overflow in StatFS (abcb5c652b).
  • Fixed a bug when use MinIO with juicefs sync (199b4d35b).
  • Fixed a bug in CopyFileRange, which may affect multipart uploads and concat of Java SDK (fb611b0825).
  • Fixed deadlock when truncate together with read (2f8a8d9).
  • Fixed stale read after truncate (226b6a7e).
  • Fixed downloading a directory using S3 gateway (#378).
juicefs - v0.12.1

Published by chnliyong over 3 years ago

JuiceFS v0.12.1 had fixed a few bugs and improvements on scalability.

Changes

  • Only cleanup leaked chunk in juicefs gc, which may overload redis on larger cluster (6358e388416c).
  • Improve sessions cleanup to avoid scanning all keys (#293).
  • Use hash set for refcount of slices to avoid scanning all keys (#294).
  • Cleanup zero refcount of slices to save memory (#295).
  • Support case insensitivity in Windows (#303).

Bugfix

  • Fixed ranged get for swift (dcd705714f8f).
  • Fixed random read benchmark on files larger than 2G (#305).
  • Fixed listing of files on root directory in Windows (#306)
juicefs - v0.12.0

Published by chnliyong over 3 years ago

JuiceFS v0.12 was arrived 1 month after v0.11, with more than 70 changes from 7 contributors (@davies @xiaogaozi @chnliyong @tangyoupeng @Arvintian @luohy15 @angristan), thanks to them.

New

  • Supports Windows (#195, #268 #271).
  • Added juicefs gc to collect garbage in object store (#248, #290).
  • Added juicefs fsck to check the consistency of file system (#253).
  • Added juicefs info to show internal information for file/directory (slow for large directory)(#288).

Changes

  • Added prefix (juicefs_) and labels (vol_name and mp) for exposed metrics.
  • Support path-style endpoint for S3 compatible storage (#175).
  • Added --verbose as an alias to --debug.
  • Support proxy setting from environment variables for OBS (#245).
  • Wait for block to be persistent in disk in writeback mode (#255).
  • Change the default number of prefetch threads to 1.
  • Fail the mount if the mount point is not ready in 10 second in daemon mode.
  • Speed up juicefs rmr by parallelizing it.
  • Limit the used memory to 200% of --buffer-size (slow down if it's above 100%).
  • Reload the Lua scripts after Redis is restarted.
  • Improved compaction to skip first few large slices to reduce traffic to object store (#276).
  • Added logging when operation is interrupted.
  • Disable compression by default (#286).
  • Limit the concurrent deletion of objects to 2 (#282).
  • Accept named options after positional argument (#274).

Bugfix

  • Accepts malformed repsonse from UFile (f4f5f53).
  • Fixed juicefs umount in Linux (#242).
  • Fixed order of returned objects from listing on SCS (#240).
  • Fixed fetching list of nodes in Java SDK when a customized URL handler is set (#247).
  • Support IPv6 address for sftp(#259).
  • Fixed build with librados (#260).
  • Supports relative path when mount in background (#266).
  • Fixed juicefs rmr with relative path.
  • Cleanup unused objects after failed compaction.
  • Fixed updated files and permissions in sftp.
juicefs - v0.11.0

Published by davies over 3 years ago

New

  • Added S3-compatible gateway based on MinIO (#178).
  • Expose metrics for Prometheus (#181), also a Grafana dashboard template.
  • Support copy_file_range in FUSE (#172) and concat in Hadoop SDK (#173).
  • Support data encryption at-rest (#185).
  • Support China Mobile Cloud (ECloud) (#206).
  • Added juicefs rmr to remove all files in a directory recursively (#207).
  • Support Redis with Senitel (#180).
  • Support multiple directories for disk cache.

Changed

  • Increased read timeout for Redis from 3 seconds to 30 seconds to avoid IO error (#196).

Bugfix

  • Fixed a bug that caused recent written data can't be read back (#157).
juicefs - v0.10.0

Published by davies over 3 years ago

New

  1. A Java SDK was released to provide HDFS compatible API for Hadoop ecosystem.
  2. Added juicefs umount to umount a volume.
  3. Added juicefs benchmark to do simple benchmark (read/write on big/small files)
  4. Merged juicesync into juicefs sync.
  5. Support Sina Cloud Storage.
  6. Support Swift as object store.

Changed

  1. Release memory in read buffer after idle for 1 minute.
  2. Ignore invalid IP (for example, IPv6) returned from DNS of object store.
  3. Improve performance for huge directory (over 1 millions)
  4. Retry operations after Redis is disconnected or restarted.
  5. Added cache for symlink.

Bugfix

  1. Fixed errno for getattr in Darwin when no extented attributes found.
  2. Updated length in readers to allow read latest data written by other clients.
juicefs - v0.9.3

Published by davies almost 4 years ago

This release includes BREAKING changes and a few bugfix:

New

  1. Support rados in Ceph.
  2. Allow update access key and secret key with existing volume.

Changes

  1. Changed to logging into syslog only when running in background.
  2. Changed to use lower cases for command options.
  3. Disable extended attributes by default, which slow down operations even it's not used.

Improvements

  1. Allow non-root to allow in macOS.
  2. Check configs of redis and show warning when there are risks for data safety or functions.
  3. Added volume name in macOS.
  4. wait the mountpoint to be ready when it's mounted in background.

Bugfix

  1. Fixed a bug in ReaddirPlus which slows down tree travelling.
  2. Fixed leaking of extended attributes in Redis.
  3. Fix a bug that truncate may delete slices written in future.
juicefs - v0.9.2

Published by chnliyong almost 4 years ago

This release addressed all the known issue reported from community since first public release.

  1. Added a flag -d to mount in background
  2. Finish compaction part 2: copy small objects into bigger one.
  3. Added pessimistic lock to reduce contention for Redis transaction, to avoid failure under high pressure.
  4. Show . and .. when list a directory with -a in Linux.
  5. Fixed errno for getxattr on internal files.
  6. Fixed local block cache.
  7. Support Scaleway and Minio as object store.
  8. Support mount using /etc/fstab and auto mount after boot.
  9. Added an option --force to overwrite existing format in Redis.
  10. log INFO into syslog, also DEBUG in daemon mode.