cubefs

UPGRAGDE NOTICE

UPGRAGDE NOTICE
If your CubeFS version is v2.3.x or before, please refer to the UPGRADE NOTICE in v2.4.0 for upgrading steps.
if you CubeFS version is v3.2.1 or before, and need to upgrade to v3.3.*,you must follow these upgrade steps:

1. When upgrading the metanode node, you need to add "raftSyncSnapFormatVersion": 0 to the configuration file.
2. After all metanodes are upgraded, remove this configuration item so that raftSyncSnapFormatVersion defaults to 1.
3. Restart all metanodes.
4. Please upgrade the client at the end, the master has enhanced the check of the client, otherwise it will cause the client to mount abnormally

If your Blobstore version is v1.1.0 or before which built with cubefs-blobstore (https://github.com/cubefs/cubefs-blobstore) , please refer to UPGRADE to v3.2.0 following these steps #1556.

Main Feature

• objectnode: ObjectNode support post object. （#2695，@yhjiango）
• objectnode: S3 sts and signature auth. (#2488, @yhjiango)
• libsdk/meta: Support dir lock. (#3344, @longerfly)
• master/data/meta: Support write disable option for volume. (#2537, @NaturalSelect)
• master: Support freeze volume and delayed deletion of volume. (#2922, @shuqiang-zheng)
• master: Support compressing client/partitions api response data to save bandwidth。 (#3314, @lily-lee)

Enhance

• objectnode: Concurrent multipart upload security mechanism (#2928, @tangdeyi)
• master: enable rename atomic operation by default (#2956, @Victor1319)
• master: Optimize the display of capacity when "used" is greater than "total". (#3315, @true1064)
• libsdk: Support more api function for libsdk. (#3316, @longerfly)
• data: Little io error only set current datapartition as broken. (#3317, @true1064)
• client: Write to tiny extent when file size less than 1M. (#3318, @leonrayang)
• master: Optimize CPU cost when invoking the volStat api (#3319, @true1064)
• master: Optimize the speed of applying raft snapshot for master module. (#3320, @NaturalSelect)
• meta: Support rotating and deleting inode files to avoid excessive size. (#3321, @NaturalSelect)
• data: Speed up the startup speed of the datanode. (#3322, @NaturalSelect)
• master: Optimize the volume deletion process. (#3323, @Victor1319)
• master: Add a rdonly flag to the client/partitions api to indicate whether a volume is read-only. (#3324, @true1064)
• data: Support asynchronous deletion of expired datapartitions. (#3325, @Victor1319)
• data: Prioritize listening to external service port when starting the datanode. (#3326, @Victor1319)
• data: Synchronously persist dp meta information when creating dp.(#3335, @bboyCH4)
• client: Support retrying requests for failed extent writes when the client encounters limit io error. ( #3338, @NaturalSelect)

Bugfix

• objectnode: PartNumber of multipart must be greater than 0. (#2885, @yhjiango）
• objectnode: Listobjectv1 with delimiter may lead to panic. (#2931, @tangdeyi)
• client: When the recycle bin is enabled, avoid deleting non-empty directories. (#3328, @bboyCH4)
• libsdk: Fix some issues encountered during the usage of the libsdk. (#3329, @longerfly)
• data: Optimize the blocking issue during data partition migration in snapshot synchronization. (#3330, @bboyCH4)
• meta: Fix the issue of possible duplication caused by the unpersisted meta unique id. (#3331, @true1064)
• master: Master follower replicas only request the leader replica to obtain the partition list.(#3332, @NaturalSelect)
• client: Fix the issue that interruption causes failure to write data to ec.(#3333, @Victor1319)
• data: Fix the issue of memory leaks caused by blocking during data transmission between datanode leader and follower. (#3336, @Victor1319)
• data: During the process of reusing network connections, there is a possibility of using incorrect network response messages. (#3337, @Victor1319)

UPGRAGDE NOTICE

UPGRAGDE NOTICE
If your CubeFS version is v2.3.x or before, please refer to the UPGRADE NOTICE in v2.4.0 for upgrading steps.
if you CubeFS version is v3.2.1 or before, and need to upgrade to v3.3.*,you must follow these upgrade steps:

1. When upgrading the metanode node, you need to add "raftSyncSnapFormatVersion": 0 to the configuration file.
2. After all metanodes are upgraded, remove this configuration item so that raftSyncSnapFormatVersion defaults to 1.
3. Restart all metanodes.
4. Please upgrade the client at the end, the master has enhanced the check of the client, otherwise it will cause the client to mount abnormally

If your Blobstore version is v1.1.0 or before which built with cubefs-blobstore (https://github.com/cubefs/cubefs-blobstore) , please refer to UPGRADE to v3.2.0 following these steps #1556.

Main Feature

• client: Add a trash feature to client (#2291, @bboyCH4)

Enhance

• datanode: Support cleaning up garbage data on the datanode (#2880, @Victor1319)
• metanode: Audit log in metanode (#2899, @NaturalSelect)
• datanode: limit datanode's disk flow and concurrent iops (#2900, @sejust)
• client: Too many waring errors log when use trash (#2915, @bboyCH4)

Bugfix

• client: Audit's writer maybe nil leadto error (#2901, @longerfly)
• gapi: fix read body entirely into memory (#2691, @tangdeyi)
• objectnode: Timing attack can leak user passwords and CubeFS leaks users key in logs from cncf security audit (#2781, @leonrayang)
• client: Optimize insecure random number generation in function util/string.go:RandomString (#2698, @true1064)
• docs: Add Security Best practice (#2853, @leonrayang)
• gapi: Timing attack can leak user passwords and CubeFS leaks users key in logs from cncf security audit (#2781, @leonrayang)
• objectnode: fix limiter lib deadlock (#2794, @tangdeyi)
• master\objectnode: some commits related with security audit (#2824, @leonrayang)
• master: qos.Lock of assignClientsNewQos forget release and trigger deadlock(#2861, @leonrayang)
• bcache: fix possible deadlock of bacahe(#2819, @longerfly)
• ClusterMgr: fix volume manager deadlock(#2793, @tangdeyi)
• blobnode: fix put shard deadlock (#2790, @mawei029)
• client: inode leak when use trash(#2911, @bboyCH4)
• client: trash encounters IO error when dealing with log file names(#2912, @bboyCH4)
• client: trash delete interval do not work(#2914, @bboyCH4)
• client: Failed to delete a large directory with trash enable(#2917, @bboyCH4)

Release v3.3.0 - 2023/08/16

UPGRAGDE NOTICE

UPGRAGDE NOTICE
If your CubeFS version is v2.3.x or before, please refer to the UPGRADE NOTICE in v2.4.0 for upgrading steps.
if you CubeFS version is v3.2.1 or before, and need to upgrade to v3.3.*,you must follow these upgrade steps:

1. When upgrading the metanode node, you need to add "raftSyncSnapFormatVersion": 0 to the configuration file.
2. After all metanodes are upgraded, remove this configuration item so that raftSyncSnapFormatVersion defaults to 1.
3. restart all metanodes.

If your Blobstore version is v1.1.0 or before which built with cubefs-blobstore (https://github.com/cubefs/cubefs-blobstore) , please refer to UPGRADE to v3.2.0 following these steps #1556.

Main Feature

• objectnode: ObjectNode Bucket Policy (#1761, @tangdeyi)
• objectnode: ObjectNode CORS(#1988, @tangdeyi)
• objectnode: ObjectNode S3 ACl (#1990, @yhjiango)
• master\metanode: quota added union directory feature (#2087, @baijiaruo)
• master: add mp step size configuration and dynamic adjustment (#2486, @baijiaruo)
• master\metanode: meta operations guarantee atomicity (#2487, @wuchunhuan
  @Victor1319)

Enhance

• master: broken disk check is wrong in some case (#1547, @Victor1319)
• master: Don't use global variable when implementing maxDpCntLimit func (#1897, @NaturalSelect)
• cfsauto: cubefs surpport autofs mount (#1977, @leonrayang)
• master\cli: The cli tool supports creating 1, 2 copies, and changing the number of copies (#1978, @leonrayang)
• master\cli: Support 2replicas volume down to 1 replica and recycling resources (#1979, @leonrayang)
• master\metanode: support uid space limitation (#1980, @leonrayang)
• client: Optimize client/partitions interface implementation to improve system stability (#1981, @leonrayang)
• master: follower support metrics report (#1982, @leonrayang)
• master: balance leader meta partitions on metanodes (#1985, @xiaocai1215)
• objectnode: ObjectNode supports large file copy (#1989, @tangdeyi)
• datanode: need a quick way to handle client retries too long when reading fault datapartitions which can't be repaired, such as mark these datapartitons as discarded and trigger client fast fail (#1993, @true1064)
• master: needs a settable swich to allow or forbid metaPartition decommission in cluster (#2139, @true1064)
• objectnode: Objnode cold volume read and write buffer optimization (#2309, @tangdeyi)
• master\metanode: add file statistics function(#2489, @liubingxing)
• master: manage access iplist to support client-side IP validity check (#2490, @leonrayang)
• datanode: Strengthen the availability of datanode, and it can still start successfully when a certain dp is abnormal (#2492, @baijiaruo)

Bugfix

• objectnode: overwrite the existing mulitpart (#1700, @tangdeyi)
• unittest: Unit Test Supplement(#1903, @NaturalSelect @jjyaoao)
• metanode: Fix the problem which snapshot was interrupted but saved the metadata (#1986, @xiaocai1215)
• master: master's metrics missing dataparition and missing metaparition not deleted after the fault replica is decommissioned (#2008, @true1064)
• objectnode: ObjectNode UploadPart Interface doesn't compare the Content-Md5 with the body checksum (#2077, @tangdeyi)
• master: A response was sent but not returned (#2084, @NaturalSelect)
• objectnode: Bucket metadata synchronization (#2106, @tangdeyi)
• metanode: not use stream conn pool when delete extents in metanode (#2134, @Victor1319)
• metanode: meta partition raft follower may apply wrong cursor when apply snapshot from leader (#2149, @true1064)
• metanode: metanode create metapartition raft panic (#2230, @Victor1319)
• objectnode: fix bucket policy and location processing (#2239, @tangdeyi)

Release v3.2.1 - 2023/03/16

UPGRAGDE NOTICE

UPGRAGDE NOTICE
If your CubeFS version is v2.3.x or before, please refer to the UPGRADE NOTICE in v2.4.0 for upgrading steps. And also please make sure that your fuse client or objectnode version is equal to or older than the servers, i.e. master, metanode and datanode. In another word, newer versioned client can not be used in a cluster with older versioned servers.

If your Blobstore version is v1.1.0 or before which built with cubefs-blobstore (https://github.com/cubefs/cubefs-blobstore) , please refer to UPGRADE to v3.2.0 following these steps #1556.

Main Feature

• meta\client: add quota of children dentries for directory (#1763, @wuchunhuan )
• master: add qps limiter for individual API on master (#1766, @wuchunhuan )
• client: add audit log for fuse client (#1764, @wuchunhuan )
• objectnode\sdk: objectnode supports docking erasure-code subsytem(blobstore) (#1765, @wuchunhuan)
• master\datanode: Add speed control of dp decommission for disk or datanode (#1778, @bboyCH4)

Enhance

• master: master support leader switching by external trigger (#1772, @leonrayang )
• metanode\master: metanode support follower read (#1775, @leonrayang)
• master: Add metrics for meta-partition consistency detection (#1776, @leonrayang)
• master: add API statistics log for master (#1767, @wuchunhuan)
• datanode: Datanode startup should not be influnced by missing data partitions (#1773, @leonrayang)
• master: volume's capacity must be bigger than 120% used size (#1723, @Victor1319)
• metanode: metanode support config memRatio of the machine. (#1725, @Victor1319)
• datanode: datanode support config diskPathPrefix to manage disk path list. (#1727, @Victor1319)
• metanode: reduce extent alloc memory when metanode snapshot (#1728, @Victor1319)
• metanode: random write may cause metanode memory grow (#1729, @Victor1319)
• datanode: bad disk space is calculated in datanode's total space (#1601, @Victor1319)
• datanode: no premission disk can't be detected (#1602, @Victor1319)
• master\cli: add display of unavailable replica for cfs-cli datapartition check (#1771 ,@true1064)
• datanode: datanode should compute datapartition used size right after loading extents from disk (#1782 ,@true1064)
• client:enhance support nfs access operation (#1798, @leonrayang )
• master: add some metircs (#1785, @liubingxing @litao)
• master\datanode\client\cli: fix the typos and format code (#1787, @liubingxing @litao)
• datanode: add lack datapartitions monitor (#1789, @guojunhao)
• master: create data partition exclude decommissioned disk (#1790 , @guojunhao)
• datanode: Optimize the qos for random write (#1791, @litao)
• metanode: speed up metanode startup (#1792, @liubingxing)
• cli: add more info to cli/fsck cmd (#1793, @liubingxing @litao)

Bugfix

• master: master snapshot recover not reset local rocksdb info (#1522, @wuchunhuan )
• master:Memory cost too fast during restart in case of data partition‘s count is magnity (#1774 , @leonrayang)
• client: Readonly dp can still accept write request from client (#1779, @bboyCH4)
• metanode: Metanode should not establish connection to blobstore for cold volume (#1781, @bboyCH4)
• client: blockcache service may be oom when the client caches many large files concurrently (#1783, @zhangtianjiong)
• datanode: too may delete requests may cause a lot of tcp connections. (#1724, @Victor1319)
• master: when master creating data partition, should use vol.dataPartitionSize as datapartition.total (#1777 ,@true1064)
• master: In addition to all replica status is readwrtie, still need to consider volume used space before check set datapartition status as readwrite (#1780 ,@true1064)
• master: master auto creates dataparitions when volume goes from full to not full (#1784 ,@true1064)
• unit test: Fix the pipeline cubefs-ci error problem (#1795, @baijiaruo)
• metanode\client:do rename on the file which is soft link not worked well (#1797 , @leonrayang)
• meta\master: fix some bug to improve cluster stability (#1786, @liubingxing @litao)

Release v3.2.0 - 2022/10/14

UPGRAGDE NOTICE

If your CubeFS version is v2.3.x or before, please refer to the UPGRADE NOTICE in v2.4.0 for upgrading steps. And also please make sure that your fuse client or objectnode version is equal to or older than the servers, i.e. master, metanode and datanode. In another word, newer versioned client can not be used in a cluster with older versioned servers.

If your Blobstore version is v1.1.0 or before which built with cubefs-blobstore (https://github.com/cubefs/cubefs-blobstore) , please refer to UPGRADE to v3.2.0 following these steps #1556.

Main Feature

• Merge data subsystem blobstore(ec) to main branch.
• blobstore: Merge some service modules #1563

Enhance

• blobstore: ClusterMgr support simple key-value persistent storage #1566
• blobstore: Remove blobstore's dependency on monogdb #1493
• blobstore: Remove blobstore's dependency on consul #1507
• blobstore: Support audit log filtering #1506

Bugfix

• blobstore: Blobnode use the same erasure encoder and buff-pool libraries as Access #1498
• blobstore: Separate flow control for repair and migration traffic #1508
• blobstore: Retry http body under rpc request #1567

UPGRAGDE NOTICE

If your CubeFS version is v2.3.x or before, please refer to the UPGRADE NOTICE in v2.4.0 for upgrading steps. And also please make sure that your fuse client or objectnode version is equal to or older than the servers, i.e. master, metanode and datanode. In another word, newer versioned client can not be used in a cluster with older versioned servers.

Bugfix

• master: add leaderSize param in create partition req #1456

Enhance

• master: support decommission broken replica for 2-replica no leader dp #1456
• datanode: in case disk full, repair size is also thought as used size #1456
• raft: raftstore.monitor can be optimized #1543

UPGRAGDE NOTICE

If your CubeFS version is v2.3.x or before, please refer to the UPGRADE NOTICE in v2.4.0 for upgrading steps. And also please make sure that your fuse client or objectnode version is equal to or older than the servers, i.e. master, metanode and datanode. In another word, newer versioned client can not be used in a cluster with older versioned servers.

Bugfix

• metanode: Show metapartition id correctly #1554
• metanode: remove redundant memcopy when reading raft snapshot #1552

Enhance

• master: update metanode or datanode id need drop old one before #1555
• metanode: add metric mpDentryCount #1553
• datanode: fallocate return interrupt err try again #1551

UPGRAGDE NOTICE

If your CubeFS version is v2.3.x or before, please refer to the UPGRADE NOTICE in v2.4.0 for upgrading steps. And also please make sure that your fuse client or objectnode version is equal to or older than the servers, i.e. master, metanode and datanode. In another word, newer versioned client can not be used in a cluster with older versioned servers.

Main Feature

• master\client\datanode: ：Provide QoS services to enhance multi-tenant isolation #1447
• client: Caching acceleration #1446
• master\datanode: Support two replicas data storage #1389
• master: Add vol label in cfs_metanode_mpInodeCount #1482
• client: Support config posixAcl for volmue #1485
• master: Add datapartition count limit for datanode. #1481

Bugfix

• datanode: TinyDeleteRecord file will be very big when one replica is down #1433
• client: EnablePosixACL is not vaild in centos system #1393
• blockcache:Blockcache memory overflow and be killed #1422
• yum source: Yum install script fails due to missing packages #1410
• unit test: S3test random path busy to produce data and timeout #1515
• master: Zone name check not strict enough while crossZone enabled #1479
• datanode\master: Avoid writing operation failed because of disk full #1519

Enhance

• master: Enable volume update from 3 to 2 replics #1516

UPGRAGDE NOTICE
If your CubeFS version is v2.3.x or before, please refer to the UPGRADE NOTICE in v2.4.0 for upgrading steps. And also please make sure that your fuse client or objectnode version is equal to or older than the servers, i.e. master, metanode and datanode. In another word, newer versioned client can not be used in a cluster with older versioned servers.

Main Feature

• master: support to delete data replica with force #1258

Bugfix

• master: revert the deletion process in the tiny extent to avoid deletion failure caused by different logical and physical sizes #1439
• master: add metapartition replica check to fix replicas status still writeable while metanode unavalible #1440

UPGRAGDE NOTICE

If your CubeFS version is v2.3.x or before, please refer to the UPGRADE NOTICE in v2.4.0 for upgrading steps. And also please make sure that your fuse client or objectnode version is equal to or older than the servers, i.e. master, metanode and datanode. In another word, newer versioned client can not be used in a cluster with older versioned servers.

Notice

This is a significant release based on master branch with a lot of code mainly to support multi layer data layer including blobstore (Erasure-Code) storage, blobstore supports erasure coding storage method(https://github.com/cubeFS/cubefs-blobstore) to support low frequency volume and reduce cost .

Before this we released v3.0.0-beta based on release v2.4.0 (https://github.com/cubeFS/cubefs/releases/tag/v3.0.0-beta) , which make volume can be create as standard(3 replica) or low frequency access type(Erasure-Code), but not support multi data layer.

Release formal version release-v3.0.0 which based on the newest master commit, below is a list of the main features.

1. BlobStore subsystem which supports erasure coding storage. #212
2. multicache layer, 1st level:client local read cache; 2nd level:hot pool replica; 3nd level:code pool based on erasure code. #1291
3. single data replica supported. releated project: https://github.com/oppo-bigdata/shuttle used for remote shuffle. #1227
4. client hot online restart(experimental).
5. optimization reduce extents fragmentation. #1318
6. manage third part library with go mod.

Please refer to the documentation for details,the content already updated.
English version : https://cubefs.readthedocs.io/en/latest/overview.html
Chinese version: https://cubefs.readthedocs.io/zh_CN/latest/

Main Feature

• log: support write ebs log to file and update ebs log level #212
• client\datanode\master: support one replica of data storage #1227
• master: support get master's raft status info #1227
• master: support preload data from blobstore to cubefs #212
• client: client local read cache #212
• client: Time-consuming statistics of client sdk interface
• client: client support blobstore #212
• client: support ebs index #212
• master: support different type dataparitition(normal dp,cache dp,preload dp) #212
• master: support mulit volume type(replcias volume,ec volume) #212
• project: enable go mod and update vendor #1343
• datanode: intruduce metrics degrade level to datanode #1438
• master: not do ttl & updateSize for hot vol #212
• compile: change go mod version from 1.14 to 1.16 and consistent with compile #1343
• compile: enable go mod and update vendor #1343
• project: change from chubaofs to cubefs #1343

Bugfix

• master: single replica create volume should set followeread with true #1227
• raft: raft snapshot still running while peer remove happened which lead to the snapshot gorutine cann't end. #1227
• master: Refactor finding partitins to be offlined #1189
• master: Avoid race syncing node's info when setting rdonly #1189
• client: follow recover handlers when closing open handler #1316
• master: remove 5 GB left limit for disk to create datapartition #1189
• server: if server recive kill-signal before being running-state, server will can't be killed graceful #1261

Enhance

• master: not do ttl & updateSize for hot vol #212
• master: support disk, datanode, metanode decommission
• client: add cli-command for cold volume #212
• master: ec-volume can't be deleted if size not equal 0 #212
• client: add cfs-cli request parameters #212
• client: add scan cli #212
• master: add volume type to newVolCreateCmd #212
• master: not support force delete cold volume #212
• datanode: add cfg diskRdonlySpace for datanode #1353
• datanode: use os.ReadDir instead of ioutil.ReadDir #1438
• datanode: improve performance of write. data allocated didn't put back to buffer pool #1438
• datanode: reduce datanode cpu usage #1438
• datanode: remove unnecessary json unmarshal when getting local extents #1438
• datanode: reduce datanode cpu usage by eliminate file seeks #1438
• client: mitigate the pain of extents fragmentation #1318

Enhance

• Improved the stability and performance of data partition migration significantly.

What's Changed

• Backport fuse client live upgrade to 2.4 by @shyodx in https://github.com/cubeFS/cubefs/pull/1322
• Backport to release 2.4 by @shuoranliu in https://github.com/cubeFS/cubefs/pull/1313
• Backport to 2.4 by @shuoranliu in https://github.com/cubeFS/cubefs/pull/1334
• backport to release 2.4 by @ZhuZhengyi in https://github.com/cubeFS/cubefs/pull/1350
• backport to release 2.4 by @shuoranliu in https://github.com/cubeFS/cubefs/pull/1363
• fix: ci not using smux connection pool by @shuoranliu in https://github.com/cubeFS/cubefs/pull/1365
• Upstream release 2.4 by @shuoranliu in https://github.com/cubeFS/cubefs/pull/1374
• enhance: use dial conn for synced master admin tasks by @shuoranliu in https://github.com/cubeFS/cubefs/pull/1375
• Upstream release 2.4 part.3 by @shuoranliu in https://github.com/cubeFS/cubefs/pull/1376
• Upstream release 2.4 part.4 by @shuoranliu in https://github.com/cubeFS/cubefs/pull/1377
• Upstream release 2.4 part.5 by @shuoranliu in https://github.com/cubeFS/cubefs/pull/1378

Full Changelog: https://github.com/cubeFS/cubefs/compare/v2.4.1...v2.4.3

UPGRAGDE NOTICE

If your CubeFS(ChubaoFS) version is v2.3.x or before, please refer to the UPGRADE NOTICE in v2.4.0 for upgrading steps. And also please make sure that your fuse client or objectnode version is equal to or older than the servers, i.e. master, metanode and datanode. In another word, newer versioned client can not be used in a cluster with older versioned servers.

notice

This is a very important release，volume can be create as standard(3 replica) or low frequency access type(Erasure-Code), we introduce a new module BlobStore subsystem which supports erasure coding storage method(https://github.com/cubeFS/cubefs-blobstore) to support low frequency volume and reduce cost.

Due to 3.0-beta based on release-v2.4 which is lag behind master which is not easy to merge into, we let the core code is temporarily merged into one commit. If you have any questions, you can consult in the community group. All releated code and detail commit will merge into master and is expected to be released in next two months as release v3.0.0.

Please refer to the documentation for details
English version : https://cubefs.readthedocs.io/en/latest/overview.html
Chinese version: https://cubefs.readthedocs.io/zh_CN/latest/

Feature

fuse client. read write blobstore(Erasure-Code) #212

master master. mulit volume type #1291
master low frequency access type volume support #1291
master volume high and low frequency datapartition offline #1291
master out scale of low frequncey volume #1291
master recycle dataparition #1291
master support different type dp in a volume #1291

meta support index of blobstore #212
meta obsolete index of eliminate data #1291

datanode support two different type dataparitition #1291
datanode obsolete and clear cache datapartition #1291
datanode offline datapartition #1291

Other

project name upgrade. chubaofs to cubefs

UPGRAGDE NOTICE

If your CubeFS(ChubaoFS) version is v2.3.x or before, please refer to the UPGRADE NOTICE in v2.4.0 for upgrading steps. And also please make sure that your fuse client or objectnode version is equal to or older than the servers, i.e. master, metanode and datanode. In another word, newer versioned client can not be used in a cluster with older versioned servers.

notice

This is a very important release with a lot of code to support mulit layer data layer such as blobstore (Erasure-Code) to adapte to datalake，volume can be create as standard(3 replica) or low frequency access type(Erasure-Code), we introduce a new module BlobStore subsystem which supports erasure coding storage method(https://github.com/cubeFS/cubefs-blobstore) to support low frequency volume and reduce cost.

Due to 3.0-beta based on release-v2.5 which is lag behind master which is not easy to merge into, we let the core code is temporarily merged into one commit. If you have any questions, you can consult in the community group. All releated code and detail commit will merge into master and release v3.0.0 is expected to be released in next two months.

Please refer to the documentation for details
English version : https://cubefs.readthedocs.io/en/latest/overview.html
Chinese version: https://cubefs.readthedocs.io/zh_CN/latest/

Feature

fuse client. read write blobstore(Erasure-Code) #212

master master. mulit volume type #1291
master low frequency access type volume support #1291
master volume high and low frequency datapartition offline #1291
master out scale of low frequncey volume #1291
master recycle dataparition #1291
master support different type dp in a volume #1291

meta support index of blobstore #212
meta obsolete index of eliminate data #1291

datanode support two different type dataparitition #1291
datanode obsolete and clear cache datapartition #1291
datanode offline datapartition #1291

Other

project name upgrade. chubaofs to cubefs

UPGRAGDE NOTICE

If your ChubaoFS version is v2.3.x or before, please refer to the UPGRADE NOTICE in v2.4.0 for upgrading steps. And also please make sure that your fuse client or objectnode version is equal to or older than the servers, i.e. master, metanode and datanode. In another word, newer versioned client can not be used in a cluster with older versioned servers.

Notice

• this release mainly used to optimize datanode cpu usage

Feature

• datanode feat: intruduce metrics degrade level to datanode [#1234]

Bugfix

• datanode remove 5 GB left limit for disk to create datapartition[#1345]
• fuse client follow recover handlers when closing open handler[#1344]

Enhance

• datanode improve performance of write. data allocated didn't put back to buffer pool [#1346]
• datanode remove unnecessary json unmarshal when getting local extents [#1346]

UPGRAGDE NOTICE

If your ChubaoFS version is v2.3.x or before, please refer to the UPGRADE NOTICE in v2.4.0 for upgrading steps. And also please make sure that your fuse client or objectnode version is equal to or older than the servers, i.e. master, metanode and datanode. In another word, newer versioned client can not be used in a cluster with older versioned servers.

Notice

• this release mainly used to optimize datanode cpu usage

Feature

• datanode feat: intruduce metrics degrade level to datanode [#1234]

Bugfix

• datanode remove 5 GB left limit for disk to create datapartition[#1345]
• fuse client follow recover handlers when closing open handler[#1344]

Enhance

• datanode improve performance of write. data allocated didn't put back to buffer pool[#1346]
• datanode remove unnecessary json unmarshal when getting local extents[#1346]

UPGRAGDE NOTICE

If your ChubaoFS version is v2.3.x or before, please refer to the UPGRADE NOTICE in v2.4.0 for upgrading steps. And also please make sure that your fuse client or objectnode version is equal to or older than the servers, i.e. master, metanode and datanode. In another word, newer versioned client can not be used in a cluster with older versioned servers.

Main Feature

• fuse client:support auto push data to push gateway #1175
• master:reduce flow from interface of "client/partiton" that follower support it and will not redirect to leader #1452
• raft:the receive buffer channel size of raft support to be configurable #1171
• metanode:implement the content summary #1161
• master:domain for cross zone #1169

Bugfix

• metanode:when kill metanode(mean while mp stoped firstly) and if apply snapshot happen at the same time, snapshot will be block, causeing metanode can't be killed #1307
• metanode:meta node migration be recorded in badmetapartitions concurrently without lock which lead to mp id miss #1259
• metanode:makes AppendExtentKeyWithCheck request idempotent #1460
• fuse client:push addr shadowed if export port is not set #1461
• objectnode:readdir gets insufficient dentries if prefix larger than marker #1462
• raft:remove redundant memcopy when reading raft snapshot #1257
• raftstart tcp listen before starting raft #1463
• ltp test:broken test case ftest01，update and unlock ltp test cases
• ltp test:fix ltptest ci bug
• docker:finstall killall command in the docker image
• cli:fix cli tool typo

Enhance

• master: support disk, datanode, metanode decommission and assign target node #1237
• master:return response cache directly, not copy again #1464
• fuse client:increase client retry times to avoid mp raft election timeout.
• grafna: change grafana volume used size rate from rate to deriv
• grafna:support push monitor data to gateway
• objectnode: add subdir authorized check for objectnode
• raft:check nil when get leader term info #1465
• raft: check size when read data from heartbeat port #1465
• raft: add getRaftStatus for metanode #1465
• style:rename cfs to cbfs & optimize the libsdk module
• docker:change: use ghcr instead of dockerhub
• build: add version information and rules for building fsck
• build:change: update build status badge

New feature brief introduction

Fault domain(Failure domain)

1. Purpose
In the cross zone scenario, the reliability need to be improved. Compared with the 2.5 version before, the number of copysets in probability can be reduced. The key point is to use fault domains to group nodesets between multiple zones.

• Reliable papers
  https://www.usenix.org/conference/atc13/technical-sessions/presentation/cidon

• Chinese can refer to
  https://zhuanlan.zhihu.com/p/28417779

2. configuration

1) Master

• Config file：master.json
  Enable faultDomain set item "faultDomain": true

• Zone count to build domain
  faultDomainGrpBatchCnt，default count:3，can also set 2 or 1

  If zone is unavaliable caused by network partition interruption，create nodeset group according to usable zone
  Set “faultDomainBuildAsPossible” true, default is false

  The distribution of nodesets under the number of different faultDomainGrpBatchCnt
  3 zone（1 nodeset per zone）
  2 zone（2 nodesets zone，1 nodeset zone，Take the size of the space as the weight, and build 2 nodeset with the larger space remaining）
  1 zone（3 nodeset in 1 zone）

• Ratio
  1) The use space threshold of the non-fault domain（origin zone）
  After the upgrade, the zone used by the previous volume can be expanded, or operated and maintained in the previous way, or the space of
  the fault domain can be used, but the original space usage ratio needs to reach a threshold, that is, the current configuration item
  The proportion of the overall space used by meta or data
  Default：0.90
  UpdateInterface：
  AdminUpdateZoneExcludeRatio = "/admin/updateZoneExcludeRatio"

  2) the use space threshold of the nodeset group in the domain
  Nodeset group will not be used in dp or mp allocation
  default：0.75
  Update interface：
  AdminUpdateDataUseRatio = "/admin/updateDomainDataRatio"

2) Datanode && metanode

After the fault domain is enabled, a minimum configuration of the fault domain is constructed under the default configuration:
Each zone contains 1 datanode and 1 metanode, and the zone name needs to be specified in the configuration file
There are 3 datanodes and 3 metanodes in 3 zones

For example, three datanodes (metanode) are configured separately:
"zoneName": "z1",
"zoneName": "z2",
"zoneName": "z3",
Start after configuration, the master will build a nodeset for z1, z2, and z3, and component a nodesetgrp

3. Note
1) After the fault domain is enabled, all devices in the new zone will join the fault domain
2) The created volume will preferentially select the resources of the original zone
3) Need add configuration items to use domain resources when creating a new volume according to the table below. By default, the original zone resources are used first if it’s avaliable

Note: the fault domain is designed for cross zone by default. The fault domain of a single zone is considered as a special case of cross zone, and the options are consistent

example : curl "http://10.177.200.119:17010/admin/createVol?name=vol_cross5&capacity=1000&owner=cfs&crossZone=true&defaultPriority=true"|jq .

Content Summary

1. Purpose
In order to query the content summary information of a directory efficiently, e.g. total file size, total files and total directories, v2.5 stores such information as the parent directory’s xattr.

The parent directory stores the files, directories and total file size of the current directory. Then only need to make recursive of the sub directories, and accumulate the information stored by the directories to query the content summary information of a directory.

2. Configuration
Client config file: fuse.json
1) Enable XAttr
”enableXattr”:”true”
2) Enable Summary
”enableSummary”:”true”
Both of xattr and summay have to be set if you want to mount a volume to the local disk.
Set summary is enough if you want to access the volume via libsdk.so.

3. How to use
There are two different ways to get the content summary of a directory.
1) Fuse mount
getfattr -n DirStat yourDirPath
getfattr can be installed by: yum install attr or apt install attr
2) libsdk.so
cfs_getsummary (libsdk/libsdk.go)

4. Note
1)The incremental files’ summary information will be held by their parent directories. But the old files will not. Use cfs_refreshsummary
(libsdk/libsdk.go) interface to rebuild the content summary information.
2)The files, directories and total file size are updated asynchronously in the background. Users are not aware of these operations, but it does
increase the requests to meta servers (usually doubled). You are recommended to evaluate the impact to your cluster before using this
feature.

UPGRAGDE NOTICE

If your ChubaoFS version is v2.3.x or before, please refer to the UPGRADE NOTICE in v2.4.0 for upgrading steps. And also please make sure that your fuse client or objectnode version is equal to or older than the servers, i.e. master, metanode and datanode. In another word, newer versioned client can not be used in a cluster with older versioned servers.

Feature

• meta&object introduce ReadDirLimit interface to retrieve partial results #1234
• fuse client use ReadDirLimit in fuse client #1244

Bugfix

• sdk makes AppendExtentKeyWithCheck request idempotent #1224
• meta start tcp listen before starting raft 1256
• raft remove redundant memcopy when reading raft snapshot to avoid snapshot hanging 1264
• object handling range read request in a behavior compatible with S3 #1286 #1298

Enhance

• sdk add version in the http requests issued to master to collect client info 1262
• sdk mitigate the pain of extents fragmentation 1282

Release v2.4.1-rc.1 - 2021/10/22

UPGRAGDE NOTICE

Please refer to the notice of release v2.4.0 for how to upgrade from previous versions.

Feature

• objectnode introduce readdirlimit which returns part of all the dentries. (#1243 )

Bugfix

• fuse client fix occasional IO errors. (#1179 #1205 #1215)
• fuse client statfs system call returns the correct inode count. (#1192)
• fuse client make AppendExtentKeyWithCheck idempotent.

Enhance

• ci unlock some useful LTP test cases; make ci results checking more strict; upload logs after ci finishes for debug convenience.
• objectnode support subdir checks. (#1208)
• ci add gofmt check. (#1233 )
• raft check size when read data from heartbeat port.
• monitor add master metrics vol_meta_count to export vol dp/mp/inode/dentry count info.

Release v2.4.1-rc.0 - 2021/09/02

UPGRAGDE NOTICE

Please refer to the notice of release v2.4.0 for how to upgrade from previous versions.

Bugfix

• fuse client fix occasional IO errors. (#1179 #1205 #1215)
• fuse client statfs system call returns the correct inode count. (#1192)

Enhance

• ci unlock some useful LTP test cases; make ci results checking more strict; upload logs after ci finishes for debug convenience.
• objectnode support subdir checks. (#1208)