v2.0.0

The Bee team is elated to announce the official v2.0.0 release. 🎉

In this release we introduce a brand new mechanism of data redundancy in Swarm with erasure coding, which, under the hood, makes use of Reed-Solomon erasure coding and dispersed replicas. This brings a whole new level of protection against potential data loss.

Erasure Coding

A new header Swarm-Redundancy-Level: n can be passed to upload requests to turn on erasure coding where n is [0, 4]. Refer to the table below for different levels of redundancy and chunk loss tolerance.

Testnet

With this milestone release, the Swarm Testnet is now officially running on the Sepolia blockchain.

Apply the configuration changes below to a fresh node to be able connect to the Sepolia Testnet.

bootnode:
- /dnsaddr/sepolia.testnet.ethswarm.org
blockchain-rpc-endpoint: {a-sepolia-rpc-endpoint}

For questions, comments, and feedback, reach out on Discord.

Features

• Uploads may now be equipped with erasure coding which brings a new level of data redundancy to Swarm. ( #4491 ).
• Added a new API endpoint to obtain the content type and length of uploads using the /bzz endpoint with the Head request type. ( #4588 )
• Re-added livesyncing to chunk syncing in the puller service. ( #4554 )
• Default testnet setting are now configured for the Sepolia blockchain. ( #4491 )
• Added a new db command that verifies the integrity of the pinned content. ( #4565 )
• The pinned content integrity verification can also be done using the API, namely with the new pins/check endpoint. ( #4573 )
• Added the ability for fresh nodes to use an external neighborhood suggester through the config options for mining the overlay address into a specific neighborhood. By default, the Swarmscanner's least populated/most optimal neighborhood suggester API is used. ( #4580 )

Bug fixes

• Localstore fixes
  • Fixed a bug where deleting a pin reference that has been pinned more than once was not removing the chunks from the localstore. ( #4558 )
  • Fixed a race condition in the cachestore that was causing refCnt inconsistencies. ( #4525 )
  • Fixed a bug in the cachestore that would not deference already cached chunks after a reserve eviction. ( #4567 )
  • Fixed a cache size bug that would undercount the number of chunks removed from the cache, leading to a cache leak until the next restart of the node. ( #4571 )
  • Fixed a leak in the upload store where the metadata of the individual chunks persists in the localstore long after the chunks have been successfully uploaded to the network. ( #4562 )
  • Fixed the storage radius metric being set incorrectly. ( #4518 )
  • Fixed a bug where the storage radius does not decrease even though the true reserve size is lower than the threshold. ( #4514 )
• Fixed a vulnerability in the encryption of uploaded data. ( #4604 )

Hardening

• Updated the btcd crypto library version. ( #4516 )
• ReserveSizeWithRadius field, which is the count of chunks in the reserve that falls under the responsibility of the node has been added to the status protocol. ( #4585 )
• Stamper changes
  • The rules for how chunks are stamped before uploading have been changed: regardless of batch type (immutable or mutable), if a chunk has been stamped before, the chunk is restamped using the old batch index and a new timestamp. ( #4556 )
  • Regardless of batch type, the reserve now overwrites chunks that have the same batch index with the higher timestamp. ( #4559 )

For a full PR rundown, please consult the v2.0.0 diff.

v2.0.0

The Bee team is elated to announce the official v2.0.0 release. 🎉

In this release we introduce a brand new mechanism of data redundancy in Swarm with erasure coding, which, under the hood, makes use of Reed-Solomon erasure coding and dispersed replicas. This brings a whole new level of protection against potential data loss.

Erasure Coding

A new header Swarm-Redundancy-Level: n can be passed to upload requests to turn on erasure coding where n is [0, 4]. Refer to the table below for different levels of redundancy and chunk loss tolerance.

Testnet

With this milestone release, the Swarm Testnet is now officially running on the Sepolia blockchain.

Apply the configuration changes below to a fresh node to be able connect to the Sepolia Testnet.

bootnode:
- /dnsaddr/sepolia.testnet.ethswarm.org
blockchain-rpc-endpoint: {a-sepolia-rpc-endpoint}

For questions, comments, and feedback, reach out on Discord.

Features

• Uploads may now be equipped with erasure coding which brings a new level of data redundancy to Swarm. ( #4491 ).
• Added a new API endpoint to obtain the content type and length of uploads using the /bzz endpoint with the Head request type. ( #4588 )
• Re-added livesyncing to chunk syncing in the puller service. ( #4554 )
• Default testnet setting are now configured for the Sepolia blockchain. ( #4491 )
• Added a new db command that verifies the integrity of the pinned content. ( #4565 )
• The pinned content integrity verification can also be done using the API, namely with the new pins/check endpoint. ( #4573 )
• Added the ability for fresh nodes to use an external neighborhood suggester through the config options for mining the overlay address into a specific neighborhood. By default, the Swarmscanner's least populated/most optimal neighborhood suggester API is used. ( #4580 )

Bug fixes

• Localstore fixes
  • Fixed a bug where deleting a pin reference that has been pinned more than once was not removing the chunks from the localstore. ( #4558 )
  • Fixed a race condition in the cachestore that was causing refCnt inconsistencies. ( #4525 )
  • Fixed a bug in the cachestore that would not deference already cached chunks after a reserve eviction. ( #4567 )
  • Fixed a cache size bug that would undercount the number of chunks removed from the cache, leading to a cache leak until the next restart of the node. ( #4571 )
  • Fixed a leak in the upload store where the metadata of the individual chunks persists in the localstore long after the chunks have been successfully uploaded to the network. ( #4562 )
  • Fixed the storage radius metric being set incorrectly. ( #4518 )
  • Fixed a bug where the storage radius does not decrease even though the true reserve size is lower than the threshold. ( #4514 )
• Fixed a vulnerability in the encryption of uploaded data. ( #4604 )

Hardening

• Updated the btcd crypto library version. ( #4516 )
• ReserveSizeWithRadius field, which is the count of chunks in the reserve that falls under the responsibility of the node has been added to the status protocol. ( #4585 )
• Stamper changes
  • The rules for how chunks are stamped before uploading have been changed: regardless of batch type (immutable or mutable), if a chunk has been stamped before, the chunk is restamped using the old batch index and a new timestamp. ( #4556 )
  • Regardless of batch type, the reserve now overwrites chunks that have the same batch index with the higher timestamp. ( #4559 )

For a full PR rundown, please consult the v2.0.0 diff.

Changelog

• 47c612aa feat: v2 (#4615)

Changelog

• ef7c97d7 fix(getter): redundancy getter cleanup (#4610)
• 776f5468 fix(seg65) (#4604)
• 4f322319 feat: alternative withdrawal address (#4606)
• 630b9b17 fix: use neighborhood suggester only on mainnet (#4612)
• c6ce523b chore: remove repetitive words (#4611)
• a54e48ac fix: store dir error info (#4605)
• d3d6685c perf(getter): cancel inflight requests if enough chunks are fetched for recovery (#4608)
• f7b3586c feat: split input file to chunks with specified redundancy (#4600)

Changelog

• be9a15b3 chore: bump github.com/quic-go/quic-go from 0.38.1 to 0.38.2 (#4534)
• b8c7e204 feat: redundancy ci (#4591)
• fd46f75f fix(redundancy): bzz unit test (#4603)
• 93ed7ed9 feat: bzz resource info API (#4588)
• 4dd83710 fix: add missing openapi spec (#4598)
• a4fa9b82 fix(redundancy): on by default when downloading (#4602)
• b49d9e89 fix(pushsync): store the chunk locally when no peers are available fo… (#4597)
• 652977f3 fix(redundancy/getter): wait for recovery and return error (#4581)
• b905de1d feat: pinned reference integrity check API (#4573)
• 5de732c2 fix: missing 200 response (#4526)
• 598bf584 feat: add reserveSizeWithinRadius to status protocol (#4585)
• e5e74a9c feat: add codeql.yml (#4334)
• 6b776052 feat: neighborhood suggester config (#4580)
• 4950c31a fix: strategy and fetch timeout parsing (#4579)
• ad0b209f fix(stamper): global lock stamper across multiple upload sessions (#4578)
• d07d458a fix(doc): minor additions (#4574)

Changelog

• 5b335eaf fix(api): lookahead buffer size header (#4575)

Changelog

• 0ece898b feat: non-local redundancy (#4491)
• d3029897 fix(cache): missing chunk (#4572)
• cd4b13e4 fix: cleanup swap-legacy-factory-addresses (#4569)
• 0d56806c fix: cache size fix (#4571)

Changelog

• ee249af6 refactor: remove legacy factory address and bytecode check (#4568)

Changelog

• b92c0486 fix(cache): deference already cached chunk during shallow copy (#4567)
• 444d365a feat: pinned content integrity cmd (#4565)
• 0a7fd5c9 fix: remove epoch migration code (#4563)
• 771a40b5 fix: upload store fixes (#4562)
• 4532f790 feat: switch to the sepolia testnet (#4465) (#4500)
• bf3a7559 feat: stamper fixes (#4556)
• cffd1cce fix(pinning): check before writing root address (#4558)
• 36855af4 fix(reserve): force overwrite index on collion (#4559)
• bf56d82e feat: re-add livesync (#4554)
• 657f7abf fix(cache): added mutex protection at chunk and eviction levels (#4546)
• 2cdfadf4 fix: tracing for pushsync and retrieval (#4281)
• 2a8c3f9f chore: update docker base image (#4536)
• 7c166ba6 chore: add tests for NewOverlayFromEthereumAddress method (#4485)
• ad2602c3 chore: use the regular split pipeline (#4524)
• 7f2438bd fix: panic on debugstore endpoint in light/ultralight mode (#4521)
• 9625e3f4 fix(postage_service): unit test (#4520)
• 1a9e1be8 chore: bump golang.org/x/crypto from 0.14.0 to 0.17.0 (#4517)
• b15f83b1 fix: radius metric (#4518)
• 6adda088 chore: bump btcd to version v2 (#4516)
• d889c216 fix: lower radius with respect to reserve size within radius (#4514)

The Bee team is excited to announce release candidate v2.0.0-rc1! 🎉

In this release we introduce a brand new mechanism of data redundancy in Swarm with erasure coding, which, under the hood, makes use of Reed-Solomon erasure coding and dispersed replicas.

Erasure Coding

A new header Swarm-Redundancy-Level: n can be passed to upload requests to turn on erasure coding where n is [0, 4]. Refer to the table below for different levels and expected error rates.

Changes for Lowering the Storage Radius

The triggering of a storage radius decrease now depends on the reserve size within the radius (chunks in the reserve with proximity order >= current radius), not the total reserve size, being lower than the 50% of the reserve capacity.

Updated Crypto Library

The btcd crypto library that Bee uses for things like signing chunks and maintaining keys has been updated.

Warning: this is an experimental release. Users only interested in trying out the features and changes in this release should update to this version.

For questions, comments, and feedback, reach out on Discord.

Features

• Uploads may now be equipped with erasure coding which brings a new level of data redundancy to Swarm. ( #4491 )

Bug fixes

• Fixed the storage radius metric being set incorrectly. ( #4518 )
• Fixed a bug where the storage radius does not decrease even though the true reserve size is lower than the threshold. ( #4514 )

Hardening

• Updated the btcd crypto library version. ( #4516 )

For a full PR rundown, please consult the v2.0.0-rc1 diff.

Building upon the previous release, the sync intervals are re-synced so that nodes may collect any potentially missing chunks from the network.

The initial syncing a node performs to collect missing chunks from peers, aka historical syncing, is now rate limited to lower and stabilize CPU usage.

For questions, comments, and feedback, reach out on Discord.

Bug fixes

• Fixed a panic when running compact with an empty db. ( #4488 )

Features

• Puller historical syncing is now rate limited to not exceed 500 chunks/second. ( #4504 )

Hardening

• Puller sync intervals are reset to sync missing chunks. ( #4499 )
• Various UX improvements. ( #4487 #4466 #4489 )

For a full PR rundown, please consult the v1.18.2 diff.

This is a patch release that properly resets the batchstore so that batches can be resynced from the new postage stamp contract.

For questions, comments, and feedback, reach out on Discord.

For a full PR rundown please consult the v1.18.1 diff.

The main theme of this release is the delivery of the last phase of storage incentives, the fourth phase, and thus the end of the storage incentive saga. For this reason, this is a breaking release, as the handshake version has been bumped. The release also includes one bug fix and minor improvements, which can be found below.

Breaking changes

• The handshake protocol has been bumped as there is a new redistribution contract release (#4490)

New features

• Introduction of a command that lists all chunk hashes for a given file (#4484)
• Swarm cache header has been added to several API endpoints (#4457, #4486)
• Phase four of storage incentives (#4373)

Bugfixes

• Re-upload of a file that was previously manually cancelled during upload (#4468)

What's Changed

• 65a45c16c6482b04361689c551d9228ac8228e6a feat: split cmd (https://github.com/ethersphere/bee/pull/4484)
• 22cd4a6b8b8dcc4cd268a32f4474c24ef02dea3f feat: added swarm-cache header to chunks get handler (https://github.com/ethersphere/bee/pull/4486)

Full Changelog: https://github.com/ethersphere/bee/compare/v1.18.0-rc7...v1.18.0-rc8

Changelog

• fbd23aad feat: batchstore cleanup for contracts migration (#4470) (#4481)

Changelog

• c98a7fde chore: bump storage-incentives abi to v0.6.0-rc16 (#4478)
• e8f09dc5 refactor: revert sepolia testnet (#4477)

Changelog

• 29d1647f feat: batchstore cleanup for sepolia migration (#4470)
• ea63f706 feat: switch to the sepolia testnet (#4465)
• aa2746cb fix: overwrite existing stampindex if they have the same timestamp (#4468)
• 3a5e25eb feat: ph4 (#4373)
• ce626eb8 feat: added swarm-cache header for download (#4457)

v1.17.6

With this release, many hardening issues were tackled. The team's focus has been mostly on improving connectivity of nodes across the network and bringing performance improvements to chunk caching operations.

Also featured is a new DB command that will perform a chunk validation of the chunkstore, similar to the optional step in the compaction command.

The retrieval protocol now has a similar multiplexing capability like pushsync, where multiple, in parallel, requests are fired from a forwarder peer that can directly access the neighborhood of a chunk.

For questions, comments, and feedback, reach out on Discord.

Bug fixes

• Fixed a bug where parallel uploads can cause a race condition in the uploadstore. ( #4434 )

Features

• Added a new DB cmd that performs a validation check of the chunks in the chunkstore. ( #4435 )
• Added multiplexing to the retrieval protocol where a forwarding peer that can reach in to the neighborhood of the chunk fires multiple attempts to different peers. ( #4405 )

Hardening

• Added extra documentation about the logger API in the CODING.md. ( #4406 )
• Fixed logs containing wrong token name. ( #4408 )
• Added metrics for zero addressed chunks received by the pullsync protocol. ( #4407 )
• Kademlia depth value is overwritten by the storage radius for full nodes. ( #4410 )
• Salud response duration check is now more stricter. ( #4417 #4426 )
• Upgraded libp2p to the latest version v0.30.0. ( #3927 )
• When batches expire and are removed the batchstore, the stamp issuer data is also removed ( #4416 #4431 #4439 )
• Add a new log to display the amount of time the postage listener will sleep for until the next blockchain sync event ( #4444 #4426 )
• API now returns 404 instead of 500 when no peer can be found for a chunk retrieval attempt. ( #4436 )
• Upgraded crypto related packages. ( #4425 )
• Added various connectivity related improvements: ( #4412 )
  • The reachability of connected peers is tested periodically instead of once at the time of the initial connection.
  • All and not a small subset of neighbor peers are broadcast to a newly connected neighbor.
  • Neighbor peers are periodically broadcast to other neighbors.
  • A peer will be re-added to the addressbook if hive detects an underlay change.

Performance

• Added a cache eviction worker so cached chunks do not need to removed immediately when adding new entries to an over-capacity cache. ( #4423 #4433 )
• The POST /pins/{ref} API endpoint now stores chunks in parallel. ( #4427 )

For a full PR rundown please consult the v1.17.6 diff.

Changelog

• 48a603c5 fix(puller): start syncing of peers in parallel (#4443)
• ed4481f6 fix(listener): log sleep duration more accurately (#4444)

Changelog

• 9221ed01 fix(postage_api): check if batch exists for stamp issuer (#4440)