The Bee team is excited to announce release candidate v2.0.0-rc1! 🎉

In this release we introduce a brand new mechanism of data redundancy in Swarm with erasure coding, which, under the hood, makes use of Reed-Solomon erasure coding and dispersed replicas.

Erasure Coding

A new header Swarm-Redundancy-Level: n can be passed to upload requests to turn on erasure coding where n is [0, 4]. Refer to the table below for different levels and expected error rates.

Changes for Lowering the Storage Radius

The triggering of a storage radius decrease now depends on the reserve size within the radius (chunks in the reserve with proximity order >= current radius), not the total reserve size, being lower than the 50% of the reserve capacity.

Updated Crypto Library

The btcd crypto library that Bee uses for things like signing chunks and maintaining keys has been updated.

Warning: this is an experimental release. Users only interested in trying out the features and changes in this release should update to this version.

For questions, comments, and feedback, reach out on Discord.

Features

• Uploads may now be equipped with erasure coding which brings a new level of data redundancy to Swarm. ( #4491 )

Bug fixes

• Fixed the storage radius metric being set incorrectly. ( #4518 )
• Fixed a bug where the storage radius does not decrease even though the true reserve size is lower than the threshold. ( #4514 )

Hardening

• Updated the btcd crypto library version. ( #4516 )

For a full PR rundown, please consult the v2.0.0-rc1 diff.

This is a patch release that properly resets the batchstore so that batches can be resynced from the new postage stamp contract.

For questions, comments, and feedback, reach out on Discord.

For a full PR rundown please consult the v1.18.1 diff.

What's Changed

• 65a45c16c6482b04361689c551d9228ac8228e6a feat: split cmd (https://github.com/ethersphere/bee/pull/4484)
• 22cd4a6b8b8dcc4cd268a32f4474c24ef02dea3f feat: added swarm-cache header to chunks get handler (https://github.com/ethersphere/bee/pull/4486)

Full Changelog: https://github.com/ethersphere/bee/compare/v1.18.0-rc7...v1.18.0-rc8

v1.17.6

With this release, many hardening issues were tackled. The team's focus has been mostly on improving connectivity of nodes across the network and bringing performance improvements to chunk caching operations.

Also featured is a new DB command that will perform a chunk validation of the chunkstore, similar to the optional step in the compaction command.

The retrieval protocol now has a similar multiplexing capability like pushsync, where multiple, in parallel, requests are fired from a forwarder peer that can directly access the neighborhood of a chunk.

For questions, comments, and feedback, reach out on Discord.

Bug fixes

• Fixed a bug where parallel uploads can cause a race condition in the uploadstore. ( #4434 )

Features

• Added a new DB cmd that performs a validation check of the chunks in the chunkstore. ( #4435 )
• Added multiplexing to the retrieval protocol where a forwarding peer that can reach in to the neighborhood of the chunk fires multiple attempts to different peers. ( #4405 )

Hardening

• Added extra documentation about the logger API in the CODING.md. ( #4406 )
• Fixed logs containing wrong token name. ( #4408 )
• Added metrics for zero addressed chunks received by the pullsync protocol. ( #4407 )
• Kademlia depth value is overwritten by the storage radius for full nodes. ( #4410 )
• Salud response duration check is now more stricter. ( #4417 #4426 )
• Upgraded libp2p to the latest version v0.30.0. ( #3927 )
• When batches expire and are removed the batchstore, the stamp issuer data is also removed ( #4416 #4431 #4439 )
• Add a new log to display the amount of time the postage listener will sleep for until the next blockchain sync event ( #4444 #4426 )
• API now returns 404 instead of 500 when no peer can be found for a chunk retrieval attempt. ( #4436 )
• Upgraded crypto related packages. ( #4425 )
• Added various connectivity related improvements: ( #4412 )
  • The reachability of connected peers is tested periodically instead of once at the time of the initial connection.
  • All and not a small subset of neighbor peers are broadcast to a newly connected neighbor.
  • Neighbor peers are periodically broadcast to other neighbors.
  • A peer will be re-added to the addressbook if hive detects an underlay change.

Performance

• Added a cache eviction worker so cached chunks do not need to removed immediately when adding new entries to an over-capacity cache. ( #4423 #4433 )
• The POST /pins/{ref} API endpoint now stores chunks in parallel. ( #4427 )

For a full PR rundown please consult the v1.17.6 diff.

v1.17.5

In this small but important release, the Bee team introduces a new db compaction command to recover disk space. To prevent any data loss, operators should run the compaction on a copy of the localstore directory and, if successful, replace the original localstore with the compacted copy. The command is available as a sub-command under db as such:

bee db compact --data-dir=

The pushsync and retrieval protocols now feature a fallback mechanism of trying un-reachable and un-healthy peers in the case that no reachable or healthy peers are left.

We've also added new logging guidelines for contributors in the readme.

For questions, comments, and feedback, reach out on Discord.

Bug fixes

• Fixed a bug where a node can get stuck syncing the same interval if the upstream peer is unable to send the chunk data. ( #4339 )

Features

• Added a new localstore compaction command that resizes sharky to the smallest size possible. ( #4329 )

Hardening

• Added a new logging guideline for contributors ( #4352)
• Improved logging of the retrieval pkg and increased the min healthy peers per bin in the salud service.
• Varying levels of peer filtering for the pushsync and retrieval protocols ( #4388 )

For a full PR rundown please consult the v1.17.5 diff.

For the past few weeks, the Bee team's focus has been on improving network health, observability, and user experience.

Node operators, can now mine an overlay address for specific neighborhoods for fresh nodes by using the new --target-neighborhood option. The new Swarm Scanner neighborhoods page displays neighborhood sizes and is a great tool to be used in tandem with this new feature.

Uploads are now by default deferred, as they were before the v1.17.0 release.

Additionally, the default postage stamp batch type is now immutable.

Another behavioral change is that swap-enable is now by default false and the bee start command without additional options starts the node in ultra-light mode. Full node operators must enable the option with swap-enable: true if not already enabled for their nodes to continue to operate as normal.

We have also improved logging across many different services and protocols.
Pushsync and retrieval protocols now report error messages back to the origin node instead of the generic "stream reset" errors. As a result, the protocol version has been bumped, making this a breaking change. It is imperative that operators update their nodes asap.

Previously, nuking a node could cause syncing problems due to the fact that
syncing intervals were never reset. This issue has now been tackled by having nodes detect that a peer's localstore has been nuked. They are able to do this by comparing the peer's localstore epoch time across connections.

For questions, comments, and feedback, reach out on Discord.

Bug fixes

• Pullsync intervals are now reset when the peer's localstore epoch timestamp changes. ( #4290)
• Fixed the issue of not being able to pass a "0" parameter in the API. ( #4301)
• Fixed a bug where the context used for cachestore operations is canceled before a chunk can be cached. ( #4307 )

Features

• Mining an overlay address for specific neighborhoods. ( #4293 )
• Pushsync and retrieval protocol propagate the message of the error that terminates the request back to the origin peer. ( #4321 )
• The naked bee start cmd starts the node in ultra-light mode. ( #4326)

Hardening

• Logging improvements. ( #4295 #4296 #4297 #4302)
• The type of refCnt field of the chunkstore changed from uint8 to uint32, increasing its capacity. ( #4299 #4309 )
• Default batch type is now immutable. ( #4304 )
• The testnet bootnode address is now supplied by default in the configuration. ( #4317 )
• Uploads are now deferred by default. ( #4318 )

For a full PR rundown please consult the v1.17.4 diff.

This is a patch release on top of the 1.17.0 release.

The main fix in this patch is the storage radius is bootstrapped manually using the current value from the mainnet, and not through any other mechanism.

Refer to the main release notes here.

v1.16.0

The bee team is excited to announce v1.16.0! 🎉

The team has been busy researching and testing ways to help swarm remain a reliable and healthy network, and to that end, we are happy to announce a brand new health service, salud.

With salud, nodes will periodically perform certain health checks on it's connected peers with data acquried from the status protocol. The checks as of this release are based on the duration of response to the status protocol msg, number of connected peers, the storage radius, and total batch commitment as computed by each peer.

For duration and number of connected peers, each peer must be within the 80th percentile to be deemed as healthy. Radius and batch commitment are measured in terms of the most common values as reported by each connected peer. Measurements are created anew for each peridoic health check.

A self check is also in place where if the node's own storage radius does not match with the rest of the networks', the node won't participate in the schelling game.

With this release, only the pushsync protocol utilizes the filtering of peers for requests based on the status of health.

For questions, comments, and feedback, reach out on discord.

Breaking changes

• The status protocol now includes the reachability of a peer and the total batch commitment of the network reported by each peer (#4054, #4042)

Bug fixes

• Fixed the last played round field in the redistribution state endpoint to report the corrent round instead of the previous round. ( #4062)

Hardening

• Loosed the requirements of the pushsync protocol so that in the case a neighborhood is down or does not exist, the chunk is pushed as close as possible to it's neighborhood. (#4064)
• Increased the wake up interval for the depthmonitor from 5 to 15 minutes. (#4068)

For a full PR rundown please consult the v1.16.0 diff.

v1.15.0

The bee team is excited to announce v1.15.0!

With this release, we introduce a new pushsync feature to improve chunk syncing and replication in the network during uploading. Peers that forward chunk into the neighborhood will fire multiple requests to target multiple storer nodes. Forwarding will also terminate at the first peer within the neighborhood and will no longer be forwarded to the most closest peer within the network.

As a result of the change in protocol logic, the protocol version has been bumped so it's important that you upgrage your nodes to the latest version.

We've added two new fields to the status protocol response: total amount from the chainstate and mode of operation of the peer (light or full mode).

For questions, comments, and feedback, reach out on discord.

Breaking changes

• Pushync forwarding nodes one hop from the neighborhood multiplexes the chunk to multiple storer nodes in the neighborhood. Forwarding terminates at the first neighbor. (#4008)

Hardening

• Blocklist endpoint now correctly display the mode and reason of the blocklisting of the peer. (#4015)
• Storage radius decreasing based on reserve size increased back to 50% threshold. (#4013)
• Chainstate total amount and peer mode added to the status protocol. (#4002 #4010)
• Connecting light nodes are not rejected based on bin size. (#4018)

Perfomance Improvements

• Initial historical syncing is limited to reduce workload. (#4000)
• Peer skiplist mechnism in retrieval and pushsync protocols now prune peers in a much more effiecient way. (#4001)

For a full PR rundown please consult the v1.15.0 diff.

v1.14.1

This one is a patch release to fix the innacurate the number of reported active nodes in the swarm scanner.

With this change, light nodes attempting to connect to a peer with a full topology bin won't get rejected.

For questions, comments, and feedback, reach out on discord.

The Bee team is excited to announce the latest release!

The main focus of the team for past few weeks has been tightening loose ends surrounding chunk syncing.
The release also features a new API endpoint that returns the status of the node related to the storage incentives, like total reward won and the last round the node has participated and won.

Hardening

• The puller also does syncing of proximity order bin with peers outside of the neighborhood and the pushync protocol uses the storage radius for chunk replication . (#3703)
• Rewrote the retrieval protocol to improve readability and maintainability. (#3671)
• Increased the size of the batch of chunks pulled per a single pullsync request. (#3754)
• Removed some stale config options, like block hash and transaction. (#3765)
• Lowered accounting package logging level to two. (#3771)
• Peers that timeout for historical pullsync requests are blocklisted for some time. (#3774)

New features

• Added a new API endpoint (/redistributionstate) that responds with the storage incentives state of node. (#3683)

Bug Fixes

• The depthmonitor resets the storage radius to the reserve radius only for fresh nodes. (#3757)
• The pusher does stamp validation of chunks in the queue just before pushing the chunk, not when entering the queue. (#3760)

For a full PR rundown please consult the v1.12.0 milestone.

The Bee team is excited to announce the latest release which mostly consists of minor fixes to chunk syncing and improved uploading experience.

The important news is uploads with mutable batches (which are the most common type of batches) are no longer terminated when max utilization of the batch is reached, whereby newer chunks that belong to the batch will simply replace older chunks in the network.

Hardening

• The node is interruptible while it's waiting for the NAT initialization. (#3688)
• Batch synchronization can now be gracefully suspended on interrupt signal. (#3671)
• Removed old overlay mining logic from the codebase. (#3684)
• Added deferred header to allowed CORs headers. (#3669)

New features

• Uploads with mutable batches are not stopped when max utilization is reached. (#3685)

Bug Fixes

• Fixed the depthmonitor to track active syncing background jobs instead of chunks per second when choosing to reduce the syncing/storage radius. (#3680)
• Lowered the minimum required reserve size to maintain a stable storage radius from 50% of the capacity to 40%. (#3682)

For a full PR rundown please consult the v1.11.0 milestone.