Changelog

• 9afd7926 fix: remove stamp data on expiration, not on node start up and API changes (#4439)

Changelog

• f32ff23d fix: cache and stamper store (#4433)
• f780811b feat: support db validate command (#4435)
• b0836ff1 fix: uploadstore (#4434)
• 6cac44cf fix: return 404 when no peers found during retrieval (#4436)
• f5acd99c fix: change beekeeper branch to master
• 60149abf fix(ci): patch max retrieval forwards (#4432)
• d0a5a523 fix(stamperstore): remove stamps belonging to expired batches (#4431)
• d324646c perf(pin_api): fetch chunks in parallel (#4428)
• 4a424d54 chore: extra batch listener log and salud tweaks (#4426)
• 58ef6f3b chore: upgrade go-ethereum and btcd modules (#4425)
• a3decdec fix(kademlia, reacher): various connection fixes (#4412)
• b6ea8b3b fix(worker): evicts batch only once (#4421)
• 75018779 fix: make cache ops independent of parent context (#4423)
• 06bc50e6 chore(linter): update version (#4422)
• 5996148f fix: delete expired stamp issuers (#4416)
• 1634ff22 chore: upgrade libp2p to v0.30.0 (#3927)
• e7906af9 chore(salud): stricter health checks (#4417)
• aa68793f fix: kademlia to use storage radius as neighborhood depth (#4410)
• 61d90a93 fix(reserve): run invalid chunk eviction only if there are no other b… (#4411)
• a7bd625f feat(retrieval): multiplex into the neighborhood (#4405)
• f469d5bf chore(pullsync): invalid chunk metric (#4407)
• 6d63ff57 fix: wrong token name in the minimum amount log messages (#4408)
• af315e20 doc: add logger API usage section (#4406)
• 9465bd61 chore: lower min depth (#4403)
• 98a595ca fix: typo in the docker README.md (#4402)
• 783d3001 chore: extra unit test for compaction (#4400)
• 847a9026 feat: stamp issuer utilization rate test (#4398)
• c375d3b9 chore: bump golang.org/x/net from 0.10.0 to 0.17.0 (#4397)
• 519c1acd fix(compact): mark already used slots (#4395)
• 8b150cf1 chore(pusher): logging address of shallow receipt (#4393)
• 5d006520 fix: remove expired batches in the worker and added sleep after to com… (#4391)
• fbf482bc fix(retrieval): wait for direct quit signal instead of context deadli… (#4392)
• 2309ec1b refactor: enrich transaction service logs with sender address (#4390)
• 414fc3b9 feat: add logs to the txstore and txchunkstore recovery method (#4387)
• 9d147362 feat(pushsync, retrieval): varying levels of filtering for the closes… (#4388)
• f42feece fix: move the postage logger under the node logger tree (#4386)
• 401e24e8 chore: added extra logging in reserve worker for invalid batches (#4384)
• cfc472a9 revert: ph4 (#4372)
• 5e6e9632 chore: increase max commit length

v1.17.5

In this small but important release, the Bee team introduces a new db compaction command to recover disk space. To prevent any data loss, operators should run the compaction on a copy of the localstore directory and, if successful, replace the original localstore with the compacted copy. The command is available as a sub-command under db as such:

bee db compact --data-dir=

The pushsync and retrieval protocols now feature a fallback mechanism of trying un-reachable and un-healthy peers in the case that no reachable or healthy peers are left.

We've also added new logging guidelines for contributors in the readme.

For questions, comments, and feedback, reach out on Discord.

Bug fixes

• Fixed a bug where a node can get stuck syncing the same interval if the upstream peer is unable to send the chunk data. ( #4339 )

Features

• Added a new localstore compaction command that resizes sharky to the smallest size possible. ( #4329 )

Hardening

• Added a new logging guideline for contributors ( #4352)
• Improved logging of the retrieval pkg and increased the min healthy peers per bin in the salud service.
• Varying levels of peer filtering for the pushsync and retrieval protocols ( #4388 )

For a full PR rundown please consult the v1.17.5 diff.

Changelog

• 519c1acd fix(compact): mark already used slots (#4395)

Changelog

• 8b150cf1 chore(pusher): logging address of shallow receipt (#4393)
• 5d006520 fix: remove expired batches in the worker and added sleep after to com… (#4391)
• fbf482bc fix(retrieval): wait for direct quit signal instead of context deadli… (#4392)
• 2309ec1b refactor: enrich transaction service logs with sender address (#4390)

Changelog

• 414fc3b9 feat: add logs to the txstore and txchunkstore recovery method (#4387)
• 9d147362 feat(pushsync, retrieval): varying levels of filtering for the closes… (#4388)
• f42feece fix: move the postage logger under the node logger tree (#4386)

Changelog

• 401e24e8 chore: added extra logging in reserve worker for invalid batches (#4384)

Changelog

• cfc472a9 revert: ph4 (#4372)
• 5e6e9632 chore: increase max commit length

Changelog

• e7f78764 feat: unwrap smart contract custom abi errors (#4367)

Changelog

• 8f17d2ac feat(compact): log retrieval item data of invalid chunks (#4354)
• 81e6e04f fix: include the reason when the redistribution contract call fails (#4358)

Changelog

• 08904602 feat(storageincentives): storage incentives phase 4 (#4345)
• 60438eb6 feat(reserve): new worker and metric for reserve size within radius (#4351)
• 15363da4 chore: added more logs and tweaked salud min peer count per bin (#4350)
• 1b088f6a chore(compaction): warning logs (#4349)
• 4ac8d9f2 fix(pullsync): swallow and log process want reserve get errs (#4339)
• 9ce6f3cf feat: db compaction cmd (#4329)
• d3377706 revert: 1f84ef4 ph4 bmt (#4338)
• 25eb2337 feat: introduce min bucket depth (#4331)
• 1f84ef45 fix(ph4): bmt (#4308)

For the past few weeks, the Bee team's focus has been on improving network health, observability, and user experience.

Node operators, can now mine an overlay address for specific neighborhoods for fresh nodes by using the new --target-neighborhood option. The new Swarm Scanner neighborhoods page displays neighborhood sizes and is a great tool to be used in tandem with this new feature.

Uploads are now by default deferred, as they were before the v1.17.0 release.

Additionally, the default postage stamp batch type is now immutable.

Another behavioral change is that swap-enable is now by default false and the bee start command without additional options starts the node in ultra-light mode. Full node operators must enable the option with swap-enable: true if not already enabled for their nodes to continue to operate as normal.

We have also improved logging across many different services and protocols.
Pushsync and retrieval protocols now report error messages back to the origin node instead of the generic "stream reset" errors. As a result, the protocol version has been bumped, making this a breaking change. It is imperative that operators update their nodes asap.

Previously, nuking a node could cause syncing problems due to the fact that
syncing intervals were never reset. This issue has now been tackled by having nodes detect that a peer's localstore has been nuked. They are able to do this by comparing the peer's localstore epoch time across connections.

For questions, comments, and feedback, reach out on Discord.

Bug fixes

• Pullsync intervals are now reset when the peer's localstore epoch timestamp changes. ( #4290)
• Fixed the issue of not being able to pass a "0" parameter in the API. ( #4301)
• Fixed a bug where the context used for cachestore operations is canceled before a chunk can be cached. ( #4307 )

Features

• Mining an overlay address for specific neighborhoods. ( #4293 )
• Pushsync and retrieval protocol propagate the message of the error that terminates the request back to the origin peer. ( #4321 )
• The naked bee start cmd starts the node in ultra-light mode. ( #4326)

Hardening

• Logging improvements. ( #4295 #4296 #4297 #4302)
• The type of refCnt field of the chunkstore changed from uint8 to uint32, increasing its capacity. ( #4299 #4309 )
• Default batch type is now immutable. ( #4304 )
• The testnet bootnode address is now supplied by default in the configuration. ( #4317 )
• Uploads are now deferred by default. ( #4318 )

For a full PR rundown please consult the v1.17.4 diff.

v1.17.3

The Bee team is excited to announce v1.17.3! 🎊

With this release, we introduce healthy peer filtering for retrieval requests to improve data download reliability and speed.

The behavior of the upload APIs are changed back to pre-1.17.0 so parallel uploads with the same batch are now allowed.

The deprecated bee-clef is now removed from all bee packaging and docker images so operators should update their setups accordingly.

The node during the initial bootup will go through a localstore recovery process that should take a few minutes to complete. A localstore leak was detected in the v1.17.0 release, so with this recovery process and the fix included in this release, disk space consumption should become stable.

Pullsync chunk intervals are reset to resync chunk data across all neighborhoods as the final step to bring the localstore release to a completely stable state.

The feeds API now features a start index hint to optimize the retrieval loop of the latest feed.

For questions, comments, and feedback, reach out on Discord.

Bug fixes

• Fixed a mutex bug in the localstore migration step. ( #4267)
• Fixed a light node bug where a panic is avoided when querying ReserveLastBinIDs on ultra-light node with no reserve. ( #4271)
• Fixed a localstore leak where sharky slots of evicted chunks were not marked as free for future chunks to be stored ( #4276 )

Features

• Added a new Feed API parameter to speed up feed lookups. ( #4244 )

Performance

• Parallel uploads with the same batch are supported again. ( #4263 )
• Retrievals only utilize healthy and reachable peers. ( #4164 )

Hardening

• Sync intervals are reset to resync reserves. ( #4257 )

For a full PR rundown please consult the v1.17.3 diff.

With the last release we upgraded the mainnet to use the new storage layer. However, seeing the results from the redistribution game, we found some issues in it that were previously missed. The main problem we saw was that some of the pullsync indexes were being overwritten and hence, although the nodes have the chunks, they were not able to sync them with the network. This is now fixed, but we will need to do a migration to reconstruct the pullsync indexes on the nodes.

We also found that the cache implementation was over-complicated, which resulted in some bugs, so this has been simplified as well. There will be another smaller migration to update the cache indexes. Node operators are requested to update their setups as soon as possible. The migrations can take up to 20-30 mins depending on the setup, so please be prepared for this as well. We also strongly advise against concurrently upgrading nodes which are running on the same physical drive since it could drasticly increase migration time.

There is another simplification on the reserve eviction and postage batch expiration interactions. In the previous design, the salud health service would check whether reserve sizes (defined as the number of chunks in reserve at storage depth) were similar within some range of tolerance, and mark the node as unhealthy if the difference in reserve size surpassed that tolerance. In the current design, we don't expect the reserve sizes to match on neighbours as this size is now defined as the total number of reserve chunks on the node, regardless of depth. This can include chunks from lower storage depths which are not yet evicted. However, this should not stop the nodes from producing matching samples for the redistribution as we only consider the storage-depth and above chunks for the sample. To address this issue we have removed the reserve-size related checks in the salud service. Nodes will no longer mark themselves as unhealthy if their reserve sizes are off, and each node can now unreserve at its own speed.

New features

• Added isHealthy to redistributionstate (#4243).

Hardening

• Simplify batch expiration and unreserve interaction. (#4245)
• Simplify cache handling. (#4242)
• Remove reserve size self-health check. (#4255)

Bugfixes

• BinID handling fixed. (#4248)
• Synchronize chunkstore invasive operations. (#4253)

For a full PR rundown please consult the v1.17.2 diff.

This is a patch release on top of the 1.17.0 release.

The main fix in this patch is the storage radius is bootstrapped manually using the current value from the mainnet, and not through any other mechanism.

Refer to the main release notes here.

Bee v1.17.0 is finally here. This release will be the most significant update to the bee client in a long time and will be a breaking release due to protocol and API changes. The Bee team has been working on the new storage layer for quite some time now. The new storage layer is the repayment of a massive technical debt for the team as the old local storage component was not in a good state and was no longer maintainable. Follow all the communication on our discord to know all the details!

The goal of this upgrade is not for any performance benefits at the moment, but to ensure all existing functionality is retained and the storage component is simplified and hence improving maintainability.

The API changes are mostly consolidations and minor changes to the response types. Users are advised to go through the API docs and update their libraries/tooling accordingly.

As mentioned earlier, this will be a breaking release on the protocols-side as well. Pullsync, hive, retrieval, and handshake versions will be bumped. As a result, all node operators are requested to update their nodes to the latest release as soon as possible. The nodes will also do a full resync of the postage stamp contract events to properly flush out all expired content from their reserves once they upgrade to the new storage layer.

The upgrade, when first deployed, will begin to migrate data from the old localstore to the new and in the best case should finish in a few minutes but may take longer on slower devices. Because data migration is involved, it is higly advised to back up everything before the upgrade.

Warning for multiple node operators: the upgrade should be done by one node at a time for nodes that share the same physical disk as the initial migration from the old localstore to the new may take up to 30 minutes on older/slower hardware.

New features

• Support multiple stamps on chunks. Chunks uploaded with different stamps will be tracked separately in the network. This means the chunk will only be removed from the network once all the stamps associated with it are gone.
• Stewardship PUT endpoint now allows users to configure a stamp to use for re-upload. This would allow users to re-upload their pinned content with new stamps if the old one was expired.

Breaking changes

• /tags endpoint is now consolidated on the main API. The debug API tags endpoint is deprecated. Also, the tag response fields have changed.
• DELETE /chunk endpoint is also deprecated. This endpoint was mainly present for testing and will no longer be supported.

Hardening

• Pinning data is now managed separately. As a result the pinned and reserve content will no longer affect each other.
• Self-health check based on reserve size of neighborhood. (#4110)
• Synchronize interval state storing. (#4115)
• Prune out-of-depth oversaturated bins based on peer health. (#4092)
• Cleanup mutex handling in batchstore and reserve. (#4159)

Bugfixes

• Unreserve batch incorrectly unpins chunks. Also other issues related to pinning. (#3037, #3224, #3302, #3514)
• Reduce light-node disk usage. (#2990)
• Reuploading content without increasing bucket collision counters. (#2851)
• Upload related memory leak. (#4009)

For a full PR rundown please consult the v1.17.0 diff.

This one is a patch release to address a bug found in the new salud service.

Bug fixes

• Fixed a panic in the salud service when the response to the status request is terminated early by the peer. ( #4077)

v1.16.0

The bee team is excited to announce v1.16.0! 🎉

The team has been busy researching and testing ways to help swarm remain a reliable and healthy network, and to that end, we are happy to announce a brand new health service, salud.

With salud, nodes will periodically perform certain health checks on it's connected peers with data acquried from the status protocol. The checks as of this release are based on the duration of response to the status protocol msg, number of connected peers, the storage radius, and total batch commitment as computed by each peer.

For duration and number of connected peers, each peer must be within the 80th percentile to be deemed as healthy. Radius and batch commitment are measured in terms of the most common values as reported by each connected peer. Measurements are created anew for each peridoic health check.

A self check is also in place where if the node's own storage radius does not match with the rest of the networks', the node won't participate in the schelling game.

With this release, only the pushsync protocol utilizes the filtering of peers for requests based on the status of health.

For questions, comments, and feedback, reach out on discord.

Breaking changes

• The status protocol now includes the reachability of a peer and the total batch commitment of the network reported by each peer (#4054, #4042)

Bug fixes

• Fixed the last played round field in the redistribution state endpoint to report the corrent round instead of the previous round. ( #4062)

Hardening

• Loosed the requirements of the pushsync protocol so that in the case a neighborhood is down or does not exist, the chunk is pushed as close as possible to it's neighborhood. (#4064)
• Increased the wake up interval for the depthmonitor from 5 to 15 minutes. (#4068)

For a full PR rundown please consult the v1.16.0 diff.

v1.15.0

The bee team is excited to announce v1.15.0!

With this release, we introduce a new pushsync feature to improve chunk syncing and replication in the network during uploading. Peers that forward chunk into the neighborhood will fire multiple requests to target multiple storer nodes. Forwarding will also terminate at the first peer within the neighborhood and will no longer be forwarded to the most closest peer within the network.

As a result of the change in protocol logic, the protocol version has been bumped so it's important that you upgrage your nodes to the latest version.

We've added two new fields to the status protocol response: total amount from the chainstate and mode of operation of the peer (light or full mode).

For questions, comments, and feedback, reach out on discord.

Breaking changes

• Pushync forwarding nodes one hop from the neighborhood multiplexes the chunk to multiple storer nodes in the neighborhood. Forwarding terminates at the first neighbor. (#4008)

Hardening

• Blocklist endpoint now correctly display the mode and reason of the blocklisting of the peer. (#4015)
• Storage radius decreasing based on reserve size increased back to 50% threshold. (#4013)
• Chainstate total amount and peer mode added to the status protocol. (#4002 #4010)
• Connecting light nodes are not rejected based on bin size. (#4018)

Perfomance Improvements

• Initial historical syncing is limited to reduce workload. (#4000)
• Peer skiplist mechnism in retrieval and pushsync protocols now prune peers in a much more effiecient way. (#4001)

For a full PR rundown please consult the v1.15.0 diff.

v1.14.1

This one is a patch release to fix the innacurate the number of reported active nodes in the swarm scanner.

With this change, light nodes attempting to connect to a peer with a full topology bin won't get rejected.

For questions, comments, and feedback, reach out on discord.