stacks-core

Changed

• • The /v2/neighbors endpoint now reports a node's bootstrap peers, so other
    nodes can find high-quality nodes to boot from (#3401)
• • If there are two or more Stacks chain tips that are tied for the canonical
    tip, the node deterministically chooses one independent of the arrival order
    (#3419).
• • If Stacks blocks for a different fork arrive out-of-order and, in doing so,
    constitute a better fork than the fork the node considers canonical, the node
    will update the canonical Stacks tip pointer in the sortition DB before
    processing the next sortition (#3419).

Fixed

• • The node keychain no longer maintains any internal state, but instead derives
    keys based on the chain tip the miner is building off of. This prevents the
    node from accidentally producing an invalid block that reuses a microblock
    public key hash (#3387).
• • If a node mines an invalid block for some reason, it will no longer stall
    forever. Instead, it will detect that its last-mined block is not the chain
    tip, and resume mining (#3406).

Changed

• The new minimum Rust version is 1.61
• The act of walking the mempool will now cache address nonces in RAM and to a
  temporary mempool table used for the purpose, instead of unconditionally
  querying them from the chainstate MARF. This builds upon improvements to mempool
  goodput over 2.05.0.4.0 (#3337).
• The node and miner implementation has been refactored to remove write-lock
  contention that can arise when the node's chains-coordinator thread attempts to store and
  process newly-discovered (or newly-mined) blocks, and when the node's relayer
  thread attempts to mine a new block. In addition, the miner logic has been
  moved to a separate thread in order to avoid starving the relayer thread (which
  must handle block and transaction propagation, as well as block-processing).
  The refactored miner thread will be preemptively terminated and restarted
  by the arrival of new Stacks blocks or burnchain blocks, which further
  prevents the miner from holding open write-locks in the underlying
  chainstate databases when there is new chain data to discover (which would
  invalidate the miner's work anyway). (#3335).

Fixed

• Fixed pow documentation in Clarity (#3338).
• Backported unit tests that were omitted in the 2.05.0.3.0 release (#3348).
• Fixed event-emitter panic on PoisonMicroblock transaction (#3356)

Full Changelog: https://github.com/stacks-network/stacks-blockchain/compare/2.05.0.4.0...2.05.0.5.1

Changed

• The new minimum Rust version is 1.61
• The act of walking the mempool will now cache address nonces in RAM and to a
  temporary mempool table used for the purpose, instead of unconditionally
  querying them from the chainstate MARF. This builds upon improvements to mempool
  goodput over 2.05.0.4.0 (#3337).
• The node and miner implementation has been refactored to remove write-lock
  contention that can arise when the node's chains-coordinator thread attempts to store and
  process newly-discovered (or newly-mined) blocks, and when the node's relayer
  thread attempts to mine a new block. In addition, the miner logic has been
  moved to a separate thread in order to avoid starving the relayer thread (which
  must handle block and transaction propagation, as well as block-processing).
  The refactored miner thread will be preemptively terminated and restarted
  by the arrival of new Stacks blocks or burnchain blocks, which further
  prevents the miner from holding open write-locks in the underlying
  chainstate databases when there is new chain data to discover (which would
  invalidate the miner's work anyway). (#3335).

Fixed

• Fixed pow documentation in Clarity (#3338).
• Backported unit tests that were omitted in the 2.05.0.3.0 release (#3348).

[2.05.0.4.0]

Fixed

• Denormalize the mempool database so as to remove a LEFT JOIN from the SQL
  query for choosing transactions in order by estimated fee rate. This
  drastically speeds up mempool transaction iteration in the miner (#3314)

Upgrade instructions can be found here: https://gist.github.com/wileyj/379e6a4d11983e15449918732d75065a

Changelog:

[2.05.0.3.0] - 2022-8-31

Added

• Added prometheus output for "transactions in last block" (#3138).
• Added envrionement variable STACKS_LOG_FORMAT_TIME to set the time format
  stacks-node uses for logging. (#3219)
  Example: STACKS_LOG_FORMAT_TIME="%Y-%m-%d %H:%M:%S" cargo stacks-node
• Added mock-miner sample config (#3225)

Changed

• Updates to the logging of transaction events (#3139).
• Moved puppet-chain to ./contrib/tools directory and disabled compiling by default (#3200)

Fixed

• Make it so that a new peer private key in the config file will propagate to
  the peer database (#3165).
• Fixed default miner behavior regarding block assembly
  attempts. Previously, the miner would only attempt to assemble a
  larger block after their first attempt (by Bitcoin RBF) if new
  microblock or block data arrived. This changes the miner to always
  attempt a second block assembly (#3184).
• Fixed a bug in the node whereby the node would encounter a deadlock when
  processing attachment requests before the P2P thread had started (#3236).
• Fixed a bug in the P2P state machine whereby it would not absorb all transient errors
  from sockets, but instead propagate them to the outer caller. This would lead
  to a node crash in nodes connected to event observers, which expect the P2P
  state machine to only report fatal errors (#3228)
• Spawn the p2p thread before processing number of sortitions. Fixes issue (#3216) where sync from genesis paused (#3236)
• Drop well-formed "problematic" transactions that result in miner performance degradation (#3212)
• Ignore blocks that include problematic transactions

Fixed

• Fixed a security bug in the SPV client whereby the chain work was not being
  considered at all when determining the canonical Bitcoin fork. The SPV client
  now only accepts a new Bitcoin fork if it has a higher chain work than any other
  previously-seen chain (#3152).

[2.05.0.2.1]

Fixed

• Fixed a security bug in the SPV client whereby the chain work was not being
  considered at all when determining the canonical Bitcoin fork. The SPV client
  now only accepts a new Bitcoin fork if it has a higher chain work than any other
  previously-seen chain (#3152).

IMPORTANT! READ THIS FIRST

Please read the following WARNINGs in their entirety before upgrading.

WARNING: Please be aware that using this node on chainstate prior to this release will cause
the node to spend up to 30 minutes migrating the data to a new schema.
Depending on the storage medium, this may take even longer.

WARNING: This migration process cannot be interrupted. If it is, the chainstate
will be irrecovarably corrupted and require a sync from genesis.

WARNING: You will need at least 2x the disk space for the migration to work.
This is because a copy of the chainstate will be made in the same directory in
order to apply the new schema.

It is highly recommended that you back up your chainstate before running
this version of the software on it.

Changed

• The MARF implementation will now defer calculating the root hash of a new trie
  until the moment the trie is committed to disk. This avoids gratuitous hash
  calculations, and yields a performance improvement of anywhere between 10x and
  200x (#3041).
• The MARF implementation will now store tries to an external file for instances
  where the tries are expected to exceed the SQLite page size (namely, the
  Clarity database). This improves read performance by a factor of 10x to 14x
  (#3059).
• The MARF implementation may now cache trie nodes in RAM if directed to do so
  by an environment variable (#3042).
• Sortition processing performance has been improved by about an order of
  magnitude, by avoiding a slew of expensive database reads (#3045).
• Updated chains coordinator so that before a Stacks block or a burn block is processed,
  an event is sent through the event dispatcher. This fixes #3015.
• Expose a node's public key and public key hash160 (i.e. what appears in
  /v2/neighbors) via the /v2/info API endpoint (#3046)
• Reduced the default subsequent block attempt timeout from 180 seconds to 30
  seconds, based on benchmarking the new MARF performance data during a period
  of network congestion (#3098)
• The blockstack-core binary has been renamed to stacks-inspect.
  This binary provides CLI tools for chain and mempool inspection.

[2.05.0.2.0]

WARNING: Please be aware that using this node on chainstate prior to this release will cause
the node to spend up to 30 minutes migrating the data to a new schema.

Changed

• The MARF implementation will now defer calculating the root hash of a new trie
  until the moment the trie is committed to disk. This avoids gratuitous hash
  calculations, and yields a performance improvement of anywhere between 10x and
  200x (#3041).
• The MARF implementation will now store tries to an external file for instances
  where the tries are expected to exceed the SQLite page size (namely, the
  Clarity database). This improves read performance by a factor of 10x to 14x
  (#3059).
• The MARF implementation may now cache trie nodes in RAM if directed to do so
  by an environment variable (#3042).
• Sortition processing performance has been improved by about an order of
  magnitude, by avoiding a slew of expensive database reads (#3045). WARNING:
  applying this change to an existing chainstate directory will take a few
  minutes when the node starts up.
• Updated chains coordinator so that before a Stacks block or a burn block is processed,
  an event is sent through the event dispatcher. This fixes #3015.
• Expose a node's public key and public key hash160 (i.e. what appears in
  /v2/neighbors) via the /v2/info API endpoint (#3046)
• Reduced the default subsequent block attempt timeout from 180 seconds to 30
  seconds, based on benchmarking the new MARF performance data during a period
  of network congestion (#3098)

WARNING: Please be aware that using this node on chainstate prior to this release will cause
the node to spend up to 30 minutes migrating the data to a new schema.

Changed

• The MARF implementation will now defer calculating the root hash of a new trie
  until the moment the trie is committed to disk. This avoids gratuitous hash
  calculations, and yields a performance improvement of anywhere between 10x and
  200x (#3041).
• The MARF implementation will now store tries to an external file for instances
  where the tries are expected to exceed the SQLite page size (namely, the
  Clarity database). This improves read performance by a factor of 10x to 14x
  (#3059).
• The MARF implementation may now cache trie nodes in RAM if directed to do so
  by an environment variable (#3042).
• Sortition processing performance has been improved by about an order of
  magnitude, by avoiding a slew of expensive database reads (#3045). WARNING:
  applying this change to an existing chainstate directory will take a few
  minutes when the node starts up.
• Updated chains coordinator so that before a Stacks block or a burn block is processed,
  an event is sent through the event dispatcher. This fixes #3015.
• Expose a node's public key and public key hash160 (i.e. what appears in
  /v2/neighbors) via the /v2/info API endpoint (#3046)
• Reduced the default subsequent block attempt timeout from 180 seconds to 30
  seconds, based on benchmarking the new MARF performance data during a period
  of network congestion (#3098)

This software update is a point-release with a large set of changes. This release's chainstate directory is compatible with chainstate directories from 2.05.0.0.0.

Added

• A new fee estimator intended to produce fewer over-estimates, by having less
  sensitivity to outliers. Its characteristic features are: 1) use a window to
  forget past estimates instead of exponential averaging, 2) use weighted
  percentiles, so that bigger transactions influence the estimates more, 3)
  assess empty space in blocks as having paid the "minimum fee", so that empty
  space is accounted for, 4) use random "fuzz" so that in busy times the fees can
  change dynamically. (#2972)
• Implements anti-entropy protocol for querying transactions from other
  nodes' mempools. Before, nodes wouldn't sync mempool contents with one another.
  (#2884)
• Structured logging in the mining code paths. This will shine light
  on what happens to transactions (successfully added, skipped or errored) that the
  miner considers while buildings blocks. (#2975)
• Added the mined microblock event, which includes information on transaction
  events that occurred in the course of mining (will provide insight
  on whether a transaction was successfully added to the block,
  skipped, or had a processing error). (#2975)
• For v2 endpoints, can now specify the tip parameter to latest. If
  tip=latest, the node will try to run the query off of the latest tip. (#2778)
• Adds the /v2/headers endpoint, which returns a sequence of SIP-003-encoded
  block headers and consensus hashes (see the ExtendedStacksHeader struct that
  this PR adds to represent this data). (#2862)
• Adds the /v2/data_var endpoint, which returns a contract's data variable
  value and a MARF proof of its existence. (#2862)
• Fixed a bug in the unconfirmed state processing logic that could lead to a
  denial of service (node crash) for nodes that mine microblocks (#2970)
• Added prometheus metric that tracks block fullness by logging the percentage of each
  cost dimension that is consumed in a given block (#3025).

Changed

• Updated the mined block event. It now includes information on transaction
  events that occurred in the course of mining (will provide insight
  on whether a transaction was successfully added to the block,
  skipped, or had a processing error). (#2975)
• Updated some of the logic in the block assembly for the miner and the follower
  to consolidate similar logic. Added functions setup_block and finish_block.
  (#2946)
• Makes the p2p state machine more reactive to newly-arrived
  BlocksAvailable and MicroblocksAvailable messages for block and microblock
  streams that this node does not have. If such messages arrive during an inventory
  sync, the p2p state machine will immediately transition from the inventory sync
  work state to the block downloader work state, and immediately proceed to fetch
  the available block or microblock stream. (#2862)
• Nodes will push recently-obtained blocks and microblock streams to outbound
  neighbors if their cached inventories indicate that they do not yet have them
  (#2986).
• Nodes will no longer perform full inventory scans on their peers, except
  during boot-up, in a bid to minimize block-download stalls (#2986).
• Nodes will process sortitions in parallel to downloading the Stacks blocks for
  a reward cycle, instead of doing these tasks sequentially (#2986).
• The node's runloop will coalesce and expire stale requests to mine blocks on
  top of parent blocks that are no longer the chain tip (#2969).
• Several database indexes have been updated to avoid table scans, which
  significantly improves most RPC endpoint speed and cuts node spin-up time in
  half (#2989, #3005).
• Fixed a rare denial-of-service bug whereby a node that processes a very deep
  burnchain reorg can get stuck, and be rendered unable to process further
  sortitions. This has never happened in production, but it can be replicated in
  tests (#2989).
• Updated what indices are created, and ensures that indices are created even
  after the database is initialized (#3029).

Fixed

• Updates the lookup key for contracts in the pessimistic cost estimator. Before, contracts
  published by different principals with the same name would have had the same
  key in the cost estimator. (#2984)
• Fixed a few prometheus metrics to be more accurate compared to /v2 endpoints
  when polling data (#2987)