Added

• Chainlink header images to the following README.md files: root, core,
  evm-contracts, and evm-test-helpers.
• Database migrations: new log_consumptions records will contain the number of the associated block.
  This migration will allow future version of chainlink to automatically clean up unneeded log_consumption records.
  This migration should execute very fast.

Fixed

Previously when the node was overloaded with heads there was a minor possibility it could get backed up with a very large head queue, and become unstable. Now, we drop heads instead in this case and noisily emit an error. This means the node should more gracefully handle overload conditions, although this is still dangerous and node operators should deal with it immediately to avoid missing jobs.

A new environment variable is introduced to configure this, called ETH_HEAD_TRACKER_MAX_BUFFER_SIZE. It is recommended to leave this set to the default of "3".

A new prometheus metric is also introduced to track dropped heads, called head_tracker_num_heads_dropped. You may wish to set an alert on a rule such as increase(chainlink_dropped_heads[5m]) > 0.

Changed

• Fix for gas bumper
• Fix for broadcast-transactions function

Changed

Performance improvements when using BulletproofTxManager.

Changed

Default max gas increased to 5000 Gwei

Fixed

Added a workaround for Infura users who are seeing "error getting balance: header not found".

This behaviour is due to Infura announcing it has a block, but when we request our balance in this block, the eth node doesn't have the block in memory. The workaround is to add a configurable lag time on balance update requests. The default is set to 1 but this is configurable via a new environment variable ETH_BALANCE_MONITOR_BLOCK_DELAY.

Added

• Job specs now support pinning to multiple keys using the new fromAddresses field in the ethtx task spec.

Changed

• Using fromAddress in ethtx task specs has been deprecated. Please use fromAddresses instead.

Breaking changes

• Support for RunLogTopic0original and RunLogTopic20190123withFullfillmentParams logs has been dropped. This should not affect any users since these logs predate Chainlink's mainnet launch and have never been used on mainnet.

IMPORTANT: The selection mechanism for keys has changed. When an ethtx task spec is not pinned to a particular key by defining fromAddress or fromAddresses, the node will now cycle through all available keys in round robin fashion. This is a change from the previous behaviour where nodes would only pick the earliest created key.

This is done to allow increases in throughput when a node operator has multiple whitelisted addresses for their oracle.

If your node has multiple keys, you will need to take one of the three following actions:

1. Make sure all keys are valid for all job specs
2. Pin job specs to a valid subset of key(s) using fromAddresses
3. Delete the key(s) you don't want to use

If your node only has one key, no action is required.

Fixed

• Incorrect sequence on keys table in some edge cases

Added

• Added a check on sensitive file ownership that gives a warning if certain files are not owned by the user running chainlink
• Added mechanism to asynchronously communicate when a job spec has an ethereum interaction error (or any async error) with a UI screen
• Gas Bumper now bumps based on the current gas price instead of the gas price of the original transaction

Fixed

• Support for multiple node addresses

Added

• ethtx tasks now support a new parameter, minRequiredOutgoingConfirmations which allows you to tune how many confirmations are required before moving on from an ethtx task on a per task basis (only works with BulletproofTxManager). If it is not supplied, the default of MIN_OUTGOING_CONFIRMATIONS is used (same as the old behaviour).

Changed

• HeadTracker now automatically backfills missing heads up to ETH_FINALITY_DEPTH

Breaking changes

• admin withdraw command has been removed. This was only ever useful to withdraw LINK if the Oracle contract was owned by the Chainlink node address. It is no longer recommended to have the Oracle owner be the chainlink node address.
• Fixed txs create to send the amount in Eth not in Wei (as per the documentation)

Added

This release contains a number of features aimed at improving the node's reliability when putting transactions on-chain.

• An experimental new transaction manager is introduced that delivers reliability improvements compared to the old one, especially when faced with difficult network conditions or spiking gas prices. It also reduces load on the database and makes fewer calls to the eth node compared to the old tx manager.
• Along with the new transaction manager is a local client command for manually controlling the node nonce - setnextnonce. This should never be necessary under normal operation and is included only for use in emergencies.
• ethtx tasks now support two new parameters (only available with BulletproofTxManager):
  • minRequiredOutgoingConfirmations - allows you to tune how many confirmations are required before moving on from an ethtx task on a per task basis. If not supplied, the default of MIN_OUTGOING_CONFIRMATIONS is used (same as the old behaviour).
  • fromAddress - enables multikey support by allowing you to peg a job spec to a particular ethereum address.
• New prometheus metrics for the head tracker:
  • head_tracker_heads_in_queue - The number of heads currently waiting to be executed. You can think of this as the 'load' on the head tracker. Should rarely or never be more than 0.
  • head_tracker_callback_execution_duration - How long it took to execute all callbacks. If the average of this exceeds the time between blocks, your node could lag behind and delay transactions.

Env var changes

• ENABLE_BULLETPROOF_TX_MANAGER - Experimental feature for transaction manager, do not enable!
• ETH_GAS_BUMP_PERCENT default value has been increased from 10% to 20%
• ETH_GAS_BUMP_THRESHOLD default value has been decreased from 12 to 3
• ETH_FINALITY_DEPTH specifies how deep protection should be against re-orgs. The default is 50. It only applies if BulletproofTxManager is enabled. It is not recommended to change this setting.
• ETH_HEAD_TRACKER_HISTORY_DEPTH specifies how many heads the head tracker should keep in the database. The default is 100. It is not recommended to change this setting.

Added

• The node now logs the eth client RPC calls
• More reliable Ethereum block header tracking
• Limit the amount of an HTTP response body that the node will read
• Make Aggregator contract interface viewable
• More resilient handling of chain reorganizations

Added

• The chainlink node can now be configured to backfill logs from n blocks after a connection to the ethereum client is reset. This value is specified with an environment variable BLOCK_BACKFILL_DEPTH.
• The chainlink node now sets file permissions on sensitive files on startup (tls, .api, .env, .password and secret)

Changed

• Solidity: Renamed the previous AggregatorInterface.sol to HistoricAggregatorInterface.sol. Users are encouraged to use the new methods introduced on the AggregatorInterface(getRoundData and latestRoundData), as they return metadata to indicate freshness of the data in a single cross-contract call.
• Solidity: Marked HistoricAggregatorInterface methods (latestAnswer, latestRound, latestTimestamp, getAnswer, getTimestamp) as deprecated on FluxAggregator, WhitelistedAggregator, AggregatorProxy, WhitelistedAggregatorProxy.

Fixed

• Fluxmonitor jobs now respect the minPayment field on job specs and won't poll if the contract does not have sufficient funding. This allows certain jobs to require a larger payment than MINIMUM_CONTRACT_PAYMENT.

Breaking changes

None

Features

• Fluxmonitor initiators may now optionally include an absoluteThreshold parameter. To trigger a new on-chain report, the absolute difference in the feed value must change by at least the absoluteThreshold value. If it is unspecified or zero, fluxmonitor behavior is unchanged.

Bugfixes

• Fix incorrect permissions on some files written by the node
• Prevent a case where duplicate ethereum keys could be added
• Improve robustness and reliability of ethtx transaction logic

Database Migrations

• Add created_at and updated_at to all tables allowing for better historical insights. This migration may take a minute or two on large databases.

Breaking changes

New cron jobs MUST now include time zone. For example if you want your jobs to run in UTC:
CRON_TZ=UTC * * * * *.

A full list can be found here: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones. You need to specify the TZ Database name.

Previously, jobs specified without a time zone would run in the server's native time zone, which in most cases is UTC but this was never guaranteed. This behaviour is now disabled for new jobs, because it is ambiguous.

Old jobs will continue to run AS IS and run on the servers timezone.

Bugfixes

• Fix crash in experimental gas updater when run on Kovan network

Database Migrations

There a number of database migrations included in this release as part of our ongoing effort to make the node even more reliable and stable, and build a firm foundation for future development.

If you have made custom edits to your database, the migration might fail. If this happens you will need to fix the offending data manually and run the migration again.

If you have very large tables, these migrations may take several minutes.

Chainlink 0.8.2 is a minor release

Features

• Several indexes and optimisations which result in improved database performance. You should see significantly reduced load on your DB running this release.
• A gas price updater is now built into core. This is designed to automatically increase the gas price in the event of network congestion, which should allow the node to more gracefully handle network congestion events such as the one we experienced on Thursday 12 March (Black Thursday). It is experimental and currently disabled by default. Tunable parameters are:
  • GAS_UPDATER_ENABLED - turns on the automatic gas updater if set to true
  • GAS_UPDATER_BLOCK_DELAY - number of blocks that the gas updater trails behind head
  • GAS_UPDATER_BLOCK_HISTORY_SIZE - number of past blocks to keep in memory
  • GAS_UPDATER_TRANSACTION_PERCENTILE - percentile gas price to choose. E.g. if the block history contains four transactions with gas prices [100, 200, 300, 400] then picking 25 for this number will give a value of 200. If the calculated gas price is higher than ETH_GAS_PRICE_DEFAULT then the higher price will be used as the base price for new transactions. Node operators are encouraged to reach out to us before experimenting with these settings.

Bugfixes

• Fix crash when attempting to save job spec with missing bridge
• The Delete Run section in Configuration section of the UI has been fixed and extended so the buttons properly remove runs and all associated data. Use this if you want to greatly reduce the size of your database. Don't forget to take a backup first.

Database Migrations

This release contains some database migrations that help with the performance improvements. We tested these migrations on our production node and it took less than 10 seconds. It may take longer if you have a very large database (millions of rows). These migrations run on application start, so be prepared for that.

Also note that as part of these migrations we clean up and remove some bad data. Be sure to take a full database backup before upgrading.

Chainlink 0.8.1 is a hotfix release. Any node operators on 0.8.0 are encourage to upgrade as soon as possible.

Bugfixes

• Fix an issue whereby fetching from an API that uses chunked transfer encoding would occasionally fail with context cancelled

Features

• Extend automatic HTTP retries to include server 500 errors

Chainlink 0.8.0 is a major release.

Breaking changes

• Support for SQLite has been dropped

Features

• More robust HTTP adapter which automatically retries requests in the event of certain types of API or network failure
• More robust retry for ethereum transactions which fail for an unknown reason
• More comprehensive prometheus metrics (eth balance, gas bumping)
• Warnings on sensitive file permissions

Bugfixes

• Multiply adapter now uses Decimal instead of Float for improved accuracy
• JSON adapter now returns large numbers as strings to avoid rounding errors in poorly behaved clients

Upgrades

• Upgrade many dependencies, notably Go 1.12 -> Go 1.13

Release 0.7.8 is a bugfix release that addresses several issues that can surface under very high network congestion:

• Fix certain cases where the node would not retry failed requests even though retrying would have caused the transaction to succeed
• Add new command ./chainlink local rebroadcast-transactions that allows node operators to manually resubmit pending transactions with a configured gas price
• Revamp gas bumping logic to be much more robust and reliable.

• Fix occasional panic under high load
• Fix a rare case where node could put 0 value on chain in certain conditions
• Make retry logic more robust in cases where nonce gets out of sync