chainlink

Added

• Error handling when arbitrum sequencer is not accessible
• Error handle for gnosis chiado for seen tx
• Gas limit estimation feature to EVM gas estimators. Introduced a new config EVM.GasEstimator.EstimateLimit to toggle this feature
• Report new heads as a telemetry to OTI
• Optimism Stack L1 Oracle to add support for Mantle
• Allow workflows to run without external registry configured
• Astar TerminallyUnderpriced error mapping
• Custom client error messages for Mantle to capture InsufficientEth and Fatal errors
• New health check that ensures RPC provides new finalized heads at least every NoNewFinalizedHeadsThreshold

Updated

• Sync feeds-manager wsrpc proto
• Add DB syncing for registry syncer
• Polygon configs to match PIP-35
• Simplify how token and gas prices are stored in the database - user upsert instead of insert/delete flow

v2.16.0 Changelog

Added

• Support for EIP-1559 transactions for Scroll

• Protocol-level support for preventing bid/ask variant violations in mercury

• Add “VerboseLogging” option to mercury. Off by default, can be enabled like so:

  [Mercury]
  VerboseLogging = true
  

Updated

• Dequeue minimum guaranteed upkeeps as a priority
• Rename the InBackupHealthReport to StartUpHealthReport and enable it for DB migrations as well. This will enable health report to be available during long start-up tasks (db backups and migrations).
• VRFV2Plus coordinator and wrapper split contracts between L1 and L2 chains
• Fix: Set LatestFinalizedBlock for finalized blocks saved by logpoller

v2.15.0 Changelog

Added

• Added Aptos Keystore to Core. This includes Aptos Key which uses ED25519, Keystore, Relevant tests
• zkSync L1 GasPrice calculation

Updated

• Refactored the BlockHistoryEstimator check to prevent excessively bumping transactions. Check no longer waits for CheckInclusionBlocks to pass before assessing an attempt.
• Fixed a bug that would use the oldest blocks in the cached history instead of the latest to perform gas estimations.
• Updated L1 gas price calculations for Optimism Ecotone and Fjord upgrades
• Fixed local finality violation caused by an RPC lagging behind on latest finalized block.
• Fixed unreachable code bug which could result in stuck txns

DB Update

• CCIP capability specs migration

v2.14.0 Changelog

Breaking Change

• Remove the xdai ChainType config option. Moving forward, only gnosis can be used.

Added

• Added a mechanism to validate forwarders for OCR2 and fallback to EOA if necessary
• Added an auto-purge feature to the EVM TXM that identifies terminally stuck transactions either through a chain specific method or heurisitic then purges them to unblock the nonce. Included 4 new toml configs under Transactions.AutoPurge to configure this new feature: Enabled, Threshold, MinAttempts, and DetectionApiUrl.
• Add option to include GasPriceSubunits pipeline to include gasPriceSubunits in median ocr2 transmission (only to be used with Starknet chain for now)
• Added config option HeadTracker.FinalityTagBypass to force HeadTracker to track blocks up to FinalityDepth even if FinalityTagsEnabled = true. This option is a temporary measure to address high CPU usage on chains with extremely large actual finality depth (gap between the current head and the latest finalized block).
• Added config option HeadTracker.MaxAllowedFinalityDepth maximum gap between current head to the latest finalized block that HeadTracker considers healthy.

Updated

• Fixed CPU usage issues caused by inefficiencies in HeadTracker.

HeadTracker's support of finality tags caused a drastic increase in the number of tracked blocks on the Arbitrum chain (from 50 to 12,000), which has led to a 30% increase in CPU usage.

The fix improves the data structure for tracking blocks and makes lookup more efficient. BenchmarkHeadTracker_Backfill shows 40x time reduction.

• Fixed metric description on mercury_transmit_queue_load

DB Update

• Improve handling of postgres connection settings and driver versions

v2.13.0 Changelog

Added

• Added a mechanism to validate forwarders for OCR2 and fallback to EOA if necessary

• Added a new CLI command, blocks find-lca, which finds the latest block that is available in both the database and on the chain for the specified chain. Added a new CLI command, node remove-blocks, which removes all blocks and logs greater than or equal to the specified block number.

• Re-enable abandoned transaction tracker

• Enable configurable client error regexes for error classification. New toml configuration options for [EVM.NodePool.Errors] to pass regexes on NonceTooLow, NonceTooHigh, ReplacementTransactionUnderpriced, LimitReached, TransactionAlreadyInMempool, TerminallyUnderpriced, InsufficientEth, TxFeeExceedsCap, L2FeeTooLow, L2FeeTooHigh, L2Full, TransactionAlreadyMined, Fatal, and ServiceUnavailable.

• Validate user email before asking for a password in the chainlink CLI.

• Add configurability to mercury transmitter

  [Mercury.Transmitter]
  TransmitQueueMaxSize = 10_000 # Default
  TransmitTimeout = "5s" # Default
  

Updated

• core/chains/evm/logpoller: Stricter finality checks in LogPoller, to be more robust during rpc failover events
• Reducing the scope of 0233 migration to include only 5th word index which is required for CCIP
• Fixed CPU usage issues caused by inefficiencies in HeadTracker. HeadTracker's support of finality tags caused a drastic increase in the number of tracked blocks on the Arbitrum chain (from 50 to 12,000), which has led to a 30% increase in CPU usage. The fix improves the data structure for tracking blocks and makes lookup more efficient. BenchmarkHeadTracker_Backfill shows 40x time reduction.
• Validate support for postgresql-client 16, and update docker image's bundled postgresql-client from 15 to 16.

Upcoming change in 2.13.0

• The xdai ChainType has been renamed to gnosis to match the chain's new name. The old value is still supported but has been deprecated and will be removed in v2.13.0.

v2.12.0 Changelog

Added

• Added a tx simulation feature to the chain client to enable testing for zk out-of-counter (OOC) errors
• Add the pool_rpc_node_highest_finalized_block metric that tracks the highest finalized block seen per RPC. If FinalityTagEnabled = true, a positive NodePool.FinalizedBlockPollInterval is needed to collect the metric. If the finality tag is not enabled, the metric is populated with a calculated latest finalized block based on the latest head and finality depth.

Updated

• Moved JuelsPerFeeCoinCacheDuration under the JuelsPerFeeCoinCache struct in config. Rename JuelsPerFeeCoinCacheDuration to updateInterval. Add stalenessAlertThreshold to JuelsPerFeeCoinCache config. The StalenessAlertThreshold config option has a default of 24 hours which means that it doesn't have to be set unless we want to override the duration after which a stale cache should start throwing errors.
• Updated config for Polygon zkEVM chains.
• HeadTracker now respects the FinalityTagEnabled config option. If the flag is enabled, HeadTracker backfills blocks up to the latest finalized block provided by the corresponding RPC call. To address potential misconfigurations, HistoryDepth is now calculated from the latest finalized block instead of the head. Note that consumers like TXM and LogPoller do not fully use the Finality Tag yet.
• Change LimitTransfer gasLimit type from uint32 to uint64

Upcoming change in v2.13.0

• The xdai ChainType has been renamed to gnosis to match the chain's new name. The old value is still supported but has been deprecated and will be removed in v2.13.0.

v2.11.0 Changelog

View the changelog at contracts/CHANGELOG.md

Added

• Gas bumping logic to the SuggestedPriceEstimator. The bumping mechanism for this estimator refetches the price from the RPC and adds a buffer on top using the greater of BumpPercent and BumpMin.
• Added a new configuration field named NodeIsSyncingEnabled for EVM.NodePool that will check on every reconnection to an RPC if it's syncing and should not be transitioned to Alive state. Disabled by default.
• Add preliminary support for "llo" job type (Data Streams V1)
• Add LogPrunePageSize parameter to the EVM configuration. This parameter controls the number of logs removed during prune phase in LogPoller. Default value is 0, which deletes all logs at once - exactly how it used to work, so it doesn't require any changes on the product's side.
• Add Juels Fee Per Coin data source caching for OCR2 Feeds. Cache is time based and is turned on by default with default cache refresh of 5 minutes. Cache can be configured through pluginconfig using "juelsPerFeeCoinCacheDuration" and "juelsPerFeeCoinCacheDisabled" tags. Duration tag accepts values between "30s" and "20m" with default of "0s" that is overridden on cache startup to 5 minutes.
• Add rebalancer support for feeds manager ocr2 plugins.

Fixed

• P2P.V2 is required in configuration when either OCR or OCR2 are enabled. The node will fail to boot if P2P.V2 is not enabled.
• Removed unnecessary gas price warnings in gas estimators when EIP-1559 mode is enabled.

Changed

• Minimum required version of Postgres is now >= 12. Postgres 11 was EOL'd in November 2023. Added a new version check that will prevent Chainlink from running on EOL'd Postgres. If you are running Postgres <= 11 you should upgrade to the latest version. The check can be forcibly overridden by setting SKIP_PG_VERSION_CHECK=true.
• Updated the LimitDefault and LimitMax configs types to uint64

Changed

• eth_call RPC requests are now sent with both input and data fields to increase compatibility with servers that recognize only one.
• GasEstimator will now include Type 0x3 (Blob) transactions in the gas calculations to estimate it more accurately.

Added

• chainlink health CLI command and HTML /health endpoint, to provide human-readable views of the underlying JSON health data.
• New job type stream to represent streamspecs. This job type is not yet used anywhere but will be required for Data Streams V1.
• Environment variables CL_MEDIAN_ENV, CL_SOLANA_ENV, and CL_STARKNET_ENV for setting environment variables in LOOP Plugins with an .env file.

  echo "Foo=Bar" >> median.env
  echo "Baz=Val" >> median.env  
  CL_MEDIAN_ENV="median.env"
  

Fixed

• Fixed the encoding used for transactions when resending in batches

Removed

• P2P.V1 is no longer supported and must not be set in TOML configuration in order to boot. Use P2P.V2 instead. If you are using both, V1 can simply be removed.
• Removed TelemetryIngress.URL and TelemetryIngress.ServerPubKey from TOML configuration, these fields are replaced by [[TelemetryIngress.Endpoints]]:

  [[TelemetryIngress.Endpoints]]
  Network = '...' # e.g. EVM. Solana, Starknet, Cosmos
  ChainID = '...' # e.g. 1, 5, devnet, mainnet-beta
  URL = '...'
  ServerPubKey = '...'

Added

• Added distributed tracing in the OpenTelemetry trace format to the node, currently focused at the LOOPP Plugin development effort. This includes a new set of Tracing TOML configurations. The default for collecting traces is off - you must explicitly enable traces and setup a valid OpenTelemetry collector. Refer to .github/tracing/README.md for more details.
• Added a new, optional WebServer authentication option that supports LDAP as a user identity provider. This enables user login access and user roles to be managed and provisioned via a centralized remote server that supports the LDAP protocol, which can be helpful when running multiple nodes. See the documentation for more information and config setup instructions. There is a new [WebServer].AuthenticationMethod config option, when set to ldap requires the new [WebServer.LDAP] config section to be defined, see the reference docs/core.toml.
• New prom metrics for mercury transmit queue:
  mercury_transmit_queue_delete_error_count
mercury_transmit_queue_insert_error_count
mercury_transmit_queue_push_error_count
  Nops should consider alerting on these.
• Mercury now implements a local cache for fetching prices for fees, which ought to reduce latency and load on the mercury server, as well as increasing performance. It is enabled by default and can be configured with the following new config variables:

  [Mercury]
  
  # Mercury.Cache controls settings for the price retrieval cache querying a mercury server
  [Mercury.Cache]
  # LatestReportTTL controls how "stale" we will allow a price to be e.g. if
  # set to 1s, a new price will always be fetched if the last result was
  # from 1 second ago or older.
  # 
  # Another way of looking at it is such: the cache will _never_ return a
  # price that was queried from now-LatestReportTTL or before.
  # 
  # Setting to zero disables caching entirely.
  LatestReportTTL = "1s" # Default
  # MaxStaleAge is that maximum amount of time that a value can be stale
  # before it is deleted from the cache (a form of garbage collection).
  # 
  # This should generally be set to something much larger than
  # LatestReportTTL. Setting to zero disables garbage collection.
  MaxStaleAge = "1h" # Default
  # LatestReportDeadline controls how long to wait for a response from the
  # mercury server before retrying. Setting this to zero will wait indefinitely.
  LatestReportDeadline = "5s" # Default
  

• New prom metrics for the mercury cache:
  mercury_cache_fetch_failure_count
mercury_cache_hit_count
mercury_cache_wait_count
mercury_cache_miss_count
• Added new EVM.OCR TOML config fields DeltaCOverride and DeltaCJitterOverride for overriding the config DeltaC.
• Mercury v0.2 has improved consensus around current block that uses the most recent 5 blocks instead of only the latest one
• Two new prom metrics for mercury, nops should consider adding alerting on these:
  • mercury_insufficient_blocks_count
  • mercury_zero_blocks_count

Changed

• PromReporter no longer directly reads txm related status from the db, and instead uses the txStore API.
• L2Suggested mode is now called SuggestedPrice
• Console logs will now escape (non-whitespace) control characters
• Following EVM Pool metrics were renamed:
  • evm_pool_rpc_node_states → multi_node_states
  • evm_pool_rpc_node_num_transitions_to_alive → pool_rpc_node_num_transitions_to_alive
  • evm_pool_rpc_node_num_transitions_to_in_sync → pool_rpc_node_num_transitions_to_in_sync
  • evm_pool_rpc_node_num_transitions_to_out_of_sync → pool_rpc_node_num_transitions_to_out_of_sync
  • evm_pool_rpc_node_num_transitions_to_unreachable → pool_rpc_node_num_transitions_to_unreachable
  • evm_pool_rpc_node_num_transitions_to_invalid_chain_id → pool_rpc_node_num_transitions_to_invalid_chain_id
  • evm_pool_rpc_node_num_transitions_to_unusable → pool_rpc_node_num_transitions_to_unusable
  • evm_pool_rpc_node_highest_seen_block → pool_rpc_node_highest_seen_block
  • evm_pool_rpc_node_num_seen_blocks → pool_rpc_node_num_seen_blocks
  • evm_pool_rpc_node_polls_total → pool_rpc_node_polls_total
  • evm_pool_rpc_node_polls_failed → pool_rpc_node_polls_failed
  • evm_pool_rpc_node_polls_success → pool_rpc_node_polls_success

Removed

• Removed Optimism2 as a supported gas estimator mode

Fixed

• Corrected Ethereum Sepolia LinkContractAddress to 0x779877A7B0D9E8603169DdbD7836e478b4624789
• Fixed a bug that caused the Telemetry Manager to report incorrect health

Upcoming Required Configuration Changes

Starting in v2.9.0:

• TelemetryIngress.URL and TelemetryIngress.ServerPubKey will no longer be allowed. Any TOML configuration that sets this fields will prevent the node from booting. These fields will be replaced by [[TelemetryIngress.Endpoints]]
• P2P.V1 will no longer be supported and must not be set in TOML configuration in order to boot. Use P2P.V2 instead. If you are using both, V1 can simply be removed.

Fixed

• Fixed a bug that caused nodes without OCR or OCR2 enabled to fail config validation if P2P.V2 was not explicitly disabled. With this fix, NOPs will not have to make changes to their config.

Fixed

• Fixed a bug that causes the node to shutdown if all configured RPC's are unreachable during startup.

Added

• Added new configuration field named LeaseDuration for EVM.NodePool that will periodically check if internal subscriptions are connected to the "best" (as defined by the SelectionMode) node and switch to it if necessary. Setting this value to 0s will disable this feature.
• Added multichain telemetry support. Each network/chainID pair must be configured using the new fields:

[[TelemetryIngress.Endpoints]]
Network = '...' # e.g. EVM. Solana, Starknet, Cosmos
ChainID = '...' # e.g. 1, 5, devnet, mainnet-beta
URL = '...'
ServerPubKey = '...'

These will eventually replace TelemetryIngress.URL and TelemetryIngress.ServerPubKey. Setting TelemetryIngress.URL and TelemetryIngress.ServerPubKey alongside [[TelemetryIngress.Endpoints]] will prevent the node from booting. Only one way of configuring telemetry endpoints is supported.

• Added bridge_name label to pipeline_tasks_total_finished prometheus metric. This should make it easier to see directly what bridge was failing out from the CL NODE perspective.

• LogPoller will now use finality tags to dynamically determine finality on evm chains if UseFinalityTags=true, rather than the fixed FinalityDepth specified in toml config

Changed

• P2P.V1 is now disabled (Enabled = false) by default. It must be explicitly enabled with true to be used. However, it is deprecated and will be removed in the future.
• P2P.V2 is now enabled (Enabled = true) by default.

Upcoming Required Configuration Changes

Starting in v2.9.0:

• TelemetryIngress.URL and TelemetryIngress.ServerPubKey will no longer be allowed. Any TOML configuration that sets this fields will prevent the node from booting. These fields will be replaced by [[TelemetryIngress.Endpoints]]
• P2P.V1 will no longer be supported and must not be set in TOML configuration in order to boot. Use P2P.V2 instead. If you are using both, V1 can simply be removed.

Removed

• Removed the ability to set a next nonce value for an address through CLI

Added

• Simple password use in production builds is now disallowed - nodes with this configuration will not boot and will not pass config validation.
• Helper migrations function for injecting env vars into goose migrations. This was done to inject chainID into evm chain id not null in specs migrations.
• OCR2 jobs now support querying the state contract for configurations if it has been deployed. This can help on chains such as BSC which "manage" state bloat by arbitrarily deleting logs older than a certain date. In this case, if logs are missing we will query the contract directly and retrieve the latest config from chain state. Chainlink will perform no extra RPC calls unless the job spec has this feature explicitly enabled. On chains that require this, nops may see an increase in RPC calls. This can be enabled for OCR2 jobs by specifying ConfigContractAddress in the relay config TOML.

Removed

• Removed support for sending telemetry to the deprecated Explorer service. All nodes will have to remove Explorer related keys from TOML configuration and env vars.
• Removed default evmChainID logic where evmChainID was implicitly injected into the jobspecs based on node EVM chainID toml configuration. All newly created jobs(that have evmChainID field) will have to explicitly define evmChainID in the jobspec.
• Removed keyset migration that migrated v1 keys to v2 keys. All keys should've been migrated by now, and we don't permit creation of new v1 keys anymore

All nodes will have to remove the following secret configurations:

• Explorer.AccessKey
• Explorer.Secret

All nodes will have to remove the following configuration field: ExplorerURL

Fixed

• Unauthenticated users executing CLI commands previously generated a confusing error log, which is now removed:
  [ERROR] Error in transaction, rolling back: session missing or expired, please login again pg/transaction.go:118
• Fixed a bug that was preventing job runs to be displayed when the job chainID was disabled.
• chainlink txs evm create returns a transaction hash for the attempted transaction in the CLI. Previously only the sender, recipient and unstarted state were returned.
• Fixed a bug where evmChainId is requested instead of id or evm-chain-id in CLI error verbatim
• Fixed a bug that would cause the node to shut down while performing backup
• Fixed health checker to include more services in the prometheus health metric and HTTP /health endpoint

Upcoming Required Configuration Change

• Starting in 2.6.0, chainlink nodes will no longer allow insecure configuration for production builds. Any TOML configuration that sets the following line will fail validation checks in node start or node validate:

AllowSimplePasswords=true

• To migrate on production builds, update the database password set in Database.URL to be 16 - 50 characters without leading or trailing whitespace. URI parsing rules apply to the chosen password - refer to RFC 3986 for special character escape rules.

Added

• Various Functions improvements

Fixed

• Updated v2/keys/evm and v2/keys/eth routes to return 400 and 404 status codes where appropriate. Previously 500s were returned when requested resources were not found or client requests could not be parsed.
• Fixed withdrawing ETH from CL node for EIP1559 enabled chains. Previously would error out unless validation was overridden with allowHigherAmounts.

Added

• Added the ability to specify and merge fields from multiple secrets files. Overrides of fields and keys are not allowed.

Upcoming Required Configuration Change

• Starting in 2.6.0, chainlink nodes will no longer allow insecure configuration for production builds. Any TOML configuration that sets the following line will fail validation checks in node start or node validate:

AllowSimplePasswords=true

• To migrate on production builds, update the database password set in Database.URL to be 16 - 50 characters without leading or trailing whitespace. URI parsing rules apply to the chosen password - refer to RFC 3986 for special character escape rules.

Added

• Add a new field called Order (range from 1 to 100) to EVM.Nodes that is used for the PriorityLevel node selector and also as a tie-breaker for HighestHead and TotalDifficulty. Order levels are considered in ascending order. If not defined it will default to Order = 100 (last level).
• Added new node selection mode called PriorityLevel for EVM, it is a tiered round-robin in ascending order of theOrder field. Example:

[EVM.NodePool]
SelectionMode = 'PriorityLevel'

[[EVM.Nodes]]
Name = '...'
WSURL = '...'
HTTPURL = '...'
Order = 5 

• The config keys WebServer.StartTimeout and WebServer.HTTPMaxSize. These keys respectively set a timeout for the node server to
  start and set the max request size for HTTP requests. Previously these attributes were set by
  JobPipeline.DefaultHTTPLimit/JobPipeline.DefaultHTTPTimeout. To migrate to these new fields, set their values to be identical to
  JobPipeline.DefaultHTTPLimit/JobPipeline.DefaultHTTPTimeout.

• Low latency oracle jobs now support in-protocol block range guarantees. This
  is necessary in order to produce reports with block number ranges that do not
  overlap. It can now be guaranteed at the protocol level, so we can use local
  state instead of relying on an unreliable round-trip to the Mercury server.

• New settings Evm.GasEstimator.LimitJobType.OCR2, OCR2.DefaultTransactionQueueDepth, OCR2.SimulateTransactions for OCR2
  jobs. These replace the settings Evm.GasEstimator.LimitJobType.OCR, OCR.DefaultTransactionQueueDepth, and OCR.SimulateTransactions
  for OCR2.

• Add new config parameter to OCR and OCR2 named TraceLogging that enables trace logging of OCR and OCR2 jobs, previously this behavior was controlled from the P2P.TraceLogging parameter. To maintain the same behavior set OCR.TraceLogging and OCR2.TraceLogging to the same value P2P.TraceLogging was set.

• Add two new config parameters WebServer.ListenIP and WebServer.TLS.ListenIP which allows binding Chainlink HTTP/HTTPS servers to a particular IP. The default is '0.0.0.0' which listens to all IP addresses (same behavior as before). Set to '127.0.0.1' to only allow connections from the local machine (this can be handy for local development).

• Add several new metrics for mercury feeds, related to WSRPC connections:

  • mercury_transmit_timeout_count
  • mercury_dial_count
  • mercury_dial_success_count
  • mercury_dial_error_count
  • mercury_connection_reset_count

Node operators may wish to add alerting based around these metrics.

Fixed

• Fixed a bug in the nodes xxx list command that caused results to not be displayed correctly

Changed

• Assumption violations for MaxFeePerGas >= BaseFeePerGas and MaxFeePerGas >= MaxPriorityFeePerGas in EIP-1559 effective gas price calculation will now use a gas price if specified
• Config validation now enforces protection against duplicate chain ids and node fields per provided TOML file. Duplicates accross multiple configuration files are still valid. If you have specified duplicate chain ids or nodes in a given configuration file, this change will error out of all node subcommands.
• Restricted scope of the Evm.GasEstimator.LimitJobType.OCR, OCR.DefaultTransactionQueueDepth, and OCR.SimulateTransactions settings so they
  apply only to OCR. Previously these settings would apply to OCR2 as well as OCR. You must use the OCR2 equivalents added above if you
  want your settings to apply to OCR2.

Removed

• Legacy chain types Optimism and Optimism2. OptimismBedrock is now used to handle Optimism's special cases.
• Optimism Kovan configurations along with legacy error messages.

Added

• Experimental support of runtime process isolation for Solana data feeds. Requires plugin binaries to be installed and
  configured via the env vars CL_SOLANA_CMD and CL_MEDIAN_CMD. See plugins/README.md.

Fixed

• Fixed a bug which made it impossible to re-send the same transaction after abandoning it while manually changing the nonce.

Changed

• Set default for EVM.GasEstimator.BumpTxDepth to EVM.Transactions.MaxInFlight.
• Bumped batch size defaults for EVM specific configuration. If you are overriding any of these fields in your local config, please consider if it is necessary:
  • LogBackfillBatchSize = 1000
  • RPCDefaultBatchSize = 250
  • GasEstimator.BatchSize = 25
• Dropped support for Development Mode configuration. CL_DEV is now ignored on production builds.
• Updated Docker image's PostgreSQL client (used for backups) to v15 in order to support PostgreSQL v15 servers.

Changed

• Database commands chainlink db ... validate TOML configuration and secrets before executing. This change of behavior will report errors if any Database-specific configuration is invalid.