chainlink

Added

• Added disk rotating logs. Chainlink will now always log to disk at debug level. The default output directory for debug logs is Chainlink's root directory (ROOT_DIR) but can be configured by setting LOG_FILE_DIR. This makes it easier for node operators to report useful debugging information to Chainlink's team, since all the debug logs are conveniently located in one directory. Regular logging to STDOUT still works as before and respects the LOG_LEVEL env var. If you want to log in disk at a particular level, you can pipe STDOUT to disk. This automatic debug-logs-to-disk feature is enabled by default, and will remain enabled as long as the LOG_FILE_MAX_SIZE ENV var is set to a value greater than zero. The amount of disk space required for this feature to work can be calculated with the following formula: LOG_FILE_MAX_SIZE * (LOG_FILE_MAX_BACKUPS + 1). If your disk doesn't have enough disk space, the logging will pause and the application will log Errors until space is available again. New environment variables related to this feature:
  • LOG_FILE_MAX_SIZE (default: 5120mb) - this env var allows you to override the log file's max size (in megabytes) before file rotation.
  • LOG_FILE_MAX_AGE (default: 0) - if LOG_FILE_MAX_SIZE is set, this env var allows you to override the log file's max age (in days) before file rotation. Keeping this config with the default value means not to remove old log files.
  • LOG_FILE_MAX_BACKUPS (default: 1) - if LOG_FILE_MAX_SIZE is set, this env var allows you to override the max amount of old log files to retain. Keeping this config with the default value means to retain 1 old log file at most (though LOG_FILE_MAX_AGE may still cause them to get deleted). If this is set to 0, the node will retain all old log files instead.
• Added support for the force flag on chainlink blocks replay. If set to true, already consumed logs that would otherwise be skipped will be rebroadcasted.
• Added version compatibility check when using CLI to login to a remote node. flag bypass-version-check skips this check.
• Interrim solution to set multiple nodes/chains from ENV. This is a temporary stand-in until configuration is overhauled and will be removed in future. Set as such: EVM_NODES='{...}' where the var is a JSON array containing the node specifications. This is not compatible with using any other way to specify node via env (e.g. ETH_URL etc). WARNING: Setting this environment variable will COMPLETELY ERASE your evm_nodes table on every boot and repopulate from the given data, nullifying any runtime modifications.

For example:

EVM_NODES='
[
	{
		"name": "primary_0_1",
		"evmChainId": "0",
		"wsUrl": "ws://test1.invalid",
		"sendOnly": false
	},
	{
		"name": "primary_0_2",
		"evmChainId": "0",
		"wsUrl": "ws://test2.invalid",
		"httpUrl": "https://test3.invalid",
		"sendOnly": false
	},
	{
		"name": "primary_1337_1",
		"evmChainId": "1337",
		"wsUrl": "ws://test4.invalid",
		"httpUrl": "http://test5.invalid",
		"sendOnly": false
	},
	{
		"name": "sendonly_1337_1",
		"evmChainId": "1337",
		"httpUrl": "http://test6.invalid",
		"sendOnly": true
	},
	{
		"name": "sendonly_0_1",
		"evmChainId": "0",
		"httpUrl": "http://test7.invalid",
		"sendOnly": true
	},
	{
		"name": "primary_42_1",
		"evmChainId": "42",
		"wsUrl": "ws://test8.invalid",
		"sendOnly": false
	},
	{
		"name": "sendonly_43_1",
		"evmChainId": "43",
		"httpUrl": "http://test9.invalid",
		"sendOnly": true
	}
]
'

Changed

• Changed default locking mode to "dual". Bugs in lease locking have been ironed out and this paves the way to making "lease" the default in future. It is recommended to set DATABASE_LOCKING_MODE=lease, default is set to "dual" only for backwards compatibility.
• It is now NOT allowed to have multiple evm RPC nodes specified with the same URL. If you see a migration error related to ERROR 0106_evm_node_uniqueness.sql: failed to run SQL migration that means you have multiple nodes specified with the same URL, and you must fix this before proceeding with the upgrade.
• EIP-1559 is now enabled by default on mainnet. To disable (go back to legacy mode) set EVM_EIP1559_DYNAMIC_FEES=false. The default settings should work well, but if you wish to tune your gas controls, see the documentation.

Note that EIP-1559 can be manually enabled on other chains by setting EVM_EIP1559_DYNAMIC_FEES=true but we only support it for official Ethereum mainnet and testnets. It is not recommended to enable this setting on Polygon since during our testing process we found that the EIP-1559 fee market appears to be broken on all Polygon chains and EIP-1559 transactions are actually less likely to get included than legacy transactions.

See issue: https://github.com/maticnetwork/bor/issues/347

Removed

• LOG_TO_DISK ENV var.

Added

• Added disk rotating logs. Chainlink will now always log to disk at debug level. The default output directory for debug logs is Chainlink's root directory (ROOT_DIR) but can be configured by setting LOG_FILE_DIR. This makes it easier for node operators to report useful debugging information to Chainlink's team, since all the debug logs are conveniently located in one directory. Regular logging to STDOUT still works as before and respects the LOG_LEVEL env var. If you want to log in disk at a particular level, you can pipe STDOUT to disk. This automatic debug-logs-to-disk feature is enabled by default, and will remain enabled as long as the LOG_FILE_MAX_SIZE ENV var is set to a value greater than zero. The amount of disk space required for this feature to work can be calculated with the following formula: LOG_FILE_MAX_SIZE * (LOG_FILE_MAX_BACKUPS + 1). If your disk doesn't have enough disk space, the logging will pause and the application will log Errors until space is available again.
• Added support for the force flag on chainlink blocks replay. If set to true, already consumed logs that would otherwise be skipped will be rebroadcasted.
• Added version compatibility check when using CLI to login to a remote node. flag bypass-version-check skips this check.
• Interrim solution to set multiple nodes/chains from ENV. This is a temporary stand-in until configuration is overhauled and will be removed in future. Set as such: EVM_NODES='{...}' where the var is a JSON array containing the node specifications. This is not compatible with using any other way to specify node via env (e.g. ETH_URL etc). WARNING: Setting this environment variable will COMPLETELY ERASE your evm_nodes table on every boot and repopulate from the given data, nullifying any runtime modifications.

For example:

EVM_NODES='
[
	{
		"name": "primary_0_1",
		"evmChainId": "0",
		"wsUrl": "ws://test1.invalid",
		"sendOnly": false
	},
	{
		"name": "primary_0_2",
		"evmChainId": "0",
		"wsUrl": "ws://test2.invalid",
		"httpUrl": "https://test3.invalid",
		"sendOnly": false
	},
	{
		"name": "primary_1337_1",
		"evmChainId": "1337",
		"wsUrl": "ws://test4.invalid",
		"httpUrl": "http://test5.invalid",
		"sendOnly": false
	},
	{
		"name": "sendonly_1337_1",
		"evmChainId": "1337",
		"httpUrl": "http://test6.invalid",
		"sendOnly": true
	},
	{
		"name": "sendonly_0_1",
		"evmChainId": "0",
		"httpUrl": "http://test7.invalid",
		"sendOnly": true
	},
	{
		"name": "primary_42_1",
		"evmChainId": "42",
		"wsUrl": "ws://test8.invalid",
		"sendOnly": false
	},
	{
		"name": "sendonly_43_1",
		"evmChainId": "43",
		"httpUrl": "http://test9.invalid",
		"sendOnly": true
	}
]
'

Changed

• Changed default locking mode to "dual". Bugs in lease locking have been ironed out and this paves the way to making "lease" the default in future. It is recommended to set DATABASE_LOCKING_MODE=lease, default is set to "dual" only for backwards compatibility.
• It is now NOT allowed to have multiple evm RPC nodes specified with the same URL. If you see a migration error related to ERROR 0106_evm_node_uniqueness.sql: failed to run SQL migration that means you have multiple nodes specified with the same URL, and you must fix this before proceeding with the upgrade.
• EIP-1559 is now enabled by default on mainnet. To disable (go back to legacy mode) set EVM_EIP1559_DYNAMIC_FEES=false. The default settings should work well, but if you wish to tune your gas controls, see the documentation.

Note that EIP-1559 can be manually enabled on other chains by setting EVM_EIP1559_DYNAMIC_FEES=true but we only support it for official Ethereum mainnet and testnets. It is not recommended to enable this setting on Polygon since during our testing process we found that the EIP-1559 fee market appears to be broken on all Polygon chains and EIP-1559 transactions are actually less likely to get included than legacy transactions.

See issue: https://github.com/maticnetwork/bor/issues/347

Added

• Added disk rotating logs. Chainlink will now always log to disk at debug level. The default output directory for debug logs is Chainlink's root directory (ROOT_DIR) but can be configured by setting LOG_FILE_DIR. This makes it easier for node operators to report useful debugging information to Chainlink's team, since all the debug logs are conveniently located in one directory. Regular logging to STDOUT still works as before and respects the LOG_LEVEL env var. If you want to log in disk at a particular level, you can pipe STDOUT to disk. This automatic debug-logs-to-disk feature is enabled by default, and will remain enabled as long as the LOG_FILE_MAX_SIZE ENV var is set to a value greater than zero. The amount of disk space required for this feature to work can be calculated with the following formula: LOG_FILE_MAX_SIZE * (LOG_FILE_MAX_BACKUPS + 1). If your disk doesn't have enough disk space, the logging will pause and the application will log Errors until space is available again.
• Added support for the force flag on chainlink blocks replay. If set to true, already consumed logs that would otherwise be skipped will be rebroadcasted.
• Added version compatibility check when using CLI to login to a remote node. flag bypass-version-check skips this check.
• Interrim solution to set multiple nodes/chains from ENV. This is a temporary stand-in until configuration is overhauled and will be removed in future. Set as such: EVM_NODES='{...}' where the var is a JSON array containing the node specifications. This is not compatible with using any other way to specify node via env (e.g. ETH_URL etc).

For example:

EVM_NODES='
[
	{
		"name": "primary_0_1",
		"evmChainId": "0",
		"wsUrl": "ws://test1.invalid",
		"sendOnly": false
	},
	{
		"name": "primary_0_2",
		"evmChainId": "0",
		"wsUrl": "ws://test1.invalid",
		"httpUrl": "https://test1.invalid",
		"sendOnly": false
	},
	{
		"name": "primary_1337_1",
		"evmChainId": "1337",
		"wsUrl": "ws://test2.invalid",
		"httpUrl": "http://test2.invalid",
		"sendOnly": false
	},
	{
		"name": "sendonly_1337_1",
		"evmChainId": "1337",
		"httpUrl": "http://test1.invalid",
		"sendOnly": true
	},
	{
		"name": "sendonly_0_1",
		"evmChainId": "0",
		"httpUrl": "http://test1.invalid",
		"sendOnly": true
	},
	{
		"name": "primary_42_1",
		"evmChainId": "42",
		"wsUrl": "ws://test1.invalid",
		"sendOnly": false
	},
	{
		"name": "sendonly_43_1",
		"evmChainId": "43",
		"httpUrl": "http://test1.invalid",
		"sendOnly": true
	}
]
'

Changed

• Changed default locking mode to "dual". Bugs in lease locking have been ironed out and this paves the way to making "lease" the default in future. It is recommended to set DATABASE_LOCKING_MODE=lease, default is set to "dual" only for backwards compatibility.
• EIP-1559 is now enabled by default on mainnet. To disable (go back to legacy mode) set EVM_EIP1559_DYNAMIC_FEES=false. The default settings should work well, but if you wish to tune your gas controls, see the documentation.

Note that EIP-1559 can be manually enabled on other chains by setting EVM_EIP1559_DYNAMIC_FEES=true but we only support it for official Ethereum mainnet and testnets. It is not recommended to enable this setting on Polygon since during our testing process we found that the EIP-1559 fee market appears to be broken on all Polygon chains and EIP-1559 transactions are actually less likely to get included than legacy transactions.

See issue: https://github.com/maticnetwork/bor/issues/347

Added

• Added disk rotating logs. Chainlink will now always log to disk at debug level. The default output directory for debug logs is Chainlink's root directory (ROOT_DIR) but can be configured by setting LOG_FILE_DIR. This makes it easier for node operators to report useful debugging information to Chainlink's team, since all the debug logs are conveniently located in one directory. Regular logging to STDOUT still works as before and respects the LOG_LEVEL env var. If you want to log in disk at a particular level, you can pipe STDOUT to disk. This automatic debug-logs-to-disk feature is enabled by default, and will remain enabled as long as the LOG_FILE_MAX_SIZE ENV var is set to a value greater than zero. The amount of disk space required for this feature to work can be calculated with the following formula: LOG_FILE_MAX_SIZE * (LOG_FILE_MAX_BACKUPS + 1). If your disk doesn't have enough disk space, the logging will pause and the application will log Errors until space is available again.
• Added support for the force flag on chainlink blocks replay. If set to true, already consumed logs that would otherwise be skipped will be rebroadcasted.
• Added version compatibility check when using CLI to login to a remote node. flag bypass-version-check skips this check.
• Interrim solution to set multiple nodes/chains from ENV. This is a temporary stand-in until configuration is overhauled and will be removed in future. Set as such: EVM_NODES='{...}' where the var is a JSON array containing the node specifications. This is not compatible with using any other way to specify node via env (e.g. ETH_URL etc).

For example:

EVM_NODES='
[
	{
		"name": "primary_0_1",
		"evmChainId": "0",
		"wsUrl": "ws://test1.invalid",
		"sendOnly": false
	},
	{
		"name": "primary_0_2",
		"evmChainId": "0",
		"wsUrl": "ws://test1.invalid",
		"httpUrl": "https://test1.invalid",
		"sendOnly": false
	},
	{
		"name": "primary_1337_1",
		"evmChainId": "1337",
		"wsUrl": "ws://test2.invalid",
		"httpUrl": "http://test2.invalid",
		"sendOnly": false
	},
	{
		"name": "sendonly_1337_1",
		"evmChainId": "1337",
		"httpUrl": "http://test1.invalid",
		"sendOnly": true
	},
	{
		"name": "sendonly_0_1",
		"evmChainId": "0",
		"httpUrl": "http://test1.invalid",
		"sendOnly": true
	},
	{
		"name": "primary_42_1",
		"evmChainId": "42",
		"wsUrl": "ws://test1.invalid",
		"sendOnly": false
	},
	{
		"name": "sendonly_43_1",
		"evmChainId": "43",
		"httpUrl": "http://test1.invalid",
		"sendOnly": true
	}
]
'

Changed

• Changed default locking mode to "dual". Bugs in lease locking have been ironed out and this paves the way to making "lease" the default in future. It is recommended to set DATABASE_LOCKING_MODE=lease, default is set to "dual" only for backwards compatibility.
• EIP-1559 is now enabled by default on mainnet. To disable (go back to legacy mode) set EVM_EIP1559_DYNAMIC_FEES=false. The default settings should work well, but if you wish to tune your gas controls, see the documentation.

Note that EIP-1559 can be manually enabled on other chains by setting EVM_EIP1559_DYNAMIC_FEES=true but we only support it for official Ethereum mainnet and testnets. It is not recommended to enable this setting on Polygon since during our testing process we found that the EIP-1559 fee market appears to be broken on all Polygon chains and EIP-1559 transactions are actually less likely to get included than legacy transactions.

See issue: https://github.com/maticnetwork/bor/issues/347

This release hotfixes issues from moving a new CI/CD system. Featurewise the functionality is the same as v1.2.0.

Fixed

• Fixed CI/CD issue where environment variables were not being passed into the underlying build

This release hotfixes issues from moving a new CI/CD system. Featurewise the functionality is the same as v1.2.0.

Fixed

• Fixed CI/CD issue where environment variables were not being passed into the underlying build

Added

• Added support for the Nethermind Ethereum client.
• Added support for batch sending telemetry to the ingress server to improve performance.
• Added v2 P2P networking support (alpha)

New ENV vars:

• ADVISORY_LOCK_CHECK_INTERVAL (default: 1s) - when advisory locking mode is enabled, this controls how often Chainlink checks to make sure it still holds the advisory lock. It is recommended to leave this at the default.
• ADVISORY_LOCK_ID (default: 1027321974924625846) - when advisory locking mode is enabled, the application advisory lock ID can be changed using this env var. All instances of Chainlink that might run on a particular database must share the same advisory lock ID. It is recommended to leave this at the default.
• LOG_FILE_DIR (default: chainlink root directory) - if LOG_TO_DISK is enabled, this env var allows you to override the output directory for logging.
• SHUTDOWN_GRACE_PERIOD (default: 5s) - when node is shutting down gracefully and exceeded this grace period, it terminates immediately (trying to close DB connection) to avoid being SIGKILLed.
• SOLANA_ENABLED (default: false) - set to true to enable Solana support
• TERRA_ENABLED (default: false) - set to true to enable Terra support
• BLOCK_HISTORY_ESTIMATOR_EIP1559_FEE_CAP_BUFFER_BLOCKS - if EIP1559 mode is enabled, this optional env var controls the buffer blocks to add to the current base fee when sending a transaction. By default, the gas bumping threshold + 1 block is used. It is not recommended to change this unless you know what you are doing.
• TELEMETRY_INGRESS_BUFFER_SIZE (default: 100) - the number of telemetry messages to buffer before dropping new ones
• TELEMETRY_INGRESS_MAX_BATCH_SIZE (default: 50) - the maximum number of messages to batch into one telemetry request
• TELEMETRY_INGRESS_SEND_INTERVAL (default: 500ms) - the cadence on which batched telemetry is sent to the ingress server
• TELEMETRY_INGRESS_USE_BATCH_SEND (default: true) - toggles sending telemetry using the batch client to the ingress server

Bootstrap job

Added a new bootstrap job type. This job removes the need for every job to implement their own bootstrapping logic.
OCR2 jobs with isBootstrapPeer=true are automatically migrated to the new format.
The spec parameters are similar to a basic OCR2 job, an example would be:

type            = "bootstrap"
name            = "bootstrap"
relay           = "evm"
schemaVersion	= 1
contractID      = "0xAb5801a7D398351b8bE11C439e05C5B3259aeC9B"
[relayConfig]
chainID	        = 4

Removed

• deleteuser CLI command.

Changed

EVM_DISABLED has been deprecated and replaced by EVM_ENABLED for consistency with other feature flags.
ETH_DISABLED has been deprecated and replaced by EVM_RPC_ENABLED for consistency, and because this was confusingly named. In most cases you want to set EVM_ENABLED=false and not EVM_RPC_ENABLED=false.

Log colorization is now disabled by default because it causes issues when piped to text files. To re-enable log colorization, set LOG_COLOR=true.

Polygon/matic defaults changed

Due to increasingly hostile network conditions on Polygon we have had to increase a number of default limits. This is to work around numerous and very deep re-orgs, high mempool pressure and a failure by the network to propagate transactions properly. These new limits are likely to increase load on both your Chainlink node and database, so please be sure to monitor CPU and memory usage on both and make sure they are adequately specced to handle the additional load.

Added

• BLOCK_HISTORY_ESTIMATOR_EIP1559_FEE_CAP_BUFFER_BLOCKS - if EIP1559 mode is enabled, this optional env var controls the buffer blocks to add to the current base fee when sending a transaction. By default, the gas bumping threshold + 1 block is used. It is not recommended to change this unless you know what you are doing.
• EVM_GAS_FEE_CAP_DEFAULT - if EIP1559 mode is enabled, and FixedPrice gas estimator is used, this env var controls the fixed initial fee cap.

Fixed

Fixed issues with EIP-1559 related to gas bumping. Due to go-ethereum's implementation which introduces additional restrictions on top of the EIP-1559 spec, we must bump the FeeCap at least 10% each time in order for the gas bump to be accepted.

The new EIP-1559 implementation works as follows:

If you are using FixedPriceEstimator:

• With gas bumping disabled, it will submit all transactions with feecap=ETH_MAX_GAS_PRICE_WEI and tipcap=EVM_GAS_TIP_CAP_DEFAULT
• With gas bumping enabled, it will submit all transactions initially with feecap=EVM_GAS_FEE_CAP_DEFAULT and tipcap=EVM_GAS_TIP_CAP_DEFAULT.

If you are using BlockHistoryEstimator (default for most chains):

• With gas bumping disabled, it will submit all transactions with feecap=ETH_MAX_GAS_PRICE_WEI and tipcap=<calculated using past blocks>
• With gas bumping enabled (default for most chains) it will submit all transactions initially with feecap=current block base fee * (1.125 ^ N) where N is configurable by setting BLOCK_HISTORY_ESTIMATOR_EIP1559_FEE_CAP_BUFFER_BLOCKS but defaults to gas bump threshold+1 and tipcap=<calculated using past blocks>

Bumping works as follows:

• Increase tipcap by max(tipcap * (1 + ETH_GAS_BUMP_PERCENT), tipcap + ETH_GAS_BUMP_WEI)
• Increase feecap by max(feecap * (1 + ETH_GAS_BUMP_PERCENT), feecap + ETH_GAS_BUMP_WEI)

[1.1.0] - 2022-01-25

Added

• Added support for Sentry error reporting. Set SENTRY_DSN at compile- or run-time to enable reporting.
• Added Prometheus counters: log_warn_count, log_error_count, log_critical_count, log_panic_count and log_fatal_count representing the corresponding number of warning/error/critical/panic/fatal messages in the log.
• The new prometheus metric tx_manager_tx_attempt_count is a Prometheus Gauge that should represent the total number of Transactions attempts that awaiting confirmation for this node.
• The new prometheus metric version that displays the node software version (tag) as well as the corresponding commit hash.
• CLI command keys eth list is updated to display key specific max gas prices.
• CLI command keys eth create now supports optional maxGasPriceGWei parameter.
• CLI command keys eth update is added to update key specific parameters like maxGasPriceGWei.
• Add partial support for Moonriver chain

Two new log levels have been added.

• [crit]: Critical level logs are more severe than [error] and require quick action from the node operator.
• [debug] [trace]: Trace level logs contain extra [debug] information for development, and must be compiled in via -tags trace.

[Beta] Multichain support added

As a beta feature, Chainlink now supports connecting to multiple different EVM chains simultaneously.

This means that one node can run jobs on Goerli, Kovan, BSC and Mainnet (for example). Note that you can still have as many eth keys as you like, but each eth key is pegged to one chain only.

Extensive efforts have been made to make migration for existing nops as seamless as possible. Generally speaking, you should not have to make any changes when upgrading your existing node to this version. All your jobs will continue to run as before.

The overall summary of changes is such:

Chains/Ethereum Nodes

EVM chains are now represented as a first class object within the chainlink node. You can create/delete/list them using the CLI or API.

At least one primary node is required in order for a chain to connect. You may additionally specify zero or more send-only nodes for a chain. It is recommended to use the CLI/API or GUI to add nodes to chain.

Creation

chainlink chains evm create -id 42 # creates an evm chain with chain ID 42 (see: https://chainlist.org/)
chainlink nodes create -chain-id 42 -name 'my-primary-kovan-full-node' -type primary -ws-url ws://node.example/ws -http-url http://node.example/rpc # http-url is optional but recommended for primaries
chainlink nodes create -chain-id 42 -name 'my-send-only-backup-kovan-node' -type sendonly -http-url http://some-public-node.example/rpc

Listing

chainlink chains evm list
chainlink nodes list

Deletion

chainlink nodes delete 'my-send-only-backup-kovan-node'
chainlink chains evm delete 42

Legacy eth ENV vars

The old way of specifying chains using environment variables is still supported but discouraged. It works as follows:

If you specify ETH_URL then the values of ETH_URL, ETH_CHAIN_ID, ETH_HTTP_URL and ETH_SECONDARY_URLS will be used to create/update chains and nodes representing these values in the database. If an existing chain/node is found it will be overwritten. This behavior is used mainly to ease the process of upgrading, and on subsequent runs (once your old settings have been written to the database) it is recommended to unset these ENV vars and use the API commands exclusively to administer chains and nodes.

Jobs/tasks

By default, all jobs/tasks will continue to use the default chain (specified by ETH_CHAIN_ID). However, the following jobs now allow an additional evmChainID key in their TOML:

• VRF
• DirectRequest
• Keeper
• OCR
• Fluxmonitor

You can pin individual jobs to a particular chain by specifying the evmChainID explicitly. Here is an example job to demonstrate:

type            = "keeper"
evmChainID      = 3
schemaVersion   = 1
name            = "example keeper spec"
contractAddress = "0x9E40733cC9df84636505f4e6Db28DCa0dC5D1bba"
externalJobID   = "0EEC7E1D-D0D2-476C-A1A8-72DFB6633F49"
fromAddress     = "0xa8037A20989AFcBC51798de9762b351D63ff462e"

The above keeper job will always run on chain ID 3 (Ropsten) regardless of the ETH_CHAIN_ID setting. If no chain matching this ID has been added to the chainlink node, the job cannot be created (you must create the chain first).

In addition, you can also specify evmChainID on certain pipeline tasks. This allows for cross-chain requests, for example:

type                = "directrequest"
schemaVersion       = 1
evmChainID          = 42
name                = "example cross chain spec"
contractAddress     = "0x613a38AC1659769640aaE063C651F48E0250454C"
externalJobID       = "0EEC7E1D-D0D2-476C-A1A8-72DFB6633F90"
observationSource   = """
    decode_log   [type=ethabidecodelog ... ]
    ...
    submit [type=ethtx to="0x613a38AC1659769640aaE063C651F48E0250454C" data="$(encode_tx)" minConfirmations="2" evmChainID="3"]
    decode_log-> ... ->submit;
"""

In the example above (which excludes irrelevant pipeline steps for brevity) a log can be read from the chain with ID 42 (Kovan) and a transaction emitted on chain with ID 3 (Ropsten).

Tasks that support the evmChainID parameter are as follows:

• ethcall
• estimategaslimit
• ethtx

Defaults

If the job- or task-specific evmChainID is not given, the job/task will simply use the default as specified by the ETH_CHAIN_ID env variable.

Generally speaking, the default config values for each chain are good enough. But in some cases it is necessary to be able to override the defaults on a per-chain basis.

This used to be done via environment variables e.g. MINIMUM_CONTRACT_PAYMENT_LINK_JUELS.

These still work, but if set they will override that value for all chains. This may not always be what you want. Consider a node that runs both Matic and Mainnet. You may want to set a higher value for MINIMUM_CONTRACT_PAYMENT on Mainnet, due to the more expensive gas costs. However, setting MINIMUM_CONTRACT_PAYMENT_LINK_JUELS using env variables will set that value for all chains including matic.

To help you work around this, Chainlink now supports setting per-chain configuration options.

Examples

To set initial configuration when creating a chain, pass in the full json string as an optional parameter at the end:

chainlink evm chains create -id 42 '{"BlockHistoryEstimatorBlockDelay": "100"}'

To set configuration on an existing chain, specify key values pairs as such:

chainlink evm chains configure -id 42 BlockHistoryEstimatorBlockDelay=100 GasEstimatorMode=FixedPrice

The full list of chain-specific configuration options can be found by looking at the ChainCfg struct in core/chains/evm/types/types.go.

Async support in external adapters

External Adapters making async callbacks can now error job runs. This required a slight change to format, the correct way to callback from an asynchronous EA is using the following JSON:

SUCCESS CASE:

{
    "value": < any valid json object >
}

ERROR CASE:

{
    "error": "some error string"
}

This only applies to EAs using the X-Chainlink-Pending header to signal that the result will be POSTed back to the Chainlink node sometime 'later'. Regular synchronous calls to EAs work just as they always have done.

(NOTE: Official documentation for EAs needs to be updated)

New optional VRF v2 field: requestedConfsDelay

Added a new optional field for VRF v2 jobs called requestedConfsDelay, which configures a
number of blocks to wait in addition to the request specified requestConfirmations before servicing
the randomness request, i.e the Chainlink node will wait max(nodeMinConfs, requestConfirmations + requestedConfsDelay)
blocks before servicing the request.

It can be used in the following way:

type = "vrf"
externalJobID = "123e4567-e89b-12d3-a456-426655440001"
schemaVersion = 1
name = "vrf-v2-secondary"
coordinatorAddress = "0xABA5eDc1a551E55b1A570c0e1f1055e5BE11eca7"
requestedConfsDelay = 10
# ... rest of job spec ...

Use of this field requires a database migration.

New locking mode: 'lease'

Chainlink now supports a new environment variable DATABASE_LOCKING_MODE. It can be set to one of the following values:

• dual (the default - uses both locking types for backwards and forwards compatibility)
• advisorylock (advisory lock only)
• lease (lease lock only)
• none (no locking at all - useful for advanced deployment environments when you can be sure that only one instance of chainlink will ever be running)

The database lock ensures that only one instance of Chainlink can be run on the database at a time. Running multiple instances of Chainlink on a single database at the same time would likely to lead to strange errors and possibly even data integrity failures and should not be allowed.

Ideally, node operators would be using a container orchestration system (e.g. Kubernetes) that ensures that only one instance of Chainlink ever runs on a particular postgres database.

However, we are aware that many node operators do not have the technical capacity to do this. So a common use case is to run multiple Chainlink instances in failover mode (as recommended by our official documentation, although this will be changing in future). The first instance will take some kind of lock on the database and subsequent instances will wait trying to take this lock in case the first instance disappears or dies.

Traditionally Chainlink has used an advisory lock to manage this. However, advisory locks come with several problems, notably:

• Postgres does not really like it when you hold locks open for a very long time (hours/days). It hampers certain internal cleanup tasks and is explicitly discouraged by the postgres maintainers.
• The advisory lock can silently disappear on postgres upgrade, meaning that a new instance can take over even while the old one is still running.
• Advisory locks do not play nicely with pooling tools such as pgbouncer.
• If the application crashes, the advisory lock can be left hanging around for a while (sometimes hours) and can require manual intervention to remove it before another instance of Chainlink will allow itself to boot.

For this reason, we have introduced a new locking mode, lease, which is likely to become the default in future. lease-mode works as follows:

• Have one row in a database which is updated periodically with the client ID.
• CL node A will run a background process on start that updates this e.g. once per second.
• CL node B will spinlock, checking periodically to see if the update got too old. If it goes more than a set period without updating, it assumes that node A is dead and takes over. Now CL node B is the owner of the row and it updates this every second.
• If CL node A comes back somehow, it will go to take out a lease and realise that the database has been leased to another process, so it will exit the entire application immediately.

The default is set to dual which used both advisory locking AND lease locking, for backwards compatibility. However, it is recommended that node operators who know what they are doing, or explicitly want to stop using the advisory locking mode set DATABASE_LOCKING_MODE=lease in their env.

Lease locking can be configured using the following ENV vars:

LEASE_LOCK_REFRESH_INTERVAL (default 1s)
LEASE_LOCK_DURATION (default 30s)

It is recommended to leave these set to the default values.

Duplicate Job Configuration

When duplicating a job, the new job's configuration settings that have not been overridden by the user can still reflect the chainlink node configuration.

Nurse (automatic pprof profiler)

Added new automatic pprof profiling service. Profiling is triggered when the node exceeds certain resource thresholds (currently, memory and goroutine count). The following environment variables have been added to allow configuring this service:

• AUTO_PPROF_ENABLED: Set to true to enable the automatic profiling service. Defaults to false.
• AUTO_PPROF_PROFILE_ROOT: The location on disk where pprof profiles will be stored. Defaults to $CHAINLINK_ROOT.
• AUTO_PPROF_POLL_INTERVAL: The interval at which the node's resources are checked. Defaults to 10s.
• AUTO_PPROF_GATHER_DURATION: The duration for which profiles are gathered when profiling is kicked off. Defaults to 10s.
• AUTO_PPROF_GATHER_TRACE_DURATION: The duration for which traces are gathered when profiling is kicked off. This is separately configurable because traces are significantly larger than other types of profiles. Defaults to 5s.
• AUTO_PPROF_MAX_PROFILE_SIZE: The maximum amount of disk space that profiles may consume before profiling is disabled. Defaults to 100mb.
• AUTO_PPROF_CPU_PROFILE_RATE: See https://pkg.go.dev/runtime#SetCPUProfileRate. Defaults to 1.
• AUTO_PPROF_MEM_PROFILE_RATE: See https://pkg.go.dev/runtime#pkg-variables. Defaults to 1.
• AUTO_PPROF_BLOCK_PROFILE_RATE: See https://pkg.go.dev/runtime#SetBlockProfileRate. Defaults to 1.
• AUTO_PPROF_MUTEX_PROFILE_FRACTION: See https://pkg.go.dev/runtime#SetMutexProfileFraction. Defaults to 1.
• AUTO_PPROF_MEM_THRESHOLD: The maximum amount of memory the node can actively consume before profiling begins. Defaults to 4gb.
• AUTO_PPROF_GOROUTINE_THRESHOLD: The maximum number of actively-running goroutines the node can spawn before profiling begins. Defaults to 5000.

Adventurous node operators are encouraged to read this guide on how to analyze pprof profiles.

merge task type

A new task type has been added, called merge. It can be used to merge two maps/JSON values together. Merge direction is from right to left such that right will clobber values of left. If no left is provided, it uses the input of the previous task. Example usage as such:

decode_log   [type=ethabidecodelog ...]
merge        [type=merge right=<{"foo": 42}>];

decode_log -> merge;

Or, to reverse merge direction:

decode_log   [type=ethabidecodelog ...]
merge        [type=merge left=<{"foo": 42}> right="$(decode_log)"];

decode_log -> merge;

Enhanced ABI encoding support

The ethabiencode2 task supports ABI encoding using the abi specification generated by solc. e.g:

{
    "name": "call",
    "inputs": [
      {
        "name": "value",
        "type": "tuple",
        "components": [
          {
            "name": "first",
            "type": "bytes32"
          },
          {
            "name": "last",
            "type": "bool"
          }
        ]
      }
    ]
}

This would allow for calling of a function call with a tuple containing two values, the first a bytes32 and the second a bool. You can supply a named map or an array.

Transaction Simulation (Gas Savings)

Chainlink now supports transaction simulation for certain types of job. When this is enabled, transactions will be simulated using eth_call before initial send. If the transaction would revert, the tx is marked as errored without being broadcast, potentially avoiding an expensive on-chain revert.

This can add a tiny bit of latency (upper bound 2s, generally much shorter under good conditions) and will add marginally more load to the eth client, since it adds an extra call for every transaction sent. However, it may help to save gas in some cases especially during periods of high demand by avoiding unnecessary reverts (due to outdated round etc).

This option is EXPERIMENTAL and disabled by default.

To enable for FM or OCR:

FM_SIMULATE_TRANSACTIONs=true
OCR_SIMULATE_TRANSACTIONS=true

To enable in the pipeline, use the simulate=true option like so:

submit [type=ethtx to="0xDeadDeadDeadDeadDeadDeadDeadDead" data="0xDead" simulate=true]

Use at your own risk.

Misc

Chainlink now supports more than one primary eth node per chain. Requests are round-robined between available primaries.

Add CRUD functionality for EVM Chains and Nodes through Operator UI.

Non fatal errors to a pipeline run are preserved including any run that succeeds but has more than one fatal error.

Chainlink now supports configuring max gas price on a per-key basis (allows implementation of keeper "lanes").

The Operator UI now supports login MFA with hardware security keys. MFA_RPID and MFA_RPORIGIN environment variables have been added to the config and are required if using the new MFA feature.

Keys and Configuration navigation links have been moved into a settings dropdown to make space for multichain navigation links.

Full EIP1559 Support (Gas Savings)

Chainlink now includes experimental support for submitting transactions using type 0x2 (EIP-1559) envelope.

EIP-1559 mode is off by default but can be enabled on a per-chain basis or globally.

This may help to save gas on spikes: Chainlink ought to react faster on the upleg and avoid overpaying on the downleg. It may also be possible to set BLOCK_HISTORY_ESTIMATOR_BATCH_SIZE to a smaller value e.g. 12 or even 6 because tip cap ought to be a more consistent indicator of inclusion time than total gas price. This would make Chainlink more responsive and ought to reduce response time variance. Some experimentation will be needed here to find optimum settings.

To enable globally, set EVM_EIP1559_DYNAMIC_FEES=true. Set with caution, if you set this on a chain that does not actually support EIP-1559 your node will be broken.

In EIP-1559 mode, the total price for the transaction is the minimum of base fee + tip cap and fee cap. More information can be found on the official EIP.

Chainlink's implementation of this is to set a large fee cap and modify the tip cap to control confirmation speed of transactions. So, when in EIP1559 mode, the tip cap takes the place of gas price roughly speaking, with the varying base price remaining a constant (we always pay it).

A quick note on terminology - Chainlink uses the same terms used internally by go-ethereum source code to describe various prices. This is not the same as the externally used terms. For reference:

Base Fee Per Gas = BaseFeePerGas
Max Fee Per Gas = FeeCap
Max Priority Fee Per Gas = TipCap

In EIP-1559 mode, the following changes occur to how configuration works:

• All new transactions will be sent as type 0x2 transactions specifying a TipCap and FeeCap (NOTE: existing pending legacy transactions will continue to be gas bumped in legacy mode)
• BlockHistoryEstimator will apply its calculations (gas percentile etc) to the TipCap and this value will be used for new transactions (GasPrice will be ignored)
• FixedPriceEstimator will use EVM_GAS_TIP_CAP_DEFAULT instead of ETH_GAS_PRICE_DEFAULT
• ETH_GAS_PRICE_DEFAULT is ignored for new transactions and EVM_GAS_TIP_CAP_DEFAULT is used instead (default 20GWei)
• ETH_MIN_GAS_PRICE_WEI is ignored for new transactions and EVM_GAS_TIP_CAP_MINIMUM is used instead (default 0)
• ETH_MAX_GAS_PRICE_WEI controls the FeeCap
• KEEPER_GAS_PRICE_BUFFER_PERCENT is ignored in EIP-1559 mode and KEEPER_TIP_CAP_BUFFER_PERCENT is used instead

The default tip cap is configurable per-chain but can be specified for all chains using EVM_GAS_TIP_CAP_DEFAULT. The fee cap is derived from ETH_MAX_GAS_PRICE_WEI.

When using the FixedPriceEstimator, the default gas tip will be used for all transactions.

When using the BlockHistoryEstimator, Chainlink will calculate the tip cap based on transactions already included (in the same way it calculates gas price in legacy mode).

Enabling EIP1559 mode might lead to marginally faster transaction inclusion and make the node more responsive to sharp rises/falls in gas price, keeping response times more consistent.

In addition, ethcall tasks now accept gasTipCap and gasFeeCap parameters in addition to gasPrice. This is required for Keeper jobs, i.e.:

check_upkeep_tx          [type=ethcall
                          failEarly=true
                          extractRevertReason=true
                          contract="$(jobSpec.contractAddress)"
                          gas="$(jobSpec.checkUpkeepGasLimit)"
                          gasPrice="$(jobSpec.gasPrice)"
                          gasTipCap="$(jobSpec.gasTipCap)"
                          gasFeeCap="$(jobSpec.gasFeeCap)"
                          data="$(encode_check_upkeep_tx)"]

NOTE: AccessLists are part of the 0x2 transaction type spec and Chainlink also implements support for these internally. This is not currently exposed in any way, if there is demand for this it ought to be straightforward enough to do so.

Avalanche AP4 defaults have been added (you can remove manually set ENV vars controlling gas pricing).

New env vars

CHAIN_TYPE - Configure the type of chain (if not standard). Arbitrum, ExChain, Optimism, or XDai. Replaces LAYER_2_TYPE. NOTE: This is a global override, to set on a per-chain basis you must use the CLI/API or GUI to change the chain-specific config for that chain (ChainType).

BLOCK_EMISSION_IDLE_WARNING_THRESHOLD - Controls global override for the time after which node will start logging warnings if no heads are received.

ETH_DEFAULT_BATCH_SIZE - Controls the default number of items per batch when making batched RPC calls. It is unlikely that you will need to change this from the default value.

NOTE: ETH_URL used to default to "ws://localhost:8546" and ETH_CHAIN_ID used to default to 1. These defaults have now been removed. The env vars are no longer required, since node configuration is now done via CLI/API/GUI and stored in the database.

Removed

• belt/ and evm-test-helpers/ removed from the codebase.

Deprecated env vars

LAYER_2_TYPE - Use CHAIN_TYPE instead.

Removed env vars

FEATURE_CRON_V2, FEATURE_FLUX_MONITOR_V2, FEATURE_WEBHOOK_V2 - all V2 job types are now enabled by default.

Fixed

• Fixed a regression whereby the BlockHistoryEstimator would use a bumped value on old gas price even if the new current price was larger than the bumped value.
• Fixed a bug where creating lots of jobs very quickly in parallel would cause the node to hang
• Propagating evmChainID parameter in job specs supporting this parameter.

Fixed LOG_LEVEL behavior in respect to the corresponding UI setting: Operator can override LOG_LEVEL until the node is restarted.

Changed

• The default GAS_ESTIMATOR_MODE for Optimism chains has been changed to Optimism2.
• Default minimum payment on mainnet has been reduced from 1 LINK to 0.1 LINK.
• Logging timestamp output has been changed from unix to ISO8601 to aid in readability. To keep the old unix format, you may set LOG_UNIX_TS=true
• Added WebAuthn support for the Operator UI and corresponding support in the Go backend

Log to Disk

This feature has been disabled by default, turn on with LOG_TO_DISK. For most production uses this is not desirable.

Added

• Improved error reporting
• Panic and recovery improvements

Fixed

• Resolved config conversion errors for ETH_FINALITY_DEPTH, ETH_HEAD_TRACKER_HISTORY, and ETH_GAS_LIMIT_MULTIPLIER
• Proper handling for "nonce too low" errors on Avalanche

Added

• chainlink node db status will now display a table of applied and pending migrations.
• Add support for OKEx/ExChain.

Changed

Legacy job pipeline (JSON specs) are no longer supported

This version will refuse to migrate the database if job specs are still present. You must manually delete or migrate all V1 job specs before upgrading.

For more information on migrating, see the docs.

This release will DROP legacy job tables so please take a backup before upgrading.

New env vars

LAYER_2_TYPE - For layer 2 chains only. Configure the type of chain, either Arbitrum or Optimism.

Misc

• Head sampling can now be optionally disabled by setting ETH_HEAD_TRACKER_SAMPLING_INTERVAL = "0s" - this will result in every new head being delivered to running jobs,
  regardless of the head frequency from the chain.
• When creating new FluxMonitor jobs, the validation logic now checks that only one of: drumbeat ticker or idle timer is enabled.
• Added a new Prometheus metric: uptime_seconds which measures the number of seconds the node has been running. It can be helpful in detecting potential crashes.

Fixed

Fixed a regression whereby the BlockHistoryEstimator would use a bumped value on old gas price even if the new current price was larger than the bumped value.

It is highly recommended to upgrade to this version before upgrading to any newer versions to avoid any complications.

Fixed

• Prevent release from clobbering newer databases

Added

• FMv2 spec now contains DrumbeatRandomDelay parameter that can be used to introduce variation between round of submits of different oracles, if drumbeat ticker is enabled.

• OCR Hibernation

Requesters/MinContractPaymentLinkJuels

V2 direct request specs now support two additional keys:

• "requesters" key which allows to whitelist requesters
• "minContractPaymentLinkJuels" key which allows to specify a job-specific minimum contract payment.

For example:

type                        = "directrequest"
schemaVersion               = 1
requesters                  = ["0xaaaa1F8ee20f5565510B84f9353F1E333E753B7a", "0xbbbb70F0e81C6F3430dfdC9fa02fB22BdD818C4e"] # optional
minContractPaymentLinkJuels = "100000000000000" # optional
name                        = "example eth request event spec with requesters"
contractAddress             = "..."
externalJobID               = "..."
observationSource           = """
...
"""

Fixed

• Resolved exiting Hibernation bug on FMv2

Fixed

• Resolved FMv2 stalling in Hibernation mode
• Resolved rare issue when the Gas Estimator fails on start
• Resolved the handling of nil values for gas price

A new configuration variable, BLOCK_BACKFILL_SKIP, can be optionally set to "true" in order to strongly limit the depth of the log backfill.
This is useful if the node has been offline for a longer time and after startup should not be concerned with older events from the chain.

• Fixes the logging configuration form not displaying the current values
• Updates the design of the configuration cards to be easier on the eyes
• View Coordinator Service Authentication keys in the Operator UI. This is hidden
  behind a feature flag until usage is enabled.

Changed

The legacy job pipeline (JSON specs) has been officially deprecated and support for these jobs will be dropped in an upcoming release.

Any node operators still running jobs with JSON specs should migrate their jobs to TOML format instead.

The format for V2 Webhook job specs has changed. They now allow specifying 0 or more external initiators. Example below:

type            = "webhook"
schemaVersion   = 1
externalInitiators = [
    { name = "foo-ei", spec = '{"foo": 42}' },
    { name = "bar-ei", spec = '{"bar": 42}' }
]
observationSource   = """
ds          [type=http method=GET url="https://chain.link/ETH-USD"];
ds_parse    [type=jsonparse path="data,price"];
ds_multiply [type=multiply times=100];
ds -> ds_parse -> ds_multiply;
"""

These external initiators will be notified with the given spec after the job is created, and also at deletion time.

Only the External Initiators listed in the toml spec may trigger a run for that job. Logged in users can always trigger a run for any job.

Migrating Jobs

• OCR
  All OCR jobs are already using v2 pipeline by default - no need to do anything here.

• Flux Monitor v1
  We have created a tool to help you automigrate flux monitor specs in JSON format to the new TOML format. You can migrate a job like this:

chainlink jobs migrate <job id>

This can be automated by using the API like so:

POST http://yournode.example/v2/migrate/<job id>

• VRF v1
  Automigration is not supported for VRF jobs. They must be manually converted into v2 format.

• Ethlog/Runlog/Cron/web
  All other job types must also be manually converted into v2 format.

Technical details

Why are we doing this?

To give some background, the legacy job pipeline has been around since before Chainlink went to mainnet and is getting quite long in the tooth. The code is brittle and difficult to understand and maintain. For a while now we have been developing a v2 job pipeline in parallel which uses the TOML format. The new job pipeline is simpler, more performant and more powerful. Every job that can be represented in the legacy pipeline should be able to be represented in the v2 pipeline - if it can't be, that's a bug, so please let us know ASAP.

The v2 pipeline has now been extensively tested in production and proved itself reliable. So, we made the decision to drop V1 support entirely in favour of focusing developer effort on new features like native multichain support, EIP1559-compatible fees, further gas saving measures and support for more blockchains. By dropping support for the old pipeline, we can deliver these features faster and better support our community.

Changed

This update will truncate pipeline_runs, pipeline_task_runs, flux_monitor_round_stats_v2 DB tables as a part of the migration.

Gas Estimation

Gas estimation has been revamped and full support for Optimism has been added.

The following env vars have been deprecated, and will be removed in a future release:

GAS_UPDATER_ENABLED
GAS_UPDATER_BATCH_SIZE
GAS_UPDATER_BLOCK_DELAY
GAS_UPDATER_BLOCK_HISTORY_SIZE
GAS_UPDATER_TRANSACTION_PERCENTILE

If you are using any of the env vars above, please switch to using the following instead:

GAS_ESTIMATOR_MODE
BLOCK_HISTORY_ESTIMATOR_BATCH_SIZE
BLOCK_HISTORY_ESTIMATOR_BLOCK_DELAY
BLOCK_HISTORY_ESTIMATOR_BLOCK_HISTORY_SIZE
BLOCK_HISTORY_ESTIMATOR_TRANSACTION_PERCENTILE

Valid values for GAS_ESTIMATOR_MODE are as follows:

GAS_ESTIMATOR_MODE=BlockHistory (equivalent to GAS_UPDATER_ENABLED=true)
GAS_ESTIMATOR_MODE=FixedPrice (equivalent to GAS_UPDATER_ENABLED=false)
GAS_ESTIMATOR_MODE=Optimism (new)

New gas estimator modes may be added in future.

In addition, a minor annoyance has been fixed whereby previously if you enabled the gas updater, it would overwrite the locally stored value for gas price and continue to use this even if it was disabled after a reboot. This will no longer happen: BlockHistory mode will not clobber the locally stored value for fixed gas price, which can still be adjusted via remote API call or using chainlink config setgasprice XXX. In order to use this manually fixed gas price, you must enable FixedPrice estimator mode.

Added

Added support for latest version of libocr with the V2 networking stack. New env vars to configure this are:

P2P_NETWORKING_STACK
P2PV2_ANNOUNCE_ADDRESSES
P2PV2_BOOTSTRAPPERS
P2PV2_DELTA_DIAL
P2PV2_DELTA_RECONCILE
P2PV2_LISTEN_ADDRESSES

All of these are currently optional, by default OCR will continue to use the existing V1 stack. The new env vars will be used internally for OCR testing.

Fixed

• Fix inability to create jobs with a cron schedule.

Changed

FMv2, Keeper and OCR jobs now use a new strategy for sending transactions. By default, if multiple transactions are queued up, only the latest one will be sent. This should greatly reduce the number of stale rounds and reverted transactions, and help node operators to save significant gas especially during times of high congestion or when catching up on a deep backlog.

Defaults should work well, but it can be controlled if necessary using the following new env vars:

FM_DEFAULT_TRANSACTION_QUEUE_DEPTH
KEEPER_DEFAULT_TRANSACTION_QUEUE_DEPTH
OCR_DEFAULT_TRANSACTION_QUEUE_DEPTH

Setting to 0 will disable (the old behaviour). Setting to 1 (the default) will keep only the latest transaction queued up at any given time. Setting to 2, 3 etc will allow this many transactions to be queued before starting to drop older items.

Note that it has no effect on FMv1 jobs. Node operators will need to upgrade to FMv2 to take advantage of this feature.

Fixed

• The HTTP adapter would remove a trailing slash on a subdirectory when specifying an extended path, so for instance http://example.com/subdir/ with a param of ?query= extended path would produce the URL http://example.com/subdir?query=, but should now produce: http://example.com/subdir/?query=.

• Matic autoconfig is now enabled for mainnet. Matic nops should remove any custom tweaks they have been running with. In addition, we have better default configs for Optimism, Arbitrum and RSK.

• It is no longer required to set DEFAULT_HTTP_ALLOW_UNRESTRICTED_NETWORK_ACCESS=true to enable local fetches on bridge tasks. Please remove this if you had it set and no longer need it, since it introduces a slight security risk.

• Chainlink can now run with ETH_DISABLED=true without spewing errors everywhere

• Removed prometheus metrics that were no longer valid after recent changes to head tracking:
  head_tracker_heads_in_queue, head_tracker_callback_execution_duration,
  head_tracker_callback_execution_duration_hist, head_tracker_num_heads_dropped

Added

• INSECURE_SKIP_VERIFY configuration variable disables verification of the Chainlink SSL certificates when using the CLI.

• JSON parse tasks (v2) now permit an empty path parameter.

• Eth->eth transfer gas limit is no longer hardcoded at 21000 and can now be adjusted using ETH_GAS_LIMIT_TRANSFER

• HTTP and Bridge tasks (v2 pipeline) now log the request parameters (including the body) upon making the request when LOG_LEVEL=debug.

• Webhook v2 jobs now support two new parameters, externalInitiatorName and externalInitiatorSpec. The v2 version of the following v1 spec:

  {
    "initiators": [
      {
        "type": "external",
        "params": {
          "name": "substrate",
          "body": {
            "endpoint": "substrate",
            "feed_id": 0,
            "account_id": "0x7c522c8273973e7bcf4a5dbfcc745dba4a3ab08c1e410167d7b1bdf9cb924f6c",
            "fluxmonitor": {
              "requestData": {
                "data": { "from": "DOT", "to": "USD" }
              },
              "feeds": [{ "url": "http://adapter1:8080" }],
              "threshold": 0.5,
              "absoluteThreshold": 0,
              "precision": 8,
              "pollTimer": { "period": "30s" },
              "idleTimer": { "duration": "1m" }
            }
          }
        }
      }
    ],
    "tasks": [
      {
        "type": "substrate-adapter1",
        "params": { "multiply": 1e8 }
      }
    ]
  }
  

  is:

  type            = "webhook"
  schemaVersion   = 1
  jobID           = "0EEC7E1D-D0D2-475C-A1A8-72DFB6633F46"
  externalInitiatorName = "substrate"
  externalInitiatorSpec = """
      {
        "endpoint": "substrate",
        "feed_id": 0,
        "account_id": "0x7c522c8273973e7bcf4a5dbfcc745dba4a3ab08c1e410167d7b1bdf9cb924f6c",
        "fluxmonitor": {
          "requestData": {
            "data": { "from": "DOT", "to": "USD" }
          },
          "feeds": [{ "url": "http://adapter1:8080" }],
          "threshold": 0.5,
          "absoluteThreshold": 0,
          "precision": 8,
          "pollTimer": { "period": "30s" },
          "idleTimer": { "duration": "1m" }
        }
      }
  """
  observationSource   = """
      submit [type=bridge name="substrate-adapter1" requestData=<{ "multiply": 1e8 }>]
  """
  

• Task definitions in v2 jobs (those with TOML specs) now support quoting strings with angle brackets (which DOT already permitted). This is particularly useful when defining JSON blobs to post to external adapters. For example:

  my_bridge [type=bridge name="my_bridge" requestData="{\\"hi\\": \\"hello\\"}"]
  

  ... can now be written as:

  my_bridge [type=bridge name="my_bridge" requestData=<{"hi": "hello"}>]
  

  Multiline strings are supported with this syntax as well:

  my_bridge [type=bridge
             name="my_bridge"
             requestData=<{
                 "hi": "hello",
                 "foo": "bar"
             }>]
  

• v2 jobs (those with TOML specs) now support variable interpolation in pipeline definitions. For example:

  fetch1    [type=bridge name="fetch"]
  parse1    [type=jsonparse path="foo,bar"]
  fetch2    [type=bridge name="fetch"]
  parse2    [type=jsonparse path="foo,bar"]
  medianize [type=median]
  submit    [type=bridge name="submit"
             requestData=<{
                            "result": $(medianize),
                            "fetchedData": [ $(parse1), $(parse2) ]
                          }>]
  
  fetch1 -> parse1 -> medianize
  fetch2 -> parse2 -> medianize
  medianize -> submit
  

  This syntax is supported by the following tasks/parameters:

  • bridge
    • requestData
  • http
    • requestData
  • jsonparse
    • data (falls back to the first input if unspecified)
  • median
    • values (falls back to the array of inputs if unspecified)
  • multiply
    • input (falls back to the first input if unspecified)
    • times

• Add ETH_MAX_IN_FLIGHT_TRANSACTIONS configuration option. This defaults to 16 and controls how many unconfirmed transactions may be in-flight at any given moment. This is set conservatively by default, node operators running many jobs on high throughput chains will probably need to increase this above the default to avoid lagging behind. However, before increasing this value, you MUST first ensure your ethereum node is configured not to ever evict local transactions that exceed this number otherwise your node may get permanently stuck. Set to 0 to disable the limit entirely (the old behaviour). Disabling this setting is not recommended.

Relevant settings for geth (and forks e.g. BSC)

[Eth.TxPool]
Locals = ["0xYourNodeAddress1", "0xYourNodeAddress2"]  # Add your node addresses here
NoLocals = false # Disabled by default but might as well make sure
Journal = "transactions.rlp" # Make sure you set a journal file
Rejournal = 3600000000000 # Default 1h, it might make sense to reduce this to e.g. 5m
PriceBump = 10 # Must be set less than or equal to chainlink's ETH_GAS_BUMP_PERCENT
AccountSlots = 16 # Highly recommended to increase this, must be greater than or equal to chainlink's ETH_MAX_IN_FLIGHT_TRANSACTIONS setting
GlobalSlots = 4096 # Increase this as necessary
AccountQueue = 64 # Increase this as necessary
GlobalQueue = 1024 # Increase this as necessary
Lifetime = 10800000000000 # Default 3h, this is probably ok, you might even consider reducing it

Relevant settings for parity/openethereum (and forks e.g. xDai)

NOTE: There is a bug in parity (and xDai) where occasionally local transactions are inexplicably culled. See: https://github.com/openethereum/parity-ethereum/issues/10228

Adjusting the settings below might help.

tx_queue_locals = ["0xYourNodeAddress1", "0xYourNodeAddress2"] # Add your node addresses here
tx_queue_size = 8192 # Increase this as necessary
tx_queue_per_sender = 16 # Highly recommended to increase this, must be greater than or equal to chainlink's ETH_MAX_IN_FLIGHT_TRANSACTIONS setting
tx_queue_mem_limit = 4 # In MB. Highly recommended to increase this or set to 0
tx_queue_no_early_reject = true # Recommended to set this
tx_queue_no_unfamiliar_locals = false # This is disabled by default but might as well make sure

• Keeper jobs now support prometheus metrics, they are considered a pipeline with a single keeper task type. Example:

pipeline_run_errors{job_id="1",job_name="example keeper spec"} 1
pipeline_run_total_time_to_completion{job_id="1",job_name="example keeper spec"} 8.470456e+06
pipeline_task_execution_time{job_id="1",job_name="example keeper spec",task_type="keeper"} 8.470456e+06
pipeline_tasks_total_finished{job_id="1",job_name="example keeper spec",status="completed",task_type="keeper"} 1

Fixed

• It is no longer required to set DEFAULT_HTTP_ALLOW_UNRESTRICTED_NETWORK_ACCESS=true to enable local fetches on bridge tasks. Please remove this if you had it set and no longer need it, since it introduces a slight security risk.
• Chainlink can now run with ETH_DISABLED=true without spewing errors everywhere

Changed

• The v2 (TOML) bridge task's includeInputAtKey parameter is being deprecated in favor of variable interpolation. Please migrate your jobs to the new syntax as soon as possible.

• Chainlink no longers writes/reads eth key files to disk

• Add sensible default configuration settings for Fantom

• Rename ETH_MAX_UNCONFIRMED_TRANSACTIONS to ETH_MAX_QUEUED_TRANSACTIONS. It still performs the same function but the name was misleading and would have caused confusion with the new ETH_MAX_IN_FLIGHT_TRANSACTIONS.

Fixed

• If a CLI command is issued after the session has expired, and an api credentials file is found, auto login should now work.

• GasUpdater now works on RSK and xDai

• Offchain reporting jobs that have had a latest round requested can now be deleted from the UI without error

Added

• Add ETH_GAS_LIMIT_MULTIPLIER configuration option, the gas limit is multiplied by this value before transmission. So a value of 1.1 will add 10% to the on chain gas limit when a transaction is submitted.

• Add ETH_MIN_GAS_PRICE_WEI configuration option. This defaults to 1Gwei on mainnet. Chainlink will never send a transaction at a price lower than this value.

• Add chainlink node db migrate for running database migrations. It's
  recommended to use this and set MIGRATE_DATABASE=false if you want to run
  the migrations separately outside of application startup.

Changed

• Chainlink now automatically cleans up old eth_txes to reduce database size. By default, any eth_txes older than a week are pruned on a regular basis. It is recommended to use the default value, however the default can be overridden by setting the ETH_TX_REAPER_THRESHOLD env var e.g. ETH_TX_REAPER_THRESHOLD=24h. Reaper can be disabled entirely by setting ETH_TX_REAPER_THRESHOLD=0. The reaper will run on startup and again every hour (interval is configurable using ETH_TX_REAPER_INTERVAL).

• Heads corresponding to new blocks are now delivered in a sampled way, which is to improve
  node performance on fast chains. The frequency is by default 1 second, and can be changed
  by setting ETH_HEAD_TRACKER_SAMPLING_INTERVAL env var e.g. ETH_HEAD_TRACKER_SAMPLING_INTERVAL=5s.

• Database backups: default directory is now a subdirectory 'backup' of chainlink root dir, and can be changed
  to any chosed directory by setting a new configuration value: DATABASE_BACKUP_DIR