chainlink

Fixed

• If a CLI command is issued after the session has expired, and an api credentials file is found, auto login should now work.

• GasUpdater now works on RSK and xDai

• Offchain reporting jobs that have had a latest round requested can now be deleted from the UI without error

Added

• Add ETH_GAS_LIMIT_MULTIPLIER configuration option, the gas limit is multiplied by this value before transmission. So a value of 1.1 will add 10% to the on chain gas limit when a transaction is submitted.

• Add ETH_MIN_GAS_PRICE_WEI configuration option. This defaults to 1Gwei on mainnet. Chainlink will never send a transaction at a price lower than this value.

• Add chainlink node db migrate for running database migrations. It's
  recommended to use this and set MIGRATE_DATABASE=false if you want to run
  the migrations separately outside of application startup.

Changed

• Chainlink now automatically cleans up old eth_txes to reduce database size. By default, any eth_txes older than a week are pruned on a regular basis. It is recommended to use the default value, however the default can be overridden by setting the ETH_TX_REAPER_THRESHOLD env var e.g. ETH_TX_REAPER_THRESHOLD=24h. Reaper can be disabled entirely by setting ETH_TX_REAPER_THRESHOLD=0. The reaper will run on startup and again every hour (interval is configurable using ETH_TX_REAPER_INTERVAL).

• Heads corresponding to new blocks are now delivered in a sampled way, which is to improve
  node performance on fast chains. The frequency is by default 1 second, and can be changed
  by setting ETH_HEAD_TRACKER_SAMPLING_INTERVAL env var e.g. ETH_HEAD_TRACKER_SAMPLING_INTERVAL=5s.

• Database backups: default directory is now a subdirectory 'backup' of chainlink root dir, and can be changed
  to any chosed directory by setting a new configuration value: DATABASE_BACKUP_DIR

Added

• Add MockOracle.sol for testing contracts

• New CLI command to convert v1 flux monitor jobs (JSON) to
  v2 flux monitor jobs (TOML). Running it will archive the v1
  job and create a new v2 job. Example:

// Get v1 job ID:
chainlink job_specs list
// Migrate it to v2:
chainlink jobs migrate fe279ed9c36f4eef9dc1bdb7bef21264

// To undo the migration:
1. Archive the v2 job in the UI
2. Unarchive the v1 job manually in the db:
update job_specs set deleted_at = null where id = 'fe279ed9-c36f-4eef-9dc1-bdb7bef21264'

• Improved support for Optimism chain. Added a new boolean OPTIMISM_GAS_FEES configuration variable which makes a call to estimate gas before all transactions, suitable for use with Optimism's L2 chain. When this option is used ETH_GAS_LIMIT_DEFAULT is ignored.

• Chainlink now supports routing certain calls to the eth node over HTTP instead of websocket, when available. This has a number of advantages - HTTP is more robust and simpler than websockets, reducing complexity and allowing us to make large queries without running the risk of hitting websocket send limits. The HTTP url should point to the same node as the ETH_URL and can be specified with an env var like so: ETH_HTTP_URL=https://my.ethereumnode.example/endpoint.

Adding an HTTP endpoint is particularly recommended for BSC, which is hitting websocket limitations on certain queries due to its large block size.

Added

• Add MockOracle.sol for testing contracts
• Cron jobs can now be created for the v2 job pipeline:

type            = "cron"
schemaVersion   = 1
schedule        = "*/10 * * * *"
observationSource   = """
ds          [type=http method=GET url="http://example.com"];
ds_parse    [type=jsonparse path="data"];
ds -> ds_parse;
"""

Changed

• Default for JOB_PIPELINE_REAPER_THRESHOLD has been reduced from 1 week to 1 day to save database space. This variable controls how long past job run history for OCR is kept. To keep the old behaviour, you can set JOB_PIPELINE_REAPER_THRESHOLD=168h
• Removed support for the env var JOB_PIPELINE_PARALLELISM.
• OCR jobs no longer show TaskRuns in success cases. This reduces
  DB load and significantly improves the performance of archiving OCR jobs.
• Archiving OCR jobs should be 5-10x faster.

Fixed

• Added GAS_UPDATER_BATCH_SIZE option to workaround websocket: read limit exceeded issues on BSC

• Basic support for Optimism chain: node no longer gets stuck with 'nonce too low' error if connection is lost

[0.10.4] - 2021-04-05

Added

• VRF Jobs now support an optional coordinatorAddress field that, when present, will tell the node to check the fulfillment status of any VRF request before attempting the fulfillment transaction. This will assist in the effort to run multiple nodes with one VRF key.

• Experimental: Add DATABASE_BACKUP_MODE, DATABASE_BACKUP_FREQUENCY and DATABASE_BACKUP_URL configuration variables

It's now possible to configure database backups: on node start and separately, to be run at given frequency.

DATABASE_BACKUP_MODE enables the initial backup on node start (with one of the values: none, lite, full where lite excludes
potentially large tables related to job runs, among others). Additionally, if DATABASE_BACKUP_FREQUENCY variable is set to a duration of
at least '1m', it enables periodic backups.

DATABASE_BACKUP_URL can be optionally set to point to e.g. a database replica, in order to avoid excessive load on the main one.

Example settings:

DATABASE_BACKUP_MODE="full" and DATABASE_BACKUP_FREQUENCY not set, will run a full back only at the start of the node.
DATABASE_BACKUP_MODE="lite" and DATABASE_BACKUP_FREQUENCY="1h" will lead to a partial backup on node start and then again a partial backup every one hour.

• Added periodic resending of eth transactions. This means that we no longer rely exclusively on gas bumping to resend unconfirmed transactions that got "lost" for whatever reason. This has two advantages:
  1. Chainlink no longer relies on gas bumping settings to ensure our transactions always end up in the mempool
  2. Chainlink will continue to resend existing transactions even in the event that heads are delayed. This is especially useful on chains like Arbitrum which have very long wait times between heads.

Periodic resending can be controlled using the ETH_TX_RESEND_AFTER_THRESHOLD env var (default 30s). Unconfirmed transactions will be resent periodically at this interval. It is recommended to leave this at the default setting, but it can be set to any valid duration or to 0 to disable periodic resending.

• Logging can now be configured in the Operator UI.

Fixed

• Chainlink node now automatically sets the correct nonce on startup if you are restoring from a previous backup (manual setnextnonce is no longer necessary).

• Flux monitor jobs should now work correctly with outlier-detection and market-closure external adapters.

• Performance improvements to OCR job adds. Removed the pipeline_task_specs table
  and added a new column dot_id to the pipeline_task_runs table which links a pipeline_task_run
  to a dotID in the pipeline_spec.dot_dag_source.

• Fixed bug where node will occasionally submit an invalid OCR transmission which reverts with "address not authorized to sign".

• Fixed bug where a node will sometimes double submit on runlog jobs causing reverted transactions on-chain

Added

• Add ADMIN_CREDENTIALS_FILE configuration variable

This variable defaults to $ROOT/apicredentials and when defined / the
file exists, any command using the CLI that requires authentication will use it
to automatically log in.

• Add ETH_MAX_UNCONFIRMED_TRANSACTIONS configuration variable

Chainlink node now has a maximum number of unconfirmed transactions that
may be in flight at any one time (per key).

If this limit is reached, further attempts to send transactions will fail
and the relevant job will be marked as failed.

Jobs will continue to fail until at least one transaction is confirmed
and the queue size is reduced. This is introduced as a sanity limit to
prevent unbounded sending of transactions e.g. in the case that the eth
node is failing to broadcast to the network.

The default is set to 500 which considered high enough that it should
never be reached under normal operation. This limit can be changed
by setting the ETH_MAX_UNCONFIRMED_TRANSACTIONS environment variable.

• Support requestNewRound in libocr

requestNewRound enables dedicated requesters to request a fresh report to
be sent to the contract right away regardless of heartbeat or deviation.

• New prometheus metric:

Name: "head_tracker_eth_connection_errors",
Help: "The total number of eth node connection errors",

• Gas bumping can now be disabled by setting ETH_GAS_BUMP_THRESHOLD=0

• Support for arbitrum

Fixed

• Improved handling of the case where we exceed the configured TX fee cap in
  geth.

Node will now fatally error jobs if the total transaction costs exceeds the
configured cap (default 1 Eth). Also, it will no longer continue to bump gas on
transactions that started hitting this limit and instead continue to resubmit
at the highest price that worked.

Node operators should check their geth nodes and remove this cap if configured,
you can do this by running your geth node with --rpc.gascap=0 --rpc.txfeecap=0 or setting these values in your config toml.

• Make head backfill asynchronous. This should eliminate some harmless but
  annoying errors related to backfilling heads, logged on startup and
  occasionally during normal operation on fast chains like Kovan.

• Improvements to the GasUpdater

Various efficiency and correctness improvements have been made to the
GasUpdater. It places less load on the ethereum node and now features re-org
detection.

Most notably, GasUpdater no longer takes a 24 block delay to "warm up" on
application start and instead loads all relevant block history immediately.
This means that the application gas price will always be updated correctly
after reboot before the first transaction is ever sent, eliminating the previous
scenario where the node could send underpriced or overpriced transactions for a
period after a reboot, until the gas updater caught up.

Changed

• Bump ORM_MAX_OPEN_CONNS default from 10 to 20
• Bump ORM_MAX_IDLE_CONNS default from 5 to 10

Each Chainlink node will now use a maximum of 23 database connections (up from previous max of 13). Make sure your postgres database is tuned accordingly, especially if you are running multiple Chainlink nodes on a single database. If you find yourself hitting connection limits, you can consider reducing ORM_MAX_OPEN_CONNS but this may result in degraded performance.

• The global env var JOB_PIPELINE_MAX_TASK_DURATION is no longer supported
  for OCR jobs.

Fixed

• Fix a potential hang in headtracker under high database load
• Fix a performance regression introduced in 0.10.0 in the v1 job runner

Fixed

• Prevent autosaving Task Spec on when Task Runs are saved to lower database load.

Fixed

• Fix a case where archiving jobs could try to delete it from the external initiator even if the job was not an EI job.
• Fixed a fluxmonitor job bug where submitting a value outside the acceptable range would stall the job permanently. Now a job spec error will be thrown if the polled answer is outside the acceptable range and no ethtx will be submitted. As additional protection, we also now check the receipts of the ethtx's and if they were reverted, we mark the ethtx task as failed.
• Improved performance of the transaction manager by fetching receipts in batches. This should help prevent the node from getting stuck when processing large numbers of OCR jobs.
• Squashed migrations into a single 1_initial migration. If you were running a version older than 0.9.10, you need to upgrade to 0.9.10 first before upgrading to the next version so that the migrations are run.

Added

• A new Operator UI feature that visualize JSON and TOML job spec tasks on a 'New Job' page.

Fixed

• Fixed a UI bug with fluxmonitor jobs where initiator params were bunched up.
• Improved performance of OCR jobs to reduce database load. OCR jobs now run with unlimited parallelism and are not affected by JOB_PIPELINE_PARALLELISM.

Added

• A new env var JOB_PIPELINE_MAX_RUN_DURATION has been added which controls maximum duration of the total run.

Added

• New CLI commands for key management:
  • chainlink keys eth import
  • chainlink keys eth export
  • chainlink keys eth delete
• All keys other than VRF keys now share the same password. If you have OCR, P2P, and ETH keys encrypted with different passwords, re-insert them into your DB encrypted with the same password prior to upgrading.

Fixed

• Fixed reading of function selector values in DB.
• Support for bignums encoded in CBOR
• Silence spurious Job spawner ORM attempted to claim locally-claimed job warnings
• OCR now drops transmissions instead of queueing them if the node is out of Ether
• Fixed a long-standing issue where standby nodes would hold transactions open forever while waiting for a lock. This was preventing postgres from running necessary cleanup operations, resulting in bad database performance. Any node operators running standby failover chainlink nodes should see major database performance improvements with this release and may be able to reduce the size of their database instances.
• Fixed an issue where expired session tokens in operator UI would cause a large number of reqeusts to be sent to the node, resulting in a temporary rate-limit and 429 errors.
• Fixed issue whereby http client could leave too many open file descriptors

Changed

• Key-related API endpoints have changed. All key-related commands are now namespaced under /v2/keys/..., and are standardized across key types.
• All key deletion commands now perform a soft-delete (i.e. archive) by default. A special CLI flag or query string parameter must be provided to hard-delete a key.
• Node now supports multiple OCR jobs sharing the same peer ID. If you have more than one key in your database, you must now specify P2P_PEER_ID to indicate which key to use.
• DATABASE_TIMEOUT is now set to 0 by default, so that nodes will wait forever for a lock. If you already have DATABASE_TIMEOUT=0 set explicitly in your env (most node operators) then you don’t need to do anything. If you didn’t have it set, and you want to keep the old default behaviour where a node exits shortly if it can’t get a lock, you can manually set DATABASE_TIMEOUT=500ms in your env.

Fixed

• An issue where the node would emit warnings on startup for fluxmonitor contracts

Added

• OCR bootstrap node now sends telemetry to the endpoint specified in the OCR job spec under MonitoringEndpoint.
• Adds "Account addresses" table to the /keys page.

Changed

• Old jobs now allow duplicate job names. Also, if the name field is empty we no longer generate a name.

Fixed

• Brings /runs tab back to the operator UI.
• Signs out a user from operator UI on authentication error.

Changes

• Removes broken ACCOUNT_ADDRESS field from /config page.

BREAKING CHANGES

• Commands for creating/managing legacy jobs and OCR jobs have changed, to reduce confusion and accomodate additional types of jobs using the new pipeline.

V1 jobs

jobs archive => job_specs archive
jobs create => job_specs create
jobs list => job_specs list
jobs show => job_specs show

V2 jobs (currently only applies to OCR)

jobs createocr => jobs create
jobs deletev2 => jobs delete
jobs run => jobs run

Changed

Numerous key-related UX improvements:

• All key-related commands have been consolidated under the chainlink keys subcommand:
  • chainlink createextrakey => chainlink keys eth create
  • chainlink admin info => chainlink keys eth list
  • chainlink node p2p [create|list|delete] => chainlink keys p2p [create|list|delete]
  • chainlink node ocr [create|list|delete] => chainlink keys ocr [create|list|delete]
  • chainlink node vrf [create|list|delete] => chainlink keys vrf [create|list|delete]
• Deleting OCR key bundles and P2P key bundles now archives them (i.e., soft delete) so that they can be recovered if needed. If you want to hard delete a key, pass the new --hard flag to the command, e.g. chainlink keys p2p delete --hard 6.
• Output from ETH/OCR/P2P/VRF key CLI commands now renders consistently.
• Deleting an OCR/P2P/VRF key now requires confirmation from the user. To skip confirmation (e.g. in shell scripts), pass --yes or -y.
• The --ocrpassword flag has been removed. OCR/P2P keys now share the same password at the ETH key (i.e., the password specified with the --password flag).
• Two new env variables are added P2P_ANNOUNCE_IP and P2P_ANNOUNCE_PORT which allow node operators to override locally detected values for the chainlink node's externally reachable IP/port.
• OCR_LISTEN_IP and OCR_LISTEN_PORT have been renamed to P2P_LISTEN_IP and P2P_LISTEN_PORT for consistency.
• Support for adding a job with the same name as one that was deleted.

Fixed

• Fixed an issue where the HTTP adapter would send an empty body on retries.

Fixed

• Hotfix to fix an issue with httpget adapter

Added

• Add new subcommand node hard-reset which is used to remove all state for unstarted and pending job runs from the database.

Changed

• Chainlink now requires Postgres >= 11.x. Previously this was a recommendation, this is now a hard requirement. Migrations will fail if run on an older version of Postgres.
• Database improvements that greatly reduced the number of open Postgres connections
• Operator UI /jobs page is now searchable
• Jobs now accept a name field in the jobspecs

Added

• Bulletproof transaction manager enabled by default
• Fluxmonitor support enabled by default
• Remove configuration option ORACLE_CONTRACT_ADDRESS, it had no effect
• Add configuration option OPERATOR_CONTRACT_ADDRESS, it filters the contract addresses the node should listen to for Run Logs
• At startup, the chainlink node will create a new funding address. This will initially be used to pay for cancelling stuck transactions.

Fixed

• Improve transaction manager architecture to be more compatible with ETH_SECONDARY_URL option (i.e. concurrent transaction submission to multiple different eth nodes). This also comes with some minor performance improvements in the tx manager and more correct handling of some extremely rare edge cases.
• As a side-effect, we now no longer handle the case where an external wallet used the chainlink ethereum private key to send a transaction. This use-case was already explicitly unsupported, but we made a best-effort attempt to handle it. We now make no attempt at all to handle it and doing this WILL result in your node not sending the data that it expected to be sent for the nonces that were used by an external wallet.
• Operator UI now shows booleans correctly

Changed

• ETH_MAX_GAS_PRICE_WEI now 1500Gwei by default

Added

• Bulletproof transaction manager enabled by default
• Fluxmonitor support enabled by default
• Remove configuration option ORACLE_CONTRACT_ADDRESS, it had no effect
• Add configuration option OPERATOR_CONTRACT_ADDRESS, it filters the contract addresses the node should listen to for Run Logs
• At startup, the chainlink node will create a new funding address. This will initially be used to pay for cancelling stuck transactions.

Fixed

• Improve transaction manager architecture to be more compatible with ETH_SECONDARY_URL option (i.e. concurrent transaction submission to multiple different eth nodes). This also comes with some minor performance improvements in the tx manager and more correct handling of some extremely rare edge cases.
• As a side-effect, we now no longer handle the case where an external wallet used the chainlink ethereum private key to send a transaction. This use-case was already explicitly unsupported, but we made a best-effort attempt to handle it. We now make no attempt at all to handle it and doing this WILL result in your node not sending the data that it expected to be sent for the nonces that were used by an external wallet.

Changed

• ETH_MAX_GAS_PRICE_WEI now 1500Gwei by default

Fixed

• Prometheus gas_updater_set_gas_price metric now only shows last gas price instead of every block since restart

[0.8.17] - 2020-09-28

Added

• Add new env variable ETH_SECONDARY_URL. Default is unset. You may optionally set this to an http(s) ethereum RPC client URL. If set, transactions will also be broadcast to this secondary ethereum node. This allows transaction broadcasting to be more robust in the face of primary ethereum node bugs or failures.

Fixed

• Gas bumper no longer hits database constraint error if ETH_MAX_GAS_PRICE_WEI is reached (this was actually mostly harmless, but the errors were annoying)

Added

• The chainlink node now will bump a limited configurable number of transactions at once. This is configured with the ETH_GAS_BUMP_TX_DEPTH variable which is 10 by default. Set to 0 to disable (the old behaviour).

Fixed

• ETH_DISABLED flag works again