NEAR Protocol Indexer pumping data in Redis for https://github.com/vgrichina/fast-near
GPL-3.0 License
NEAR State Indexer is built on top of NEAR Indexer microframework to watch the network and store all the state changes in the Redis database. It can be consumed by https://github.com/vgrichina/fast-near to run NEAR RPC service.
Before you proceed, make sure you have the following software installed:
Clone this repository and open the project folder
$ git clone [email protected]:near/near-indexer-for-explorer.git
$ cd near-indexer-for-explorer
You need to provide database credentials in .env
file like below (replace Redis connection string as needed):
$ echo "REDIS_URL=redis://127.0.0.1/" > .env
To connect the specific chain you need to have necessary configs, you can generate it as follows:
$ cargo run --release -- --home-dir ~/.near/testnet init --chain-id testnet --download-config
The above code will download the official genesis config and generate necessary configs. You can replace testnet
in the command above to different network ID (betanet
, mainnet
).
NB! According to changes in nearcore
config generation we don't fill all the necessary fields in the config file.
While this issue is open https://github.com/nearprotocol/nearcore/issues/3156 you need to download config you want and replace the generated one manually.
Configs for the specified network are in the --home-dir
provided folder. We need to ensure that NEAR State Indexer follows
all the necessary shards, so "tracked_shards"
parameters in ~/.near/testnet/config.json
needs to be configured properly.
For example, with a single shared network, you just add the shard #0 to the list:
...
"tracked_shards": [0],
...
Command to run NEAR State Indexer have to contain sync mode.
You can choose NEAR State Indexer sync mode by setting what to stream:
sync-from-latest
- start indexing blocks from the latest finalized blocksync-from-interruption --delta <number_of_blocks>
- start indexing blocks from the block NEAR Indexer was interrupted last time but earlier for <number_of_blocks>
if providedsync-from-block --height <block_height>
- start indexing blocks from the specific block heightOptionally you can tell Indexer to store data from genesis (Accounts and Access Keys) by adding key --store-genesis
to the run
command.
NEAR State Indexer works in strict mode by default, but you can disable it for specific amount of blocks. The strict mode means that every piece of data will be retried to store to database in case of error. Errors may occur when the parent piece of data is still processed but the child piece is already trying to be stored. So Indexer keeps retrying to store the data until success. However if you're running Indexer not from the genesis it is possible that you really miss some of parent data and it'll be impossible to store child one, so you can disable strict mode for 1000 blocks to ensure you've passed the strong relation data area and you're running Indexer where it is impossible to loose any piece of data.
To disable strict mode you need to provide:
--non-strict-mode
Sometimes you may want to index block while sync process is happening, by default an indexer node is waiting for full sync to complete but you can enable indexing while the node is syncing by passing --stream-while-syncing
By default NEAR State Indexer processes only a single block at a time. You can adjust this with the --concurrency
argument (when the blocks are mostly empty, it is fine to go with as many as 100 blocks of concurrency).
So final command to run NEAR State Indexer can look like:
$ cargo run --release -- --home-dir ~/.near/testnet run --store-genesis --stream-while-syncing --non-strict-mode --stop-after-number-of-blocks 1000 --concurrency 1 sync-from-latest
After the network is synced, you should see logs of every block height currently received by NEAR State Indexer.
Whenever you run NEAR State Indexer for any network except localnet you'll need to sync with the network. This is required because it's a natural behavior of nearcore
node and NEAR State Indexer is a wrapper for the regular nearcore
node. In order to work and index the data your node must be synced with the network. This process can take a while, so we suggest to download a fresh backup of the data
folder and put it in you --home-dir
of your choice (by default it is ~/.near
)
Running your NEAR State Indexer node on top of a backup data will reduce the time of syncing process because your node will download only missing data and it will take reasonable time.
All the backups can be downloaded from the public S3 bucket which contains latest daily snapshots:
It's not necessary but in order to index everything in the network it is better to do it from the genesis. nearcore
node is running in non-archival mode by default. That means that the node keeps data only for 5 last epochs. In order to index data from the genesis we need to turn the node in archival mode.
To do it we need to update config.json
located in --home-dir
or your choice (by default it is ~/.near
).
Find next keys in the config and update them as following:
{
...
"archive": true,
"tracked_shards": [0],
...
}
The syncing process in archival mode can take a lot of time, so it's better to download a backup provided by NEAR and put it in your data
folder. After that your node will need to sync only missing data and it should take reasonable time.
All the backups can be downloaded from the public S3 bucket which contains latest daily snapshots:
See https://docs.near.org/docs/roles/integrator/exchange-integration#running-an-archival-node for reference
If you want to play with the code locally, it's better not to copy existing mainnet/testnet (it requires LOTS of memory), but to have your own small example.
Go through steps above until (by not including) init
command.
Then use init
command with different arguments,
$ cargo run --release -- --home-dir ~/.near/localnet init --chain-id localnet
Edit ~/.near/localnet/config.json
by adding tracking shards and archiving option (see example above).
$ cargo run -- --home-dir ~/.near/localnet run --store-genesis sync-from-latest
Congrats, the blocks are being produced right now! There should be some lines in the DB. Now, we need to generate some activity to add new examples.
$ npm i -g near-cli
$ NEAR_ENV=local near create-account awesome.test.near --initialBalance 30 --masterAccount test.near --keyPath=~/.near/localnet/validator_key.json
$ NEAR_ENV=local near send test.near awesome.test.near 5
All available commands are here.
You can stop and re-run the example at any time. Blocks will continue producing from the last state.
When operating normally, you should see "INFO indexer_for_explorer: Block height ..." messages in the logs.
Make sure the blocks you want to save exist on the node. Check them via JSON RPC:
curl http://127.0.0.1:3030/ -X POST --header 'Content-type: application/json' --data '{"jsonrpc": "2.0", "id": "dontcare", "method": "block", "params": {"block_id": 9820214}}'
NOTE: Block #9820214 is the first block after genesis block (#9820210) on Mainnet.
If it returns an error that the block does not exist or missing, it means that your node does not have the necessary data. Your options here are to start from the blocks that are recorded on the node or start an archival node (see above) and make sure you have the full network history (either use a backup or let the node sync from scratch (it is quite slow, so backup is recommended))