ESP RainMaker Admin CLI

Introduction

ESP RainMaker Admin CLI is a tool offered by Espressif Rainmaker for admin users to be able to perform mass manufacturing of nodes of ESP32-S2 and ESP32 based products. This tool will enable you to perform node id generation and certificate registration operations required for the manufacturing process.

Getting the Admin CLI

Clone this project using:

git clone https://github.com/espressif/esp-rainmaker-admin-cli.git

Setup Build Environment

Note: If you are using esp-idf, the python and virtual environment requirement would already be fulfilled (via install.sh), and you can just install the dependencies using python -m pip install -r requirements.txt and move on to the next section which speaks about the workflow.

Operating System requirements

Linux / MacOS / Windows (standard distributions)

Other requirements

To setup your build environment, please make sure you have the following installed on your host machine:

python (If not installed, please refer to https://www.python.org/)
pip (If not present, please refer to https://pip.pypa.io/en/stable/)
virtualenv (You can install using command - pip install virtualenv). This is not mandatory, but recommended so that rest of your python based utilities do not break.

The following python versions are supported:

python 3.5.x
python 3.6.x
python 3.7.x
python 3.8.x

Installing dependencies

Once python and pip are installed, set up the virtual environment by following the instructions here. Thereafter, please enter the directory where this tool is installed (using terminal) and execute the below to install the dependencies:

pip install -r requirements.txt

python -m pip install -r requirements.txt

Workflow

You need to perform the following steps to generate and register node credentials. To know more about the commands in detail, please refer the Usage section below.

Set Server Configuration:
python rainmaker_admin_cli.py account serverconfig --endpoint <endpoint>
Login:
python rainmaker_admin_cli.py account login --email <email_id>
Generate Device Certificate(s):
python rainmaker_admin_cli.py certs devicecert generate --count <count>
Register Generated Device Certificate(s):
python rainmaker_admin_cli.py certs devicecert register --inputfile <inputfile>
Check Device Certificate Registration Status:
python rainmaker_admin_cli.py certs devicecert getcertstatus --requestid <request_id>

Usage

To get help information for any RainMaker Admin CLI command or sub-command, you can use the -h option at various levels

Eg.

python rainmaker_admin_cli.py -h
python rainmaker_admin_cli.py account -h
python rainmaker_admin_cli.py account login -h

The Admin CLI commands are divided into 2 broad categories

Account Operations
Certificate Operations

You need to setup the account before you can move on to the Certificate Operations

Account Operations

Server Config

You need to setup server configuration to get started. The endpoint would be your deployment's Base URL of the form https://xxxx/amazonaws.com/dev which you would have received on the super admin email configured during RainMaker deployment.

Usage:

python rainmaker_admin_cli.py account serverconfig --endpoint <endpoint>

Login

You need to login to get started and use the subsequent APIs. The email id for login would be the super admin user email configured during RainMaker deployment. The password should have been already received on that email at the end of the backend deployment process.

Usage:

python rainmaker_admin_cli.py account login --email <emailid>

Note: Login configuration will be stored at location ~/.espressif/rainmaker/rainmaker_admin_config.json

You can now use the rest of the commands once you have logged in successfully.

Device Certificate Operations

You can perform the following operations for the device certificate.

generate - You can generate multiple device certificates at a time.
register - You can register multiple generated device certificates.
getcertstatus - Once you register the device certificates, you can check the device certificate registration status.

Generate Device Certificates

This will generate the private keys and certificates required by the RainMaker nodes to connect to your deployment. It will also set other information like the node ids, mqtt endpoint, etc.

Notes:

This will also create the CA key and certificate that would be used for signing the device certificates. If you already have your own CA key and certificate, you can provide it explicitly.

If you want the Provisioning QR codes to be generated as well, please use the --prov option, and pass appropriate transport. Generally, the default is "ble" for all chips that support BLE (ESP32, ESP32-C3) and "softap" for the ones that do not (ESP32-S2). However, this primarily depends on what you have used in your firmware.

Usage:

python rainmaker_admin_cli.py certs devicecert generate [-h] [--outdir <outdir>] [--count <count>]
                                                        [--cacertfile <cacertfile>] [--cakeyfile <cakeyfile>]
                                                        [--prov <prov_type>] [--fileid <fileid>]
                                                        [--local <local>] [--inputfile <inputfile>]

optional arguments:
  -h, --help            show this help message and exit
  --outdir <outdir>     Path to output directory. Files generated will be saved in <outdir>
                        If directory does not exist, it will be created
                        Default: current directory
  --count <count>       Number of Node Ids for generating certificates
                        Default: 0
  --cacertfile <cacertfile>
                        Path to file containing CA Certificate
  --cakeyfile <cakeyfile>
                        Path to file containing CA Private Key
  --prov <prov_type>    Provisioning type to generate QR code 
                        (softap/ble)
  --fileid <fileid>     File identifier 
                        Used to identify file for each node uniquely (used as filename suffix)
                        Default: <node_id> (The node id's generated)
                        If provided, eg. `mac_addr`(MAC address),
                        must be part of ADDITIONAL_VALUES file (provided in config)
                        and must have <count> values in the file (for each node)
  --local <local>       This is to determine whether or not to generate node ids locally.
                        Default: false
  --inputfile <csvfile> This is the node_ids.csv file containing pre-generated node ids.

For generating the node Ids locally without the rainmaker login: python rainmaker_admin_cli.py certs devicecert generate --count 5 --prov ble --outdir test --local true

For generating the node certificates by providing pre-generated node ids csv file: python rainmaker_admin_cli.py certs devicecert generate --count 5 --prov ble --outdir test --local true --inputfile <node_ids.csv>

Note that in this command, count and local argument will be ignored and inputfile will get the precendence.

For simplest use case, the usage is as given below. If you want to add some custom data or customise some other parameters, please refer the subsequent sections.

Note that it is better to first create a small set of certificates, say 5, so that you get an idea about how the tool works.

Example: python rainmaker_admin_cli.py certs devicecert generate --count 5 --prov ble --outdir test --local true --inputfile <node_ids.csv>

Sample result for 2 nodes is as below :

  test
  └── 2020-11-29
     └── Mfg-00001
        ├── bin
        │   ├── node-00001-T2uNDXPMS9nj9vpKjs2QG8.bin
        │   └── node-00002-dRagJ6GBim2HE5ENQ5nbYG.bin
        ├── common
        │   ├── ca.crt
        │   ├── ca.key
        │   ├── config.csv
        │   ├── endpoint.txt
        │   ├── node_certs.csv
        │   ├── node_ids.csv
        │   └── values.csv
        ├── node_details
        │   ├── node-00001-T2uNDXPMS9nj9vpKjs2QG8
        │   │   ├── node.crt
        │   │   ├── node.key
        │   │   ├── node_info.csv
        │   │   ├── qrcode.txt
        │   │   └── random.txt
        │   └── node-00002-dRagJ6GBim2HE5ENQ5nbYG
        │       ├── node.crt
        │       ├── node.key
        │       ├── node_info.csv
        │       ├── qrcode.txt
        │       └── random.txt
        └── qrcode
              ├── node-00001-T2uNDXPMS9nj9vpKjs2QG8.png
              └── node-00002-dRagJ6GBim2HE5ENQ5nbYG.png

The output directory will have the following sub-directory structure:

<outdir>/<current_date>/Mfg-<no>
- Sub-directory with the current date is created.
- A Mfg-<no> sub-directory will be created where <no> is the batch number (which increments on each CLI run).

The output directory contains the following files:

bin/: For each device certificate, the corresponding NVS partition binaries are generated in this directory, which can be used to flash onto the device. File format: node-<index>-<node_id>.bin
common/: This has some common files that are generated during the process
- ca.crt: CA Certificate.
- ca.key: CA Key.
- endpoint.txt: MQTT Endpoint for this deployment.
- node_certs.csv : CSV for all the Node Certificates in this batch to be registered to the cloud.
- node_ids.csv : CSV for all node ids generated in this batch.
- config.csv : The NVS configuration file as per the format defined here for the IDF Manufacturing Utility.
- values.csv : Master file with all the values for all the nodes as per the format defined here for the IDF Manufacturing Utility.
node_details/: All node details are stored in this directory. Following details for each node are stored in node_details/node-<index>-<node_id> directory:
- node.crt: Device Certificates.
- node.key: Private key for each device certificate.
- node_info.csv: The csv file used as configuration to generate the binary.
- qrcode.txt: The QR code payload (used during provisioning, available only if --prov is given).
- random.txt: The random bytes information (used to generate device name and PoP, available only if --prov is given).
- node_encr_key.bin: Encryption key (if encryption is enabled in config).
qrcode/: QR code images for all nodes are stored in this directory (used during provisioning, available only if --prov is given). File format: node-<index>-<node_id>.png

Adding Custom Data

There could often be a requirement to add some custom data to the nvs binaries generated. Such custom data can be added using the formats specified by the ESP IDF Manufacturing Utility. The config file and values file can be given as input by setting the ADDITIONAL_CONFIG and ADDITIONAL_VALUES fields in config/binary_config.ini. Please check out samples for such files at samples/extra_config.csv and samples/extra_values.csv

Register Device Certificates

Once the device certificates are generated, they also need to be registered with the cloud service using the register command.

Usage:

python rainmaker_admin_cli.py certs devicecert register [-h] --inputfile <csvfilename>
                                                        [--groupname <nodegroupname>] [--type <nodetype>]
                                                        [--model <nodemodel>] [--parent_groupname <parent_groupname>][--subtype <nodesubtype>]
                                                        [--tags <nodetags>]

optional arguments:
  -h, --help            show this help message and exit
  --inputfile <csvfilename>
                        Name of file containing node ids and certs
  --groupname <nodegroupname>
                        Name of the group to which node are to be added after successful registration
  --type <nodetype>     Node type
  --model <nodemodel>   Node model
  --parent_groupname <parent_groupname> 
                        Name of the parent group to which this newly created group will be a child group      
  --subtype <nodesubtype> Node SubType
  --tags <nodetags> Comma separated strings of tags to be attached to the nodes.(eg: location:Pune,office:espressif)

For the example in device certificate generation section the node_certs_file file would be test/2020-11-29/Mfg-00001/common/node_certs.csv. This command will give a request id in response, which can be used for monitoring the status.

Adding Tags

Basic Tags can be added to nodes using the --tags option. Using --tags loc:Amsterdam will add this same tag to all the nodes in node_certs.csv.

Note that for adding tags, minimum rainmaker supported version is 1.1.27.

If different tag values are to be used for each node in the node_certs.csv file, use --tags loc:@city, where city should be a column in the node_certs.csv.

For example: node_certs.csv:

node_id	certs	city
node1	cert1	Amsterdam
node2	cert2	Barcelona

After passing --tags loc:@city,

node1 will get the tag: loc:Amsterdam.
node2 will get the tag: loc:Barcelona.

To use CSV based tags, i.e city:@loc where loc is a column in the CSV, the minimum rainmaker supported version is 1.1.28.

Check Device Certificate Registration Status

The certificate registration process can take significant time. One it is finished, the super admin user will get an email with the status. The same can also be checked using the getcertstatus command.

Usage:

python rainmaker_admin_cli.py certs devicecert getcertstatus --requestid XXXXXXX

Please check the output of the register command above to get the request id.

CA Certificate Operations

The steps here would generally not be required. However, if you want to explicitly create a new CA certificate and use it for signing the device certificates, you can use this command.

Usage:

python rainmaker_admin_cli.py certs cacert generate --outdir <outdir>

This will generate the CA key and certificate at following locations:

<outdir>/<current_date>/Mfg-<no>/common/ca.key
<outdir>/<current_date>/Mfg-<no>/common/ca.crt

These can be used for signing the device certificates by passing these via the --cacertfile and --cakeyfile options for rainmaker_admin_cli.py certs devicecert generate

Flashing

To flash binary generated onto the device, you can use the following command:

esptool.py --port <port> write_flash <fctry_address> <outdir>/bin/<filename>.bin

Note: The <fctry_address> is typically 0x340000. However, please check your partition table to find the appropriate address.

The esptool.py would be available in your PATH only if you have esp-idf set up, else, please find it at esp-idf/components/esptool_py/esptool/esptool.py and use from there.