Using Vault by HashiCorp to secure your deployment secrets

Educational repository demonstrating approaches for safe secure deployment of passwords, api keys etc

Updates

Batches were updated to conform with vault API versions 0_9 and 0_10 ; If your vault is less than 0.10.0 , - you need to checkout 0.9.0 branch.

Background

"Dont Check Passwords into Source Control or Hard-Code Them in Your Application Operations staff will remove your eyes with a spoon if they catch you doing this. Dont give them the pleasure Passwords should always be entered by the user performing the deployment. There are several acceptable ways to handle authentication for a multilayer system. You could use certificates, a directory service, or a single sign-on system". This quote is taken from Chapter 2 of the Continuous Delivery: Reliable Software Releases Through Build, Test, And Deployment Automation (Addison-Wesley Signature Series (Fowler)) book by and David Farley, Jez Humble

Vault by HashiCorp is one of the tools that might provide acceptable level of security for devops engineers for enterprise scenarios as well as for smaller teams like startups.

Challenges to address

At the end of the article we should be able

• install vault on a ubuntu 14.04 :TS server
• initialize vault
• store secrets in vault
• access secrets in vault

Installing Manually

Formal installation steps are covered by this article: https://vaultproject.io/docs/install/ For purposes of the demo article let me provide semi automated script, that installs vault 0.1.2 into /opt/vault_0.1.2 folder , configures it to listen on localhost port 8200 and registers it as a service called vault-server

Installing automatically with Ansible

See https://github.com/softasap/sa-hashicorp-vault Ansible Role for unattended Vault installation

Check the installation:

Message means, that vault was installed and configured correctly, but needs to be initialized. Initialization happens once when the server is started against a new backend that has never been used with Vault before. During initialization, the encryption keys are generated, unseal keys are created, and the initial root token is setup. To initialize Vault use vault init. This is an unauthenticated request, but it only works on brand new Vaults with no data

Let's init. Important influence on security has number of key shares to generate and number of key shares provided to unlock the seal.

How does it work: the key used to encrypt the data is also encrypted using 256-bit AES in GCM mode. This is known as the master key. The encrypted encryption key is stored in the backend storage. The master key is then split using Shamir's Secret Sharing. Shamir's Secret Sharing ensures that no single person (including Vault) has the ability to decrypt the data. To decrypt the data, a threshold number of keys (by default three, but configurable) are required to unseal the Vault. Thesekeys are expected to be with three different places / individuals.

It has full analogy to secure bank cell where one key has bank personnel and one is yours.In case of vault you might have much higher level of security.

For demo purposes I will use single key only.

Initial Root Token must be immediately saved in a secure location.

Using vault

Unsealing

When a Vault server is started, it starts in a sealed state. Unsealing is the process of constructing the master key necessary to read the decryption key to decrypt the data, thus prior to unsealing, almost no operations are possible with Vault.

Let's unseal:

Note, if you had higher threshold set, all the key holders would need to perform unseal operation with their parts of the key. That's provides additional level of security for accessing the data

Authorization

In order to continue working with vault, you should first identify yourself. Let's use auth command to do this by providing our initial root token

Policies

Access control policies in Vault control what a user can access.When initializing Vault, only the "root" policy is present. It gives superuser access to everything in Vault.

As we plan to store secrets for saying multiple projects, we should be able to clearly separate access to secrets that belong to different projects. And this is where policies do their job.

Policies in Vault are formatted with HCL. HCL is a human-readable configuration format that is also JSON-compatible, so you can use JSON as well. An example policy is shown below:

It specify path, like we have in some tree structure, wildcards are supported. If you provide access to specific part of the tree, you also provide the same access to all subnodes, unless you override it.

Policy is registered with policy-write command

Deployment tokens

Now it is time to create deployment token. In our case, this is token that would allow us to read the secret deployment value from vault, and does not have any additional privileges except this.

In order to do so, we are using creating token with policy command

Storing data

Now it is time to store some secrets for deployment. For purposes of the demo, let it be some api key and private key used for deployment.

Command write is used to write the secrets

Important

Binary file storing is not supported as for now, but you always can store base64 encoded file, like the MIME attachments are stored in mails. Fortunately, for most deployments we have api keys and private keys that are text files.

Retrieving the data

There are two ways to access your data. First is using vault client itself

second is using http based API. For that scenario you will need to authorize via deployment token we allocated previously.

Securing vault http api

Vault supports https itself, but I believe for production deployment it would be better to hide it behind web server as a proxy.

Below is an example of nginx configuration

Code in action

Code can be downloaded from repository https://github.com/Voronenko/hashi_vault_utils

Some files just help using existing vault functionality in a more handy way:

• vault_status.sh - gets status of the vault
• vault_policy.sh - lists known policies, or shows details of the policy provided as a first parameter
• vault_create_token_with_policy.sh creates and returns token with policy provided as a first parameter.
• vault_read.sh reads secret by key (first parameter)
• vault_write.sh writes secret by key (first parameter) and set's it's value (second parameter)
• vault_write_file.sh writes secret by key (first parameter) and stores content's of text file provided as second parameter
• vault_curl.sh can be used to test http api. First parameter - access token, second parameter secret key to read

Devops: usage with ansible

Lookup plugin: github.com/Voronenko/ansible-developer_recipes/blob/master/ansible_extras/lookup_plugins/sa_hashi_vault.py

Action module: github.com/Voronenko/ansible-developer_recipes/blob/master/ansible_extras/action_plugins/sa_hashi_vault.py

Galaxy role: softasap/sa-hashicorp-vault

Standalone ansible deployment: sa-hashicorp-vault box example

Points of interest

This covers only very basic aspects to start using vault in your organization, but could be a nice first step to move forward.