Configure Vault for Storing Secrets

Follow the steps in this guide to configure NGINX Management Suite to use Hashicorp’s Vault for storing secrets.


This documentation applies to NGINX Management Suite Instance Manager 2.6.0 and later.


Hashicorp’s Vault is a popular solution for storing secrets. While NGINX Management Suite provides encryption-at-rest for secrets stored on disk, if you have an existing Vault installation, you may prefer to store all secrets in one place. NGINX Management Suite provides a driver you can use to connect to existing Vault installations and store secrets.

Before You Begin

To complete the steps in this guide, you need the following:


Steps

Create Periodic Service Tokens

Access to a vault requires a renewable token.

Note:
If you attempt to use the vault’s Root Token, NGINX Management Suite will not be able to start the secrets driver, as that token is not renewable.

To create a periodic service token for NGINX Management Suite, take the following steps:

  1. Use the Vault user interface to create a new policy.

    The “default” policy has no access to store or retrieve secrets, and the root policy is too broad. We recommend creating a policy called nms_secrets with these capabilities:

    path "secret/*" {
      capabilities = ["create", "read", "update", "delete", "list"]
    }
    
  2. Create an initial service token that will last so long as it’s renewed on time until it’s manually revoked. We recommend a period of 24 hours, which is used in the following example. NGINX Management Suite will always attempt to renew tokens before expiring, so shorter times also work.

    To create a token, take the following step, substituting your vault’s Root Token for $VAULT_ROOT_TOKEN and your vault’s address for $VAULT_ADDR:

    curl -X POST --header "X-Vault-Token: $VAULT_ROOT_TOKEN" \
      --data '{"policies": "nms_secrets", "period": "24h"}' \
      $VAULT_ADDR/v1/auth/token/create | jq -r ".auth.client_token" > periodic_token.txt
    
  3. Verify your token works:

    curl --header "X-Vault-Token: $(cat periodic_token.txt)" \
      $VAULT_ADDR/v1/auth/token/lookup-self | jq .data
    
  4. If everything looks good, stop the nms-core service and configure NGINX Management Suite to use your token:

    sudo systemctl stop nms-core
    sudo NMS_VAULT_TOKEN=$(cat periodic_token.txt) nms-core secret vault-token
    sudo systemctl restart nms-core
    

Set core_vault_addr

  1. To use the new token on the NGINX Management Suite server, open the /etc/nms/nms.conf file for editing.

  2. Add a new line that sets core_vault_addr to tell NGINX Management Suite how to reach your running vault instance. For example, an internal development installation of Vault might use:

    core_vault_addr = http://127.0.0.1:8200
    
  3. Save the changes and close the file.

Switching to or from Vault

After setting core_vault_addr as detailed above, restart NGINX Management Suite in the normal way to start using Vault for storing secrets.

sudo systemctl restart nms

To switch back to using local disk encryption, remove the definition of core_vault_addr from /etc/nms/nms.conf, and then restart NGINX Management Suite in the normal way.

Warning:
Any and all secrets that have been previously stored will be lost when switching to or from Vault! The secrets will still exist on disk (if switching to Vault) or in Vault (if switching to disk), but will be inaccessible to NGINX Management Suite.

Troubleshooting

Token has expired

If the vault service token is manually revoked or expires before renewal – possibly because NGINX Management Suite was shut down and was unavailable to renew the token, all access to stored secrets will fail.

To resolve this problem, you need is to supply a new service token using nms-core secret vault-token. See the Create Periodic Service Tokens section above for details on generating and supplying a new token.


Certs are missing after switching to/from Vault

When configuring Vault for storing secrets, NGINX Management Suite assumes that no secrets have been stored previously and won’t migrate any existing stored secrets. All existing certs must be uploaded again.

Stored secrets are not deleted: secrets remain in the encrypted disk storage or vault. We can’t guarantee that the secrets will remain accessible forever. If you want to recover the missing secrets, you can switch back to the other method for storing secrets following the instructions above. Then restart NGINX Management Suite to see the old secrets again.