Vault and Long Running Apps

Solving Secure Token Introduction

Assuming the applications have a client token, Chef cookbooks can leverage the Vault Ruby gem, direct API calls, native language integrations or the Vault client installed in the VM.

Traditionally, to deliver credentials to retrieve this client token, the Trusted Entity model is used. This is great when deploying in the cloud (AWS, Azure, GCP), using Kubernetes or Jenkins as part of a CI/CD pipeline.

However for applications with no guarantee of ever being redeployed, but that have Chef agents running at a recurring interval, there are at least two potential approaches:

Periodic Token deployed manually:

Benefits: simplicity Drawbacks: token exposed to Vault admin upon creation and to whomever deploys to VMs

Create Vault ACL policy using web UI or cli:

# app1.hcl
path "secret/app1" {
  capabilities = ["create", "update", "read"]
}

path "/auth/token/renew-self" {
  capabilities = ["update"]
}

$ vault policy write app1 app1.hcl

Create a periodic token associated with this policy Note that a token can only create a child that has a subset of its own policies. In order to allow a more flexible behavior, we will create a token role, which list a possible subset of policies that a child token can have, without having to associate all of those policies to the admin token:

3.1 Create a token role:

vault write auth/token/roles/app1 allowed_policies="app1, default" period=768h orphan=true

3.2 Create child token with desired behavior:

$ vault token create -role=app1

3.4 Validate child token:

vault token lookup CHILD-TOKEN
# Expected output:
Key                 Value
---                 -----
accessor            2u8dpPOEsrYYia2iT2VKAVij.9QrMy
creation_time       1535731222
creation_ttl        768h
display_name        token
entity_id           n/a
expire_time         2018-10-02T16:00:22.685686227Z
explicit_max_ttl    0s
id                  2zhtlFPW17953H4JwX4ywNdo.9QrMy
issue_time          2018-08-31T16:00:22.685685192Z
meta                <nil>
namespace_path      chef/
num_uses            0
orphan              true
path                auth/token/create/app1
policies            [app1 default]
renewable           true
role                app1
ttl                 767h59m36s

Deploy Vault Address and token to app1 vms
Update cookbook to configure recurring agent (CRON or similar): Every 15 days issue the command:

# cli
$ vault token renew

# alternatively, through API:
curl \
    --header "X-Vault-Token: $VAULT_TOKEN" \
    --request POST \
    $VAULT_ADDR/v1/auth/token/renew-self

A Vault admin can revoke this token by issuing:

$ vault token revoke <TOKEN>

This is how an access using this token will look like in the logs:

{
  "time": "2018-08-24T20:25:06.146914356Z",
  "type": "response",
  "auth": {
    "client_token": "hmac-sha256:abbda8fa9172e9ee66d61eb7e463c5a396f291547c1fea2213009bede3da4b38",
    "accessor": "hmac-sha256:cac82ee495cb2fc07f9b09461a9c29aa1c1d53cd28327a4d21621a046c247dd2",
    "display_name": "my-app-1",
    "policies": [
      "app1"
    ],
    "token_policies": [
      "app1"
    ],
    "metadata": null,
    "entity_id": ""
  },
  "request": {
    "id": "04869041-4c4f-c6a4-2384-c1aeb090cc5d",
    "operation": "read",
    "client_token": "hmac-sha256:abbda8fa9172e9ee66d61eb7e463c5a396f291547c1fea2213009bede3da4b38",
    "client_token_accessor": "hmac-sha256:cac82ee495cb2fc07f9b09461a9c29aa1c1d53cd28327a4d21621a046c247dd2",
    "path": "secret/user1",
    "data": null,
    "policy_override": false,
    "remote_address": "127.0.0.1",
    "wrap_ttl": 0,
    "headers": {}
  },
  "response": {
    "secret": {
      "lease_id": ""
    },
    "data": {
      "password": "hmac-sha256:c4441253e513752b6adb6e88dc3b1ed19029943e6bf4d1d4ac914c9e4e50a0b9"
    }
  },
  "error": ""
}

# Example command to find in the log file:
$ cat vault_audit.log |  jq 'select(.auth.display_name | startswith("my-app-1"))'

AppRole with Periodic Token, deployed manually:

Benefits: Client token only visible to the VM, easier to track roles while allowing to revoke access to individual instances if needed Drawbacks: Additional step to create credentials and deploy initial login token

Create Vault ACL policy using web UI or cli:

# app1.hcl
path "secret/app1" {
  capabilities = ["create", "update", "read"]
}

path "/auth/token/renew-self" {
  capabilities = ["update"]
}

$ vault policy write app1 app1.hcl

Create another policy, that will allow the VM to authenticate using AppRole:

# login-appRole.hcl
path "auth/approle/login" {
  capabilities=["update"]
}

$ vault policy write login-appRole login-appRole.hcl

Enable AppRole and create role associated with this policy:

$ vault auth enable approle

# This assumes you want to create one SecretId per VM
$ vault write auth/approle/role/app1role \
    period=768h \
    secret_id_num_uses=1 \
    policies=app1

Retrieve RoleId and SecretId. This SecretId will only be able to be used once, so if there is no authentication failure in the VM you can guarantee that the subsequent Client Token generated hasn't been exposed.

$ vault read auth/approle/role/app1role/role-id
# Output RoleId

$ vault write -f auth/approle/role/app1role/secret-id
# Output SecretId

Generate a token to allow logging with AppRole (this can be used multiple times in different VMs, since it does not specify RoleId and SecretId will only be used once):

$ vault token create -policy=login-appRole -display-name=login-appRole

Deploy the login-appRole token to "~/.vault-token" in the VM
Deploy the Vault Address, RoleId and SecretId to VM
Update Chef cookbook to include AppRole authentication logic using Vault Ruby gem

# vault_auth.rb:
require 'vault'

vault_token = Vault.auth.approle(ENV['ROLE_ID'], ENV['SECRET_ID'])
vault_client = Vault::Client.new(address: ENV["VAULT_ADDR"], token: vault_token.auth.client_token)

Update cookbook to configure recurring agent (CRON or similar): Every 15 days issue the command:

# cli:
$ vault token renew

# alternatively, through API:
curl \
    --header "X-Vault-Token: $VAULT_TOKEN" \
    --request POST \
    $VAULT_ADDR/v1/auth/token/renew-self

A Vault admin can revoke the client token of a specific VM by identifying IP, doing reverse hmac on the token (available through Vault web UI) and issuing

$ vault token revoke <TOKEN>

This is how an access using AppRole will look like in the logs:

{
  "time": "2018-08-28T19:10:16.292180298Z",
  "type": "response",
  "auth": {
    "client_token": "hmac-sha256:883d7702eb6b48067dc6482a32af1e049a301dd0722fe6d95d19d4534a3609bf",
    "accessor": "hmac-sha256:49bdb7bf983f1fc72d3443c58147db1e1ada2b715894e6384a4c0208925ee99c",
    "display_name": "approle",
    "policies": [
      "app1",
      "default"
    ],
    "token_policies": [
      "app1",
      "default"
    ],
    "metadata": {
      "role_name": "app1role"
    },
    "entity_id": "aa709fab-816f-4ec6-b6d0-ace9766b347d"
  },
  "request": {
    "id": "da41f356-f0dd-ac3a-19c3-3f3c210c3f23",
    "operation": "update",
    "client_token": "hmac-sha256:883d7702eb6b48067dc6482a32af1e049a301dd0722fe6d95d19d4534a3609bf",
    "client_token_accessor": "hmac-sha256:49bdb7bf983f1fc72d3443c58147db1e1ada2b715894e6384a4c0208925ee99c",
    "path": "secret/app1",
    "data": {
      "value": "hmac-sha256:68370cdb9216366c2dfc537a5a921ef9e92970a721aeb33996e7827b783b7a92"
    },
    "policy_override": false,
    "remote_address": "205.178.21.228",
    "wrap_ttl": 0,
    "headers": {}
  },
  "response": {},
  "error": ""
}

# Example command to find in the log file:
$ cat vault_audit.log |  jq 'select(.auth.metadata.role_name | startswith("app1role"))'

stenio123/README.md

Vault and Long Running Apps

Solving Secure Token Introduction

Periodic Token deployed manually:

AppRole with Periodic Token, deployed manually: