Using Concourse and Hashicorp Vault for a pipeline for private repositories

concourse-vault.md

Setup a Hashicorp Vault instance

One way is to follow the instructions at https://gist.github.com/chris-moreton/f523650c1863f0181e22e2020d0f2268

Create a token to allow Concourse to talk to the vault

Create concourse-policy.hcl

path "concourse/*" {
  policy = "read"
}

Then run

vault policy write concourse ./concourse-policy.hcl
vault token create --policy concourse --period 24h

Put a deploy key in the vault which can be accessed by ((github_deploy_key)) in the pipeline config

vault write concourse/main/my_project/github_deploy_key value=@/path/to/deploy_key

Note, that in the above, the "my_project" element will need to be exactly the name of the pipeline you create for the project in Concourse.

Bring up the docker image

Create a docker-compose.yml as below, adding VAULT_ADDR and TOKEN

version: '3'

services:
  concourse-db:
    image: postgres
    environment:
      - POSTGRES_DB=concourse
      - POSTGRES_PASSWORD=concourse_pass
      - POSTGRES_USER=concourse_user
      - PGDATA=/database

  concourse:
    image: concourse/concourse:4.2.1
    command: quickstart
    privileged: true
    depends_on: [concourse-db]
    ports: ["8080:8080"]
    environment:
      - CONCOURSE_POSTGRES_HOST=concourse-db
      - CONCOURSE_POSTGRES_USER=concourse_user
      - CONCOURSE_POSTGRES_PASSWORD=concourse_pass
      - CONCOURSE_POSTGRES_DATABASE=concourse
      - CONCOURSE_EXTERNAL_URL
      - CONCOURSE_ADD_LOCAL_USER=admin:admin
      - CONCOURSE_MAIN_TEAM_LOCAL_USER=admin
      - CONCOURSE_VAULT_URL=<VAULT_ADDR>
      - CONCOURSE_VAULT_CLIENT_TOKEN=<TOKEN>

Bring up a local Concourse instance

docker-compose up -d

Create a pipeline.yml and add the following

---
resources:
- name: my_project
  type: git
  source:
    uri: [email protected]:my-github-name/my_project
    branch: master
    private_key: ((github_deploy_key))
jobs:
- name: my_project
  public: false
  plan:
  - get: my_project
  - task: build
    config:
      platform: linux
      image_resource:
        type: docker-image
        source: {repository: busybox}
      inputs:
        - name: my_project
      run:
        path: cat
        args: [my_project/devops/ci/build.sh]

Set the pipeline

fly --target my_project login --concourse-url http://127.0.0.1:8080 -u admin -p admin
fly --target my_project sync
fly --target my_project set-pipeline -c pipeline.yml -p my_project

Configuring Vault

vault secrets enable -path=concourse kv
vault kv put /concourse/main/my_project bitbucket_private_key=/path/to/deployment.key