Abusing Docker Compose environment variables for fun and profit

abusing-docker-compose-environment-variables-for-fun-and-profit-env-file-env_file.md

Abusing Docker Compose environment variables for fun and profit

Are you using Docker Compose but annoyed at the convoluted management and maintenance of runtime environment variables for your containers? Then this is for you.

TLDR;

Easily manage env vars across environments and containers:

1. Update your docker-compose.yml:

...
  myservice:
    image: myservice:latest
    env_file:
      - ${ENV_FILE} # <-- add this
    ...
     ports:
      - "${HTTP_PORT}:8080"

2. Create a test.env file with a ENV_FILE key

Add vars for the container, the compose file, or both

ENV_FILE="./my/path/test.env" # <-- set this to be a path to the env file itself
...
HTTP_PORT="9000"  # <-- DOCKER-COMPOSE subsitution
...
MY_API_KEY="i-am-a-key" # <-- CONTAINER runtime env

3. Run docker-compose with the --env-file flag

$ docker-compose --env-file=./my/path/test.env up

4. Profit

your docker-compose.yml will substitute to:

...
  myservice:
    image: myservice:latest
    env_file:
      - ./my/path/test.env
    ...
     ports:
      - "9000:8080"

and your container's environment will contain:

MY_API_KEY="i-am-a-key"

The Problem OR What is going on here?

Let's assume you have a microservice. The microservice wants an API key which you are going to pass to it from the environment variable MY_API_KEY in the container at runtime.

So in your docker-compose.yml file you add something like this:

...
  myservice:
    image: myservice:latest
    environment:
      - MY_API_KEY="I am a production api key"
    ports:
      - "9000:8080"

Of course for testing you don't the production key, you want the test key. At this point you might make docker-compose-test.yml which overrides MY_API_KEY with a test key and maybe ports with a test port. This works all right, but tracking and maintaining those files quickly become a pain.

Add to this, every time you add a new env var, you need to go back and update everything:

...
  myservice:
    image: myservice:latest
    environment:
      - MY_API_KEY="i-am-a-key"
      - MY_OTHER_API_KEY="i-am-also-a-key"
    ports:
      - "9000:8080"

⚲ Put a pin in that for second.

Another option: you might use environment variables and substitution in the docker-compose.yml itself:

...
  myservice:
    image: myservice:latest
    environment:
      - MY_API_KEY="${MY_API_KEY}"
      - MY_OTHER_API_KEY="${MY_OTHER_API_KEY}"
    ports:
      - "${HTTP_PORT}:8080"

To more easily support this, docker-compose lets you pass an env file via the --env-flag

HTTP_PORT="9000"
MY_API_KEY="i-am-a-key"
MY_OTHER_API_KEY="i-am-also-a-key"

Running

$ docker-compose --env-file=./my/path/test.env up

will result in the following substitution:

...
  myservice:
    image: myservice:latest
    environment:
      - MY_API_KEY="i-am-a-key"
      - MY_OTHER_API_KEY="i-am-also-a-key"
    ports:
      - "${HTTP_PORT}:8080"

The critical thing here is that the --env-file flag passes vars to the env of Docker Compose...

and not the env of the container. If you add or change an env var being passed to the container you still need to update/maintain both the env file and the docker-compose.yml.

But wait, doesn't docker-compose provide a way to pass an env file to the container?

Yes! We can use env_file in the docker-compose.yml:

...
  myservice:
    image: myservice:latest
    env_file:
      - ./my/path/test.env
    ports:
      - "${HTTP_PORT}:8080"

Ok, that works all right. Now, when we run $ docker-compose --env-file=./my/path/test.env up the HTTP_PORT will be correct, and the container will have the right MY_API_KEY... but now we're back to needing a docker-compose-test.yml to override the env_file and pass the correct env. Ugh.

There has to be a better way!

First change your docker-compose.yml to make the value of env_file itself environment variable:

...
  myservice:
    image: myservice:latest
    env_file:
      - ${ENV_FILE} # <-- add this
    ...
     ports:
      - "${HTTP_PORT}:8080"

Next, add an entry for ENV_FILE in your test.env and set its value to be a path to itself:

ENV_FILE="./my/path/test.env" # <-- set this to be a path to the env file itself
...
HTTP_PORT="9000"  # <-- DOCKER-COMPOSE subsitution
...
MY_API_KEY="i-am-a-key" # <-- CONTAINER runtime env
MY_OTHER_API_KEY="i-am-also-a-key" # <-- CONTAINER runtime env

Now Bring the Magic

Run

$ docker-compose --env-file=./my/path/test.env up

Your docker-compose.yml will substitute to:

...
  myservice:
    image: myservice:latest
    env_file:
      - ./my/path/test.env // <-- Oh look, I am the correct env for the container
    ...
     ports:
      - "9000:8080" // <-- *And* I am the correct port

And your container will start with the correct keys:

MY_API_KEY="i-am-a-key"
MY_OTHER_API_KEY="i-am-also-a-key"

All your environment vars now live one set of files prod.env, test.env, etc. with a single docker-compose.yml, which you no longer have to sync or touch as often.

Now when you (or your deployment toolchain) needs to switch to prod?

Just use:

$ docker-compose --env-file=prod.env up

Easy peasy.