Skip to content

Instantly share code, notes, and snippets.

@aanand
Last active June 4, 2018 00:58
Show Gist options
  • Save aanand/9e7ac7185ffd64c1a91a to your computer and use it in GitHub Desktop.
Save aanand/9e7ac7185ffd64c1a91a to your computer and use it in GitHub Desktop.

Preview build: Container grouping and stack composition

NOTE: this is out of date - refer to moby/moby#9694

Here is a preview build of two new features we’re working on concurrently: container grouping (docker groups) and stack composition (docker up). Together, they will eventually form a complete replacement for Fig.

Here's how to test it out if you're running boot2docker. First, replace the binary in your VM:

$ boot2docker ssh
docker@boot2docker:~$ sudo -i
root@boot2docker:~# curl -LO http://cl.ly/2c3p40251H11/download/docker-1.3.1-dev-linux
root@boot2docker:~# /etc/init.d/docker stop
root@boot2docker:~# mv /usr/local/bin/docker ./docker-stable
root@boot2docker:~# mv ./docker-1.3.1-dev-linux /usr/local/bin/docker
root@boot2docker:~# chmod +x /usr/local/bin/docker
root@boot2docker:~# /etc/init.d/docker start
root@boot2docker:~# docker version   # both "Git commit"s should be 56d8c9e
root@boot2docker:~# exit
docker@boot2docker:~$ exit

Next, replace your client binary:

$ curl -LO http://cl.ly/1U1A0W2e0Y3u/download/docker-1.3.1-dev-darwin-amd64
$ mv /usr/local/bin/docker ./docker-stable
$ mv ./docker-1.3.1-dev-darwin-amd64 /usr/local/bin/docker
$ chmod +x /usr/local/bin/docker
$ docker version                     # both "Git commit"s should be 56d8c9e

The idea of stack composition is that with a very simple group.yml which describes what containers you want your application to consist of, you can type docker up and Docker will do everything necessary to get it running - including building or pulling any required images.

I demonstrated this functionality a couple of weeks ago in this screencast; this build is similar to that demoed but differs in that the shared-hosts-file approach has been ditched in favour of a Fig-style links keyword. More on that in the proposal discussion.

Here’s a sample app you can try:

app.py:

from flask import Flask
from redis import Redis

app = Flask(__name__)
redis = Redis(host="redis", port=6379)

@app.route('/')
def hello():
    redis.incr('hits')
    return 'Hello World! I have been seen %s times.' % redis.get('hits')

if __name__ == "__main__":
    app.run(host="0.0.0.0", debug=True)

requirements.txt:

flask
redis

Dockerfile:

FROM python:2.7
ADD . /code
WORKDIR /code
RUN pip install -r requirements.txt

group.yml:

name: counter

containers:
  web:
    build: .
    command: python app.py
    ports:
      - "5000:5000"
    volumes:
      - .:/code
    links:
      - redis
    environment:
      - PYTHONUNBUFFERED=1
  redis:
    image: redis:latest
    command: redis-server --appendonly yes

If you put those four files in a directory and type docker up, you should see everything start up:

docker up example

It'll build the web image, pull the redis image, start both containers and stream their aggregated output. If you Ctrl-C, it'll shut them down.

Your app will run in a group, which is a new way of keeping multiple related containers organised. The docker groups command lets you list, create and delete groups, and also bulk start/stop/kill the containers within them. To find out more about that, run docker groups -h.

Groups are created by posting JSON to the daemon with the configuration info for all containers. docker up posts to that endpoint with a JSON payload generated from group.yml, and is responsible for automatically building and pulling images before doing so. To get a feel for the separation, try this:

$ docker up --parse > group.json
$ docker groups create - < group.json
$ docker groups start counter

That's essentially equivalent to docker up -d.

A group has a unique name which exists in the same namespace as containers. This app's group is called counter, as specified in group.yml. Its containers are called counter/web and counter/redis, and you can perform all the normal docker commands on them (e.g. docker inspect counter/web).

You can also start a new container in an existing group with docker run --group, e.g. docker run --group counter busybox echo hello world.

You can list just the containers in a specific group with docker ps GROUPNAME.

Finally, some docker commands have been augmented with an experimental syntax for picking up names from group.yml:

# list containers within the group defined in group.yml
$ docker ps :

# start/stop/kill/remove all containers in the current group
$ docker groups [start|stop|kill|rm] :

# start/stop/kill/remove the web container
$ docker [start|stop|kill|rm] :web

# run a bash shell in the web container
$ docker exec -ti :web bash

# pull the image specified under 'redis' in group.yml, i.e. 'redis:latest'
$ docker pull :redis

# build image for 'web' using the directory specified under 'web' in group.yml, i.e. '.'
$ docker build :web

There are a few things left to do before this constitutes a Fig replacement, including:

  • Supporting volumes_from
  • An equivalent to fig run, which runs a one-off container with image and configuration picked up from group.yml. (docker exec covers some of the use cases but not all, and requires a container to already be running)
  • An equivalent to fig logs, which streams the logs of already-running containers.
  • docker groups (pause|unpause|restart)
  • Validation of the YAML file
  • Nicer output for ps (giving more prominence to names over IDs, and perhaps doing away with the long list of names in the default display)
  • We should think about scaling. I don't think an equivalent to fig scale is necessary on day one, but it will eventually be needed as Docker becomes a multi-host platform, so there shouldn't be anything in the design that'll make that difficult to implement later.
  • Code cleanup
  • Tests!

To get hacking, check out the composition branch on my fork:

$ git checkout -b composition master
$ git pull [email protected]:aanand/docker.git composition
@saifikhan
Copy link

Hi:
The docker source tree on my box is updated as of 2014-11-10. Building the docker-1.3.1-dev binary on CentOS 7 does not show the 'group' and 'up' options when i run docker.

@ailjushkin
Copy link

Hey @aanand,
I'm interested in creating a web service which have to produce gui for my colleagues.
Can you suggest how could this be done ? (I didn't find anything in my docker distrib connected with container groups)

My gui must implement:

  • login free - use corporate hostname to link with inner logic (only local web) and this should provide information which belongs only for exact user (container groups)
  • container groups: admin (me) will maintain docker imfrastructure. I need to create groups of containers. for ex.:

group_app_v25:
app_v25: localhost:5000/app:v25
srv_v30: localhost:5000/app:v30

group_app_v26:
app_v26: localhost:5000/app:v26
srv_v31: localhost:5000/app:v31

So when user comes into a site page, he will see this groups. Below this he should see his running groups, like

jason_statham_app_v25: destroy, restart, recreate
jason_statham_app_v26....

so I could filter container names to show only those groups which belongs to the user by his hostname.

The problem is that i need something between docker logic and the user, because I want to simplify connection of user to his group container. for instance, using standard port, like 1234, like if two users would try to connect to a server on port 1234, they would connect to different containers.

On the other hand, i want to simplify the connection between containers in the container groups, where would be standard ports too, like

  1. user1 ->host:1234 -> forward -> host:48123 -> forward -> container_group1 -> container_app:1234 -> srv:1235.
  2. user2 ->host:1234 -> forward -> host:48224 -> forward -> container_group2 -> container_app:1234 -> srv:1235.

It's very untrivial task for me. Maybe you could help me with decisions. Thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment