Information sharing / hooks notes

gistfile1.md

R&D Notes: Information Sharing & Hooks

This gist captures the results of a couple sprints of R&D on information sharing and hooks.

How gears are linked

We discussed two models of how gears are linked:

• Services (containing only a single container type) and links - easy to model multiple web containers and DBs - but you may wind up with a complex configuration of containers in domain. We discussed a possible subdomain concept to manage groups of containers in a domain more easily.

• Like the current openshift model, but with cross-app linking between gears

Shortcomings of V2 connection hooks

We have a lot of experience as a group at this point with V2 connection hooks and we know there are a number of shortcomings:

1. No way to do decision making with connection hooks (no second pass)
2. They’re used to send information around that we already know at a higher level
3. Some things (single IP/port) only make sense in a flat network model
4. No way to control how results of hooks are summarized

Ideally, we want a system without connection hooks that have to be orchestrated. For now we will concentrate on information sharing using environment variables. Each image can be used with a descriptor that provides metadata about what env vars an image publishes and which it wants to consume from other containers. Container descriptors will be modified to allow the specification of rules for the generations of secrets such as session keys and passwords. Evironment variables published by one container will be translated by consumers to any desired name.

Hooks in other PaaSs and Docker orchestrators

We surveyed several frameworks in the docker-orchestration/paas space to understand what they provide:

	Links?	Secret Values?	Docker Registry Auth?	Health Checks?	Life Cycle Hooks?
maestro-ng	Supports deps between services; using external IPs with the docker link env scheme.,Allows soft links that allow a container to consume information shared by another container but do not affect start/stop ordering.see: https://github.com/signalfuse/maestro-ng#how-maestro-orchestrates-and-service-auto-configuration	No	Yes	Checks for 'running' and 'stopped' containers with port pinging and script execution	No
fig	Docker links	No	No	No	No
heroku app-setups	N/A	Yes	N/A	No	Post-deploy hook

Requirements of popular packages

Our model must be robust enough to accomodate clustered configuration for arbitrary software packages. In our research we looked at the clustering configuration requirements of a number of popular packages:

1. Mongod
2. MySQL read replication (master/slave)
3. Etcd
4. Zookeeper
5. Cassandra
6. ActiveMQ
7. RabbitMQ
8. Solr

We found that a limited set of dimensions can capture the necessary functionality to deploy these packages in a clustered configuration:

1. Does the port in the container have to match the port in the host?
2. If the size parameters of the cluster are known, is a pre-installation hook need to generate a clustered configuration artifact?
3. Is a hook required to run in any instances in order to initialize the cluster?
4. Is some other type of complex orchestration required?
5. Do instances in a cluster share a unique value, such as a token or password?
6. Do clients of the cluster need aggregated endpoints in a single value?
7. Do the instances in a cluster need aggregated endpoints in a single value?
8. Are instances in a cluster configured with a specific role?

	Port match?	Pre-install hook?	Cluster setup hook?	Complex orchestration?	Shared unique value?	Exposes aggregated endpoints?	Instances need aggregated endpoints?	Role-aware instances?
mongodb	Yes		In master
mysql master/slave				Lock required in master during slave setup			Slaves need the coordinates of the master	Yes
etcd		Yes			Unique token from discovery API	Yes	Yes
zookeeper		Yes				Yes	Yes	Yes
cassandra		Yes				Yes	Each instance needs the coordinates for at least one seed
activemq						Yes
rabbitmq		Yes			Instances share a password	Yes	Yes
solr		Yes	In all instances			Yes

Most of the above packages can simply start in clustered mode using generated secrets and other shared environment variables if the run command generates a configuration before the initial start.

Differences from that formula are as follows:

1. Solr requires a special initialize hook
2. Mongodb requires an initialization hook to be run for a single instance to initialize a replica set.
3. MySQL requires information passing from the master to the slave and coordination between the two to setup read slave replication.

Scale up workflow of popular packages

We also looked at the scaling properties of the packages we surveyed. We looked at the following aspects of their scaling workflows:

1. What is the minimum increment of cluster size?
2. Is a hook required in any instances when scaling?
3. Is there a more complex orchestration required when scaling?

TODO: table

User hook requirements of popular OpenShift quickstarts

Many existing OpenShift quickstarts and instant apps make use of action hooks. We must offer functional equivalents for these where applicable and support the necessary granularity (eventually) for scaling distributed databases.

The following hooks are proposed:

Hook	Runs When	Runs In
pre-build	Before an image build occurs	build container
post-deploy	After a new deployment occurs	namespace of container - all
pre-start	Before the main command runs in the container	container
post-start	After the container is started	namespace of container
pre-stop	Before the container is stopped	namespace of container

Different use-cases have different granularity/location requirements for hooks. We propose:

first: hook runs on the first instance of a container in the pod any: hook runs on any instance of a container in the pod all: hook runs on all instance of a container in the pod

Pod Descriptor Additions

The Kubernetes pod descriptor describes the containers in a pod and which ports they expose, volumes they have, etc. Since Kubernetes will be the basis of the next version of OpenShift, we propose some extensions to the pod descriptor format so that the format may capture all the necessary data to express the information sharing features we need.

Environment Variables

The Env section of a Container specification describes the environment a container should run with. We propose to extend the Kubernetes EnvVar type to allow the specification of rules for generating secrets and specify whether that environment variable is shared across containers in a pod or across service boundaries. The generate field takes an argument which is a format string. The format string will be simple syntax to express rules for generating secrets, such as a subset of the regex grammar and allow function calls for convenience (viz: ‘password’, ‘uuid’) or make REST calls (viz, ‘GET:https://discovery.etcd.io/new’).

“Env”: [
{
    “name”: “DB_PASSWORD”
    “generate”: “[a-zA-Z0-9]{8}”,
    “shared”: true
},
{
    “name”: “SHARED_SECRET”
    “generate”: “uuid”,
    “shared”: true
}
]

There are two use-cases that container links must be considered in: intra-pod and between pods/services.

Within a pod, each container shares a single network namespace. So, no special network setup is needed in order for containers to contact one another. Each container in a pod receives environment variables shared by other containers in the pod. (TODO: say how this is different from the current Kube model) Between services, we will establish a notion of links. Each container in a pod will get every published environment variable from every service is it linked to and network rules to provide connectivity. Environment collisions across containers groups will be handled by namespacing the variable names with the name of the container group in the app template. For example, if you have services A, B, and C, and both B and C publish SOME_VAR, then the environments (before translations are applied) of containers in pod A will contain B_SOME_VAR and C_SOME_VAR.

Pod Descriptor as Template

We introduce a use-case for Kubernetes pod descriptors where the descriptors are stored and used to compose new applications or add functionality to existing applications. When altering existing applications, the environment and network configuration of containers in the pods of the services the new pods are linked to will be updated accordingly.

Hooks

We propose a new element in the Kubernetes container descriptor to declare a git repository url or local path from which hooks can be downloaded:

“Hooks”: {
    “url”: “git://github.com/user/container-hooks.git”
}

The hooks element also allows the description of commands and locations (todo: better word) for hooks explicitly. The default location for a hook is ‘all’, meaning it runs in every container of the associated type in a pod. The where field can also be “any” or “first”.

If a hook script exists in a URL referenced by a hooks element, and that element also specifies a command for that hook, and the command begins with a path separator, the command in the element will override the hook in the URL:

“Hooks”: {
    “url”: “git://github.com/user/container-hooks.git”,
    “prestart”: {
    “command”: “/path/to/script.sh arg1 arg2”,
    “where”: “any”
}

If the command for a hook present in a URL does not start with a path separator, it represents arguments to that hook.

“Hooks”: {
    “url”: “git://github.com/user/container-hooks.git”,
    “prestart”: {
        “command”: “arg1 arg2”,
        “where”: “first”
    }
}