Fact: docker is an abstraction on top of LXC Containers. This means you can run an isolated “virtual machine” (container) within your Linux-based distro. Containers are extremely lightweight and you can start a new container very quickly. You can use containers to run your websites, databases, etc, in a isolated environment. In theory you should be able to use containers to run low-latency operations such as an individual database transaction
-
Worth Reading! 12 Factor
It is quite common for our customers to use a DTAP environment to deploy applications. DTAP stands for Development, Test, Acceptance and Production.
http://www.hokstad.com/docker/patterns
- Ansible was a great introduction to automated server configuration and deployment. It was easy to get up and running and has great documentation.
- deployment and orchestration
- Docker
- Vagrant
- Packer
- Chef / Puppet / Ansible / Salt
- Jenkins
The most flexible setup is the one where you deploy one app container + one MySQL container for each app.
http://martinfowler.com/bliki/ContinuousDelivery.html
Continuous Delivery includes following:
- Continuous Deployment: Continuous Deployment means that every change goes through the pipeline and automatically gets put into production, resulting in many production deployments every day. Continuous Delivery just means that you are able to do frequent deployments but may choose not to do it, usually due to businesses preferring a slower rate of deployment. In order to do Continuous Deployment you must be doing Continuous Delivery.
- Continuous Integration: Continuous Integration usually refers to integrating, building, and testing code within the development environment. Continuous Delivery builds on this, dealing with the final stages required for production deployment.
http://martinfowler.com/bliki/DeploymentPipeline.html
Backing services, such as the app’s database, queueing system, or cache, is one area where dev/prod parity is important.
A backing service is any service the app consumes over the network as part of its normal operation. Examples include datastores (such as MySQL or CouchDB), messaging/queueing systems (such as RabbitMQ or Beanstalkd), SMTP services for outbound email (such as Postfix), and caching systems (such as Memcached).
An app’s config is everything that is likely to vary between deploys (staging, production, developer environments, etc). This includes:
- Resource handles to the database, Memcached, and other backing services
- Credentials to external services such as Amazon S3 or Twitter
- Per-deploy values such as the canonical hostname for the deploy
Each distinct backing service is a resource. For example, a MySQL database is a resource; two MySQL databases (used for sharding at the application layer) qualify as two distinct resources. The twelve-factor app treats these databases as attached resources, which indicates their loose coupling to the deploy they are attached to.
Docker is an open source framework that automates the deployment of applications in lightweight and portable containers. The Docker framework is modelled on the concept of the standard shipping containers that are used to transport much of the world’s goods. Like shipping containers you can build, fill, open and transport Docker containers. These containers can then be run in a wide variety of places: on your laptop, in the Cloud, on a virtual machine or even on physical hardware.
Docker is an open platform for developers and sysadmins to build, ship, and run distributed applications, whether on laptops, data center VMs, or the cloud.
-
It can take too long to set it up How long does it takes for a new developer to setup your current project's development environment? The answer may depends on many factors (the project age, the number of developers that have worked on it, etc...) but half a day or more is not an uncommon one. Hey! It should be much faster than that: checkout a script and execute it. That's all. Two steps should be sufficient to setup your environment and get ready for development.
-
It can differ too much from testing and production environments Have you ever skipped your build's automated tests because they failed on your machine? Or even worst, have you ever been the cause of a build break even if your changes compiled smoothly on your machine but failed consistently on CI server? Any slight difference can result in an unexpected behaviour. Diverging can be as simple as giving a try to the last version of a framework or switching to a different project for half a day. Finding out what makes your system behave differently is an annoying task every developer should avoid.
As a consequence development environment should have two characteristics:
- Isolated: you don't want to mess it up when testing some new tool or a different project.
- Repeatable: the same environment should be consistently reproducible on every team member machine and on CI and production servers.
Virtual environments ensure these features. But classic VMs are resource consuming. Developers need to code/build/test every few minutes and won't accept the virtualization overhead.
"the platform’s ability to separate the concerns of application development management from those of infrastructure provisioning, configuration, and operations. Docker gave these early users a new, faster way to build distributed apps as well as a “write once, run anywhere” choice of deployment from laptops to bare metal to VMs to private and public clouds."
configuration management tools like Puppet.
Compared to other tools that can help running Docker on non Linux platforms (e.g. boot2docker), Vagrant has some important advantages:
- Configure it once and run it everywhere: Vagrant is just a Docker wrapper on systems that support Docker natively while it spins up a "host VM" to run containers on systems that don't support it. User don't have to bother wether Docker is supported natively or not : the same configuration will work on every OS.
- Docker hosts are not limited to boot2docker (a Virtualbox image of Tiny Core Linux) but Debian, Ubuntu, CoreOS and other Linux distros are supported too. And can run can run on more stable VM managers than Virtualbox (e.g. VMWare).
- Vagrant can orchestrate Docker containers: run multiple containers concurrently and link them together
That question naturally breaks into two issues:
- On the one hand, Docker hosts need to be managed, Docker Engine needs to be installed, and containers need to be started.
- On the other hand, building Docker images and handling the inside of running containers could easily send us back to the unhappy days of golden images.
When running immutable infrastructure with containers, this core problem of composing the setup from disparate sources is not eliminated, but merely moved from the time containers are run to the time when container images are built.
http://puppetlabs.com/blog/docker-and-puppet-for-application-management
Before designing an approach to managing property files, it's critical first to understand the types of properties are to be managed. Here are four characteristics to consider:
- Are some properties environment specific – in other words, do they have a different value in one environment than another? For example, a database connection string would differ in the TEST environment than it would the PROD environment.
- Do any properties contain sensitive information, like passwords or credentials, that can’t sit in plain text in the deployment unit (e.g. WAR, EAR)?
- Do any properties need to be located external from the deployment unit (e.g. on the file system rather than in the WAR, perhaps to be managed by a systems administrator)?
- Should properties be dynamic, in the sense that they can be updated at runtime (i.e. without requiring a restart of the application)?
Each of these special characteristics adds a degree of complexity to applications!
- an-introduction-to-tracking-statistics-with-graphite-statsd-and-collectd
- uptime-monitoring
- The main tool in our arsenal is a simple tool that we wrote that simply pings REST endpoints and URLs to ensure they are up and available, displaying any failed responses on a simple dashboard. With so many endpoints, this simple dashboard is proving invaluable in quickly checking the health of the estate.
- With some relatively minor changes to our deployment process, our CI (continuous integration) servers now build complete containers—with all of our assets and code baked in—rather than copying files in place on the host machine. These containers are shipped over and booted up next to the old containers on our web machines, and whenever they are ready to start answering queries they register with the locally-running instance of Hipache to start forwarding requests to the new system. The result: no downtime!
- We took this responsibility seriously and launched the “Zero Downtime Push” project with the goal of being able to push out product updates without interrupting our users’ experience on the site. Today, we’re happy to say it’s helping us make constant improvements to our technology without inconveniencing you. Here’s how we did it.
that consists out of the following:
- CoreOS (A stripped Linux distribution that has the unneeded software removed that other Linux distributions have)
- Docker (Container management tool that uses Linux LVM)
- Vagrant (Virtual Machine manager, mostly used for development environments).
Every environment has prerequisites that have to be installed, we can not avoid this but we can keep it to a minimum. This is why a new developer only has to install the following when they come in:
- VirtualBox (Virtualization Software)
- Vagrant
- Git (Version Control System)
The advantages of using this development environment are:
- Small initial setup time allowing the newly hired development too start developing within 10 minutes.
- Almost an exact clone of the production environment, this allows the developer too test changes that are not dependent on the OS.
- Docker has the advantage over Chef / Puppet / Ansible / ... that it is way faster and smaller.
- Separation of concerns, we keep the complete development environment on a VM so that the developer does not need to install unneeded software on their own machine.
- CoreOS ships Docker by default, which is one package less to install.
CoreOS gives you three essential tools:
- service discovery,
- container management and
- process management.
Vagrant is a simple-to-use command line virtual machine manager. There are install packages available for Windows, Linux and OSX. Find the latest installer on the Vagrant downloads page. Be sure to get version 1.6.3 or greater.
Vagrant can use either the free VirtualBox provider or the commercial VMware provider. For the VirtualBox provider, version 4.3.10 or greater is required.
- Step 1: Install Vagrant - Download Vagrant - Be sure to get version 1.6.3 or greater.
- Step 2: Install VirtualBox - Download VirtualBox - Be sure to get version 1.6.3 or greater.