environments instead of stages

webzone-puppet-stages-environements.science

Environments Instead of Stages
==============================

This WebZone post recommends (ab)using puppet environments to do bootstrap/staging.
-----------------------------------------------------------------------------------

Puppet stages is a pretty important idea. Its existence accepts the idea that configuration management is hard. Since configuration management is hard some advanced tools are needed. Advanced tools will likely not be installed by default so we need some concept of staging in order to get the system to a state where the advanced tool works properly. 

The problem is Puppet stages come up short in a couple of places:

1. The catalog seems to be compiled all at once instead of in stages
2. Circular dependencies grow organically from the ooze

Example problem of (1) is when you're using something like hiera. 
------

If there is:

```puppet
stage { 'setup': before => Stage['main'] }

class {
'setup':  stage => 'setup';
}

class {'workstation::role::development':
username => 'nibz',
}
```
as site.pp, and: 

```puppet

package {
  'hiera':
    provider => gem,
    ensure   => present,
}
```
as modules/setup/manifests/init.pp, and:

```puppet
$pkgs = hiera('pkgs')

package {[$pkgs]:
  ensure  => latest;
}
```
as modules/workstation/manifests/role/development/nibz.pp

Then `puppet apply` gives:

> /usr/lib/ruby/vendor_ruby/1.8/rubygems/custom_require.rb:36:in `gem_original_require': no such file to load -- hiera (LoadError)

And a stack trace. :(

What's going on here is puppet wants to compile the catalog for all stages before running anything. This is kinda cool, since syntax errors will abort the run before the system is [splinched](http://harrypotter.wikia.com/wiki/Splinching). It hurts here because the error would be resolved on the next run if only the first stage would complete!


Example of problem (2) is when out-of-nowhere a circular dependency.
------

This problem is not very well understood by the author of this WebZone. It was less work to do it right and blag about it than to trace it all back. But steps to reproduce.

<pre>

git clone https://github.com/nibalizer/puppet-manifests-oslcat 
git checkout blag-6-10
puppet apply --modulepath=modules setup_env/manifests/nodes/interceptor.pp
</pre>

When run on a host 'interceptor' or a munged interceptor.pp, a confusing error is presented:

<pre>
err: Could not apply complete catalog: Found 1 dependency cycle:
(File[puppetlabs.list] => Apt::Source[puppetlabs] => Class[Mirror::Source::Puppetlabs] => Stage[setup] => Stage[main] => Class[Apt] => File[sources.list.d] => File[puppetlabs.list])
Try the '--graph' option and opening the resulting '.dot' file in OmniGraffle or GraphViz
</pre>

Tracking this down proved to me more difficult than the real solution to the problem.

Using environments as a bootstrap/staging
------

Many readers of this WebZone will recognize the generic [bootstrap.pp](https://gist.github.com/2780516)

```shell
echo "comfiguring apt"
gpg --recv-key 4BD6EC30
gpg --recv-key 4BD6EC30
gpg -a --export 4BD6EC30 | sudo apt-key add -
echo '# puppetlabs' >> /etc/apt/sources.list
echo 'deb http://apt.puppetlabs.com precise main' >> /etc/apt/sources.list
echo 'deb-src http://apt.puppetlabs.com precise main' >> /etc/apt/sources.list

echo "aptitude update && aptitude upgrade"
aptitude update && aptitude upgrade

echo "aptitude install puppet"
aptitude install puppet
```

This work needs to be done. Where to do it is the question. 4 places come to mind. 

1. Default to whatever the distribution ships.
2. Do some/all of bootstrap.sh in kickstart/cobbler/razor
3. Run bootstrap.sh manually or from a script that ssh's in.
4. Use puppet to install puppet.

The recommendation is to use an environment called bootstrap to work with 1/2/3 to get 4 done. 

Implementation
-----

For puppet-manifests-oslcat(which runs entirely by puppet apply), a setup_env directory with a duplicate manifests/site.pp is provided. Each node definition is code that will bring that node into compliance with the newest puppet and install whatever extras are needed(hiera for example). The use of this is pretty obvious:


<pre>

puppet apply --modulepath=modules setup_env/manifests/nodes/interceptor.pp
puppet apply --modulepath=modules --config=/home/nibz/.puppet/puppet.conf manifests/site.pp
</pre>


The same thing can be done in master/agent setups.

Configure bare metal provisioner to include a basic puppet(can be old and janky, since the setup_env manifest will be very basic) and a puppet.conf that points it at your puppetmaster. Point the environment at bootstrap.

<pre>
[main]
environment=bootstrap
logdir=/var/log/puppet
vardir=/var/lib/puppet
ssldir=/var/lib/puppet/ssl
rundir=/var/run/puppet
factpath=$vardir/lib/facter
pluginsync=true
runinterval=1380
configtimeout=600
splay=true
report=true
</pre>

Setup the bootstrap environment on the puppet master next to your other production/testing branches:

<pre>
[main]
modulepath=/etc/puppet/environments/$environment/modules/
manifest=/etc/puppet/environments/$environment/manifests/site.pp
logdir=/var/log/puppet
vardir=/var/lib/puppet
ssldir=/var/lib/puppet/ssl
rundir=/var/run/puppet
factpath=$vardir/lib/facter
pluginsync=true
runinterval=1380
configtimeout=600
report=true
</pre>

<pre>
mkdir -p /etc/puppet/environments/bootstrap/manifests
touch /etc/puppet/environments/bootstrap/manifests/site.pp
ln -s /etc/puppet/environments/production/modules /etc/puppet/environments/bootstrap/modules
</pre>

The bare minimum must go in these manifests. The point here is to make sure whatever git-hook-thing-script-derp is maintaining git/production is also making sure that the same modules are in production as are in bootstrap. Define a module setup. Do bootstraping in setup, include setup in production and in bootstrap. 

```puppet
class setup{
  include mirror::source::puppetlabs
  
  package {
   'rubygems':
      ensure   => present,
   'hiera':
      provider => gem,
      ensure   => present,
      require  => Package['rubygems'],
   'hiera-puppet':
      provider => gem,
      ensure   => present,
      require  => Package['rubygems'],
  }

  file {
    '/etc/puppet/hiera.yaml':
      ensure  => present,
      content => template('setup/hiera.yaml.erb'),
      require  => Package['hiera-puppet','hiera'],
    '/etc/puppet/puppet.conf':
      ensure  => present,
      content => template('setup/puppet.conf.erb'),
      require  => Package['hiera-puppet','hiera'],
  }
}
```


Where the puppet.conf.erb sets the puppet environment to production.

So this is a neat little setup where puppet bootstraps itself into working order, then points itself back at production. You don't have a problem where a provider is broken because puppet on default oneric is too old but can't upgrade puppet becuase a provider is broken.

If you're running puppet with a cron, you can detect the output of a puppet run, and if its at all weird, you can drop into maintenance mode by sed -i ing puppet.conf back into bootstrap.