High Availability Recipe

Problem:

Sometimes, when configuration management is a crucial part of a bigger picture, e.g. server management, you just can't rely on one puppet master that stores your resources (fileservers) and configuration data. The easy solution would be having two puppet masters running which stores the same resources and configuration data. When doing this, different puppet masters will have issues with trusting certificates signed by puppetmasters different from themselfves, and clients will store their signed certificates which probably aren't trusted by other puppetmasters. This is not a bug or error, it's just what these certificates guarantee, having a trust relationship between a puppetmaster and a client.

This recipe will handle a - possible, but certainly not the only one - way to set up high availability with Puppet.

Solution

The tools which I will use to have HA are:

• heartbeat: provides death-of-node detection and communication.
• mon: provides death-of-service detection (in our situation: local on the machine). When a death-of-service is detected, it will stop the heartbeat process on the current machine so the heartbeat on the other server will detect a death-of-node.
• a redundant IP: this is specified as a heartbeat resource. When a death-of-node is detected, it will assign this resource to another active node (in our case: the second puppet master).

The certificates have a following way of living:

• When a puppet client pulls configuration from a puppet master, it will check for a self created certificate.
  • When this certificate isn't available, it will create one and send it to the master. The master will sign it automatically and send it back to the puppet client, who will store it in his certificate repository. When this handshake is done, the client knows that the master is trusted, and the master will permit the client to pull configuration from him.

To ensure that the problem (as described above) will not occur in the future, follow this (chronological) way to start this high availability service:

1. Set up the puppet master (including the CA service) on the first server.

   firstpuppetmasterhost# /etc/init.d/puppetmaster start
   

2. Start a client on the second server (which will generate certificates and let them get signed by the first puppet master.

   secondpuppetmasterhost# puppetd --test --server firstpuppetmasterhost
   

3. Copy the CRL certificate (/var/lib/puppet/ssl/ca/ca_crl.pem) from the first server to the second server (same directory). Start the puppet master (with no CA service) on the second server.

   firstpuppetmasterhost# scp /var/lib/puppet/ssl/ca/ca_crl.pem $user@secondpuppetmasterhost:/var/lib/puppet/ssl/ca/
   secondpuppetmasterhost# puppetmasterd --noca
   

4. All clients can now pull configuration from both servers, certificates will still need to be validated by the first server.

   puppetclient# puppetd --test --serve firstpuppetmasterhost --server [firstpuppetmasterhost|secondpuppetmasterhost]
   

You probably want autosigning enabled, you can do this by adding the following lines to the puppetd.conf:

[ca]
   autosign = true

The CRL certificate is shared by the two puppet masters, which will grant all certificates signed by the first puppet master.

To ensure that both puppetmasters share the same puppet configuration data, you can let puppet handle the consistency of the configuration data itself. This can be done by running a puppet client on both two puppetmasterservers. Below is a bash script which can help in converting the configuration data files into a puppet configuration file:

#!/bin/bash
# 07/12/2006 v0.1
# koen.vereeken at gmail dot com

TMPFILE=/tmp/$RANDOM.pp
PUPPETDIR=/etc/puppet/
HOSTNAME=firstpuppetmasterhost
OTHERPUPPETMASTER=secondpuppetmasterhost

echo "class puppetcf {" > $TMPFILE

for directory in `find $PUPPETDIR -type d` ; do
        echo "  file { \"$directory\":
                owner => root,
                group => root,
                ensure => directory,
                mode => 644
        }" >> $TMPFILE
done

for file in `find $PUPPETDIR -type f` ; do

        filestripped=`echo ${file}|sed -e 's/\/etc\/puppet//'`

        echo "  file { \"$file\":
                owner => root,
                group => root,
                mode => 644,
                source => \"puppet://$secondpuppetmasterhost/puppetcf${filestripped}\"
        }" >> $TMPFILE
done

echo "}" >> $TMPFILE

mv $TMPFILE $PUPPETDIR/manifests/projects/puppetcf.pp

Everything is set up now to use HA. Now we have to configure the HA tools. I'm assuming you know how to use these tools, I'm only going to describe and list some scripts and configuration that I've used to let it work with Puppet.

• mon: put this in the /etc/mon/mon.cf:

  hostgroup puppetmaster  localhost
  
  watch puppetmaster
          service puppetmasterd
                  interval 30s
                  monitor puppet.monitor
                  period wd {Mon-Sun}
                          alert stop-heartbeat.alert
  

and /usr/lib/mon/alert.d/stop-heartbeat.alert:

/usr/lib/heartbeat/hb_standby

and /usr/lib/mon/mon.d/puppet.monitor

#!/bin/bash
# koen.vereeken at gmail dot com
# this can probably be far more fine grained
# but it does the thrick now
exitstatus=0
processid=`ps -u puppet|grep puppetmasterd|awk '{print $1}'`

if [ "$processid" == "" ] || test `netstat -apn|grep "${processid}/ruby"|wc -l` -eq 0  ; then
        exitstatus=1
fi

exit $exitstatus

• heartbeat: add a redundant IP into the /etc/ha.d/haresources file:

  firstpuppetmasterhost       172.16.32.40/21/eth0:0
  

Just start up the daemons and let the HA do its work

 firstpuppetmasterhost# /etc/init.d/heartbeat start && /etc/init.d/mon start
 secondpuppetmasterhost# /etc/init.d/heartbeat start && /etc/init.d/mon start

Of course you can use other HA tools, as discussed in the puppet list (round robin DNS, UCARP, ...) but this is what I've been using and it works great. Thanks to Luke for helping me out with the certificate problem.

- Koen Vereeken?