Testing against a Branch Revision
This document describes how to quickly set-up additional puppetmasterd servers for testing purposes.
Author: Jeff McCune <mccune@math.ohio-state.edu>
Motivation
We all know it's a good idea to test changes before deploying them to a production environment. This can be difficult in a site configuration system because of the overhead setting up a new server, authentication, etc...
With Puppet, it's relatively easy to fork off a branch of your production configuration and start an additional service, serving only the testing manifest. This allows you to test a limited number of clients against the untested policy before merging it back into the production tree.
Assumptions
- Your Puppet configuration files and manifests are under revision control.
- Your revision control system supports branching and merging. e.g. subversion.
- You're running Puppet in a client-server environment.
- /etc/puppet is a subversion working copy on the master.
Module Organisation
As of version 0.22.2, Puppet now supports a configurable search path to load modules. This greatly helps the management of site specific manifest alongside manifest information from external sources. This document should be updated to reflect this new feature. Generally speaking, the module search path allows us to direct a puppet master directly to VCS working copies of isolated pieces of our manifest.
Please see Module Organisation for more information on this excellent feature.
Procedure
Fork production configuration
I assume the puppetmasterd's confdir is under revision control, and is a svn working copy.
In our personal working copy, we need to copy the production configuration to a testing one. We'll make our changes in testing, test them, then merge them back into production once we're satisfied.
svn copy puppet puppet.testing
Once you do this, you should have a layout that looks like this:
puppet/manifests/site.pp puppet/puppetca.conf puppet/puppetd.conf puppet/puppetmasterd.conf puppet.testing/manifests/site.pp puppet.testing/puppetca.conf puppet.testing/puppetd.conf puppet.testing/puppetmasterd.conf
My repository looks like:
Path: . URL: file:///Volumes/svn/repos/management/master Repository Root: file:///Volumes/svn/repos/management Revision: 1 Node Kind: directory Schedule: normal Last Changed Author: mccune Last Changed Rev: 1 Last Changed Date: 2007-02-11 12:23:34 -0500 (Sun, 11 Feb 2007)
Make changes to testing configuration
At this point, you need to edit puppetmasterd.conf to reflect the new location of your testing configuration.
I change all instances of /etc/puppet to /etc/puppet.testing, and other runtime directories. You should keep the same SSL setup so you don't have to re-sign client keys.
Finally, change masterport = 8141. This allows you to run a second server on a non-default port.
Commit these changes back into the repository.
Second puppetmasterd
Now that we have a second copy of the configuration, with relevant changes in the repository, we can start the second server:
svn co file:///Volumes/svn/repos/management/master/puppet.testing /etc/puppet.testing
I include a simple shell script to start the test puppetmasterd in the foreground:
#!/bin/bash export RUBYLIB="/etc/puppet.testing/lib/ruby:$RUBYLIB" exec puppetmasterd --verbose --confdir=/etc/puppet.testing --masterport=8141 $*
Making changes
Now that you have a branched configuration, you make changes in your personal svn working copy within the puppet.testing tree, commit them to the repository, then update the runtime configuration with svn up /etc/puppet.testing. This represents your basic work cycle.
Client Runs
To test a client against this isolated server, you merely need to call it as normal, adding --masterport=8141.
puppetd --test --masterport=8141
Merging back into Production
It's been suggested that Svnmerge.py might be better at handling merges like this procedure describes. You might be better off taking a look at it: http://www.orcaware.com/svn/wiki/Svnmerge.py
Now here's the slightly tricky part. We'll assume that the testing basic work cycle has gone on for some time, and changes have been occurring in both puppet and puppet.testing.
This merge procedure is well documented in the SVN book:
http://svnbook.red-bean.com/nightly/en/svn.branchmerge.html
First, we need to find the revision we forked puppet.testing from.
svn log --verbose --stop-on-copy file:///Volumes/svn/repos/management/master/puppet.testing
------------------------------------------------------------------------ r3 | mccune | 2007-02-11 12:53:56 -0500 (Sun, 11 Feb 2007) | 1 line Changed paths: A /master/puppet.testing (from /master/puppet:1) A /master/puppet.testing/manifests/classes (from /master/puppet/manifests/classes:2) A /master/puppet.testing/manifests/site.pp (from /master/puppet/manifests/site.pp:2) A /master/puppet.testing/puppetd.conf (from /master/puppet/puppetd.conf:2) D /master/puppet.testing/ssl forked puppet to testing branch ------------------------------------------------------------------------
So, we can see that !r3 is the epoch of the testing branch. --stop-on-copy is very useful here.
Here's the final procedure for merging back:
$ cd master/puppet $ svn update At revision 4. $ svn merge -r3:4 file:///Volumes/svn/repos/management/master/puppet.testing U manifests/site.pp
At this point, your personal working copy of the production configuration has been modified with the changes you made to the testing branch. You should review any conflicts, make sure everything merged cleanly with changes that happened to the production branch after the revision you copied from, then commit your changes back:
svn ci -m 'Merged testing back into production'
That's it.
Regarding the Configuration Files
If you're using branch testing, you may only want to merge certain sections of the tree back and forth.
For example, I personally maintain discrete copies of the configuration files, but try and keep my manifest directory merged together.
I'll first commit a change for testing into puppet.testing/manifests, test the change on the second puppetmaster, then merge it into puppet.production/manifests.
This allows me to keep discrete configuration files; /etc/puppet/*.conf and /etc/puppet.testing/*.conf for each of the puppetmaster daemons as working copies on the server. Only the manifest directory is a moving target, the configuration files remain unchanged and discrete.
Persistent Branches
I maintain two branches in subversion, and I always test and commit against the puppet.testing branch. Once I'm satisfied, I merge. This makes for a lot of merging over time, so it helps to keep the last merged revision and the URL of where you're merging from as SVN properties.
I find this incredibly useful for the manifests directory.
For example: (Set last revision of testing which was merged into production)
cd /Volumes/svn/puppet/branches/puppet.production/ svn propset merge:rev '2432' manifests svn propset merge:url 'https://manage.math.ohio-state.edu/svn/puppet/branches/puppet.testing/manifests' manifests
Then, to actually merge, it's as simple as: (Note, we're in the production working directory, merging manifests from the testing repository.)
cd /Volumes/svn/puppet/branches/puppet.production/manifests svn merge -r$(svn propget merge:rev):HEAD $(svn propget merge:url)
Once you've tested your merged changes in the production branch, you can commit and update the merge revision property:
cd /Volumes/svn/puppet/branches/puppet.production/manifests
PRODUCT_REVISION=$(svn propget merge:rev)
TESTING_REVISION=$(svn info $(svn pg merge:url) | grep Revision | awk '{print $2}')
svn propset merge:rev $TESTING_REVISION ./
svn update
svn status
svn commit -m "MERGED: ${PRODUCT_REVISION}:${TESTING_REVISION} from $(svn pg merge:url)"
If you do this each time, it's trivial to track what has and has not been merged into production from testing.
