Change Management

Many environments require some kind of change management workflow -- changes must pass through multiple phases (e.g., development, testing, production) and often require approval to move between phases. How might such a system be built into Puppet, or built externally and integrated with Puppet?

This problem is usually known as "Change Management", and it must be solved for multiple problem spaces, including software development. We should be able to take examples from those other problem spaces, but I have not done sufficient research on those other spaces at this point (other than trawling the 'net but not finding much in the way of examples).

Potential Solutions

Rather than going through a complicated analysis of all of the options and what effect the different choices have, I'm just going to lay out what I consider to be the basic options, to provide a framework for discussion. From there, we can hopefully reduce the scope of discussion to the more obviously appropriate solutions.

Parse Trees in a Database

Because Puppet's language is declarative, it's relatively straightforward to model it with a database. It wouldn't necessarily be easy, but it would at least be straightforward. With everything in a database, you could provide an interface for selecting values based on whatever criteria you want, and those decisions could be implemented via conditional structures in the database. The approval process could also be relatively straightforward, because they become approvals of an SQL statement.

The downside of this solution, of course, is that it's a heckuva lot of work. First is creating the database modelling and the necessary interfaces into it, but you are also suddenly limited in code-sharing because you have to translate back and forth between SQL and Puppet code, and the scope of your solution is almost entirely open-ended -- where does your change management application end and Puppet begin?

Because of the open-ended nature of this solution, and the large amount of work involved, I do not think this is a good idea.

Functional Approval Lookup

The simplest solution is just to connect Puppet to an external approval mechanism via a Puppet function and simple keyword lookup. The approval system would have registered values that could be modified as necessary, and then the appropriate values could be looked up in the Puppet language:

package { apache:
  ensure => approved("apache-version")
}

The biggest problem with this solution is that systems in different service levels vary by much more than just simple values -- sometimes a different package version has different dependencies or renames a file, and re-factoring an implementation often looks different enough from the original that the difference cannot be modelled with simple values.

The function could be enhanced to support returning code segments to apply, but then you're back to modelling Puppet elements in the database.

Version Control Solutions

Probably the easiest way to handle differences between machines with slightly different configurations is to create a branch in your version control system for each service level, and then create a mechanism for approving individual changes and merges between branches. This allows you to reuse the version control model for everything except the approval process. This approval process should not be too complicated, although it will almost certainly be tied to a specific version control system. You could use post-commit hooks for registering changes and merges and then have the approval system migrate approved changes and merges into a separate branch.

The big upside to this approach is that it's the most minimalist approach and thus the easiest, and it should support creating a decoupled approval system that could be used with any text-based system, and it should be relatively easy to hook into different version control systems. The downside is that it's entirely unsemantic -- approvers will get plain diffs, rather than semantic modifications (e.g., they'll get two lines mentioning the package version, rather than a single statement along the lines of "changing version from X to Y").

Filtering

This isn't really a complete solution, but it's an idea, anyway. There are multiple points at which a configuration could be filtered through an approval process.

The parse tree could be passed through an approval filter that modified it in some way, probably something like creating conditional structures where there previously were none. Alternatively, each individual host's compiled configuration could be passed through a filter which could fill in values based on the results of approvals. This option probably limits what can be done via the approval mechanism, because again it mostly affects values, not the elements themselves.

You could also potentially directly modify the database cache of host configurations via the approval mechanism, but I think that would again limit your flexibility while requiring too much knowledge of Puppet's internals.

Conclusion

This document is meant more as an entry point to discussion, rather than a real solution, so it should not be seen as a complete listing of options nor should it be expected to be complete. Hopefully it's sufficient to continue discussion, anyway.