Regenerating Puppet SSL Certificates

A bug in the Debian shipped OpenSSL libraries has caused many of us to need to entirely regenerate all of Puppet's SSL Certificates.

The general flow for doing this:

1. Update your openssl packages using apt-get or aptitude
2. Stop your Puppetmaster(s)
3. Stop Puppetd clients
4. Remove (or rename) the certificate directories ($ssldir)
5. Start a webrick Puppetmaster to regenerate your certificates
6. Restart your Puppetmaster
7. Start puppetd on all your clients
8. Sign the certificates

Automation

Below is a Capistrano task that will do this for you, in parallel. It works for us, but since we're doing things like deleting directories in parallel, make sure you understand and trust it before you use it.

What follows are instructions on using this Capistrano task.

Install Capistrano

$ gem install capistrano

Create puppet_ssl_cleanup.rb

Create a file called "puppet_ssl_cleanup.rb", using the contents found here. (Or download the file attached to this page.)

#
# Make sure you set these constants properly!
#

# Set this to true if you are autosigning your certificates
AUTOSIGN = false

# Set this to the commands you need to run to stop your puppetmasterd
PUPPETMASTERD_STOP = [
  "/etc/init.d/puppetmasterd stop",
  "/etc/init.d/mongrel-puppetmasterd stop",
]

# Set this to the commands you need to start your puppetmasterd
PUPPETMASTERD_START = [
  "env SVWAIT=30 /etc/init.d/mongrel-puppetmasterd start",
  "env SVWAIT=30 /etc/init.d/puppetmasterd start",
]

# Set this to the commands you need to stop puppetd on the clients
PUPPETD_STOP = [ "/etc/init.d/puppetd stop" ]

# Set this to the commands you need to start puppetd on the clients
PUPPETD_START = [ "/etc/init.d/puppetd start" ]

# Set this to the location of your puppet SSL directories
PUPPET_SSL_LOCATION = "/etc/puppet/ssl"

# Set this to the URL of your iclassify server, if you have one
ICLASSIFY_SERVER = "https://iclassify.sfo.trusera.com"

has_iclassify = false
begin
  require '/srv/icagent/lib/iclassify'
  has_iclassify = true
rescue
end

default_run_options[:pty] = true

if has_iclassify
  set(:query, ENV["QUERY"]) if ENV.has_key?("QUERY")
  set(:query) do 
    Capistrano::CLI.ui.ask "iClassify Query: "
  end unless exists?(:query)

  set(:password, ENV["PASSWORD"]) if ENV.has_key?("PASSWORD")

  set(:ic_user, ENV["USER"]) unless exists?(:ic_user)
  if ENV.has_key?('IC_SERVER')
    set(:ic_server, ENV["IC_SERVER"])
  else
    set(:ic_server, ICLASSIFY_SERVER)
  end

  ic = IClassify::Client.new(ic_server, ic_user, password)
  ic_nodes = ic.search(query, [ 'fqdn' ])

  ic_nodes.each do |node|
    role :clients, node.attrib?('fqdn')
  end
else
  set(:clients) do
    Capistrano::CLI.ui.ask "Comma Seperated list of Clients to clean: "
  end unless exists?(:clients)
  clients.split(",").each do |c|
    role :clients, c
  end
end

# State which system the Puppet Master is
set(:master) do 
  Capistrano::CLI.ui.ask "Puppet Master FQDN:"
end unless exists?(:master)

role :master, master

default_run_options[:pty] = true

task :stop_puppetmasterd, :roles => :master do
  run_command(PUPPETMASTERD_STOP)
end

task :start_puppetmasterd, :roles => :master do
  run_command(PUPPETMASTERD_START)
end

task :stop_puppetd do
  run_command(PUPPETD_STOP)
end

task :start_puppetd do
  run_command(PUPPETD_START)
end

task :rm_certs do
  sudo("rm -rf #{PUPPET_SSL_LOCATION}")
end

# Oh, what a dirty, dirty thing this is. 
# If you are running mongrel, though, your puppetmasterd will never re-generate your certs
# So this is going to do the right thing for you
# Please forgive me.
task :generate_ca_cert, :roles => :master do
  sudo("puppetmasterd --daemonize")
  logger.info("Waiting 30 seconds for the Puppetmaster to start and generate CA") 
  sleep 30
  sudo("killall -9 puppetmasterd") 
end

task :generate_certs, :roles => :clients do
  run(%{ruby -e 'i = rand(60); puts "Sleeping " + i.to_s; sleep i'})
  sudo("sh -c 'puppetd --onetime --debug --ignorecache --no-daemonize --server #{master}; exit 0'")
end

task :sign_all, :roles => :master do

  sudo("puppetca --sign --all") if AUTOSIGN != true
end

task :rebuild_certs do
  logger.info("Stopping Puppetmasterd")
  stop_puppetmasterd
  logger.info("Stopping Puppetd")
  stop_puppetd
  logger.info("Removing Certificates")
  rm_certs
  logger.info("Regenerating CA Certificates")
  generate_ca_cert
  logger.info("Starting Puppetmasterd")
  start_puppetmasterd
  logger.info("Running puppetd to generate certificates")
  generate_certs
  logger.info("Signing all waiting requests")
  sign_all
  logger.info("Starting Puppetd")
  start_puppetd
  logger.info("Certificates regenerated!")
end

def run_command(const)
  const.each do |cmd|
    sudo(cmd)
  end
end

At the top of the script are a set of constants, which you need to edit for your environment. They involve how to start/stop your puppetmasterd, puppetd, and where your SSL certificates are stored on the filesystem. Edit those now.

Running the Task

If you are using iClassify, you can run the task as follows:

$ cap -f puppet_ssl_cleanup.rb -S master=fqdn_puppetmaster -S query="domain:example.com"

If you are not running iClassify (which I expect is most of you, at this point) you will need to specify the clients to clean as well:

$ cap -f puppet_ssl_cleanup.rb -S master=fqdn_puppetmaster -S client=fqdn1,fqdn2,fqdn3

If you need to run the script as a user other than yourself (as root, say) add -S user=root to the command.