Using Mongrel with Apache

Alternatively, see: UsingMongrelOnDebian UsingMongrelPound UsingMongrelNginx UsingPassenger

Puppet daemons default to using WEBrick for http serving, but puppetmasterd can be used with Mongrel instead. This requires setting up a web proxy in front of Mongrel to handle the SSL connections, and then you can start as many puppetmasterd instances as you need. Note that Puppet uses Mongrel differently than most -- usually, you would have a mongrel_cluster command that managed multiple Mongrel instances. I planned to implement it this way, but it ended up being easier in the short term to stick with puppetmasterd. Please contact me (Luke) if you want to work on getting mongrel_cluster to work with Puppet, as it could be used to manage multiple Mongrel instances at once.

See the following diagram which shows how it works: http://reductivelabs.com/trac/puppet/attachment/wiki/UsingMongrel/puppetmaster-mongrel.png

Version 0.23.1 or higher of puppetmasterd is required. (Earlier versions couldn't do certificate signing because they didn't support the X-Client-Verify header.)

This document only describes setting up Apache as a proxy in front of Mongrel. Any other http proxy should work, as long as it supports validating client certificates.

Why You'd Do This

Mongrel scales much better than WEBrick, at least partially because it allows you to run multiple processes serving the same pool of clients on the same host. WEBrick only uses Ruby's threading, which does not scale beyond one processor, and it appears that WEBrick starts dropping connections beyond about 2 concurrent connections.

If you're getting connection-reset or End-of-file errors, you should try Mongrel. As more people try it and it proves to be stable, it will eventually become the preferred serving platform for the master.

Known Issues

There is a problem with Apache and the certificate revocation list (CRL) as generated by the puppet CA. If you encounter this problem, you are likely to see lines like this in your error log:

[Mon Nov 10 08:47:20 2008] [warn] Invalid signature on CRL

[Mon Nov 10 08:47:20 2008] [error] Certificate Verification: Error (8): CRL signature failure

If so, comment out the line containing !SSLCARevocationFile from your apache config.

Apache Configuration

Here is a complete apache configuration, intended for use only with puppet:

# Jeff McCune <mccune@math.ohio-state.edu>
# 2007-09-14
#
# Minimal Apache Configuration for Apache+Mongrel+Puppetmaster
#
#  Host System Setup and Configuration:
#  - Add puppet/puppet user/group
#  - Use the following configuration file.
#  - /Library/Puppet/Generated/Server is owned by puppet/puppet
#
#  - If you have a system which doens't provide Apache 2.2.X packages:
#    Apache may be built with:
#
#    ./configure --enable-so \
#       --enable-ssl=shared --enable-proxy=shared --enable-proxy_http=shared \
#       --enable-proxy_balancer=shared --enable-headers=shared \
#       --enable-authz_host=shared --enable-log_config=shared \
#       --prefix=/Library/Puppet/Resources/httpd-2.2.4
#
# - Finally, start the SSL proxy with:
#   /Library/Puppet/Resources/httpd-2.2.4/bin/httpd -f \
#   /Library/Puppet/Versioned/Server/cluster-orange/httpd/puppetmasterd.conf

Listen 8140
PidFile /Library/Puppet/Generated/Server/balancer.pid
User puppet
Group puppet

LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
LoadModule headers_module modules/mod_headers.so
LoadModule ssl_module modules/mod_ssl.so
LoadModule authz_host_module modules/mod_authz_host.so
LoadModule log_config_module modules/mod_log_config.so

<Directory />
    Options FollowSymLinks
    AllowOverride None
    Order deny,allow
    Deny from all
</Directory>

<Proxy balancer://puppetmaster>
  BalancerMember http://127.0.0.1:18140 keepalive=on max=2 retry=30
  BalancerMember http://127.0.0.1:18141 keepalive=on max=2 retry=30
</Proxy>

<VirtualHost *:8140>
    SSLEngine on
    SSLCipherSuite SSLv2:-LOW:-EXPORT:RC4+RSA
    SSLCertificateFile      /Library/Puppet/Generated/Server/SSL/host_cert.pem
    SSLCertificateKeyFile   /Library/Puppet/Generated/Server/SSL/host_key.pem
    SSLCertificateChainFile /Library/Puppet/Generated/Server/SSL/ca/ca_crt.pem
    SSLCACertificateFile    /Library/Puppet/Generated/Server/SSL/ca/ca_crt.pem
    SSLCARevocationFile     /Library/Puppet/Generated/Server/SSL/ca/ca_crl.pem
    SSLVerifyClient optional
    SSLVerifyDepth  1
    SSLOptions +StdEnvVars

    # The following client headers allow the same configuration to work with Pound.
    RequestHeader set X-SSL-Subject %{SSL_CLIENT_S_DN}e
    RequestHeader set X-Client-DN %{SSL_CLIENT_S_DN}e
    RequestHeader set X-Client-Verify %{SSL_CLIENT_VERIFY}e

    <Location />
        SetHandler balancer-manager
        Order allow,deny
        Allow from all
    </Location>

    ProxyPass / balancer://puppetmaster:8140/ timeout=180
    ProxyPassReverse / balancer://puppetmaster:8140/
    ProxyPreserveHost on
    SetEnv force-proxy-request-1.0 1
    SetEnv proxy-nokeepalive 1

    ErrorLog  /Library/Puppet/Generated/Server/balancer_error.log
    CustomLog /Library/Puppet/Generated/Server/balancer_access.log combined
    CustomLog /Library/Puppet/Generated/Server/balancer_ssl_request.log \
                  "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"
</VirtualHost>

Note that you'll have to customize it to match the name of your puppet server. Note also the cluster member declarations -- this is where you tell Apache how to contact the Mongrel instances -- you need a separate line for every instance, and (of course) each instance needs to be started on a separate port.

This configuration starts Apache on Puppet's normal server port, so clients do not need to be reconfigured.

Another Apache configuration

Here's another happy apache configuration, without all the OSXey stuff ('''Note:''' You still need to load the apache modules from the configuration above. Except for ''log_config'' which seems to be OS X specific):

Listen 8140

ProxyRequests Off

<Proxy balancer://puppetmaster>
    BalancerMember http://127.0.0.1:18140
</Proxy>

<VirtualHost *:8140>
    SSLEngine on
    SSLCipherSuite SSLv2:-LOW:-EXPORT:RC4+RSA
    SSLCertificateFile /etc/puppet/ssl/certs/backup4.foo.com.pem
    SSLCertificateKeyFile /etc/puppet/ssl/private_keys/backup4.foo.com.pem
    SSLCertificateChainFile /etc/puppet/ssl/ca/ca_crt.pem
    SSLCACertificateFile /etc/puppet/ssl/ca/ca_crt.pem
    SSLCARevocationFile /etc/puppet/ssl/ca/ca_crl.pem
    SSLVerifyClient optional
    SSLVerifyDepth  1
    SSLOptions +StdEnvVars

    RequestHeader set X-Client-DN %{SSL_CLIENT_S_DN}e
    RequestHeader set X-Client-Verify %{SSL_CLIENT_VERIFY}e

    <Location />
       SetHandler balancer-manager
       Order allow,deny
       Allow from all
    </Location>

    ProxyPass / balancer://puppetmaster:8140/
    ProxyPassReverse / balancer://puppetmaster:8140/
    ProxyPreserveHost on

</VirtualHost>

Configuring Puppet

The only change to Puppet is that you need to start puppetmasterd with --servertype mongrel and choose a separate masterport for each instance. You will also need to specify a separate pidfile for each instance.

The attached example init script for RedHat systems, or this helper script might be useful:

#!/bin/bash
#
# Start a Puppet Master Server instance.
EZPUPPET_CONFDIR="${EZPUPPET_CONFDIR:=${EZPUPPET_BASE:=/Library/Puppet}/Versioned/Server/${EZPUPPET_SERVER:=cluster-orange}/puppet}"

if ! [[ "$1" -gt 0 ]]; then
  echo "ERROR: You must provide a port to run this puppet master on."
  echo "Start at 18140 and increment upward."
  echo "  The apache load balancer is configured to talk to these servers"
  echo "  so the port numbers are NOT arbitrary."
  exit 1
fi

MASTERPORT="$1"
shift

puppetmasterd \
  --confdir="${EZPUPPET_CONFDIR}" \
  --pidfile=${EZPUPPET_BASE}/Generated/Server/var/run/puppetmaster."${MASTERPORT}".pid \
  --servertype=mongrel \
  --masterport="${MASTERPORT}" \
  $*

As mentioned earlier, this would be easier if it were done with mongrel_start, but it will take a significant amount of work to get all of Puppet's existing code working in that framework, and I'm unlikely to do it except as part of contract development. Clearly, patches that provided this would be accepted.

'''NOTE''': The current (as of at least 0.24.6) /etc/init.d/puppetmaster script on RedHat and compatible systems already supports multiple mongrel instances in a similar way to above. Simply specify your Mongrel instance ports in /etc/sysconfig/puppetmaster in BASH array format as per the following example:

PUPPETMASTER_PORTS=([0]=18140 [1]=18141 [2]=18142 [3]=18143 [4]=18144)

Generating Certificates

In version 0.23.2 I found that puppetmasterd would not generate certificates when started using Mongrel. I worked around the problem by using puppetca to generate the certificates as part of the install process, like this:

puppetca --generate <DOMAIN>