Allow signed manifests to eliminate single point of compromise

I don't see how this helps prevent a compromised Puppet master from being used for malicious purposes?

This is definitely a much more difficult problem to solve than static packages, because all of the responses from the Puppet server are obviously not going to be the same and can't just be signed with the key on a different server. But something like this is what I'm thinking:

1. A public/private key pair is generated. The public keys are distributed to the Puppet clients. The private key is encrypted with a passphrase.
2. A secure mode is turned on for the Puppetmaster. In this mode, Puppetmaster does not automatically reload manifests. It must be restarted manually. When restarted, the Puppetmaster's startup procedure prompts the user for a passphrase, like Apache when using encrypted certificates. This key is used to sign all communications between the Puppetmaster and clients.
3. The clients verify each message against the public key. If the signature doesn't match the signature the client is expecting, the manifest is discarded and a warning is raised.

It seems like this kind of model wouldn't be difficult to shoehorn into Puppet's existing SSL model, though I don't know anything about how it ties into WEBrick/Mongrel internally.

This still has the following weaknesses:

1. The server can be silently compromised, and before attempting to replace any manifests, the Puppetmaster binary can be replaced with one that fishes the passphrase or stores the unencrypted key for the attacker to use later.
2. Once rooted, the key can be fished out of memory, whether obfuscated or not.

However, assuming a workable security infrastructure is already in place, and the administrator is capable of automatically verifying system binaries before reloading the manifests, this allows an administrator to detect the intrusion and simply correct the problem and reinstall the server before malicious manifests can be deployed across the network.

Regardless of approach, Puppet could really benefit from some mechanism to mitigate cross-network compromises.

I somehow neglected to mention in that whole spiel that the manifests for the production environment could be encrypted using the same key, which would allow the Puppetmaster to verify their authenticity.

Given my efforts to use puppet across customer networks (ie: managed remotely over the net) I've been working on using encrypted and signed manifests from tarball distributed outside puppet. Unfortunately since the responses from the puppetmaster are dynamically generated it makes signing impractical. I sidestepped the issue by not using puppetmaster and just running puppet locally from authenticated manifests.

I am very interested in seeing where this discussion leads.