The static compiler was added to Puppet in the 2.7.0 release. 1
The static compiler is intended to produce a catalog that requires fewer fileserver API calls than the regular compiler. The implementation in Puppet 2.7 and 3.0 should result in the elimination of all fileserver metadata calls originating from the agent.
The static compiler replaces all source => "puppet://..." parameters of
file resources with a content parameter containing the checksum of the file
content. The Puppet agent will retrieve the file content from the file bucket
instead of the file server. This behavior eliminates fileserver metadata API
calls because the owner, group, mode, and checksum of the file is contained
directly in the catalog.
The static compiler is included in Puppet 2.7.0 and later. No additional software is required beyond Puppet itself.
The static compiler is under active development. We hope the static compiler will become the default compiler. The underlying implementation and the user-facing configuration of the static compiler will change substantially in the 3.x series. If you plan to use the static compiler in production please be aware configuration changes may be required when upgrading Puppet, even in minor releases.
Specifically, the Filebucket[puppet] resource configures the agent to obtain
file content from a specific remote host. This user-facing configuration
resource will change. Please update and watch
16188 and related tickets for
current information about the agent configuration settings for the static
compiler.
Configure agent nodes to obtain file content from the server file bucket by adding this resource to the configuration catalog of all your managed nodes:
filebucket { puppet:
server => $server,
path => false,
}
The resource title puppet is required. The resource in this example
configures the agent to talk to the puppet master to obtain file contents.
The agent uses the local client bucket if the Filebucket[puppet] resource is
not declared in the catalog. If you receive unexpected messages about file
contents not being found please check to make sure the filebucket resource
specifies the puppet master using the server parameter.
Please watch 16188 for information about when we change this configuration behavior.
Configure the Puppet master to use the static compiler by setting the
catalog_terminus setting to static_compiler.
# /etc/puppet/puppet.conf
[master]
catalog_terminus=static_compiler
Alternatively, you may start a master process directly from the command line as in the following example:
$ puppet master --verbose --no-daemonize --catalog_terminus=static_compiler
This is an example of what to expect in Puppet 2.7:
The agent:
$ puppet agent --test
info: Loading facts in /workspace/puppet-2.7.x/etc/puppet/modules/stdlib/lib/facter/puppet_vardir.rb
info: Loading facts in /workspace/puppet-2.7.x/etc/puppet/modules/stdlib/lib/facter/root_home.rb
info: Caching catalog for mccune.agent
info: Applying configuration version '1346369441'
notice: /Stage[main]/Filetest/File[/tmp/configtest.txt]/ensure: defined content as '{md5}6e2ffdbadb73ae81b65a451cf6c87d12'
notice: /Stage[main]/Filetest/File[/tmp/filetest]/ensure: created
notice: /Stage[main]/Filetest/File[/tmp/filetest/rubygems]/ensure: created
notice: /Stage[main]/Filetest/File[/tmp/filetest/rubygems/README]/ensure: defined content as '{md5}f4d73936a5c4e309ee1b9e99bed57837'
notice: /Stage[main]/Filetest/File[/tmp/filetest/rubygems/bin]/ensure: created
notice: /Stage[main]/Filetest/File[/tmp/filetest/rubygems/Manifest.txt]/ensure: defined content as '{md5}d1c0b4d3aaf9864891d6bf6aa694b192'
...
The master will display checksums when storing the file content into the file bucket:
$ puppet master --verbose --no-daemonize --trace --catalog_terminus=static_compiler
notice: Starting Puppet master version 2.7.19
info: Inserting default '~ ^/catalog/([^/]+)$' (auth true) ACL because /workspace/puppet-2.7.x/etc/puppet/auth.conf doesn't exist
info: Inserting default '~ ^/node/([^/]+)$' (auth true) ACL because /workspace/puppet-2.7.x/etc/puppet/auth.conf doesn't exist
info: Inserting default '/file' (auth ) ACL because /workspace/puppet-2.7.x/etc/puppet/auth.conf doesn't exist
info: Inserting default '/certificate_revocation_list/ca' (auth true) ACL because /workspace/puppet-2.7.x/etc/puppet/auth.conf doesn't exist
info: Inserting default '/report' (auth true) ACL because /workspace/puppet-2.7.x/etc/puppet/auth.conf doesn't exist
info: Inserting default '/certificate/ca' (auth any) ACL because /workspace/puppet-2.7.x/etc/puppet/auth.conf doesn't exist
info: Inserting default '/certificate/' (auth any) ACL because /workspace/puppet-2.7.x/etc/puppet/auth.conf doesn't exist
info: Inserting default '/certificate_request' (auth any) ACL because /workspace/puppet-2.7.x/etc/puppet/auth.conf doesn't exist
info: Inserting default '/status' (auth true) ACL because /workspace/puppet-2.7.x/etc/puppet/auth.conf doesn't exist
info: Expiring the node cache of mccune.agent
info: Not using expired node for mccune.agent from cache; expired at Thu Aug 30 16:29:41 -0700 2012
info: Caching node for mccune.agent
notice: Compiled catalog for mccune.agent in environment production in 0.05 seconds
info: FileBucket read 6e2ffdbadb73ae81b65a451cf6c87d12
info: Content for 'puppet:///modules/filetest/configtest.txt' already exists
info: Metadata for File[/tmp/configtest.txt] in catalog for 'mccune.agent' added from 'puppet:///modules/filetest/configtest.txt'
info: FileBucket read 837ed0f02f9b9a1ec2fb9c00402b2f62
info: Content for 'puppet:///modules/filetest/rubygems-1.3.5/lib/rubygems/commands/contents_command.rb' already exists
info: Metadata for File[/tmp/filetest/rubygems/lib/rubygems/commands/contents_command.rb] in catalog for 'mccune.agent' added from 'puppet:///modules/filetest/rubygems-1.3.5/lib/rubygems/commands/contents_command.rb'
info: FileBucket read bd8dbc44044061597cb06fff3635b7c7
info: Content for 'puppet:///modules/filetest/rubygems-1.3.5/test/mockgemui.rb' already exists
....
info: FileBucket read 56bd3048694a2406ad4e4e4cb674ea49
info: FileBucket read 4d4fb01a806cfe92865ffd5cd9226e88
info: FileBucket read bf573bf37b75ec556095e56be4ee1716
info: FileBucket read dac3a0b4e7551b8a675c5848f8026508
info: FileBucket read 7afedddcf16b6eb33fcd9a5b47b9529f
The static compiler writes to the master's file bucket. In a multi-master setup, the file bucket may not contain the content the agent is requesting if the request is sent to a master that did not compile the catalog for the agent.
We recommend load balancer affinity for all requests coming from a discrete node. Affinity will ensure an agent is able to obtain the file content for all resources contained in the catalog.
Releases of Puppet 2.7.x greater than or equal to 2.7.17 and prior to 2.7.20 are susceptible to issue 15193 where by the following error is raised:
NoMethodError: undefined method `find' for Puppet::FileServing::Content:Class
This special filebucket resource named "puppet" will cause the agent to fetch file contents specified by checksum from the remote filebucket instead of the default clientbucket.
The static compiler increases catalog compilation when compared to the normal compiler. We do not consider this a performance issue per-se because we expect the performance hit to be re-gained in the elimination of metadata API calls from the agent. Please keep an eye on the performance impact of the static compiler on catalog compilation time. We anticipate the need to re-implement the behavior of reading the current metadata server-side, checking the filebucket for the checksum, and storing the content if it doesn't exist. Doing this in-line in catalog compilation may increase compilation time unacceptably long.
Please keep an eye out for response timeouts when agents request their catalogs. Catalog compilation timing out is a clear signal the compilation time is taking too long.