Cleaning up Puppet code

After months and years of using Puppet, the code base becomes increasingly complex and cluttered. How can you ensure its quality, as well as clean up unused code?

puppet-lint

In the Puppet world, puppet-lint is the reference for code quality. It is used as a standard to check that modules follow the style guide, ensuring consistency in coding style and practices.

puppet-lint can also be used in your control repository, to check your private modules (such as roles & profiles).

There’s at least three ways of achieving this: using PDK, a Rakefile, or onceover along with its code quality plugin.

PDK

PDK, the standard tool to manage Puppet modules in a standard way, also works with control repositories. Once installed, you can convert your control repository and run validation tests with:

$ pdk convert $ pdk validate

Rakefile method

The Rakefile method is a easy way to automate puppet-lint:

Rake::Task[:lint].clear

PuppetLint::RakeTask.new :lint do |config|

  config.ignore_paths = [
    'modules/**/*.pp',
    'vendor/**/*',
  ]

  config.disable_checks = [
    '80chars',
    'documentation',
  ]

  config.fail_on_warnings = true
  config.fix = true if ENV['PUPPETLINT_FIX'] == 'yes'
end

Add a Gemfile in your repository to install puppet-lint:

source ENV['GEM_SOURCE'] || "https://rubygems.org"

group :development, :test do
  gem 'rake', :require => false
  gem 'puppet-lint', :require => false

  # Other lint plugins (optional)
  gem 'puppet-lint-spaceship_operator_without_tag-check', :require => false
  gem 'puppet-lint-unquoted_string-check', :require => false
  gem 'puppet-lint-undef_in_function-check', :require => false
  gem 'puppet-lint-leading_zero-check', :require => false
  gem 'puppet-lint-trailing_comma-check', :require => false
  gem 'puppet-lint-file_ensure-check', :require => false
  gem 'puppet-lint-version_comparison-check', :require => false

  # You can also use the voxpupuli-test gem,
  # which pulls rake, puppet-lint & plugins as dependencies
  gem 'voxpupuli-test', :require => false
end

You can then run the lint with:

$ bundle install --path vendor/bundle
$ bundle exec rake lint

# And if you want to autofix the detected mistakes
$ PUPPETLINT_FIX=yes bundle exec rake lint

Onceover Code Quality

Onceover is a toolbox to automate tasks for Puppet control repositories. Among other things, its code quality plugin allows to run syntax checks and invoke puppet-lint.

In order to use it, update your Gemfile, e.g.:

source ENV['GEM_SOURCE'] || "https://rubygems.org"

group :development, :test do
  gem 'voxpupuli-test', :require => false
  gem 'onceover', :require => false
  gem 'onceover-codequality', :require => false
end

Refresh your bundle and run onceover:

$ bundle update
$ bundle exec onceover run codequality --no-docs

Ideally, run this on every commit in a Continuous Integration/Continuous Deployment setup. At Camptocamp, we use a GitLab CI pipeline to check our control repo using Onceover before deploying it with r10k (also running a GitLab CI runner).

Getting rid of dead code

You’ve checked the quality of your existing code. Good!

But what if you’re actually maintaining and cleaning code that you don’t use anymore? This would be quite the waste of time…

At Camptocamp, we’ve built on puppet-lint to provide a system to detect unused code and help us clean it up. This is what the puppet-ghostbuster project is for.

Under the hood, puppet-ghostbuster is a collection of puppet-lint plugins, distributed in a single puppet-ghostbuster gem. These plugins analyze your Puppet code and then connect to your PuppetDB to check if that code is actually used for any known node. It can also check Hiera data for unused keys.

Just as previously, you can set it up as a Rake task, but our current setup requires a patch to puppet-lint in order to whitelist the puppet-lint checks activated (the current release of puppet-lint only supports blacklisting checks in Rake tasks).

source ENV['GEM_SOURCE'] || "https://rubygems.org"

group :development, :test do
  gem 'rake', :require => false
  gem 'puppet-lint', :require => false,
    :git => 'https://github.com/raphink/puppet-lint',
    :ref => '2cac4fb' # Includes patch for whitelisting checks

  gem 'puppet-ghostbuster', :require => false
end

You can then set up a Rake task such as this one:

PuppetLint::RakeTask.new :ghostbuster do |config|
  config.pattern = ['./site/**/*']
  config.only_checks = [
    'ghostbuster_classes',
    'ghostbuster_defines',
    'ghostbuster_facts',
    'ghostbuster_files',
    'ghostbuster_functions',
    'ghostbuster_hiera_files',
    'ghostbuster_templates',
    'ghostbuster_types',
  ]

  config.fail_on_warnings = true
end

puppet-ghostbuster requires info to connect to your PuppetDB, so you need to provide the following environment variables:

PUPPETDB_URL: URLs to PuppetDB
PUPPETDB_CERT_FILE: path to the certificate to use to connect to PuppetDB
PUPPETDB_KEY_FILE: path to the key to use to connect to PuppetDB
PUPPETDB_CACERT_FILE: path to the Puppet CA certificate
HIERA_YAML_PATH: path to hiera.yaml to use

If you don’t want to provide certificates and keys, you can connect to the PuppetDB through the unencrypted port 8080, for example by forwarding it through SSH.

At Camptocamp, we’re automating this setup by using summon as a wrapper to launch the command. We store the certificates and keys to connect to PuppetDB in gopass, then provide a secrets.yml file like so:

PUPPETDB_URL: https://puppetdb.example.com:8081
PUPPETDB_CERT_FILE: !var:file path/to/secret:cert
PUPPETDB_KEY_FILE: !var:file path/to/secret:key
PUPPETDB_CACERT_FILE: !var:file path/to/secret:cacert
HIERA_YAML_PATH: ./hiera.yaml

Which allows to run: