Puppet agents force dozens of errors: “Could not set ‘file’ on ensure: incorrect header check”

Recently I had the problem that my homelab’s Linux machines managed by Puppet were generating dozens of error messages when updating the catalog:

# puppet agent --test
Error: /File[/var/lib/puppet/lib/puppet/parser/functions/load_module_metadata.rb]/ensure: change from absent to file failed: Could not set 'file' on ensure: incorrect header check
Error: Could not set 'file' on ensure: incorrect header check
Error: Could not set 'file' on ensure: incorrect header check
Wrapped exception:
incorrect header check
...

This issue first occurred when upgrading Katello to version 3.5. Back then, I also upgraded Puppet from version 3 to 4:

katello# puppet --version
4.10.12

In accordance with the documentation Puppet 3 clients are still supported. After numerous agent re-installations and further analysis, I had no other ideas.

My colleague Mirko Schmidt gave me the helpful advise of upgrading the Puppet agent – ignoring the official support matrix. My systems were using the Puppet agent from the EPEL repository – in a very old version:

# rpm -qa | grep puppet
puppet-3.6.2-3.el7.noarch
# yum info puppet|grep -i "from repo"
From repo   : EPEL7 x86_64

With Puppet Collections, Puppet offers a software repository with more updated packages:

# yum localinstall http://yum.puppetlabs.com/puppetlabs-release-pc1-el-7.noarch.rpm

Upgrading the agent was easy – the package puppet is replaced by puppet-agent:

# systemctl stop puppet
# kill $(cat /var/run/puppet/agent.pid) ; killall puppet
# yum update -y puppet

With this change, also configuration file paths change. Files below /etc/puppet can be found under /etc/puppetlabs/puppet after the upgrade.

As the whole agent is replaced it is necessary to re-configure the software. Existing configuration files might be used furthermore. I suggest creating a backup of the new agent’s configuration files before moving the old configuration:

# cp /etc/puppetlabs/puppet/puppet.conf /etc/puppetlabs/puppet/puppet.conf.initial
# cp /etc/puppet/puppet.conf.rpmsave /etc/puppetlabs/puppet/puppet.conf

Before starting the new agent, make sure to remove the former certificate from the Smart Proxy. For this, click Infrastructure -> Smart Proxies -> Proxy -> Puppet CA -> Certificates -> Revoke in the Foreman interface.

Afterwards, run the Puppet agent in order to create and submit a new SSL certificate for the Smart Proxy.

# /opt/puppetlabs/bin/puppet agent --test --noop
...
Exiting; no certificate found and waitforcert disabled

Move to the Foreman interface and click Sign to import the certificate. Enable the agent:

# systemctl enable puppet ; systemctl start puppet

Afterwards, error messages disappeared and the infrastructure status was back to normal again: