Puppet agents force dozens of errors: "Could not set 'file' on ensure: incorrect header check"

Recently I had the problem that my homelab's Linux machines managed by Puppet were generating dozens of error messages when updating the catalog:

1# puppet agent --test
2Error: /File[/var/lib/puppet/lib/puppet/parser/functions/load_module_metadata.rb]/ensure: change from absent to file failed: Could not set 'file' on ensure: incorrect header check
3Error: Could not set 'file' on ensure: incorrect header check
4Error: Could not set 'file' on ensure: incorrect header check
5Wrapped exception:
6incorrect header check
7...

This issue first occurred when upgrading Katello to version 3.5. Back then, I also upgraded Puppet from version 3 to 4:

1katello# puppet --version
24.10.12

In accordance with the documentation Puppet 3 clients are still supported. After numerous agent re-installations and further analysis, I had no other ideas.

My colleague Mirko Schmidt gave me the helpful advise of upgrading the Puppet agent - ignoring the official support matrix. My systems were using the Puppet agent from the EPEL repository - in a very old version:

1# rpm -qa | grep puppet
2puppet-3.6.2-3.el7.noarch
3# yum info puppet|grep -i "from repo"
4From repo   : EPEL7 x86_64

With Puppet Collections, Puppet offers a software repository with more updated packages:

1# yum localinstall http://yum.puppetlabs.com/puppetlabs-release-pc1-el-7.noarch.rpm

Upgrading the agent was easy - the package puppet is replaced by puppet-agent:

1# systemctl stop puppet
2# kill $(cat /var/run/puppet/agent.pid) ; killall puppet
3# yum update -y puppet

With this change, also configuration file paths change. Files below /etc/puppet can be found under /etc/puppetlabs/puppet after the upgrade.

As the whole agent is replaced it is necessary to re-configure the software. Existing configuration files might be used furthermore. I suggest creating a backup of the new agent's configuration files before moving the old configuration:

1# cp /etc/puppetlabs/puppet/puppet.conf /etc/puppetlabs/puppet/puppet.conf.initial
2# cp /etc/puppet/puppet.conf.rpmsave /etc/puppetlabs/puppet/puppet.conf

Before starting the new agent, make sure to remove the former certificate from the Smart Proxy. For this, click Infrastructure -> Smart Proxies -> Proxy -> Puppet CA -> Certificates -> Revoke in the Foreman interface.

Afterwards, run the Puppet agent in order to create and submit a new SSL certificate for the Smart Proxy.

1# /opt/puppetlabs/bin/puppet agent --test --noop
2...
3Exiting; no certificate found and waitforcert disabled

Move to the Foreman interface and click Sign to import the certificate. Enable the agent:

1# systemctl enable puppet ; systemctl start puppet

Afterwards, error messages disappeared and the infrastructure status was back to normal again: