Puppet agents force dozens of errors: "Could not set 'file' on ensure: incorrect header check"
Recently I had the problem that my homelab's Linux machines managed by Puppet were generating dozens of error messages when updating the catalog:
1# puppet agent --test 2Error: /File[/var/lib/puppet/lib/puppet/parser/functions/load_module_metadata.rb]/ensure: change from absent to file failed: Could not set 'file' on ensure: incorrect header check 3Error: Could not set 'file' on ensure: incorrect header check 4Error: Could not set 'file' on ensure: incorrect header check 5Wrapped exception: 6incorrect header check 7...
This issue first occurred when upgrading Katello to version 3.5. Back then, I also upgraded Puppet from version 3 to 4:
1katello# puppet --version 24.10.12
In accordance with the documentation Puppet 3 clients are still supported. After numerous agent re-installations and further analysis, I had no other ideas.
My colleague Mirko Schmidt gave me the helpful advise of upgrading the Puppet agent - ignoring the official support matrix. My systems were using the Puppet agent from the EPEL repository - in a very old version:
1# rpm -qa | grep puppet 2puppet-3.6.2-3.el7.noarch 3# yum info puppet|grep -i "from repo" 4From repo : EPEL7 x86_64
With Puppet Collections, Puppet offers a software repository with more updated packages:
1# yum localinstall http://yum.puppetlabs.com/puppetlabs-release-pc1-el-7.noarch.rpm
Upgrading the agent was easy - the package
puppet is replaced by
1# systemctl stop puppet 2# kill $(cat /var/run/puppet/agent.pid) ; killall puppet 3# yum update -y puppet
With this change, also configuration file paths change. Files below
/etc/puppet can be found under
/etc/puppetlabs/puppet after the upgrade.
As the whole agent is replaced it is necessary to re-configure the software. Existing configuration files might be used furthermore. I suggest creating a backup of the new agent's configuration files before moving the old configuration:
1# cp /etc/puppetlabs/puppet/puppet.conf /etc/puppetlabs/puppet/puppet.conf.initial 2# cp /etc/puppet/puppet.conf.rpmsave /etc/puppetlabs/puppet/puppet.conf
Before starting the new agent, make sure to remove the former certificate from the Smart Proxy. For this, click Infrastructure -> Smart Proxies -> Proxy -> Puppet CA -> Certificates -> Revoke in the Foreman interface.
Afterwards, run the Puppet agent in order to create and submit a new SSL certificate for the Smart Proxy.
1# /opt/puppetlabs/bin/puppet agent --test --noop 2... 3Exiting; no certificate found and waitforcert disabled
Move to the Foreman interface and click Sign to import the certificate. Enable the agent:
1# systemctl enable puppet ; systemctl start puppet
Afterwards, error messages disappeared and the infrastructure status was back to normal again: