If you need to protect a Linux host against unauthorized access, fail2ban is a very handy service. The application scans log files of various services, automatically detects failed logins and blocks attacker's IP addresses. Especially on public hosts, it is essential to implement a protection like this on prominent services like SSH.

For monitoring bans, the project offers a Nagios / Icinga plugin named check_fail2ban: [click me!]

This plugin uses the fail2ban-client utility for checking bans - for this, a sudo rule is required for NRPE, e.g. for EL7:

1nrpe ALL = (root) NOPASSWD: /usr/lib64/nagios/plugins/check_fail2ban

Unfortunately, the plugin was not working as expected in my environment.

Troubleshooting

Manually running the configured check on the monitoring server was not successful:

1mon$ lib/nagios/plugins/check_nrpe -H myhost.localdomain.loc -c check_fail2ban
2NRPE: Unable to read output

Well, this error message does not help at all - the NRPE log file should deliver more information. In the appropriate configuration file, a debug logging can be enabled:

1# vi /etc/nagios/nrpe.cfg
2...
3debug=1
4
5ESC ZZ
6# service nrpe restart

Depending on your rsyslog (or yet another logging service) configuration, those messages are stored in a separated log file. Under EL7 you will need to check the file /var/log/messages:

1# tail -f /var/log/messages
2...
3May 27 23:59:12 myhost nrpe[11919]: Allowing connections from: 127.0.0.1,xx.xx.xx.xx
4May 27 23:59:12 myhost systemd: Started NRPE.

Under EL7, there are no information about started processes and checks at all - for other Linux distributions this seems to work, a bug?

Next, I enabled a login shell for the NRPE service user and started the command manually using runuser:

1# chsh -s /bin/bash nrpe
2# runuser -l nrpe -c "/usr/lib64/nagios/plugins/check_fail2ban"

This time I stubmled upon a hint - it seems like sudo crashed the process:

1sudo: sorry, you must have a tty to run sudo

sudo requires a fully-featured terminal session for granting extended permissions - which is not the case for NRPE.

The behavior can be disabled in the sudo configuration:

1# visudo
2...
3#Defaults    requiretty
4
5ESC ZZ

Alternatively, you can specify an exception for the NRPE user (nrpe in this case) - e.g. in the file that is required for the check_fail2ban plugin:

1nrpe ALL = (root) NOPASSWD: /usr/lib64/nagios/plugins/check_fail2ban
2Defaults:nrpe !requiretty

After altering the configuration file, the test was executed again - again, this was not successful. Time to have a look at SELinux and disable it temporary:

1# setenforce 0
2# service nrpe restart

Now, the plugin was working:

1mon$ lib/nagios/plugins/check_nrpe -H myhost.localdomain.loc -c check_fail2ban
2CHECK FAIL2BAN ACTIVITY - OK - 1 detected jails with 0 current banned IP(s)

So it seems like a valid SELinux module is missing for the monitoring plugin - we will focus on this in a moment.

Since the NRPE user was not the reason for the failure, it is recommended to change the login shell to its default value and enable SELinux again:

1# chsh -s /sbin/nologin nrpe
2# setenforce 1

Creating a SELinux module

Often, SELinux is the cause for strange errors. In this case, SELinux is not aware of the plugin's behavior - and therefore drops the calls. Fortunately there are plenty of tools to define exceptions. All breaches of SELinux rules are documented in the file /var/log/audit/audit.log - based on this information it is possible to create SELinux modules that allow access. Using the audit2allow utility module drafts can be created - they are compiled into binary modules with the semodule_package command.

Before creating the module draft, it is a good idea to clear the audit log and restart the rebellious utility again - this will ensure that only required exceptions relevant to our plugin are defined. If additional software is dropped by SELinux at the same time, it is possible, that too many exceptions are created. The checkmodule command checks a module's syntax, semodule installs the module.

1# > /var/log/audit/audit.log
2# service nrpe restart
3(Run check)
4# audit2allow >> nrpe_fail2ban.te << /var/log/audit/audit.log
5# checkmodule -M -m -o nrpe_fail2ban.mod nrpe_fail2ban.te
6# semodule_package -o nrpe_fail2ban.pp -m nrpe_fail2ban.mod
7# semodule -i nrpe_fail2ban.pp

semodule can also be used to verify whether the module was loaded successfully:

1# semodule -l | grep nrpe
2nrpe_fail2ban   1.0

Time to run the test once again - after restarting NRPE:

1# service nrpe restart

In my case, additional calls were dropped by SELinux - so let's repeat the process again:

1# > /var/log/audit/audit.log ; service nrpe restart
2(Run check)
3# audit2allow >> nrpe_fail2ban.te << /var/log/audit/audit.log

By executing the audit2allow command, we converted breach of rules into module syntax and appended the information into the pre-existing module draft (*.te file). To ensure syntax compliance we need to move the new rules to the top of the file. Afterwards, the module is created and installed after checking the module syntax - just like in the previous listing.

1# checkmodule -M -m -o nrpe_fail2ban.mod nrpe_fail2ban.te
2# semodule_package -o nrpe_fail2ban.pp -m nrpe_fail2ban.mod
3# semodule -r nrpe_fail2ban ; semodule -i nrpe_fail2ban.pp

I repeated this 8 times until I had documented all the required permissions:

 1# cat nrpe_fail2ban.te
 2
 3module nrpe_fail2ban 1.0;
 4
 5require {
 6  type nrpe_t;
 7  class unix_dgram_socket sendto;
 8  class file execute;
 9  class file getattr;
10  class file { read getattr open };
11  class file execute_no_trans;
12  type fail2ban_client_exec_t;
13  class file { ioctl getattr };
14  class file { read open };
15  class file execute_no_trans;
16  type fail2ban_var_run_t;
17  class sock_file write;
18  class file ioctl;
19  type fail2ban_t;
20  class unix_stream_socket connectto;
21}
22
23#============= nrpe_t ==============
24allow nrpe_t self:unix_dgram_socket sendto;
25allow nrpe_t fail2ban_client_exec_t:file getattr;
26allow nrpe_t fail2ban_client_exec_t:file execute;
27allow nrpe_t fail2ban_client_exec_t:file { read open };
28allow nrpe_t fail2ban_client_exec_t:file execute_no_trans;
29allow nrpe_t fail2ban_client_exec_t:file ioctl;
30allow nrpe_t fail2ban_var_run_t:sock_file write;
31allow nrpe_t fail2ban_t:unix_stream_socket connectto;