CentOS 7.4, Spacewalk 2.7 and Reposync: Internal Server Error

After my last Spacewalk 2.7 installation I had the problem that synchronising repositories using the web interface failed with the following error:

Reposync error in Spacewalk 2.7 web interface

The system protocol (/var/log/messages) included the following finding:

1Jan 24 21:02:13 st-spacewalk03 server: Caused by: java.lang.RuntimeException: File not found: /var/log/rhn/reposync/icinga2.log
2Jan 24 21:02:13 st-spacewalk03 server: at com.redhat.rhn.common.util.FileUtils.readStringFromFile(FileUtils.java:101)
3Jan 24 21:02:13 st-spacewalk03 server: at com.redhat.rhn.frontend.action.channel.manage.SyncRepositoriesAction.getLastSyncLog(SyncRepositoriesAction.java:215)
4Jan 24 21:02:13 st-spacewalk03 server: at com.redhat.rhn.frontend.action.channel.manage.SyncRepositoriesAction.parseSyncLog(SyncRepositoriesAction.java:227)
5Jan 24 21:02:13 st-spacewalk03 server: at com.redhat.rhn.frontend.action.channel.manage.SyncRepositoriesAction.execute(SyncRepositoriesAction.java:84)
6Jan 24 21:02:13 st-spacewalk03 server: at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:425)
7Jan 24 21:02:13 st-spacewalk03 server: ... 45 more

Strangely, the file existed:

1# cd /var/log/rhn
2# ll -d reposync
3drwxr-xr-x. 2 root apache 4096 18. Jan 23:21 reposync
4# ll reposync
5total 3244
6-rw-rw----. 1 apache apache   94706 24. Jan 20:57 icinga2.log
7-rw-rw----. 1 apache apache 3203348 24. Jan 20:57 opensuse-42.3.log
8-rw-rw----. 1 apache apache    7915 24. Jan 20:50 spacewalk-27-client.log

Also, the log file's SELinux type was noticeable:

1# ll -Z /var/log/rhn/reposync
2-rw-rw----. apache apache unconfined_u:object_r:unconfined_t:s0 icinga2.log
3-rw-rw----. apache apache unconfined_u:object_r:unconfined_t:s0 opensuse-42.3.log
4-rw-rw----. apache apache unconfined_u:object_r:unconfined_t:s0 spacewalk-27-client.log

Restoring the SELinux type changed it to spacewalk_log_t:

1# restorecon -Rv /var/log/rhn/reposync
2restorecon reset icinga2.log context unconfined_u:object_r:unconfined_t:s0->unconfined_u:object_r:spacewalk_log_t:s0
3restorecon reset opensuse-42.3.log context unconfined_u:object_r:unconfined_t:s0->unconfined_u:object_r:spacewalk_log_t:s0
4restorecon reset spacewalk-27-client.log context unconfined_u:object_r:unconfined_t:s0->unconfined_u:object_r:spacewalk_log_t:s0

Unfortunately, it didn't fix the issue - time to have a deeper look at SELinux. With having SELinux disabled, synchronising repositories worked - like mentioned in a mailing list. Of course this cannot be considered as persistent solution - missing permissions can be fixed with a customised SELinux module. For creating a this module, I cleared the audit protocol before reloading the erroneous web site. In the next step, the protocol is read again to ensure that only the last missing permissions are documented in a module code draft (which is named reposync_tomcat in this case):

1# > /var/log/audit/audit.log
2# audit2why -i /var/log/audit/audit.log
3type=AVC msg=audit(1516824133.699:59063): avc:  denied  { read } for  pid=3498 comm="java" name="icinga2.log" dev="dm-3" ino=2409 scontext=system_u:system_r:tomcat_t:s0 tcontext=unconfined_u:object_r:spacewalk_log_t:s0 tclass=file
5        Was caused by:
6                Missing type enforcement (TE) allow rule.
8                You can use audit2allow to generate a loadable module to allow this access.
9# audit2allow -i /var/log/audit/audit.log -m reposync_tomcat > reposync_tomcat.te

The module source code quickly showed that the Tomcat type lacks the permissions to read and write files of the spacewalk_log_t type:

 1# cat reposync_tomcat.te
 3module reposync_tomcat 1.0;
 5require {
 6        type tomcat_t;
 7        type spacewalk_log_t;
 8        class file { open read };
11#============= tomcat_t ==============
12allow tomcat_t spacewalk_log_t:file open;
13allow tomcat_t spacewalk_log_t:file read;

For compiling the module, it might be necessary to install the policycoreutils-devel package:

1# yum install policycoreutils-devel

Afterwards, the module can be compiled and installed like this:

1# make -f /usr/share/selinux/devel/Makefile reposync_tomcat.pp
2# semodule -i reposync_tomcat.pp

Using semodule it is possible to check whether the module has been loaded:

1# semodule -l | grep reposync
2reposync_tomcat 1.0

Tadaaa - working:

Reposync in Spacewalk 2.7 web interface

If you still have issues with syncing repositories, you might also want to enable the following SELinux setting (thanks to Rick from the comment section below):

1# setsebool -P tomcat_read_rpm_db 1

Regarding this issue, there is also a bug in the Red Hat bug tracker: "1522939: Internal Server Error - Syncing Repos to Channel".