If you're maintaining multiple Red Hat Enterprise Linux systems (or equivalent offsets like CentOS or Scientific Linux) your administration work with the particular hosts will gain in a routine. Because even the best administrator might forget something it would be advantageously to have a central software and configuration management solution. Chef and Puppet are two very mighty and popular mangement tools for this application. Depending on your system landscape and needs these tools might also be oversized though - RPM can emerge as a functional alternative in this case.

It is often forgotten that RPM can be used for sharing own software and configurations as well. If you're not managing huge system landscapes with uncontrolled growth of software and want to have a easy-to-use solution, you might want to have a look at RPM.

I'm myself using RPM to maintain my whole Red Hat Enterprise Linux system landscape - this article will show you how easy RPM can be used to simplify system management.

Platform

The core of this scenario is a web-service which can be implemented on a dedicated host or on a pre-existing server (e.g. web server) as well. This web-service offers RPM packages for downloading and doesn't even need to be a RHEL or CentOS system because the server only serves the RPM files (and ideally doesn't create them).

RPM packages are created and replicated to the web server using SSH and Rsync on dedicated RHEL or CentOS systems depending on the distribution releases you want to maintain (e.g. RHEL 5 and 6). A YUM repository is created of the RPM packages collection preliminary using createrepo. The YUM repository can be used by additional servers and other clients and consists of multiple sub-directories that are named after the supported processor architectures. After the repository had been configured on the client it can be used for downloading and installing additional software packages. If you only have to maintain RPM packages for one distribution release you can scale down your test environment.

In this case a YUM repository for the Red Hat Enterprise Linux releases 5 and 6 is created.

Web server file structure

The YUM repository directory (myrepo in this case) consists of multiple sub-directories containing the software packages per supported distribution release and processor architecture. The names of these folders are very important - the name has to be the same like the value of the appropriate YUM variable $releasever (discussed later!).

A table of popular RPM-based Linux distributions:

Example: If you want to serve software packages for the Red Hat Enterprise Linux releases 5 and 6 you'll have to create two sub-directories: 5Server and 6Server.

This implies the following directory structure:

 1myrepo/
 2       /5Server
 3            .../repodata
 4            .../noarch
 5                      ...
 6            .../i686
 7                    ...
 8            .../x86_64
 9                      ...
10       /6Server
11               (see above)
12               ...

The appropriate main directories are created on the web server - the sub-directories and further contents are copied to the machine using SSH / Rsync later:

1# mkdir -p /var/www/html/myrepo/{5,6}Server

Preparation

Before RPM packages can be created and served to other hosts using a YUM repository several development tools need to be installed:

1# yum install rpm-build createrepo rpmdevtools

I suggest to create the RPM packages on dedicated hosts or virtual machines and copy the packages to the web server using SSH and Rsync afterwards. The web server should never be used as development environment additionally, due to security reasons. Especially if you want to serve packages for multiple distribution releases (RHEL5, RHEL6) you will definitely need appropriate test environments.

The web server host needs to be prepared for serving the data (if not done yet) - e.g. for a EL system:

1# yum install httpd
2# chkconfig httpd on
3# system-config-firewall-tui
4# service httpd start

Due to security reasons, a dedicated service user for creating the packages is created on the development machines. RPM packages should never be created under the user context of root! Afterwards the needed directory structures are created using rpmdev-setuptree (this command doesn't exist under EL5):

1# useradd su-rpmdev
2# passwd su-rpmdev
3# su - su-rpmdev
4$ rpmdev-setuptree
5$ ln -s /usr/src/redhat ~su-rpmdev/rpmbuild    #symb. link under EL5

If you're using EL5 you'll find the needed structures below /usr/src/redhat - the directory permissions need to be set for the created user su-rpmdev. In the new created folder rpmbuild (respectively below /usr/src/redhat) consists of the following sub-directories:

• BUILD - folder in which source code archives are extracted and compiled.
• RPMS - created RPM packages are stored here in appropriate sub-directories per supported processor architecture (x86_64, i686, noarch,...).
• SOURCES - contains source code archives that are needed for creating RPM binary packages; e.g. software source codes or archives containing configuration files (see example below).
• SPECS - contains structure files (.spec files) that define the content and structure of RPM packages; this file is needed to create the RPM file.
• SRPMS - source code RPM packages; are created on demand. Enables clients to also download source codes and compile packages manually using yumdownloader.

SOURCES and SPECS are the most important directories - most of the time the RPM farmer is moving around there. 😉

Example 1: configuration files

The most administrators will customize the standard configuration of a Enterprise Linux systems to fit their needs. Some popular examples of customizable configuration files are:

• NTP
• sudo
• environment profiles (below /etc/profile.d)
• WINS and DNS configuration

As a matter of principle, creating RPM packages is a complex topic which is only raised roughly in this article. If you want to go deeper with this topic you might want to have a look at the Fedora wiki and documentation - there are plenty useful information:

• http://fedoraproject.org/wiki/How_to_create_an_RPM_package
• http://docs.fedoraproject.org/en-US/Fedora_Draft_Documentation/0.1/html/RPM_Guide/ch-creating-rpms.html

In thie case a NTP configuration shall be packaged in a RPM file and rolled out on all hosts in the company. First of all, a specfile is created for the future RPM package:

1$ cd ~su-rpmdev/rpmbuild/SPECS
2$ rpmdev-newspec mycompany-ntp
3Skeleton specfile (minimal) has been created to "mycompany-ntp.spec".

All rpmdev tools doesn't exist under EL5 so that you'll have to create the specfile on your own!

If you're on EL6, the previous command created a skeleton of a RPM specfile - this file is edited using an editor now:

 1$ vim mycompany-ntp.spec
 2Name:           mycompany-ntp
 3Version:
 4Release:        1%{?dist}
 5Summary:
 6
 7Group:
 8License:
 9URL:
10Source0:
11
12BuildRequires:
13Requires:
14
15%description
16
17%prep
18%setup -q
19
20%build
21%configure
22make %{?_smp_mflags}
23
24%install
25rm -rf $RPM_BUILD_ROOT
26make install DESTDIR=$RPM_BUILD_ROOT
27
28%clean
29rm -rf $RPM_BUILD_ROOT
30
31%files
32%defattr(-,root,root,-)
33%doc
34
35%changelog

Beside meta information about the application, additional scripts for compiling and creating the package are included. The particular meta variables are largely self-explanatory - some explanations:

• Name - name of the software package (mycompany-ntp-1.0-1)
• Version - version of the software package (my-company-ntp-1.0-1)
• Release - release of the software package (my-company-ntp-1.0-1)
• Summary - short package description
• Group - software group the package belongs to. A full listing of possible groups can be found below: /usr/share/doc/rpm-*/GROUPS
• License - assigned license (GPL, AGPL, MIT,...)
• URL - URL of the project website
• Source0 - relative path starting in SOURCES to the source code
• BuildRequires - dependencies that are needed to compile the source code (e.g. header files or library source codes)
• Requires - dependencies that are needed to run the application (in this case: the NTP daemon)
• Conflicts - defines third-party software packages and version that are incompatible with the customized software packages (missing in template - but very important)
• %description - long description of the software package
• %changelog - changelog, applied changes, authors, and so on...

Some of the additional script or macro segments:

• %prep - preparation before compiling the source code; e.g. applying patches
• %build - compiling the source code
• %install - creating the RPM package; copying of created binary files and directory structures to the RPM package

To be honest - it's not hard to lose track in the beginning. I suggest to have a look at finished RPM specfiles - often you'll learn some "tricks" by having a look at other's work.

There are two ways to have a look at finished RPM specfiles. Some additional repositories are serving their specfiles over SVN or GIT. As an example, Repoforge has a public GIT mirror for this: [click me!]

Another possibility is to include the optional source code channels of additional repositories - like EPEL - and download the source code packages:

 1# vi /etc/yum.repos.d/epel.repo
 2...
 3[epel-source]
 4...
 5enabled=1
 6
 7ESC ZZ
 8
 9# yum install yum-utils
10# yumdownloader --source nrpe

The RPM packages can be adapted using cpio - another more comfortable way is to use Midnight Commander to examine the package. The RPM package includes a CPIO archive named CONTENTS.cpio - the specfile is stored there:

To build a preferably "clean" package it is important to create a source code archive - even if you only want to share configuration files. It is also possible to create those files directly in the install macro of the specfile - but especially if you want to share multiple or long configuration files you'll lose track . In this case an archive containing the NTP configuration is created:

 1$ mkdir ~/rpmbuild/SOURCES/mycompany-ntp-1.0
 2$ cd ~/rpmbuild/SOURCES/mycompany-ntp-1.0
 3$ vi ntp.conf
 4driftfile /var/lib/ntp/drift
 5server localserver.loc
 6
 7ESC ZZ
 8
 9$ cd ..
10$ tar cf mycompany-ntp-1.0.tar.gz mycompany-ntp-1.0/*

Afterwards the RPM specfile is modified - my version looks like this:

 1$ cd ../SPECS
 2$ cat mycompany-ntp.spec
 3Name:           mycompany-ntp
 4Version:        1.0
 5Release:        1%{?dist}
 6Summary:        MyCompany customized NTP configuration
 7
 8Group:          System Environment/Daemons
 9License:        GPL
10Source0:        %name-%version.tar.gz
11
12Requires:       ntp
13
14%description
15This package includes MyCompany customized NTP configuration files.
16Feel free to delete this package if received outside the MyCompany network.
17
18%prep
19%setup -q
20
21%build
22
23%install
24rm -rf $RPM_BUILD_ROOT
25install -m 0755 -d %{buildroot}%{_sysconfdir}/mycompany
26install -m 0644 ntp.conf %{buildroot}%{_sysconfdir}/mycompany/ntp.conf
27
28%clean
29rm -rf $RPM_BUILD_ROOT
30
31%files
32%config(noreplace) %{_sysconfdir}/mycompany/ntp.conf

I'm sure you notices that the modified NTP configuration file isn't stored at its accurate place (/etc/ntp.conf) - it is rather saved in an alternative directory (/etc/mycompany/ntp.conf). The reason for this is that the former configuration file (which is provided by the ntp package) can be replaced because of the noreplace flag:

1#code quote of the ntp RPM specfile
2%files
3%config(noreplace) %{_sysconfdir}/ntp.conf

This package stores its configuration file in an alternative directory which can't be overwritten by other RPM packages.

You'll have to help yourself by using a "t_rigger trick_" that saves the former configuration and creates a symbolic link to the new configuration after the installation. After the uninstallation of the package this step is rolled back. For implement this you'll have to add the following macros to your RPM specfile:

 1%triggerin -- mycompany-ntp
 2if [ ! -h /etc/ntp.conf -o ! "`readlink /etc/ntp.conf`" = "/etc/mycompany/ntp.conf" ] ; then
 3        if [ -e /etc/ntp.conf ] ; then
 4                mv -f /etc/ntp.conf /etc/ntp.conf.orig
 5        fi
 6        ln -s /etc/mycompany/ntp.conf /etc/ntp.conf
 7fi
 8
 9%triggerun -- mycompany-ntp
10if [ $1 -eq 0 -a $2 -gt 0 -a -e /etc/ntp.conf.orig ] ; then
11        mv -f /etc/ntp.conf.orig /etc/ntp.conf
12fi
13
14%triggerpostun -- mycompany-ntp
15if [ $2 -eq 0 ]; then
16        rm -f /etc/ntp.conf.rpmsave /etc/ntp.conf.orig
17fi
18if [ -e /etc/ntp.conf.rpmnew ] ; then
19        mv /etc/ntp.conf.rpnnew /etc/ntp.conf.orig
20fi
21
22%postun
23if [ -e /etc/ntp.conf.orig -a -h /etc/ntp.conf -a ! -e "`readlink /etc/ntp.conf`" ] ; then
24        mv -f /etc/ntp.conf.orig /etc/ntp.conf
25fi

Simplified summarization of the triggers functions:

1. Installation: if the former configuration file exists  ".orig" is appended to the file name and a symbolic link to the new configuration file is created
2. Deinstallation of the customized NTP configuration: if the former configuration file still exists the file name is reset
3. After uninstalling the customized NTP configuration: remaining additional or newly added NTP configuration files are deleted
4. After uninstalling: resetting the file name of the former NTP configuration file

I abdicated the URL and BuildRequires tags in my specfile because there are no websites or special compiling dependencies for a customized NTP configuration. 😉

Example 2: Meta packages

There are plenty of applications and configuration that are part of a senseful customized system installation - to name some practical examples: sudo configuration, GNU Screen (of course!), and customized application profiles.

To avoid doing the application installation manually everytime, meta packages can be built to simplify the process. These packages don't have their own files - they only have dependencies to other packages. Because of this, RPM meta packages aren't assigned to special processor architectures (x86_64, i686, s390,...) - an additional specification "BuildArch noarch" is added in the specfile.

Another practical example: a meta package for installing NTP including the customized configuration and telnet for checking the daemon's function:

 1$ cd SOURCES
 2$ mkdir mycompany-ntp-full-1.0
 3$ tar cvfz mycompany-ntp-full-1.0.tar.gz mycompany-ntp-full-1.0
 4$ cd ../SPECS
 5$ cat mycompany-ntp-full.spec
 6Name:           mycompany-ntp-full
 7Version:        1.0
 8Release:        1%{?dist}
 9Summary:        MyCompany customized NTP configuration and netstat utility
10
11Group:          System Environment/Daemons
12License:        GPL
13Source0:        %name-%version.tar.gz
14
15BuildArch:      noarch
16Requires:       ntp mycompany-ntp net-tools
17
18%description
19This package includes MyCompany customized NTP configuration files.
20Feel free to delete this package if received outside the MyCompany network.
21
22%prep
23%setup -q
24
25%build
26
27%install
28
29%clean
30rm -rf $RPM_BUILD_ROOT
31
32%files

Using rpm you can list the package's dependencies:

1$ rpmbuild -bb mycompany-ntp-full.spec
2$ rpm --query -Rp ../RPMS/noarch/mycompany-ntp-full-1.0-1.el6.noarch.rpm
3ntp
4mycompany-ntp
5net-tools
6rpmlib(PayloadFilesHavePrefix) < = 4.0-1
7rpmlib(CompressedFileNames) <= 3.0.4-1

Package dependencies can very depending on the distribution release. In this example between RHEL5 and 6 - the packages providing the telnet command are different here:

1rhel5 # rpm -qf $(which telnet)
2krb5-workstation-1.6.1-70.el5_9.2
3rhel6 # rpm-qf $(which telnet)
4net-tools-1.60-110.el6_2.x86_64

You might want to consider this in the specfile:

1Requires:       ntp mycompany-ntp
2%{?el5:Requires: krb5-workstation}
3%{?el6:Requires: net-tools}

The first dependeny line is significant for all releases, the following lines are considered under RHEL5 or RHEL6.

You can also define particular versions in combination with the Requires and Conflicts tags - for example, if you want to reference a myapp package which is at least version 1.1. one of the following lines can be used:

1Requires:        myapp >= 1.1
2Conflicts:       myapp < 1.1

If you want to reference a special version of a package there are also two possibilities - choose one:

1Requires:        myapp = 1.1
2Conflicts:       myapp < 1.1 myapp > 1.1

It's a kind of philosophical question which of the two possibilites is used - like the question if the glass is half-full or half-empty. Either a package is excluded or referenced explicitely. 😉

Further information regarding RPM dependencies can be found on the official RPM website: [click me!]

Let's go back to the former motivation of this meta package: as an alternative you can also define package groups in your own YUM repository. If you have already worked with the YUM commands grouplist, groupinstall and groupremove, you might know this logical grouping of software packages. You can find an interesting article about this in the YUM wiki: [click me!]

Package and repository creation

Okay - we have a RPM specfile now, what's next? RPM packages are creating using the rpmbuild utility. This tool has a plenty of swithces and arguments - as an example you can also create source code packages or packages for additional processor architectures.

Important parameters:

Some examples:

1# Creates a binary and source code package of myapp
2$ rpmbuild -ba myapp.spec
3# Creates a 32-bit binary package of myapp
4$ rpmbuild -bb --target=i686 myapp.spec

Afterwards you'll find a RPM package below RPMS depending on your processor architecture (noarch, i686 or x86_64). If you're a proud owner of a IBM System Z machine you might want to have a look below s390 or s390x. 😉

1$ ls RPMS/*/*.rpm
2RPMS/i386/mycompany-ntp-1.0-1.el6.i386.rpm

This RPM package could now be installed using YUM:

1# yum localinstall --nogpgcheck mycompany-ntp-1.0-1.el6.i386.rpm

Synchronization and automation

After creating the RPM packages on the particular test machines (e.g. RHEL6 and RHEL5) the packages need to be copied to the web server. I suggest using SSH and Rsync for a synchronization between the test machines, the main test machine and the web server.

If you don't want to do this manually every time you can automate this using a small script:

1.Synchronization between the EL5 machine and the main test machine:

 1$ ln -s /usr/src/redhat /home/su-rpmdev/rpmbuild
 2$ vi /home/su-rpmdev/export_repo.sh
 3#!/bin/sh
 4rsync -avz --delete /home/su-rpmdev/rpmbuild/RPMS/ /opt/myrepo
 5createrepo --database /opt/myrepo/
 6rsync -avz --delete -e ssh /opt/myrepo/ su-rpmdev@MAIN:/opt/myrepo/5Server
 7
 8ESC ZZ
 9
10$ chmod +x /home/su-rpmdev/export_repo.sh
11$ ./home/su-rpmdev/export_repo.sh

The first rsync command copied all RPM packages of all processor architectures below  /opt/myrepo - if a RPM package is deleted in the source, it is also deleted below /opt/myrepo. createrepo created a SQLite database for the YUM repository (myrepo) below /opt/myrepo. The second rsync command copies the local YUM repository to the main test machine (MAIN).

2.Synchronization between the main test machine (EL6) and the web server:

 1$ vi /home/su-rpmdev/export_repo.sh
 2#!/bin/sh
 3rsync -avz --delete /home/su-rpmdev/rpmbuild/RPMS/ /opt/myrepo/6Server
 4createrepo --database /opt/myrepo/6Server
 5rsync -avz --delete -e ssh /opt/myrepo/ su-rpmdev@WEB:/var/www/html/myrepo
 6
 7ESC ZZ
 8
 9$ chmod +x /home/su-rpmdev/export_repo.sh
10$ ./home/su-rpmdev/export_repo.sh

The first rsync command copies all (EL6) RPM packages below /opt/myrepo/6Server. After that createrepo creates a SQLite-Datenbank for the EL6 repository. The second rsync command copies the whole repository (including the other EL5 repository) to the web server (WEB).

Usage and test

To use the new YUM repository on other hosts, an appropriate YUM configuration needs to be created. In this file the repository URL and other parameters like package signing are defined. The syntax looks a bit like good-old Windows .ini files:

1# vi /etc/yum.repos.d/myrepo.repo
2[myrepo]
3name=mycompany packages for EL
4baseurl=http://server01/$releasever
5enabled=1
6gppgcheck=0
7
8ESC ZZ

You might see the variable $releasever - it was mentioned in the table above. This variable is replaced by another value depending on your distribution release - in this case by 5Server or 6Server. These directories had been filled with the appropriate RPM packages from the test machines.

After that, all available packages of the repository can be listed:

1# yum --disablerepo='*' --enablerepo='myrepo' makecache
2# yum --disablerepo='*' --enablerepo='myrepo' list available
3...
4mycompany-ntp                      1.0-1                      myrepo

If you have multiple repository web servers (e.g. because of big setups for processing the amount of requests or compensate failures) you can assign a mirror list:

 1# vi /etc/yum.repos.d/myrepo.repo
 2[myrepo]
 3...
 4#baseurl=...
 5mirrorlist=myrepo.mirror
 6
 7ESC ZZ
 8
 9# vi /etc/yum.repos.d/myrepo.mirror
10http://server01/$releasever
11http://server02/$releasever
12http://server03/$releasever

For every download YUM uses one of these servers - beginning with the first one. If this server fails or doesn't have the file YUM will select the next server.

By the way - these configuration files could be shared using a RPM package, too. If this is done, you only need to install a RPM package to access the YUM repository zugreifen. This is how access to Fedora's Extra Packages For Enterprise Linux (EPEL) is provided:  [click me!]

This package could look like this:

 1# vi SPECS/myrepo-release.spec
 2Name:           myrepo-release
 3Version:        1.0
 4Release:        1%{?dist}
 5Summary:        mycompany Packages for Enterprise Linux repository configuration
 6
 7Group:          System Environment/Base
 8BuildArch:      noarch
 9License:        GPL
10URL:            http://www.mycompany.com
11Source0:        myrepo-release-%{version}.tar.gz
12BuildRoot:      %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n)
13
14%description
15This package contains the mycompany customized Packages for Enterprise Linux repository.
16
17%prep
18%setup -q
19
20%build
21
22%install
23install -m 0755 -d %{buildroot}%{_sysconfdir}/yum.repos.d/
24install -m 0755 myrepo.repo %{buildroot}%{_sysconfdir}/yum.repos.d/myrepo.repo
25install -m 0755 myrepo.mirror %{buildroot}%{_sysconfdir}/yum.repos.d/myrepo.mirror
26
27%clean
28rm -rf $RPM_BUILD_ROOT
29
30%files
31%config(noreplace) %{_sysconfdir}/yum.repos.d/myrepo.repo
32%config(noreplace) %{_sysconfdir}/yum.repos.d/myrepo.mirror
33
34%changelog
35* Sat Jun 29 2013 FirstName LastName  - 1.0-1
36- initial release
37
38ESC ZZ
39
40# tar tvfz SOURCES/myrepo-release-1.0.tar.gz
41-rw-r--r-- su-rpmdev/su-rpmdev 94 2013-06-28 18:12 myrepo-release-1.0/myrepo.mirror
42-rw-r--r-- su-rpmdev/su-rpmdev 189 2013-06-28 18:09 myrepo-release-1.0/myrepo.repo
43
44# tar xvfz SOURCES/myrepo-release-1.0.tar.gz -C SOURCES/
45
46# cat SOURCES/myrepo-release-1.0/myrepo.mirror
47http://server01/$releasever
48http://server02/$releasever
49http://server03/$releasever
50
51# cat SOURCES/myrepo-release-1.0/myrepo.repo
52[myrepo]
53name=mycompany packages for EL
54#baseurl=http://server01/$releasever
55mirrorlist=/etc/yum.repos.d/myrepo.mirror
56enabled=1
57gppgcheck=0

After the RPM specfile had been created the package can be created and distributed easily. Finally, you only have to install the RPM package to use the YUM repository:

1# rpmbuild -bb SPECS/myrepo-release.spec
2# scp RPMS/noarch/myrepo-release-1.0-1.rpm root@host:/root
3root@host # yum localinstall --nogpgcheck myrepo-release*.rpm
4root@host # yum repolist
5repoid            reponame
6...
7myrepo            mycompany packages for EL

Et voilà! 🙂

Perspective / Additional ideas

Of course there is a lot that can be optimized. To name some of these things:

• SSH public-key authentification - while synchronizing the RPM packages between the test machines and the web server the password of the service user has to be entered twice. This can be avoided by implementing SSH public-key authentification.
• Theme for viewing the repository in web browsers - when the repository is accessed using a normal web browser the directory contents are displayed. Depending on your web server this might look ugly - a header including your company's logo and usage information could be used in addition.
• Version control using SVN / GIT - especially if you're working in a team on your repository it might be very handy to use a version control system like SVN or GIT.
• Integration in Red Hat Network Satellite, Spacewalk or SuSE Manager - if you're maintaining a big amount of hosts you (hopefully) have the management suite of your preferred distributor. You can include your repository to simply share your packages.
• Package signing - To avoid manipulated packages, packages can be signed using a key which has to be copied to every client. Using this mechanism, clients can verify that the downloaded packages are authentic.

To manage a plenty of systems you don't need a "egg-laying, milk-bearing woolly sow" - RPM is a mighty tool which can also be used for software and configuration management. If you're engaged with Red Hat Enterprise Linux, you will often into a situation where you have to automate something quickly. And that's where RPM can also help you to reach your goal quickly. 😉