System management with Foreman/Katello – Part 2: Products, Repositories and Content Views

The last post of this series introduced the software projects Foreman and Katello. If you followed the instructions, you were able to login to a fresh Foreman installation.

In this part, the new installation is filled with live: first software contents are added to deploy and manage systems later.

If you already had a chance to work with Spacewalk (which was the upstream project for the previous Red Hat Satellite major version) might be puzzled by the new terms that are used within Katello:

  • Repository
  • GPG ley
  • Product
  • Content View
  • Composite Content View
  • Lifecycle Path

Repository

Let’s explain them bit by bit. The first term in this list might be self-explanation: Repository. A repository mostly stands for a YUM mirror, something like:

http://mirror.centos.org/centos/7/updates/x86_64/

In this example you will find all CentOS 7 updates for the x86_64 architecture. Keep in mind that Katello not only allows YUM repositories – Docker, Puppet and conventional file repositories are also possible. In other words: you can also distribute Docker and Puppet modules – we will focus on this in a later blog post. Debian packages are not officially supported yet – but it is planned to change this; there is already an implementation in progress (ATIX AG is major contributor). So it is quite realistic that Katello will also be able to maintain Debian and Ubuntu systems in near future.

GPG key

If you ever imported a YUM repository, you already know that they are mostly signed. Therefore, it is required to import a key before the first use:

# yum install foobar
...
Importing GPG key 0x0608B895:
 Userid: "EPEL (6) <epel@fedoraproject.org>"
 From : http://download.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-6
Is this ok [y/N]:

Within Katello, GPG keys are created per repository so that packages can verified before importing them. In addition to that, systems installed by Foreman can import the required keys enabling automatic package installation.

GPG keys are maintained via the web interface below the menu Content > GPG Keys, they can be used for multiple repositories.

Product

A product basically consists of the following three elements:

  • Repository
  • GPG key
  • Synchronisation plan

It makes sense to group associated software content in a product – e.g. the operating system and all dependent repositories (updates, extras, source codes,…). The synchronisation plan defines how often content updates are downloaded. Beside a time frame, it is also possible to define whether synchronisation should perform weekly, daily or hourly.

During product creation, available repositories can be discovered automatically after specifying the YUM mirror – this feature is called Repo Discovery.

Products are defined within the web interface below the menu Content > Products.

An example for a valid product definition:

  • Product: CentOS 7 x86_64
  • GPG key: http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-7
  • Repositories:
    • OS: http://mirror.centos.org/centos/7/os/x86_64/
    • Updates: http://mirror.centos.org/centos/7/updates/x86_64/
  • Synchronisation: daily, 03:00am

Content View

A Content View (CV) represents a snapshot that can include the following:

  • YUM repositories and filter
  • Puppet modules
  • Docker and OStree content
Create YUM filter

Create YUM filter

YUM filter help with including or excluding packages by means of the certain criteria such as:

  • Name
  • Package group membership
  • Errata ID, date or type

Especially the last filters can be of major interest – e.g. if systems must only be updated quarterly. Using this filter, updates can be “freezed” to a particular date so that only package updates that have been released in this time frame can be installed. Another example would be excluding development utilities per definition on production machines.

Content Views are released in versions – a version includes the summary of all packages that were available during creation. On the other hand, this also means that Content Views need to be updated if recently added packages should be integrated. This is huge difference in comparison with Spacewalk where imported packages were available immediately. It might sound confusing – but this change totally makes sense in the section after next – I promise! 🙂

If you want to automatically update some CVs (e.g. nightly for development machines), you can automate this using the hammer cli or katello-cvmanager.

When publishing new versions, comments can assist with following changes – which is mandatory for bigger environments. If colleague A is wondering about plenty of new packages in a CV, a hint “Product XYZ added” by colleague B might be very helpful.

Once a CV version was saved, it can be presented to a Lifecycle Environment – more on that later.

Composite Content View

Composite Content View (CCV) is a combination of multiple Content Views. This mostly applies to bigger setups in order to increase clarity and reduce maintenance overhead. Using this, it is possible to create CCVs that include all necessary operating system and application repositories per function and host. If operating system and application are updated in independent cycles (e.g. application is updated faster/more frequent than operating system), maintaining CCVs can be very valuable.Another use case could be that operating system and application maintenance is done by different teams.

The following graphics show the differences between conventional CVs and CCVs in an environment with the following products:

  • Operationg system: CentOS 7
  • Applications: App1, App2, App3

Operating system and applications are updated independently. Some systems host an application, some not.

Dedicated CVs per OS and application

Dedicated CVs per OS and application

With conventional CVs it quickly figures out, that this means more maintenance effort. Per application dedicated CVs need to be created – every CV also contains the operating system product. Because operating system and applications are updated independently, this results in more CV version updates. If the operating system is updated, it is important to keep in mind that all other application CVs (App1, App2, App3) also needs to be updated as they also contain the operating system product.

CVs per application, CCVs per use-case

CVs per application, CCVs per use-case

With coming CCVs into picture, this maintenance effort can be reduced. In this case, CVs are maintained per operating system and application; those can be updated independently – also from different teams. Operating systems and applications are linked in a CCV – it is also possible to refer to specific CV versions. In other words: it is possible to combine a newer application CV version with an older operation system CV version. If one of the referred CVs is updated, the effort to update the CCV is significantly lower im comparison to conventional CVs (see above).

Lifecycle Environments and Path

In my opinion, another benefit in comparison with Spacewalk is the implementation of Lifecycle Paths and Environments. Systems can be grouped by their purpose – the following three-system-landscape is quite common and can be represented perfectly with Lifecycle Environments:

  • Development (Dev)
  • Quality assurance (QA)
  • Production (Prod)

Keeping the example from above in mind and thinking about adding recent package updates, leads into a new use case. Usually, new patches should not be available to all systems in the first place. It is a good idea to test them on development and QA systems before installing them on production systems – otherwise, things could get horribly wrong.

Lifecycle Path describes the sequence in which CV updates are released (“promoted”) – e.g. for the previous example:

  1. Dev
  2. QA
  3. Prod

Using the web interface, this order needs to be obeyed; when using the command-line utility hammer, this order can be bypassed.

It is possible to define multiple Lifecycle Environments and Paths – e.g., if you need to maintain multiple applications with different staging environments.

Example: CentOS 7

Time to turn theory into effective practice. In this example, all the necessary preparations to deploy CentOS 7 along with an application requiring the PostgreSQL 9.6 database will be done. As CentOS 7 only offers version 9.2 of the PostgreSQL database, a third-party repository needs to be added.This process should undergo the entire Lifecycle Paths Dev, QA and Prod. To summarize, we need:

  • Lifecycle Paths Dev, QA and Prod
  • Synchronisation plan
  • CentOS 7 product including GPG key, repositories and CV
  • PostgreSQL 9.6 product including GPG key, repository and CV
  • application CCV of both products
  • promote CV and CCV verions

The first step is to create Lifecycle Paths – move to the menu  Content > Lifecycle Environments. By clicking Add New Environment, a dialog is opened; fill it like following:

Create Lifecycle Environment

Create Lifecycle Environment

  • Name: environment name
  • Label: appropriate internal name
  • Description: short description

For Dev, QA and Prod, appropriate LIfecycle Environments are created.

The next step is to create GPG keys – move to the menu Content > GPG Keys. Open the next dialog by clicking Create GPG Key:

Import GPG key

Import GPG key

  • Name: GPG key name, might want to pick file name
  • GPG Key Contents: GPG key content

Alternatively, you can also upload a GPG key as file.

For this example, the following two GPG keys are needed:

  • http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-7
  • https://download.postgresql.org/pub/repos/yum/RPM-GPG-KEY-PGDG-96
Create synchronisation plan

Create synchronisation plan

To enable Foreman respectively Katello to automatically download packages, synchronisation plans are needed. To start the appropriate assistant, click Content > Sync Plans > Create Sync Plan. Enter the following information:

  • Name: Rule name
  • Description: Short description
  • Interval: hourly, daily or weekly
  • Start Date: Date the rule will be activated
  • Start Time: Execution time

Afterwards, products are created. To avoid the need of defining repositories manually, we will leverage the Repo Discovery functionality. Clicking Content > Products > New Product will start the assistant. This time, enter:

Create Puppet product

Create Puppet product

  • Name: Product name
  • Label: Appropriate internal name
  • GPG Key: GPG key just created
  • Sync Plan: Synchronisation plan just created
  • Description: Short description

In the next step, repositories can be added to the product. By clicking New Repository, they could be created in a manual manner. könnte man manuell diese manuell anlegen. To browser a mirror recursively for repositories, start the appropriate assistant by clicking Content > Product > Repo Discovery. Especially, this is quite useful when multiple YUM repositories are on one server and you only need some of them – e.g. like for the CentOS repositories. Fill the dialog like this:

  • Repository Type: Yum Repositories
  • URL to Discover: search entry point, in this case http://mirror.centos.org/centos-7/7/

By clicking Discover, search is started; afterwards detected repositories can be checked and assigned to the product just created by clicking Create Selected. Keep in mind to check the option Existing Product and select the appropriate product:

Now, a first CV on product base can be created: Content > Content Views > Create New View; enter:

  • Name: CentOS
  • Label: <generated automatically>
  • Description: CentOS product
  • Composite View: No

By clicking Save all available repositories are listed automatically. Check the CentOS repositories and click Add Repositories. Afterwards, a new CV version can be stored by clicking Publish New Version – after entering an optional description click Save.

Beside CentOS, another product for PostgreSQL is needed – click Content > Product > Create Product again. This time, the required repository is created manually. After clicking New Repository, the form is filled as follows:

Create repository

Create repository

  • Name: PostgreSQL 9.6
  • Label:  <generated automatically>
  • Type: yum
  • URL: https://download.postgresql.org/pub/repos/yum/9.6/redhat/rhel-7-x86_64/
  • Download Policy: On Demand
  • Mirror on Sync: yes

Foreman offers three different download policies:

  • Immediate: Metadata and packages are downloaded immediately
  • On Demand: Metadata is downloaded immediately; packages are downloaded when a client requests them (default)
  • Background: Metadata is downloaded immediately; packages are downloaded in the background

Specifying the GPG key is optional here as we already supplied the key at product level.

The option Mirror on Sync can be helpful for repositories that often update program versions and remove old versions completely. In other words: removed outdated packages are also removed locally.

Another CV is required for  PostgreSQL – repeat the steps above for this product.

At least, a CCV combining both CVs is . created – Content > Content View > Create New View:

  • Name: CentOS + PGSQL 9.6
  • Label: <generated automatically>
  • Description: CentOS and PostgreSQL 9.6
  • Composite View: Ja
Promote version

Promote version

After creating the CCVs, make sure to pick it from the list. Click Content Views > Add to see available CVs that can be added. In this example, the two available CVs are selected. Click Publish New Version to start the assistant for publishing new versions.

Yay! The last step is to present the new version to the appropriate Lifecycle Environments. To manage this, click Promote – in the dialog, select the appropriate environment. Repeat this step for every environment stage. 🙂

The next part of this post series will focus on creating and maintaining the first host.

19 comments Write a comment

  1. I’ve got a question. I’m trying to wrap my head around Content Views. I understand that they are released in versions, and as you said a version includes the summary of all packages that were available during creation.

    So in my current understanding, when you sync repositories for a particular product and then add said product to some Content View and publish that Content View, what happens with the repository files that were synced the following day? They should not be available in the Content View since they were not there yesterday when it was published. This would suggest that you need to publish a new version of the Content View every time you want new packages to be available/visible. From my experience it takes a fair amount of time to publish/promote content and I find this approach rather awkward. On a similar note, if you are developing Puppet modules, does it mean that you have to publish a new version of the Content View every time you update a module?

    • Good morning Tomas!
      You’re absolutely right. Package updates that has been downloaded after you published a CV version are not automatically included. As updating CVs and CCVs takes some times, there are ways to automate this (e.g. nightly cronjob) – you might want to have a look at katello-cvmanager or the hammer cli which can be used to automate this via a script. It is a common procedure to automate this for development systems but do this manually for production machines – katello-cvmanager offers appropriate configuration options to implement this.
      Thanks for the feedback! Just added this information to the blog post.
      The same applies for Puppet modules. So when starting with Foreman/Katello and implementing basic content it’s quite awkward until you have a fully working environment – I agree with that!

      • Thank you. I am aware of the Hammer CLI tool and I use it extensively. My concern was with regards to Content Views as I was wondering whether I got it right. It’s not a problem to use Hammer CLI to kick a repository sync on a regular basis (e.g. once a week) and then publish a new Content View and promote to a test/qa/prod lifecycle environments.

        My issue is related to Puppet modules and their development as these modules change rather frequently and I cannot keep up with publishing new Content Views every time there is a change made. I believe I may be using the wrong tool (Katello) for the job here, or not using it the right way, not sure yet. Thanks to your response I see that I’m not the only one having this issue.

      • You’re welcome!
        I know the issue with handling software content and Puppet with one tool. I think the toolset depends on your system landscape’s size. If you have really big environments, utilising Puppet Enterprise is a common approach – it might be a besser solution as it also comes with a newer Puppet version. Maybe it would also be worth a try managing Puppet modules in a dedicated CV and assigning CCVs to your systems.

        Best wishes,
        Christian!

      • I think you misunderstood. Puppet version is not a problem, nor is the environment size, the problem is Katello’s way of presenting Puppet modules to subscribers. Migrating to Puppet Enterprise solves nothing as long as you use Katello. I came to realise that Katello isn’t the tool that works well in an environment with rapid Puppet module development. Don’t get me wrong, Katello is perfect for production use. If you have a stable environment with infrequent changes, then publishing CVs works really well.

  2. Good morning. Good morning.

    I really liked the article because it addresses directly the installation and principles of the foreman with katello.

    What I missed was an approach to creating a Debian repository.

    • Hello Julio,
      Thanks for the feedback!

      You’re right – totally missed that because there is currently no stable implementation for managing Debian machines with Foreman and Katello. You can manage Debian with Foreman, but staging DEB packages is not stable at this point. Once this is stable (there are first implementations for Pulp being implemented right now) I will create a post about that. 🙂

      Best wishes,
      Christian.

  3. Hi Christian,

    Thanks for the great blog post. You have clearly demonstrated the process but I’m trying to play around with it but I don’t want to have all the services running on a single server.

    I already have a dhcpd server in my environment and I was trying to install katello to first start with baremetal OS provisioning but I still couldn’t figure out the best way to do it. Hope you can share some insight.

    Thanks

    • Hey Vim,
      thanks for the feedback!

      That’s actually a good point. I will focus on that in a later post of this series. 🙂

      I will keep you updated!

      Best wishes,
      Christian.

Leave a Reply