Forcefully stop hanging Foreman tasks

Aug 19, 2017 · 2 min read · katello red-hat-satellite-6 ·

After a system crash I recently had the problem that Foreman had several orphaned tasks that could not be stopped from the web interface:

Because I planned a Foreman upgrade this was pretty bad as it prevented the pre-upgrade check:

1# foreman-rake katello:upgrade_check
2...
3Checking upgradeability...
4Checking for running tasks...
5[ERROR] - There are 8 active tasks.

To fix this it is necessary to kill the orphaned tasks with fire. To do so, you will need to retrieve the task ID from the PostgreSQL database:

 1# su - postgres
 2$ psql foreman
 3foreman=# select id, label, started_at, state, result from foreman_tasks_tasks where state != 'stopped';
 4                  id                  |                     label                     |     started_at      |  state  | result
 5--------------------------------------+-----------------------------------------------+---------------------+---------+---------
 6 e140a579-a4a2-4a5f-9c32-97b6ae3700e8 | Actions::Candlepin::ListenOnCandlepinEvents   | 2017-01-11 12:38:12 | paused  | error
 7 e955c5a3-8367-4cea-b468-14f89416eaea | Actions::Candlepin::ListenOnCandlepinEvents   | 2017-04-12 08:42:00 | paused  | error
 8 73ac6137-e4f3-4092-940f-1b0fda065b63 | Actions::Katello::Host::GenerateApplicability | 2017-04-30 11:36:18 | paused  | error
 9 45ecf092-30f5-4d4d-b8db-4bb920fe6bec | Actions::Candlepin::ListenOnCandlepinEvents   | 2017-05-04 08:57:57 | paused  | pending
10 db1711f0-80d8-4cae-8659-c81918c0db25 | Actions::Candlepin::ListenOnCandlepinEvents   | 2017-05-08 07:08:45 | paused  | error
11 4ca57d8f-8b1c-4837-ae1a-580a775f51a8 | Actions::Katello::Host::GenerateApplicability | 2017-08-16 21:12:37 | paused  | error
12 b96b45f7-4c75-4a66-9216-04bfd8de7b50 | Actions::Katello::Host::GenerateApplicability | 2017-08-17 02:31:56 | paused  | error
13 5d3aff81-f169-47c9-adba-752d074a3400 | Actions::Katello::Host::GenerateApplicability | 2017-08-17 10:56:08 | paused  | error
14 4faa5498-aeaa-462f-9c0c-f2ddfe2d403d | Actions::Candlepin::ListenOnCandlepinEvents   | 2017-08-19 17:08:43 | paused  | pending
15 56509b4c-16e1-4167-bcf1-20682cbd23cb | Actions::Katello::EventQueue::Monitor         | 2017-08-19 17:08:44 | paused  | pending
16 404cf497-6cc2-4fbd-8016-cb04fa1423e7 | Actions::Candlepin::ListenOnCandlepinEvents   | 2017-02-21 10:23:01 | running | error
17 8e1be320-bead-4c64-a477-b46604c2bcd7 | Actions::Katello::Host::Erratum::Install      | 2017-03-10 14:44:30 | paused  | error
18(12 rows)

The first column (id) of the dump shows the task ID; the third column (started_at) is also important. All tasks that were started before the last restart of Foreman need to be stopped.

It is a good idea to copy the appropriate IDs - the next step is to abort these tasks using foreman-rake:

1# foreman-rake console
2irb(main)> ForemanTasks::Task.find("e140a579-a4a2-4a5f-9c32-97b6ae3700e8").destroy
3irb(main)> ForemanTasks::Task.find("e955c5a3-8367-4cea-b468-14f89416eaea").destroy
4...

Afterwards it is recommended to check and fix database integrity:

1# foreman-rake katello:reimport
2...
3Importing Katello::ContentViewPuppetEnvironment
4Importing Activation Key Subscriptions

On older systems the command might differ:

1# foreman-rake katello:reindex

Afterwards I was able to start the upgrade:

1# foreman-rake katello:upgrade_check
2...
3Checking upgradeability...
4Checking for running tasks...
5[PASS] - There are 0 active tasks.
6 You may proceed with the upgrade.

Forcefully stop hanging Foreman tasks

Translations: