koji

#2963 image build processes can die without failing task
Opened by mikem. Modified

Open
Close Issue

The image build tasks that use imagefactory currently use the imgfac library directly in the task process. This creates some rare cases where imagefactory can fail in a way that python exception handling does not occur. When this happens, the task process never calls failTask.

This causes the task to become "stale". The main kojid process will free it. If the failure repeats, this process can loop indefinitely as the task fails over and over without being marks as such. E.g.

2021-07-22 10:55:11,885 [INFO] koji.TaskManager: Attempting to take task 38306083
2021-07-22 11:05:23,990 [INFO] koji.TaskManager: Task 38306083 (pid 719522) was killed by signal 6
2021-07-22 11:05:24,186 [INFO] koji.TaskManager: Expiring subsession 16475411 (task 38306083)
2021-07-22 11:06:11,378 [INFO] koji.TaskManager: freeing stale tasks: [38306083]
2021-07-22 11:06:13,701 [INFO] koji.TaskManager: Attempting to take task 38306083
2021-07-22 11:15:31,090 [INFO] koji.TaskManager: Task 38306083 (pid 720029) was killed by signal 6
2021-07-22 11:15:31,285 [INFO] koji.TaskManager: Expiring subsession 16476693 (task 38306083)
2021-07-22 11:16:17,184 [INFO] koji.TaskManager: freeing stale tasks: [38306083]
2021-07-22 11:16:18,548 [INFO] koji.TaskManager: Attempting to take task 38306083
2021-07-22 11:26:17,166 [INFO] koji.TaskManager: Task 38306083 (pid 720696) was killed by signal 6

My first instinct was that we should refactor things so that imagefactory is isolated in its own process, but that may be a lot of work.

Alternately, it might be reasonable for the main process to fail the task in updateTasks when the task process ends prematurely.

Metadata Update from @mikem:
- Custom field Size adjusted to None

Side note: this is the sort of issue that can trigger this behavior:
https://bugzilla.redhat.com/show_bug.cgi?id=1923971

The discussion there is interesting and it seems that bug is tickled by the interaction of imports between koji and libguestfs. It may be the the process isolation approach I suggest above might also solve this issue.

@mikem this issue is simply highlighted by the failure in BZ1923971, right? Should we open a separate issue to discuss a mitigation for BZ1923971?

Sure separate issue, though possibly overlapping solution.

Ok, opened #2964.

This issue has been migrated to Fedora Forge:
https://forge.fedoraproject.org/koji/koji/issues/2963

Please continue any further discussion there.
Metadata
None

discussion

None
None
Low
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.