#3217 Adding Driver Update Disk building support
Merged by tkopecek. Opened by djuarezg.
djuarezg/koji dud_building  into  master

Download 3217.patch

This is a version of our Driver Update Disk building for https://pagure.io/koji/issue/2998. @tkopecek

This is still a preliminary version that has the following issues/questions:

  • It is unable to build for tags with multiple archs, just as https://pagure.io/koji/issue/3210. I see no obvious fixing for this due to the relevant code being part of kojihub.py.
  • One cannot download the resulting file from the client or at least I did not find any way yet.
  • Should it create any type of logs at all to be attached to the task?

The plugin allows to include multiple packages inside the ISO, as well as optionally including non-rpm content from a SCM URL.

From the logic we use it allows you to either get the latest package version or to optionally include the full NVR. tag needs to have the configured repos to be able to retrieve the requested package(s).

mkisofs and genisoimage are dead. Please use xorriso instead.

You could probably get the arch from Koji for this...

Please use xorriso (standard CLI) or xorrisofs (mkisofs compatible CLI)

  • About multiple archs - see changes I've did for kiwi #3211 (e.g. https://pagure.io/koji/pull-request/3211#_1__44) I'm just rewriting filenames to include arch. For log files on hub side there is an exception in https://pagure.io/koji/blob/master/f/hub/kojihub.py#_10385 I don't think it is a good idea to expand it more in that part of the code. It should be more near to the log/file source.
  • download image artifacts from the build: koji download-build <nvr> --type=image
  • logs - it would be nice to run mkisofs/xorrisof in verbose mode and capture that log separately (if it can provide more info than what will be present in mock_output.log

It raises a lot of flake8 errors (especially incoherent indenting) but it is of course simple to fix. I would also rename group dud to dud-build to be consistent with the rest of build groups. (I did it for kiwi in the last PR).

/me thinking: I've written kiwi plugin as py3 only as originally kiwi was meant to be installed on builder and it doesn't have py2 version. Anyway, now it runs inside the mock so this requirement is lifted. I wonder if I should it also backport to be 2.7 compatible...

2 new commits added

  • Rename pkg group to dud-build
  • Apply xorriso and flake8 suggestions
  • About multiple archs - see changes I've did for kiwi #3211 (e.g. https://pagure.io/koji/pull-request/3211#_1__44) I'm just rewriting filenames to include arch. For log files on hub side there is an exception in https://pagure.io/koji/blob/master/f/hub/kojihub.py#_10385 I don't think it is a good idea to expand it more in that part of the code. It should be more near to the log/file source.
  • download image artifacts from the build: koji download-build <nvr> --type=image
  • logs - it would be nice to run mkisofs/xorrisof in verbose mode and capture that log separately (if it can provide more info than what will be present in mock_output.log

It raises a lot of flake8 errors (especially incoherent indenting) but it is of course simple to fix. I would also rename group dud to dud-build to be consistent with the rest of build groups. (I did it for kiwi in the last PR).

/me thinking: I've written kiwi plugin as py3 only as originally kiwi was meant to be installed on builder and it doesn't have py2 version. Anyway, now it runs inside the mock so this requirement is lifted. I wonder if I should it also backport to be 2.7 compatible...

Regarding the xorriso logs, I think it is not worth extracting them apart, they do not provide much more info and can be checked in the mock_output.log.

Regarding the hub side logs, I think I did not get your message. Do you mean we should treat those logs inside the plugin itself? Something like moving or renaming them before the hub part does its log thing?

Regarding the xorriso logs, I think it is not worth extracting them apart, they do not provide much more info and can be checked in the mock_output.log.

agreed

Regarding the hub side logs, I think I did not get your message. Do you mean we should treat those logs inside the plugin itself? Something like moving or renaming them before the hub part does its log thing?

Yes, builder should rename the file before uploading it to the hub (adding architecture in the name).

Please update the documentation to note we're using xorrisofs.

Yes, builder should rename the file before uploading it to the hub (adding architecture in the name).

I may need some help in here. The logfiles (build.log, hw_info.log, mock_output.log, root.log and state.log) are created (and listed on the koji task UI) as soon as the child task starts running. Is it actually possible to rename them during the builder BuildRoot handler?

Before expiring my BuildRoot I can get to rename them, using workpath = koji.pathinfo.task(self.id), which is where they are ending up. The problem is that at this point I have no permissions to rename files in there.

For reference, I see in my tests that this comment does not seem to be true with the current code: https://imgur.com/a/698GsFW

I've lookg into that again and now I'm more persuaded to do system-level change supporting this. Filing #3229

You can try #3230

2 new commits added

  • Update NVR option checking to match kiwi's
  • Change dud plugin doc to mention xorrisofs

You can try #3230

Working fine with that. No problem building multiarch DUD's in both scratch and non-scratch builds.

1 new commit added

  • Remove trailing whitespaces

Metadata Update from @tkopecek:
- Pull-request tagged with: testing-ready

s/dug-build-tag/dud-target/ or something similar to not confuse build tag with target.

I wonder if there is some reason for that (except of clarity). Code is not using anything specific to image channel (appliance-tools, imagefactory, ...). So it can run on any builder now, not? Maybe just rephrase it to as a hint but not strict requirement.

I wonder if there is some reason for that (except of clarity). Code is not using anything specific to image channel (appliance-tools, imagefactory, ...). So it can run on any builder now, not? Maybe just rephrase it to as a hint but not strict requirement.

Isn't this required when dealing with tasks based on BuildImageTask ? See https://pagure.io/koji/blob/master/f/plugins/builder/kiwi.py + https://pagure.io/koji/blob/master/f/docs/source/plugins.rst. This is where I took this from.

rebased onto def681d2185708553a363927d478292d050b6d4d

1 new commit added

  • Do not confuse dud-build-tag with dud-target

You've got me :-) It is a leftover of original kiwi work. In that moment I've had installed kiwi outside of buildroot (on builder) as there were no other way how to parse results file. Nevertheless, it is not true anymore and it is completely independent. I've created #3315 for myself.

1 new commit added

  • better description for dud channel requirements

You've got me :-) It is a leftover of original kiwi work. In that moment I've had installed kiwi outside of buildroot (on builder) as there were no other way how to parse results file. Nevertheless, it is not true anymore and it is completely independent. I've created #3315 for myself.

Done, I had forgotten about it.

Metadata Update from @mfilip:
- Pull-request tagged with: testing-done

Commit f61fbeb6 fixes this pull-request

Pull-Request has been merged by tkopecek

Metadata