PR#1273: Allow generating separate src repo for build repos -

koji

#1273 Allow generating separate src repo for build repos

Merged May 23, 2019 by mikem. Opened Feb 21, 2019 by mizdebsk.

mizdebsk/koji separate-src-repo into master

mizdebsk commented Feb 21, 2019

Fixes #1266

Currently Koji has an option that can be used to include source rpms in each of generated arch repos. Howewer it makes repositories significantly bigger and slower to generate. Source metadata is duplicated across all arch repos. Increased metadata size makes builds slower.

This pull request takes a slightly different approach - it makes it possible to generate build repos with source rpms in separate repos. Such repos don't change size of each arch repo and can be generated faster - generation is done in a separate createrepo task that can run on separate host. For example:

newRepo
  ├  createrepo (src)
  ├  createrepo (aarch64)
  ├  createrepo (ppc64le)
  └  createrepo (x86_64)

CC @ignatenkobrain

ignatenkobrain commented Feb 21, 2019

======================================================================
FAIL: test_legacy_data (tests.test_builder.test_taskparams.TestParseTaskParams)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/srv/jenkins/workspace/koji/label/EL7/tests/test_builder/test_taskparams.py", line 61, in test_legacy_data
    self.assertIn(list(spec), koji.tasks.LEGACY_SIGNATURES[method])
AssertionError: [['tag', 'event', 'src', 'debuginfo', 'separate_src'], None, None, (None, False, False, False)] not found in [[['tag', 'event', 'src', 'debuginfo'], None, None, (None, False, False)]]
======================================================================
FAIL: Test handle_regen_repo help message
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/srv/jenkins/workspace/koji/label/EL7/tests/test_cli/test_regen_repo.py", line 176, in test_handle_regen_repo_help
    """ % self.progname)
  File "/srv/jenkins/workspace/koji/label/EL7/kojienv/lib/python2.7/site-packages/mock/mock.py", line 1305, in patched
    return func(*args, **keywargs)
  File "/srv/jenkins/workspace/koji/label/EL7/tests/test_cli/utils.py", line 209, in assert_help
    exit_code=0)
  File "/srv/jenkins/workspace/koji/label/EL7/tests/test_cli/utils.py", line 190, in assert_system_exit
    self.assert_console_message(stdout, **message['stdout'])
  File "/srv/jenkins/workspace/koji/label/EL7/tests/test_cli/utils.py", line 86, in assert_console_message
    self.assertMultiLineEqual(output, message)
AssertionError: "Usag[136 chars]          show this help message and exit\n  -[344 chars]po\n" != "Usag[136 chars]     show this help message and exit\n  --targ[211 chars]po\n"
  Usage: nosetests regen-repo [options] <tag>
  (Specify the --help global option for a list of other help options)
  Options:
-   -h, --help            show this help message and exit
?                    -----
+   -h, --help       show this help message and exit
-   --target              Interpret the argument as a build target name
?           -----
+   --target         Interpret the argument as a build target name
-   --nowait              Don't wait on for regen to finish
?                    -----
+   --nowait         Don't wait on for regen to finish
-   --debuginfo           Include debuginfo rpms in repo
?                    -----
+   --debuginfo      Include debuginfo rpms in repo
-   --source, --src       Include source rpms in each of repos
?                  -----                          ------     -
+   --source, --src  Include source rpms in the repo
?                                           ++
-   --separate-source, --separate-src
-                         Include source rpms in separate src repo

You need to adjust tests :)

mizdebsk commented Feb 21, 2019

You need to adjust tests :)

Thanks. I forgot that Koji now has test suite - it didn't have them for long time.

rebased onto 3590ab0a355adaf63c7a5f068f7a6ac0eb5e8b05

Feb 21, 2019

mizdebsk commented Feb 21, 2019

Test failures are fixed now.

ngompa commented Feb 22, 2019

:thumbsup:

ngompa commented Feb 22, 2019

@mizdebsk Is there a reason we wouldn't just make this the only behavior? I feel like that would make more sense than allowing the current behavior to persist...

mizdebsk commented Feb 22, 2019

I kept current behaviour for backwards-compatibility. I don't know of any Koji instance using it and personally I would be fine with dropping it. But it's not my call to make.

ngompa commented Feb 22, 2019

@mikem @tkopecek I think we should drop the old behavior as this new behavior is more efficient.

mikem commented Mar 4, 2019

I have put a few general comments on the issue (#1266).

My concern with the current PR is how we expect to manage this. Is a set of glob patterns in kojira.conf the right way to control which tags get this behavior?

The existing source_tags option in kojira is pretty old (2010), and I think we have better options now. Consider how the repo_include_all option is handled (PR#590). Would it be better to have this live in tag config? I probably depends on how folks intend to use this. So, it would really be nice if we could get more detail on use cases in issue #1266.

mizdebsk commented Mar 7, 2019

For my use cases source_tags option in kojira config should be enough. When implementing this PR I tried to be consistent with current approach (src repos combined with arch repos), but I can move configuration to tag_config in database if @mikem thinks it would be better.

ignatenkobrain commented Mar 20, 2019

@mikem, what do you think?

ignatenkobrain commented Apr 23, 2019

@mikem @tkopecek what is blocking this? I really need this functionality and I would like to not ask @nirik to patch this in Fedora infra.

ignatenkobrain commented Apr 23, 2019

@kevin I obviously meant :)

mikem commented May 22, 2019

Sorry for the delay. I'm not super happy with managing this in the kojira config, but I guess that's good enough for now.

mikem commented May 22, 2019

Rebased with a couple fixes here: