From e32879c7312bc5e2770a744b2a7c4aa2039fed0c Mon Sep 17 00:00:00 2001 From: Tomas Kopecek Date: Mar 08 2024 11:47:18 +0000 Subject: [PATCH 1/2] docs: mock's configuration Related: https://pagure.io/koji/issue/4019 --- diff --git a/docs/source/hub_conf.rst b/docs/source/hub_conf.rst index d8c13bf..0edb099 100644 --- a/docs/source/hub_conf.rst +++ b/docs/source/hub_conf.rst @@ -233,6 +233,8 @@ Notification options For more on notifications in Koji, see :ref:`notification-basics` +.. _hub_resource_limits: + Resource limits ^^^^^^^^^^^^^^^ If configured, the following resource limits are applied by the hub at startup. diff --git a/docs/source/index.rst b/docs/source/index.rst index 1d7da3d..b43ea86 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -43,6 +43,7 @@ Contents database_howto kojid_conf hub_conf + mock_config using_the_koji_build_system setting_rpm_macros profiles diff --git a/docs/source/kojid_conf.rst b/docs/source/kojid_conf.rst index dbc6747..af7e046 100644 --- a/docs/source/kojid_conf.rst +++ b/docs/source/kojid_conf.rst @@ -1,8 +1,10 @@ kojid.conf Options ------------------ -kojid.conf is standard .ini-like configuration file. Its main section is -called ``[kojid]`` and contains following options. They are split by function -to section but can occur anywhere in ``[kojid]`` section in config file. + +``/etc/kojid/kojid.conf`` is standard .ini-like configuration file. Its main +section is called ``[kojid]`` and contains following options. They are split by +function to the following sections but can occur anywhere in ``[kojid]`` block +in config file. General ^^^^^^^ @@ -214,6 +216,23 @@ RPM Builds The vendor to use in rpm headers. Value is propagated via macros to rpmbuild. +.. _rlimits_kojid: + +General RLIMIT options +^^^^^^^^^^^^^^^^^^^^^^ +.. glossary:: + RLIMIT_* + Same resource limit options as in the :ref:`hub config + `. Note, then when ``mock.new_chroot`` is set, those + are not propagated to the chroot and must be set separately in + ``/etc/mock/site-defaults.cfg`` as e.g. ``config_opts['nspawn_args'] = + ['--rlimit=RLIMIT_NOFILE=16384:16384']``. Note, that in case both old/new + chroots are used, this must be set in ``kojid.conf`` and also in + ``site-defaults.cfg`` to retain consistent behaviour across the + buildroots. For other recommended ``site-defaults.cfg`` settings see + :doc:`mock`. + + Mock ^^^^ .. glossary:: @@ -234,6 +253,10 @@ Mock yum_proxy=None Address of proxy server which will be passed via mock to yum. + RLIMIT_* + These options affect mock's behaviour and are described :ref:`here + `. + Notifications ^^^^^^^^^^^^^ .. glossary:: diff --git a/docs/source/mock_config.rst b/docs/source/mock_config.rst new file mode 100644 index 0000000..87d371e --- /dev/null +++ b/docs/source/mock_config.rst @@ -0,0 +1,46 @@ +Interaction with Mock +--------------------- + +`Mock`_ is crucial component for building in Koji. Every build runs the mock to +prepare fresh isolated environment and builds rpm or other content in chroot +handled by Mock. + +For each buildroot (and thus every build) koji prepares configuration file for +mock to be used. Content of the file depends on some administrator's decisions +and can be tweaked by: + + * ``/etc/mock/site-defaults.cfg`` on builder. This file is always consulted by + mock, so it is strongly recommended not to use default configuration but + deploy this file with more strict settings. + Typical values which should be changed: + + - ``use_bootstrap = False`` - Bootstrap is a) not needed in most cases b) + slows down the build c) e.g. in combination with next option could lead to + unreproducible build. While this feature is pretty useful in some cases, + it should be turned off as default and could be selectively turned on for + specific buildroots per tag configuration. + + - ``use_bootstrap_image`` - As noted before, this is generally unsafe + option, especially if used with image tags which are expected to be moved + (e.g. "latest"). Note, that to use this option (even via tag's + configuration), you need to explicitly allow it in :doc:`kojid.conf + ` via ``mock_bootstrap_image = True``. + + - ``nspawn_args`` - for old chroot, :ref:`RLIMIT_* ` support + setting various RLIMIT values to the build process. In case of + nspawn-based chroot (``mock.new_chroot`` tag extra option) these are + handled by mock itself and must be set separately there. + + - ``macros`` - some rpm macros could be defined every time for every build + based on builder itself. Typical usecase is to limit usage of CPUs on + given builder by setting ``config_opts['macros']['%_smp_ncpus_max'] = 2``. + + - ``plugin_conf`` - Various plugins could be enabled and configured to be + available in build environment. Example is `rpmautospec`_ which allows + various ``%auto*`` macros in spec files. + + * Another selective variant of altering mock's behaviour is setting its + options per buildtag. This is more described in :ref:`tuning-mock-per-tag`. + +.. _Mock: https://github.com/rpm-software-management/mock/wiki +.. _rpmautospec: https://docs.pagure.org/fedora-infra.rpmautospec/ diff --git a/docs/source/server_howto.rst b/docs/source/server_howto.rst index f87a82f..ea4205e 100644 --- a/docs/source/server_howto.rst +++ b/docs/source/server_howto.rst @@ -1054,9 +1054,12 @@ Configuration Files * ``/etc/kojid/kojid.conf`` - Koji Daemon Configuration * ``/etc/sysconfig/kojid`` - Koji Daemon Switches +* ``/etc/mock/site-defaults.cfg`` - Mock's base configuration file All options for `kojid.conf` are described :doc:`here `. +Mock-tuning options are described :doc:`here `. + Install kojid ------------- diff --git a/docs/source/using_the_koji_build_system.rst b/docs/source/using_the_koji_build_system.rst index c3c3dbf..603efcc 100644 --- a/docs/source/using_the_koji_build_system.rst +++ b/docs/source/using_the_koji_build_system.rst @@ -390,11 +390,17 @@ environment follows: * ``mock.package_manager`` - If this is set, it will override mock's default package manager. Typically used with ``yum`` or ``dnf`` values. + * ``mock.new_chroot`` - 0/1 value. If it is set, ``--new-chroot`` or - `--old-chroot` option is appended to any mock call. If it is not set, - mock's default behavior is used. + `--old-chroot` option is appended to any mock call. If it is not set, mock's + default behavior is used. Note, that in the case when new chroot (nspawn) is + used, some of the options in ``kojid.conf`` will not be propagated to chroot + env properly, typically ``RLIMIT_*``, see :ref:`kojid.conf's options + `. + * ``mock.releasever`` - When doing cross-compiles it may be necessary to explicitly set the ``releasever`` to be used. + * ``mock.use_bootstrap`` - 0/1 value. If it is set, ``--bootstrap-chroot`` is appended to the mock init call. This tells mock to build in two stages, using chroot rpm for creating the build chroot. If it is not set, mock's @@ -402,6 +408,7 @@ environment follows: `_. Note, that it is not turn on by default by koji, as it is often not needed and it consumes additional resources (larger buildroot, downloading more data). + * ``mock.bootstrap_image`` - set to name of image, which can builder's podman download (e.g. ``fedora:32``). See mock's `doc `_ @@ -470,6 +477,9 @@ do: koji edit-tag dnf-fedora-tag -x rpm.env.CC=clang +Note, that mock's behaviour is always impacted by builder's default values in +``/etc/mock/site-defaults.cfg``. Check :doc:`mock` for details. + Using Koji to control tasks ^^^^^^^^^^^^^^^^^^^^^^^^^^^ From 9605ba02bf3d0de36d051835c815e8e050b9aa52 Mon Sep 17 00:00:00 2001 From: Mike McLean Date: Mar 19 2024 10:08:01 +0000 Subject: [PATCH 2/2] doc updates --- diff --git a/docs/source/index.rst b/docs/source/index.rst index b43ea86..61bda74 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -11,7 +11,7 @@ Koji ==== Koji is the software that builds `RPM packages for the Fedora project`_. It -uses `Mock`_ to create chroot environments to perform builds. To download the +uses :doc:`Mock ` to create chroot environments to perform builds. To download the source code, report bugs, join the mailing list etc., see the `Koji project website`_. diff --git a/docs/source/kojid_conf.rst b/docs/source/kojid_conf.rst index af7e046..45fb92d 100644 --- a/docs/source/kojid_conf.rst +++ b/docs/source/kojid_conf.rst @@ -1,13 +1,15 @@ kojid.conf Options ------------------ -``/etc/kojid/kojid.conf`` is standard .ini-like configuration file. Its main -section is called ``[kojid]`` and contains following options. They are split by -function to the following sections but can occur anywhere in ``[kojid]`` block -in config file. +``/etc/kojid/kojid.conf`` is a standard .ini-like configuration file. Its main +section is named ``[kojid]`` and can contain the following options. +For readability, we have broken them into categories in this document, +but they should all go under the ``[kojid]`` block of the config file. + +General Options +^^^^^^^^^^^^^^^ +These options affect overall kojid behavior: -General -^^^^^^^ .. glossary:: chroot_tmpdir=/chroot_tmpdir The temporary directory in buildroot. It is advised to not use ``/tmp`` @@ -33,7 +35,7 @@ General buildroot_final_cleanup_delay buildroot_final_cleanup_delay=86400 - Time after which buildroot (pre-cleand after + Time after which buildroot (pre-cleaned after ``buildroot_basic_cleanup_delay``) is deleted completely. (1 day in seconds) @@ -98,8 +100,10 @@ General workdir=/tmp/koji The directory root for temporary storage on builder. -Building -^^^^^^^^ +Build Options +^^^^^^^^^^^^^ +The following options affect various build processes in kojid: + .. glossary:: allowed_scms=scm.example.com:/cvs/example git.example.org:/example svn.example.org:/users/\*:no Controls which source control systems the builder will accept. It is a @@ -194,8 +198,10 @@ Building the image. Default value is compromise between speed and resource usage. Only one option (not space-separated) is allowed here for now. -RPM Builds -^^^^^^^^^^ +RPM Build Options +^^^^^^^^^^^^^^^^^ +The following options affect rpms builds: + .. glossary:: distribution=Koji The distribution to use in rpm headers. Value is propagated via macros @@ -220,21 +226,23 @@ RPM Builds General RLIMIT options ^^^^^^^^^^^^^^^^^^^^^^ +Kojid accepts the same rlimit options as the hub. + .. glossary:: RLIMIT_* Same resource limit options as in the :ref:`hub config - `. Note, then when ``mock.new_chroot`` is set, those - are not propagated to the chroot and must be set separately in - ``/etc/mock/site-defaults.cfg`` as e.g. ``config_opts['nspawn_args'] = - ['--rlimit=RLIMIT_NOFILE=16384:16384']``. Note, that in case both old/new - chroots are used, this must be set in ``kojid.conf`` and also in - ``site-defaults.cfg`` to retain consistent behaviour across the - buildroots. For other recommended ``site-defaults.cfg`` settings see - :doc:`mock`. - - -Mock -^^^^ + `. Note, that when ``mock.new_chroot`` is set, those + are not propagated to the mock chroot. To apply rlimits to mock in + such cases, you can set them separately in + ``/etc/mock/site-defaults.cfg``. See :doc:`mock_config` for more information. + + +.. _mock_kojid_conf: + +Mock Options +^^^^^^^^^^^^ +The following options affect how Koji uses Mock: + .. glossary:: mockdir=/var/lib/mock The directory root for mock. @@ -253,24 +261,26 @@ Mock yum_proxy=None Address of proxy server which will be passed via mock to yum. - RLIMIT_* - These options affect mock's behaviour and are described :ref:`here + rlimits + The ``RLIMIT_*`` options can affect mock's behaviour and are described :ref:`here `. -Notifications -^^^^^^^^^^^^^ +Notification Options +^^^^^^^^^^^^^^^^^^^^ .. glossary:: admin_emails='' Space-separated list of addresses for sending logs. - from_addr=Koji Build System + from_addr=Koji Build System The From address used when sending email notifications. smtphost=example.com The mail host to use for sending email notifications. -Kerberos Authentication -^^^^^^^^^^^^^^^^^^^^^^^ +Kerberos Options +^^^^^^^^^^^^^^^^ +The following options control Kerberos authentication: + .. glossary:: ccache=/var/tmp/kojid.ccache Credentials cache used for krbV login. @@ -283,8 +293,10 @@ Kerberos Authentication Location of the keytab. -SSL Authentication -^^^^^^^^^^^^^^^^^^ +SSL Options +^^^^^^^^^^^ +The following options control SSL authentication: + .. glossary:: ca='' noop, obsoleted, will be removed soon. diff --git a/docs/source/mock_config.rst b/docs/source/mock_config.rst index 87d371e..471a512 100644 --- a/docs/source/mock_config.rst +++ b/docs/source/mock_config.rst @@ -1,18 +1,47 @@ Interaction with Mock --------------------- -`Mock`_ is crucial component for building in Koji. Every build runs the mock to -prepare fresh isolated environment and builds rpm or other content in chroot -handled by Mock. +`Mock`_ is crucial component for building in Koji. Most build types use mock to +prepare a fresh, isolated environment for building content. -For each buildroot (and thus every build) koji prepares configuration file for -mock to be used. Content of the file depends on some administrator's decisions -and can be tweaked by: +For further reading on the mock tool itself, +see the `mock documentation `_. - * ``/etc/mock/site-defaults.cfg`` on builder. This file is always consulted by - mock, so it is strongly recommended not to use default configuration but - deploy this file with more strict settings. - Typical values which should be changed: + + +Mock config files +================= + +For each buildroot Koji prepares a configuration file for mock to use. +The content of the file depends in part on some administrator's decisions +and can be tweaked by options in the build tag. +These files are managed by kojid and stored under ``/etc/mock/koji`` on +the build host. + +Since Koji 1.34.1, builds will save the mock configuration file as a log named +``mock_config.log`` and store it alongside other logs from the build. + +The contents of this file are generated by Koji and are affected by: + +* :ref:`kojid configuration ` +* the build tag +* the build arch +* the repo used for the build +* build tag options, see :ref:`tuning-mock-per-tag` +* the controlling build task (e.g. buildArch for rpms) + +You can generate mock configs locally in the same way that kojid does. +See :ref:`cli_mock_config` + + +Site defaults +============= + +In addition to the mock configuration file for the buildroot, the mock command +also consults ``/etc/mock/site-defaults.cfg`` on the build host. +You can use this file to set global defaults on your builders. + +The following settings are recommended for use with Koji: - ``use_bootstrap = False`` - Bootstrap is a) not needed in most cases b) slows down the build c) e.g. in combination with next option could lead to @@ -26,10 +55,8 @@ and can be tweaked by: configuration), you need to explicitly allow it in :doc:`kojid.conf ` via ``mock_bootstrap_image = True``. - - ``nspawn_args`` - for old chroot, :ref:`RLIMIT_* ` support - setting various RLIMIT values to the build process. In case of - nspawn-based chroot (``mock.new_chroot`` tag extra option) these are - handled by mock itself and must be set separately there. + - ``nspawn_args`` - if you want to apply rlimit settings for mock with nspawn + isolation, you will need to use this option. See :ref:`rlimits_mock` below. - ``macros`` - some rpm macros could be defined every time for every build based on builder itself. Typical usecase is to limit usage of CPUs on @@ -39,8 +66,66 @@ and can be tweaked by: available in build environment. Example is `rpmautospec`_ which allows various ``%auto*`` macros in spec files. - * Another selective variant of altering mock's behaviour is setting its - options per buildtag. This is more described in :ref:`tuning-mock-per-tag`. -.. _Mock: https://github.com/rpm-software-management/mock/wiki +.. _rlimits_mock: + +RLIMIT settings +^^^^^^^^^^^^^^^ + +When mock uses simple isolation (aka old-chroot), the +:ref:`RLIMIT_* settings ` from ``kojid.conf`` will propagate +to mock. + +However, when mock uses nspawn isolation (aka new-chroot, controlled by the +``mock.new_chroot`` tag extra option), these limits are not propagated automatically. +To apply such limits to the mock nspawn chroots, you can add them to the +``nspawn_args`` option in ``site-defaults.cfg``. +For example, you might set: + +:: + + config_opts['nspawn_args'] = ['--rlimit=RLIMIT_NOFILE=16384:16384'] + +If your system uses a mix of mock isolation settings, then you will likely want +to apply rlimit settings in *both* ``kojid.conf`` and ``site-defaults.cfg`` for +consistency. + + +.. _cli_mock_config: + +Local mock configs +================== + +The ``koji mock-config`` command can be used to generate a mock config file locally +in the same way that the builders do. These can be useful for local testing or debugging. + +:: + + $ koji mock-config --help + Usage: koji mock-config [options] + (Specify the --help global option for a list of other help options) + + Options: + -h, --help show this help message and exit + -a ARCH, --arch=ARCH Specify the arch + -n NAME, --name=NAME Specify the name for the buildroot + --tag=TAG Create a mock config for a tag + --target=TARGET Create a mock config for a build target + --task=TASK Duplicate the mock config of a previous task + --latest use the latest redirect url + --buildroot=BUILDROOT + Duplicate the mock config for the specified buildroot + id + --mockdir=DIR Specify mockdir + --topdir=DIR Specify topdir, topdir tops the topurl + --topurl=URL URL under which Koji files are accessible, when topdir + is specified, topdir tops the topurl + --distribution=DISTRIBUTION + Change the distribution macro + --yum-proxy=YUM_PROXY + Specify a yum proxy + -o FILE Output to a file + + +.. _Mock: https://rpm-software-management.github.io/mock/ .. _rpmautospec: https://docs.pagure.org/fedora-infra.rpmautospec/ diff --git a/docs/source/using_the_koji_build_system.rst b/docs/source/using_the_koji_build_system.rst index 603efcc..c77f252 100644 --- a/docs/source/using_the_koji_build_system.rst +++ b/docs/source/using_the_koji_build_system.rst @@ -478,7 +478,7 @@ do: Note, that mock's behaviour is always impacted by builder's default values in -``/etc/mock/site-defaults.cfg``. Check :doc:`mock` for details. +``/etc/mock/site-defaults.cfg``. Check :doc:`mock_config` for details. Using Koji to control tasks ^^^^^^^^^^^^^^^^^^^^^^^^^^^