From 886ccf71d14458dcd276e739ae51bffa3f9ecec3 Mon Sep 17 00:00:00 2001 From: Adam Williamson Date: Apr 19 2024 19:41:24 +0000 Subject: Various updates and enhancements for the Package Update Guide This is mainly about emphasizing the use of side tag builds and updating the information on automated tests and waivers. I tried to reflow the sections in a way that makes a bit more sense. I also threw out the whole long complicated bit on waiverdb-cli because it should rarely be used any more, and it's a bit of a footgun because it doesn't handle scenarios properly. It was written back when we first tried to enable universal gating for Fedora CI tests, and we did not have the waive button in Bodhi at that time. Signed-off-by: Adam Williamson --- diff --git a/modules/ROOT/pages/Package_Update_Guide.adoc b/modules/ROOT/pages/Package_Update_Guide.adoc index b19550c..ad67d70 100644 --- a/modules/ROOT/pages/Package_Update_Guide.adoc +++ b/modules/ROOT/pages/Package_Update_Guide.adoc @@ -9,7 +9,7 @@ It is not a guide to using the Fedora package source control system: see xref:Package_Maintenance_Guide.adoc[Package Maintenance Guide] for that. * For details of the policy on requirements for updates -at various stages of the https://fedoraproject.org/wiki/Fedora_Release_Life_Cycle[Fedora Release Life Cycle], +at various stages of the xref:releases::lifecycle.adoc[Fedora Release Life Cycle], refer to the xref:fesco::Updates_Policy.adoc[Updates Policy]. == Overview @@ -30,11 +30,20 @@ and stable releases. The repository layouts differ somewhat for Rawhide, Branched and stable releases, but the update workflows split up as described above. -[#rawhide_and_early_branched] -== Rawhide and early Branched +There is also a difference between single-package updates +and multi-package updates. + +[#rawhide_and_early_branched] [#single_packages] -=== Single Packages +== Single-package updates for Rawhide and early Branched + +Most packages can be updated alone, +without needing to be grouped with other packages +for consistent dependencies. +These are "single package builds". +If the change you are making affects other packages, +see xref:multiple_packages[Multi-package updates]. The update workflow for single package builds in Rawhide and Branched before updates-testing activation @@ -44,28 +53,32 @@ is simple: This is all you need to do, a Bodhi update will be created automatically, -from which potential tests will be run. +from which potential tests will be run, +some of which may gate the update. +In the Bodhi web UI, gating tests are indicated by an asterisk. -* If the built package doesn't have tests, +* If the built package doesn't have gating tests, or if they succeed, the update will be marked as _stable_ and your package will appear in subsequently created build roots, as well as in the next daily compose of Rawhide or Branched and will be used in any image composes built from that tree. -* If the built package has tests which fail, +* If the built package has gating tests which fail, this will be recorded in the update. -You can now either xref:waive_a_result[waive the failing tests] -if you're sure that the test shouldn't fail, -or fix whatever is wrong with the package, -build it again, -which in turn will create an update, +You can now fix whatever is wrong with the package and build it again, +which in turn will create a new update, running the tests (on the now hopefully fixed package), and so forth. +If you're absolutely sure that the test shouldn't fail, +you can xref:waive_a_result[waive the failing tests], +but please check with the xref:ci::index.adoc#_contact[Fedora CI] and/or https://fedoraproject.org/wiki/QA#Communicate[Quality team] +before waiving a failure. + [#later_branched_and_stable_releases] -== Later Branched and stable releases +== Single-package updates for later Branched and stable releases At the https://docs.fedoraproject.org/en-US/fesco/Updates_Policy/#updates-testing-activation[updates-testing activation] point, the Bodhi update feedback system is enabled by https://docs.pagure.org/releng/[Release Engineering] @@ -93,6 +106,13 @@ submit the update to xref:quick-docs::repositories.adoc#stable-is-not-a-reposito with `bodhi updates request stable` or the web interface. +Just like early Branched and Rawhide updates, +gating tests may be run on the update. +If any fail, +you should fix the problem, +send a new build, +and edit it into the update. + [#update_attributes] === Update attributes @@ -132,200 +152,31 @@ as your update reaches various points in the process. Using this mechanism can be very useful both for you and for users of your package. -You may set a _karma_ (feedback) level -at which the update will automatically be submitted to _stable_. -This is optional. -If you choose to use it, +You may set thresholds for automatic push to stable +based on positive feedback (karma) +and/or time spent in updates-testing. +If your update's total karma +(items of positive feedback minus items of negative feedback) +reaches the karma threshold, it will be pushed stable automatically. +If your update reaches the threshold for time in updates-testing +without the builds changing +or the total karma going so low it is unpushed, +it will be pushed stable automatically. + +There are defaults for these thresholds +which vary at different points in the release process. + +If you choose to use the karma auto-push threshold, please carefully consider an appropriate feedback level. For a relatively obscure package which is quite stable, 1 or 2 may be an appropriate value. For a popular, sensitive and complex package such as `firefox` or `kernel`, -the default of 3 may be insufficient +the default may be insufficient and a choice of 5 or even 10 may be appropriate. -[#who_will_receive_your_update_when] -=== Who will receive your update, when? - -When a release is in Branched state, -the _updates-testing_ repository is enabled by default -so most users will see the package, -but only packages from the stable _fedora_ repository -are used in building milestone releases (Beta and Final) -and nightly images. - -Where a package goes when it is marked as _stable_ -differs between Branched and stable releases. -In Branched releases, -_stable_ packages are pushed to the base _fedora_ repository. -In stable releases, -_stable_ packages are pushed to the _updates_ repository. -However, from the point of view of the packager, -this is an insignificant implementation detail. -For more details, see xref:quick-docs::repositories.adoc[Repositories]. - -When a release is in stable state, -the _updates-testing_ repository is disabled by default, -but QA team members and others run with it enabled -in order to provide testing and Bodhi feedback. -The main user population will see your update only when it passes Bodhi, -is marked as _stable_ -and reaches the _updates_ repository. - -[#handling_feedback_from_automated_tests] -=== Handling feedback from automated tests - -Fedora's automated testing system, OpenQA, -may run automated tests on your update. -The OpenQA tests are functional tests -of some critical Workstation and Server features. - -In the Bodhi web interface, updates have an _Automated Tests_ tab -which displays the results of all automated tests. -Tests shown with a red background failed. -The tests are not all 100% accurate, -but they are fairly often correct. -If you see a failure, it is a very good idea to click on the result -(which will take you to a detailed log) -and investigate the issue. -If you are unsure what the test indicates, -you can contact the QA team for help. - -[#waive_a_result] -==== Waive a result - -At present, a failure of the _dist.rpmdeplint_, _dist.abicheck_, or _org.centos.prod.ci.pipeline.complete_ tests -will prevent your update from being released. -On the update's _Details_ page in the Bodhi web interface, -the *Test Gating Status* will be shown as -_N of N required tests failed_. -If you are sure such a failure is a false one, -you can 'waive' the result using the tool `waiverdb-cli`. - -You can submit a waiver for a failing result with `waiverdb-cli` -specifying the `subject` and the `testcase`: - -[subs="attributes+"] -.... -waiverdb-cli -t YOUR_TESTCASE_HERE -s '{"item": "this-is-the-subject", "type": "also-this-is-part-of-the-subject"}' -p "fedora-{MAJOROSVER}" -c "This is fine"` -.... - -Example: - -[subs="attributes+"] -.... -waiverdb-cli -t dist.rpmdeplint -s '{"item": "python-requests-1.2.3-1.fc{MAJOROSVER}", "type": "koji_build"}' -p "fedora-{MAJOROSVER}" -c "This is fine"` -.... - -You can also waive a failing result by result's id, -which you can retrieve from resultsdb with curl. -To do that, you'll need the `testcase` name and the `nvr`. For example: - -[subs="attributes+"] -.... -curl "https://taskotron.fedoraproject.org/resultsdb_api/api/v2.0/results?testcases=dist.python-versions&item=python-alembic-0.9.7-1.fc{MAJOROSVER}" | jq ".data[0].id" -.... - -This should print out the of the failing result. -You can then submit a waiver for this failing result with - -[subs="attributes+"] -.... -waiverdb-cli -p fedora-{MAJOROSVER} -r YOUR_ID_HERE -c "This is fine." -.... - -Also, if you enabled automatic stable push at a karma threshold, -this will be disabled if any automated test fails. -If you have examined the result -and you are sure it is a false one -and there is no problem with the package, -you may re-enable the automatic push mechanism -or submit the package to _stable_ manually -once it meets the other requirements of the xref:fesco::Updates_Policy.adoc[Updates Policy]. - -[#waive_the_absence_of_a_result] -==== Waive the absence of a result - -Submitting a waiver using subject/testcase -allows to waive the absence of a result -(e.g. the test never ran for some reason, so there is no result item). - -If it is not clear which testcase you should specify, -the answer can be fetched with this Python script: - -[source,python] ----- -#!/usr/bin/env python -""" Ask a question of greenwave. """ -# Usage: either modify and set PRODUCT_VERSION and NVR_LIST and run, or pass version as first arg and then NVRs as further args -import pprint -import requests -import sys -PRODUCT_VERSION = 'fedora-27' if len(sys.argv) == 1 else sys.argv[1] -NVR_LIST = [] or sys.argv[2:] # Insert your NVRs here, or pass them via command line args -for nvr in NVR_LIST: - url = ( - 'https://greenwave-web-greenwave.app.os.fedoraproject.org/' - 'api/v1.0/decision') - payload = dict( - #verbose=True, - decision_context='bodhi_update_push_stable', - product_version=PRODUCT_VERSION, - subject=[{'item': nvr, 'type': 'koji_build'}], - ) - response = requests.post(url, json=payload) - print("-" * 40) - print(nvr, response, response.status_code) - data = response.json() - print(pprint.pformat(data)) ----- - -The output will show that -Greenwave is requiring a specific testcase to run, -but it cannot find a result for it -(neither pass nor failure). -So now it is possible to submit a waiver -with the specified testcase in the output -and the subject already known. - -==== Troubleshooting - -If you run the tool and it gives you the following error: - -.... -Error: The config option "resultsdb_api_url" is required -.... - -Edit `/etc/waiverdb/client.conf` and add the following line: - -.... -resultsdb_api_url=https://taskotron.fedoraproject.org/resultsdb_api/api/v2.0 -.... - -[#branched_milestone_freezes] -=== Branched milestone freezes - -For a short period before each milestone release, -the stable xref:quick-docs::repositories.adoc#the-fedora-repository-in-branched-releases[fedora] repository is frozen. -These periods are shown as _Milestone freezes_ -(Post-branch Freeze, Beta Freeze, Final Freeze) -on schedules. -During these periods, builds will not be marked _stable_ -and pushed from _updates-testing_ to _fedora_ -even after being submitted manually or automatically. -In the normal course of events, -they will be pushed after the milestone release is approved -at a https://fedoraproject.org/wiki/Go_No_Go_Meeting[Go No Go Meeting]. -If you believe your update deserves to break a milestone freeze, -a _freeze exception_ may be granted -through the https://fedoraproject.org/wiki/QA:SOP_freeze_exception_bug_process[freeze exception process]. -Accepted release blocking bugs are granted the same status -through the https://fedoraproject.org/wiki/QA:SOP_blocker_bug_process[blocker bug process]. - -For more on the Fedora development process, see https://fedoraproject.org/wiki/Fedora_Release_Life_Cycle[Fedora Release Life Cycle]. - [#multiple_packages] == [[updating_inter_dependent_packages]] Multi-package updates @@ -342,6 +193,8 @@ Fedora has the concept of using side tags for these situations, which means the builds are done "on the side" and do not affect packages out of the side tag, nor are they available for installing until the side tag is merged. +Updates are never automatically created for side tag builds. +Using a side tag is the best and recommended way to do multi-package updates. Packagers can create side tags on their own, allowing them to build disruptive components in isolation @@ -392,6 +245,9 @@ that the respective build is available in the build root for subsequent builds. When you're done building all packages you want in a side tag, you have to submit them as an update to Bodhi before they can be made available generally to be installed and built upon. +Even for Rawhide and early Branched, +when using a side tag, +you must create the update manually. In the _Create New Update_ form in Bodhi, choose the _Use Side-Tag_ drop-down @@ -401,13 +257,18 @@ image:Bodhi-builds-from-side-tag.png[Bodhi-builds-from-side-tag.png,title="Bodhi As with single packages, tests will be run whose result affects if the update can be moved to stable. -The difference is that tests have to succeed (or be waived) for all builds in the update. +The difference is that gating tests have to succeed (or be waived) for all builds in the update. If you have to update the list of builds, e.g. to fix problems found during testing, -edit the update and refresh the list of builds using the 🔃 (refresh) button: +send the new build to your side tag, +then edit the update and refresh the list of builds using the 🔃 (refresh) button: image:Bodhi-builds-refresh-from-side-tag.png[Bodhi-builds-refresh-from-side-tag.png,title="Bodhi-builds-refresh-from-side-tag.png"] +This will change the update to use the latest builds from the side tag. +If there are multiple builds for the same package in the side tag, +it will always use the most recently tagged one. + The web interface only works if you are the creator of the side tag. If you are a proven packager submitting an update for a side tag, you currently need to use the bodhi cli: @@ -417,6 +278,16 @@ you currently need to use the bodhi cli: bodhi updates new --from-tag --notes "whatever" f{NEXTOSVER}-build-side-7863 ---- +For more general information on creating updates, +see xref:update_attributes[Update attributes] above. +Once the update is created, +it will behave similarly to a single-package update. +For Rawhide and early Branched, +it will go to stable as soon as any gating tests pass. +For later Branched and stable releases, +it will require positive feedback or some time in updates-testing, +as explained in xref:later_branched_and_stable_releases[the earlier section]. + Once the update moves to stable, the builds will be tagged to the main tag of the release, i.e. are available for the general public. @@ -442,19 +313,6 @@ Removing the last build in a side-tag will automatically remove/delete the side ========= -[#removing_side_tag] -==== Removing a side tag -To conserve build system resources, -side tags that are not needed anymore should be removed. -This can be achieved with `fedpkg` as follows: - ----- -fedpkg remove-side-tag ----- - -Bodhi automatically removes side tags used for updates pushed to stable, -removing them manually is not necessary. - ==== Using macros in a side tag Koji allows macros to be set in a side tag. @@ -489,11 +347,32 @@ macros must be approved by FESCo and the list of currently supported macros is available in the xref:fesco::Policy_for_buildroot_macros.adoc[Policy for Buildroot Macros]. -=== Bodhi multi package updates +[#removing_side_tag] +==== Removing a side tag +To conserve build system resources, +side tags that are not needed anymore should be removed. +This can be achieved with `fedpkg` as follows: + +---- +fedpkg remove-side-tag +---- + +Bodhi automatically removes side tags used for updates pushed to stable, +so removing them manually is usually not necessary. + +=== Bodhi multi-package updates You can also create an update with multiple packages directly in Bodhi, either with the https://bodhi.fedoraproject.org[Bodhi web application], or with the `bodhi` command line tool. +This is usually a worse method than using a side tag, +because managing the build root is more difficult +and will affect packages outside of the group. +It may be appropriate if the packages logically belong together as an update, +but do not have any build-time interdependencies. +You cannot use this method for Rawhide or early Branched updates, +because of the automatic update creation system. + You can pass multiple package names to `bodhi updates new` command to create a new multi-package update, or use `bodhi updates edit` to edit an existing update. @@ -506,13 +385,150 @@ you should contact the https://docs.pagure.org/releng/[release engineering] team or a proven packager for help. [#buildroot_overrides] -=== Buildroot overrides - -Yet another way to build multiple packages together are -https://fedoraproject.org/wiki/Bodhi/BuildRootOverrides[buildroot overrides]. -Unlike side tags, buildroot overrides are not isolated +==== Buildroot overrides + +When trying to do a multi-package update without using side tags, +you may need to use https://fedoraproject.org/wiki/Bodhi/BuildRootOverrides[buildroot overrides]. +A buildroot override puts a build that is not yet stable +into the main build root for the release. +Before on-demand side tags, +this was the usual way to create a multi-package update +when the packages have build-time interdependencies. +Unlike with side tags, buildroot overrides are not isolated and thus should be only used in special cases where side tags do not suffice. +[#handling_feedback_from_automated_tests] +=== Handling feedback from automated tests + +Fedora's automated testing systems, +including Fedora CI, openQA, and Fedora CoreOS CI, +may run automated tests on your update. + +In the Bodhi web interface, updates have an _Automated Tests_ tab +which displays the results of all automated tests. +Tests with an asterisk (star) at the left-hand side of the row are "gating" tests. +These are the tests which prevent your update going stable if they fail. +If a test does not have an asterisk, it is not gating, +and does not affect whether your update can be pushed stable. +For updates in the critical path, +some gating tests are always present. +For updates not in the critical path, +there may be gating tests +if any package in the update has configured them in its package-level gating configuration. +If a failed gating test is waived, +the asterisk is replaced with a thumbs-up. + +* Tests shown with a red background failed. +* Tests shown with a green background passed. +* Tests shown with a blue background and an "i" icon had some minor issue, but effectively passed. +* Tests shown with a blue background and an hourglass icon are pending. + +The tests are not all 100% accurate, +but they are fairly often correct. +Especially failures of openQA tests - +those whose name starts with _update._ - +should always be investigated and resolved +before an update is pushed stable. +If you see a failure, it is a very good idea to click on the result +(which will take you to a detailed log) +and investigate the issue. +If you are unsure what the test indicates, +you can contact the relevant team for help. +When a test has failed gating, a box is shown with contact details. + +The Fedora Quality team proactively investigates failed openQA tests +and will usually resolve them or post a comment on the update +explaining the problem and suggesting a resolution. + +[#waive_a_result] +==== Waive a result + +Any failure of a gating test will prevent your update from being released. +On the update's _Details_ page in the Bodhi web interface, +the *Test Gating* status will be shown as +_N of N required tests failed_, +and one or more red rows with asterisks should be present on the _Automated Tests_ tab. +If you are absolutely sure such a failure is a false one, +you can 'waive' all failed gating tests from the Bodhi web interface +by clicking the *Waive Test Results* button. +This button is only present when an update has failed gating tests, +and you have privileges to control the update. +At present it is not possible to waive the absence of a result +for a gating test in this way. +If needed, this can be done with `waiverdb-cli`. +Please do not waive failures of openQA tests - +those whose names start with _update._ - +without checking with the https://fedoraproject.org/wiki/QA#Communicate[Fedora Quality team]. + +A tool called `waiverdb-cli` exists for filing waivers manually, +though this should rarely be necessary. +The tool is also missing some features needed to correctly specify waivers in some cases, +like the ability to specify a scenario. +It is not currently packaged in Fedora. +You can get it from the https://pagure.io/waiverdb[the waiverdb repository], +which also has some instructions on usage. + +[#who_will_receive_your_update_when] +=== Who will receive your update, when? + +For Rawhide and Branched before updates-testing activation, +when an update goes _stable_ it is tagged for inclusion in the next compose. +Composes are usually run daily at 05:15 UTC and take several hours. +After a compose completes, +it takes some more time for it to reach the public mirrors. +All users will see your update when they update their systems +after a successful compose containing your update +has reached the mirrors. + +For Branched releases after updates-testing activation, +the _updates-testing_ repository is enabled by default +so most users will get your update soon after it is pushed there, +but only packages that have been pushed to _stable_ +are used in building composes +(both nightlies and the Beta and Final candidates and releases). +The _updates-testing_ repository is regenerated daily. + +When a release is in stable state, +the _updates-testing_ repository is disabled by default, +but Quality team members and others run with it enabled +in order to provide testing and Bodhi feedback. +The main user population will see your update only when it passes Bodhi, +is marked as _stable_ +and reaches the _updates_ repository. +Both _updates-testing_ and _updates_ repositories are regenerated daily. + +Where a package goes when it is marked as _stable_ +differs between Branched and stable releases. +In Branched releases, +_stable_ packages are pushed to the base _fedora_ repository. +In stable releases, +_stable_ packages are pushed to the _updates_ repository. +However, from the point of view of the packager, +this is an insignificant implementation detail. +For more details, see xref:quick-docs::repositories.adoc[Repositories]. + +[#branched_milestone_freezes] +=== Branched milestone freezes + +For a short period before each milestone release, +the stable xref:quick-docs::repositories.adoc#the-fedora-repository-in-branched-releases[fedora] repository is frozen. +These periods are shown as _Milestone freezes_ +(Post-branch Freeze, Beta Freeze, Final Freeze) +on schedules. +During these periods, builds will not be marked _stable_ +and pushed from _updates-testing_ to _fedora_ +even after being submitted manually or automatically. +In the normal course of events, +they will be pushed after the milestone release is approved +at a https://fedoraproject.org/wiki/Go_No_Go_Meeting[Go No Go Meeting]. +If you believe your update deserves to break a milestone freeze, +a _freeze exception_ may be granted +through the https://fedoraproject.org/wiki/QA:SOP_freeze_exception_bug_process[freeze exception process]. +Accepted release blocking bugs are granted the same status +through the https://fedoraproject.org/wiki/QA:SOP_blocker_bug_process[blocker bug process]. + +For more on the Fedora development process, see xref:releases::lifecycle.adoc[Fedora Release Life Cycle]. + [#security_updates] == Security updates