From 3e55f5c7765e13dd9baa6a930b1436f79c1f2668 Mon Sep 17 00:00:00 2001 From: Diego Herrera Date: Mar 10 2025 17:21:08 +0000 Subject: [PATCH 1/23] Start EPEL10+ minor branching draft Signed-off-by: Diego Herrera --- diff --git a/modules/ROOT/nav.adoc b/modules/ROOT/nav.adoc index 7db8ae4..b9b40f6 100644 --- a/modules/ROOT/nav.adoc +++ b/modules/ROOT/nav.adoc @@ -41,6 +41,7 @@ *** xref:release_guide:sop_file_ftbfs.adoc[File FTBFS bugs] *** xref:release_guide:mass_branching_checklist.adoc[Mass Branching Checklist] *** xref:release_guide:sop_mass_branching.adoc[Mass Branching] +*** xref:release_guide:sop_epel_minor_mass_branching.adoc[EPEL Minor Version Mass Branching] *** xref:release_guide:beta_freeze.adoc[Beta freeze and updates-testing activation] *** xref:release_guide:beta_RC_compose.adoc[Beta RC compose] *** xref:release_guide:beta_release.adoc[Beta release] diff --git a/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc b/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc new file mode 100644 index 0000000..df508e3 --- /dev/null +++ b/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc @@ -0,0 +1,284 @@ +include::_partials/attributes.adoc[] + += EPEL Minor Version Mass Branching + +== Description + +WARNING: TODO: Talk about the minor branching process + +== Send announcement + +One day before the mass branching, we send out announcemt because during mass branching, +new koji builds for rawhide are disabled. + +== Disable EPEL10 builds in koji + +. + +WARNING: TODO: Check if they are the same servers and if it will imply an outage for Fedora too. + +The recommended method to achieve this is by adding firewall rules to both koji01 and koji02 servers, effectively blocking connections from proxy01 and proxy10. This can be achieved with the following iptables commands: + +[source,bash,subs="attributes"] +---- +iptables -I INPUT -m tcp -p tcp --dport 80 -s proxy01.iad2.fedoraproject.org -j REJECT +iptables -I INPUT -m tcp -p tcp --dport 80 -s proxy10.iad2.fedoraproject.org -j REJECT +---- + +These commands reject incoming traffic on port 80 from the specified proxies, preventing external submissions. Internal connections routed via proxy101 and proxy110 will continue to function as expected. + +To reverse the firewall changes and allow external submissions again, use: + +[source,bash,subs="attributes"] +---- +iptables -D INPUT -m tcp -p tcp --dport 80 -s proxy01.iad2.fedoraproject.org -j REJECT +iptables -D INPUT -m tcp -p tcp --dport 80 -s proxy10.iad2.fedoraproject.org -j REJECT + +---- + +This change should be implemented on both koji01 and koji02 machine. + +=== Cancel all running build for EPEL10 + +List all running tasks and select only those relevant for current branching. + +WARNING: TODO: Check if grep requires to filter minor commits. + +[source,bash,subs="attributes"] +---- +$ koji list-builds --state=0 --type=rpm | grep epel10 | awk '{print $1}' +---- + +Cancel each of those tasks + +``` +$ koji cancel +``` + +== dist-git + +Now we need to update dist-git in two steps: + +* Create the new branch in git +* Update the gitolite.conf to allow user to push to this new branch + +For both of these actions you will need the file generated by pdc above. + +=== Create the git branches + +On `pkgs01.stg` (for testing) or `pkgs02` (for production), run: + +[WARNING] +==== +TODO: + +* Check if this works for EPEL. +* Check how to generate input file, since we are not using pdc +==== + +``` +$ sudo -u pagure python /usr/local/bin/mass-branching-git.py +``` + +Where `` will be like `epel10.` and the +`` the path to the file generated by pdc above. + +== Ansible + +Apps in https://pagure.io/fedora-infra/ansible[ansible] need to be updated to be aware of a new branch. + +=== Bodhi + +WARNING: TODO: Associate epel10 to the new minor + +Bodhi needs to be updated to add new release. This needs to be done in +https://pagure.io/fedora-infra/ansible/blob/main/f/roles/bodhi2[bodhi2 role] in +infra ansible repo. This change includes, updating `koji-sync-listener.py`, +`new-updates-sync`, pungi configs for rpm updates, bodhi templates. + +* `roles/bodhi2/backend/files/new-updates-sync` +* `roles/bodhi2/backend/tasks/main.yml` +* `roles/bodhi2/backend/templates/pungi.rpm.conf.j2` +* `roles/bodhi2/backend/templates/koji_sync_listener.toml` + +Please check these files from the https://pagure.io/fedora-infra/ansible/c/549e5d3ace41c04fdbef9d81f359f16c2fe0c2fa?branch=main[commit] for your reference. + +== Toddlers + +WARNING: TODO: Check if any of these apply for EPEL10 + +=== Add new SLA to the toddlers App + +Use https://pagure.io/fedora-infra/ansible/pull-request/1527[this PR] for reference and add new version to the config. + +=== Fedora Branched + +. Set https://pagure.io/fedora-infra/ansible/blob/main/f/vars/all/FedoraBranched.yaml[FedoraBranched] to True. +. Set https://pagure.io/fedora-infra/ansible/blob/main/f/vars/all/FedoraBranchedBodhi.yaml[FedoraBranchedBodhi] to preenable. + +Please check the file `FedoraBranched.yaml` and `FedoraBranchedBodhi.yaml` from the https://pagure.io/fedora-infra/ansible/c/549e5d3ace41c04fdbef9d81f359f16c2fe0c2fa?branch=main[commit] for your reference. + +=== Koji hub + +Update the koji hub config to allow side tags for new koji rawhide (currently f{rawhide}) tag. + +Please check the file `roles/koji_hub/templates/hub.conf.j2` from the https://pagure.io/fedora-infra/ansible/c/549e5d3ace41c04fdbef9d81f359f16c2fe0c2fa?branch=main[commit] for your reference. + +=== Push the changes + +When done editing the files, commit, push and apply them via the corresponding +ansible playbook: + +``` +$ sudo rbac-playbook groups/koji-hub.yml +$ sudo rbac-playbook groups/releng-compose.yml +$ sudo rbac-playbook groups/bodhi-backend.yml +$ sudo rbac-playbook openshift-apps/greenwave.yml +$ sudo -i ansible-playbook /srv/web/infra/ansible/playbooks/$ groups/proxies.yml -t pkgdb2 +$ sudo rbac-playbook groups/mbs.yml -t mbs +``` + +Ask someone in fedora infra to run the robosignatory playbook. + +== Koji + +WARNING: TODO: Check if any of these applies for EPEL10 + +The koji build system needs to have some tag/target work done to handle +builds from the new branch and to update where builds from rawhide go. + +Run +https://pagure.io/releng/blob/main/f/scripts/branching/make-koji-release-tags[make-koji-release-tags] +script in https://pagure.io/releng[pagure releng] repo + +== Bodhi + +=== Linking Empty Repos + +WARNING: TODO: Check if any of this applies for EPEL10 + +We need to link empty repos so that new-updates-sync wont complain about +missing repos. The following commands should be run on +*bodhi-backend01.phx2.fedoraproject.org* + + +[source,bash,subs="attributes"] +---- +$ sudo ln -s /mnt/koji/compose/updates/empty-repo/ /mnt/koji/compose/updates/f{branched}-updates +$ sudo ln -s /mnt/koji/compose/updates/empty-repo/ /mnt/koji/compose/updates/f{branched}-updates-testing +---- + +=== Creating Empty Repos + +WARNING: TODO: Check if any of this applies for EPEL10 + +To create empty repos on the master mirror, run +https://pagure.io/releng/blob/main/f/scripts/branching/create_empty_repos.sh[create_emtpy_repos.sh] +from https://pagure.io/releng[pagure releng] repo. This should be run on +*bodhi-backend01.phx2.fedoraproject.org* + +[source,bash,subs="attributes"] +---- +$ sudo -u ftpsync sh scripts/branching/create_empty_repos.sh {branched} +---- + +[NOTE] +==== +Update the link in /mnt/koji/repos/rawhide/latest as per https://pagure.io/releng/issue/12255. +==== + +[NOTE] +==== +Please verify the repo permissions that are created under +/pub/fedora/linux/development/ and +/pub/fedora-secondary/development/. They should +be owned by _ftpsync:ftpsync_ +Check directory permissions (should be "0755") to ensure new composes synchronize correctly. +==== + +=== Creating EPEL10. release + +To create a new minor release in bodhi, you need to run: + +WARNING: TODO: Update scripts for EPEL10 (check hackmd) + +[source,bash,subs="attributes"] +---- +$ bodhi releases create \ + --name "F{rawhide}" --long-name "Fedora {rawhide}" \ + --id-prefix FEDORA --version {rawhide} --branch f{rawhide} \ + --dist-tag f{rawhide} \ + --stable-tag f{rawhide} \ + --testing-tag f{rawhide}-updates-testing \ + --candidate-tag f{rawhide}-updates-candidate \ + --pending-stable-tag f{rawhide}-updates-pending \ + --pending-testing-tag f{rawhide}-updates-testing-pending \ + --pending-signing-tag f{rawhide}-signing-pending \ + --state pending \ + --override-tag f{rawhide}-override \ + -create-automatic-updates \ + --not-composed-by-bodhi +---- + +WARNING: TODO: Check if any of this applies for EPEL10 + +You need to run the `bodhi openshift` playbook, so that UI will know +about the new release. Then, you need to restart +*fm-consumer@config.service* and *bodhi-celery.service* services on +*bodhi-backend01.phx2.fedoraproject.org*: + +``` +$ sudo rbac-playbook openshift-apps/bodhi.yml +$ sudo systemctl restart fm-consumer@config.service bodhi-celery.service +``` + +=== Update rawhide koji repo + +WARNING: TODO: Check if any of this applies for EPEL10 + +We need to point the _rawhide_ buildroot repo to the newly created +rawhide buildroot. This way kojira doesn't make a newrepo for _rawhide_ +target as often as fxx-build (new rawhide buildroot). + +Run the following commands from any of the compose boxes: + +[source,bash,subs="attributes"] +---- +$ cd /mnt/koji/repos/rawhide +$ rm -f latest +$ ln -s ../f{rawhide}-build/latest ./latest +---- + +=== Update block_retired.py script + +WARNING: TODO: Check if any of this apply for EPEL10 + +https://pagure.io/releng/blob/main/f/scripts/block_retired.py[block_retired.py] +script in releng repo should be updated with rawhide release and also +branched release should be added to the script. + +Please look at this +https://pagure.io/releng/c/9eb97f491f7a767ab8b90498adfa3b34ee235247?branch=main[block_retired.py +commit] as an example. + +=== Updating MirrorManager + +WARNING: TODO: Check if any of this applies for EPEL10 + +We need to update the mirrormanager so that it will point EPEL10 to the +new EPEL10 minor. + +Please follow the instructions in the +https://pagure.io/fedora-infrastructure/issue/9239#comment-671446[fedora +infra ticket] to update the database of mirrormanager. + +=== Branch new rawhide in Koschei + +WARNING: TODO: Check if any of this applies for EPEL10 + +Branch new fedora rawhide in +https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/koschei/#_branching_a_new_fedora_release[koschei]. + +== Consider Before Running + +NOTE: FIXME: Need some love here From e5c887ed118cd12555663e9f3c5952ea0fac4a14 Mon Sep 17 00:00:00 2001 From: Diego Herrera Date: Mar 10 2025 17:21:08 +0000 Subject: [PATCH 2/23] Resolve some of the TODOs Signed-off-by: Diego Herrera --- diff --git a/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc b/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc index df508e3..922d1b3 100644 --- a/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc +++ b/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc @@ -4,19 +4,15 @@ include::_partials/attributes.adoc[] == Description -WARNING: TODO: Talk about the minor branching process +Originally, the EPEL repository had packages that built against the latest RHEL minor, but since EPEL10, the EPEL repository now builds against the latest Centos Stream release. To keep consistency for RHEL releases, EPEL repository goes through a mass branching process to generate a new minor release branch. This process let's the developers continue to maintain their packages against Centos Stream without risking to break packages that already work on the current active RHEL minor target. == Send announcement One day before the mass branching, we send out announcemt because during mass branching, -new koji builds for rawhide are disabled. +new koji builds for EPEL are disabled. This will imply a koji outage in Fedora too. == Disable EPEL10 builds in koji -. - -WARNING: TODO: Check if they are the same servers and if it will imply an outage for Fedora too. - The recommended method to achieve this is by adding firewall rules to both koji01 and koji02 servers, effectively blocking connections from proxy01 and proxy10. This can be achieved with the following iptables commands: [source,bash,subs="attributes"] @@ -42,8 +38,6 @@ This change should be implemented on both koji01 and koji02 machine. List all running tasks and select only those relevant for current branching. -WARNING: TODO: Check if grep requires to filter minor commits. - [source,bash,subs="attributes"] ---- $ koji list-builds --state=0 --type=rpm | grep epel10 | awk '{print $1}' @@ -64,17 +58,19 @@ Now we need to update dist-git in two steps: For both of these actions you will need the file generated by pdc above. -=== Create the git branches +=== Get current EPEL active packages -On `pkgs01.stg` (for testing) or `pkgs02` (for production), run: +To get the list of all the current active packages, you must run the `scripts/branching-epe/get_all_active_packages_branching.sh` script. -[WARNING] -==== -TODO: +``` +sh scripts/branching-epel/get_all_active_packages_branching.sh 10.{next_minor} +``` -* Check if this works for EPEL. -* Check how to generate input file, since we are not using pdc -==== +The results will be saved to the components_epel10.{next_minor}.txt + +=== Create the git branches + +On `pkgs01.stg` (for testing) or `pkgs02` (for production), run: ``` $ sudo -u pagure python /usr/local/bin/mass-branching-git.py @@ -89,7 +85,7 @@ Apps in https://pagure.io/fedora-infra/ansible[ansible] need to be updated to be === Bodhi -WARNING: TODO: Associate epel10 to the new minor +WARNING: TODO: This is a step from Fedora, check which scripts are impacted by EPEL branching Bodhi needs to be updated to add new release. This needs to be done in https://pagure.io/fedora-infra/ansible/blob/main/f/roles/bodhi2[bodhi2 role] in @@ -105,14 +101,14 @@ Please check these files from the https://pagure.io/fedora-infra/ansible/c/549e5 == Toddlers -WARNING: TODO: Check if any of these apply for EPEL10 - === Add new SLA to the toddlers App -Use https://pagure.io/fedora-infra/ansible/pull-request/1527[this PR] for reference and add new version to the config. +Make sure that the SLAs are added, check https://pagure.io/fedora-infra/ansible/pull-request/2190#request_diff[this PR] for reference. === Fedora Branched +WARNING: TODO: Confirm if we should skip this part for EPEL + . Set https://pagure.io/fedora-infra/ansible/blob/main/f/vars/all/FedoraBranched.yaml[FedoraBranched] to True. . Set https://pagure.io/fedora-infra/ansible/blob/main/f/vars/all/FedoraBranchedBodhi.yaml[FedoraBranchedBodhi] to preenable. @@ -122,10 +118,12 @@ Please check the file `FedoraBranched.yaml` and `FedoraBranchedBodhi.yaml` from Update the koji hub config to allow side tags for new koji rawhide (currently f{rawhide}) tag. -Please check the file `roles/koji_hub/templates/hub.conf.j2` from the https://pagure.io/fedora-infra/ansible/c/549e5d3ace41c04fdbef9d81f359f16c2fe0c2fa?branch=main[commit] for your reference. +Please check the file `roles/koji_hub/templates/hub.conf.j2` from https://pagure.io/fedora-infra/ansible/c/9860fa89b5a9e223fc7a42299fa30a8ffdbfbd77[this PR] for your reference. === Push the changes +WARNING: TODO: Check this PR to see if there are changes required to be done in this regard. + When done editing the files, commit, push and apply them via the corresponding ansible playbook: @@ -138,50 +136,43 @@ $ sudo -i ansible-playbook /srv/web/infra/ansible/playbooks/$ groups/proxies.yml $ sudo rbac-playbook groups/mbs.yml -t mbs ``` -Ask someone in fedora infra to run the robosignatory playbook. - == Koji -WARNING: TODO: Check if any of these applies for EPEL10 - The koji build system needs to have some tag/target work done to handle builds from the new branch and to update where builds from rawhide go. Run -https://pagure.io/releng/blob/main/f/scripts/branching/make-koji-release-tags[make-koji-release-tags] +https://pagure.io/releng/blob/main/f/scripts/branching-epel/make-koji-release-tags[make-koji-release-tags] script in https://pagure.io/releng[pagure releng] repo == Bodhi === Linking Empty Repos -WARNING: TODO: Check if any of this applies for EPEL10 - We need to link empty repos so that new-updates-sync wont complain about missing repos. The following commands should be run on *bodhi-backend01.phx2.fedoraproject.org* - [source,bash,subs="attributes"] ---- -$ sudo ln -s /mnt/koji/compose/updates/empty-repo/ /mnt/koji/compose/updates/f{branched}-updates -$ sudo ln -s /mnt/koji/compose/updates/empty-repo/ /mnt/koji/compose/updates/f{branched}-updates-testing +$ sudo ln -s /mnt/koji/compose/updates/empty-repo/ /mnt/koji/compose/updates/epel10.1-updates +$ sudo ln -s /mnt/koji/compose/updates/empty-repo/ /mnt/koji/compose/updates/epel10.1-updates-testing ---- === Creating Empty Repos -WARNING: TODO: Check if any of this applies for EPEL10 - To create empty repos on the master mirror, run -https://pagure.io/releng/blob/main/f/scripts/branching/create_empty_repos.sh[create_emtpy_repos.sh] +https://pagure.io/releng/blob/main/f/scripts/branching-epel/create_empty_repos.sh[create_emtpy_repos.sh] from https://pagure.io/releng[pagure releng] repo. This should be run on *bodhi-backend01.phx2.fedoraproject.org* [source,bash,subs="attributes"] ---- -$ sudo -u ftpsync sh scripts/branching/create_empty_repos.sh {branched} +$ sudo -u ftpsync sh scripts/branching-epel/create_empty_repos.sh 10.1 ---- +WARNING: TODO: Check if these notes apply for EPEL10 + [NOTE] ==== Update the link in /mnt/koji/repos/rawhide/latest as per https://pagure.io/releng/issue/12255. @@ -196,28 +187,30 @@ be owned by _ftpsync:ftpsync_ Check directory permissions (should be "0755") to ensure new composes synchronize correctly. ==== -=== Creating EPEL10. release +=== Creating EPEL10.1 release To create a new minor release in bodhi, you need to run: -WARNING: TODO: Update scripts for EPEL10 (check hackmd) - [source,bash,subs="attributes"] ---- $ bodhi releases create \ - --name "F{rawhide}" --long-name "Fedora {rawhide}" \ - --id-prefix FEDORA --version {rawhide} --branch f{rawhide} \ - --dist-tag f{rawhide} \ - --stable-tag f{rawhide} \ - --testing-tag f{rawhide}-updates-testing \ - --candidate-tag f{rawhide}-updates-candidate \ - --pending-stable-tag f{rawhide}-updates-pending \ - --pending-testing-tag f{rawhide}-updates-testing-pending \ - --pending-signing-tag f{rawhide}-signing-pending \ - --state pending \ - --override-tag f{rawhide}-override \ - -create-automatic-updates \ - --not-composed-by-bodhi + --name "EPEL-10.1" \ + --long-name "Fedora EPEL 10.1" \ + --version 10 \ + --branch epel10 \ + --id-prefix FEDORA-EPEL \ + --dist-tag epel10.1 \ + --stable-tag epel10.1 \ + --testing-tag epel10.1-testing \ + --candidate-tag epel10.1-testing-candidate \ + --pending-signing-tag epel10.1-signing-pending \ + --pending-testing-tag epel10.1-testing-pending \ + --pending-stable-tag epel10.1-pending \ + --override-tag epel10.1-override \ + --mail-template fedora_epel_legacy_errata_template \ + --state current \ + --composed-by-bodhi \ + --no-create-automatic-updates ---- WARNING: TODO: Check if any of this applies for EPEL10 @@ -234,14 +227,14 @@ $ sudo systemctl restart fm-consumer@config.service bodhi-celery.service === Update rawhide koji repo -WARNING: TODO: Check if any of this applies for EPEL10 - -We need to point the _rawhide_ buildroot repo to the newly created -rawhide buildroot. This way kojira doesn't make a newrepo for _rawhide_ -target as often as fxx-build (new rawhide buildroot). +We need to point the _epel10_ buildroot repo to the newly created +epel buildroot. This way kojira doesn't make a newrepo for _epel10_ +target as often as epel10.xx-build (new epel10 buildroot). Run the following commands from any of the compose boxes: +WARNING: TODO: Check if any of this applies for EPEL10 + [source,bash,subs="attributes"] ---- $ cd /mnt/koji/repos/rawhide @@ -251,8 +244,6 @@ $ ln -s ../f{rawhide}-build/latest ./latest === Update block_retired.py script -WARNING: TODO: Check if any of this apply for EPEL10 - https://pagure.io/releng/blob/main/f/scripts/block_retired.py[block_retired.py] script in releng repo should be updated with rawhide release and also branched release should be added to the script. @@ -263,8 +254,6 @@ commit] as an example. === Updating MirrorManager -WARNING: TODO: Check if any of this applies for EPEL10 - We need to update the mirrormanager so that it will point EPEL10 to the new EPEL10 minor. From 301b26c50905311d40ff8fd955634a9779fee3cb Mon Sep 17 00:00:00 2001 From: Diego Herrera Date: Mar 10 2025 17:21:08 +0000 Subject: [PATCH 3/23] Update some steps of the process Signed-off-by: Diego Herrera --- diff --git a/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc b/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc index 922d1b3..eeb6223 100644 --- a/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc +++ b/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc @@ -171,18 +171,15 @@ from https://pagure.io/releng[pagure releng] repo. This should be run on $ sudo -u ftpsync sh scripts/branching-epel/create_empty_repos.sh 10.1 ---- -WARNING: TODO: Check if these notes apply for EPEL10 - [NOTE] ==== -Update the link in /mnt/koji/repos/rawhide/latest as per https://pagure.io/releng/issue/12255. +Update the link in /mnt/koji/repos/epel10/latest use https://pagure.io/releng/issue/12255 as reference. ==== [NOTE] ==== Please verify the repo permissions that are created under -/pub/fedora/linux/development/ and -/pub/fedora-secondary/development/. They should +/pub/epel/. They should be owned by _ftpsync:ftpsync_ Check directory permissions (should be "0755") to ensure new composes synchronize correctly. ==== @@ -233,13 +230,11 @@ target as often as epel10.xx-build (new epel10 buildroot). Run the following commands from any of the compose boxes: -WARNING: TODO: Check if any of this applies for EPEL10 - [source,bash,subs="attributes"] ---- -$ cd /mnt/koji/repos/rawhide +$ cd /mnt/koji/repos/epel10 $ rm -f latest -$ ln -s ../f{rawhide}-build/latest ./latest +$ ln -s ../epel10.1-build/latest ./latest ---- === Update block_retired.py script From 31bc81d135f5dd8c0226bc71d30fa24388960bfb Mon Sep 17 00:00:00 2001 From: Diego Herrera Date: Mar 10 2025 17:21:08 +0000 Subject: [PATCH 4/23] Add extra note about testing packages Signed-off-by: Diego Herrera --- diff --git a/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc b/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc index eeb6223..b5ade28 100644 --- a/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc +++ b/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc @@ -145,6 +145,12 @@ Run https://pagure.io/releng/blob/main/f/scripts/branching-epel/make-koji-release-tags[make-koji-release-tags] script in https://pagure.io/releng[pagure releng] repo +[NOTE] +==== +Make sure that things tagged on epel10.0-testing are tagged correctly. +https://bodhi.fedoraproject.org/updates/?search=&status=pending&status=testing&releases=EPEL-10.0 +==== + == Bodhi === Linking Empty Repos From 27e1b6f27eaf193e1c5e37d1a1e843d38fa8803d Mon Sep 17 00:00:00 2001 From: Diego Herrera Date: Mar 10 2025 17:21:08 +0000 Subject: [PATCH 5/23] Drop more stuff that isn't required for EPEL Signed-off-by: Diego Herrera --- diff --git a/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc b/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc index b5ade28..9f285b5 100644 --- a/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc +++ b/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc @@ -54,7 +54,6 @@ $ koji cancel Now we need to update dist-git in two steps: * Create the new branch in git -* Update the gitolite.conf to allow user to push to this new branch For both of these actions you will need the file generated by pdc above. @@ -105,15 +104,6 @@ Please check these files from the https://pagure.io/fedora-infra/ansible/c/549e5 Make sure that the SLAs are added, check https://pagure.io/fedora-infra/ansible/pull-request/2190#request_diff[this PR] for reference. -=== Fedora Branched - -WARNING: TODO: Confirm if we should skip this part for EPEL - -. Set https://pagure.io/fedora-infra/ansible/blob/main/f/vars/all/FedoraBranched.yaml[FedoraBranched] to True. -. Set https://pagure.io/fedora-infra/ansible/blob/main/f/vars/all/FedoraBranchedBodhi.yaml[FedoraBranchedBodhi] to preenable. - -Please check the file `FedoraBranched.yaml` and `FedoraBranchedBodhi.yaml` from the https://pagure.io/fedora-infra/ansible/c/549e5d3ace41c04fdbef9d81f359f16c2fe0c2fa?branch=main[commit] for your reference. - === Koji hub Update the koji hub config to allow side tags for new koji rawhide (currently f{rawhide}) tag. @@ -177,19 +167,6 @@ from https://pagure.io/releng[pagure releng] repo. This should be run on $ sudo -u ftpsync sh scripts/branching-epel/create_empty_repos.sh 10.1 ---- -[NOTE] -==== -Update the link in /mnt/koji/repos/epel10/latest use https://pagure.io/releng/issue/12255 as reference. -==== - -[NOTE] -==== -Please verify the repo permissions that are created under -/pub/epel/. They should -be owned by _ftpsync:ftpsync_ -Check directory permissions (should be "0755") to ensure new composes synchronize correctly. -==== - === Creating EPEL10.1 release To create a new minor release in bodhi, you need to run: @@ -228,21 +205,6 @@ $ sudo rbac-playbook openshift-apps/bodhi.yml $ sudo systemctl restart fm-consumer@config.service bodhi-celery.service ``` -=== Update rawhide koji repo - -We need to point the _epel10_ buildroot repo to the newly created -epel buildroot. This way kojira doesn't make a newrepo for _epel10_ -target as often as epel10.xx-build (new epel10 buildroot). - -Run the following commands from any of the compose boxes: - -[source,bash,subs="attributes"] ----- -$ cd /mnt/koji/repos/epel10 -$ rm -f latest -$ ln -s ../epel10.1-build/latest ./latest ----- - === Update block_retired.py script https://pagure.io/releng/blob/main/f/scripts/block_retired.py[block_retired.py] From 4590f20cf9efa47885c7875f4fd12f199f9c7fe9 Mon Sep 17 00:00:00 2001 From: Diego Herrera Date: Mar 10 2025 17:21:08 +0000 Subject: [PATCH 6/23] Remove final TODO's Signed-off-by: Diego Herrera --- diff --git a/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc b/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc index 9f285b5..62caea5 100644 --- a/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc +++ b/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc @@ -84,8 +84,6 @@ Apps in https://pagure.io/fedora-infra/ansible[ansible] need to be updated to be === Bodhi -WARNING: TODO: This is a step from Fedora, check which scripts are impacted by EPEL branching - Bodhi needs to be updated to add new release. This needs to be done in https://pagure.io/fedora-infra/ansible/blob/main/f/roles/bodhi2[bodhi2 role] in infra ansible repo. This change includes, updating `koji-sync-listener.py`, @@ -93,9 +91,15 @@ infra ansible repo. This change includes, updating `koji-sync-listener.py`, * `roles/bodhi2/backend/files/new-updates-sync` * `roles/bodhi2/backend/tasks/main.yml` -* `roles/bodhi2/backend/templates/pungi.rpm.conf.j2` * `roles/bodhi2/backend/templates/koji_sync_listener.toml` +[NOTE] +==== +In this step in Fedora, `roles/bodhi2/backend/templates/pungi.rpm.conf.j2` +would also need to be modified. But here it's not required since the signing key doesn't +change between minor versions +==== + Please check these files from the https://pagure.io/fedora-infra/ansible/c/549e5d3ace41c04fdbef9d81f359f16c2fe0c2fa?branch=main[commit] for your reference. == Toddlers @@ -106,14 +110,12 @@ Make sure that the SLAs are added, check https://pagure.io/fedora-infra/ansible/ === Koji hub -Update the koji hub config to allow side tags for new koji rawhide (currently f{rawhide}) tag. +Update the koji hub config to allow side tags for new koji epel10 minor (currently epel10.1) tag. Please check the file `roles/koji_hub/templates/hub.conf.j2` from https://pagure.io/fedora-infra/ansible/c/9860fa89b5a9e223fc7a42299fa30a8ffdbfbd77[this PR] for your reference. === Push the changes -WARNING: TODO: Check this PR to see if there are changes required to be done in this regard. - When done editing the files, commit, push and apply them via the corresponding ansible playbook: @@ -129,7 +131,7 @@ $ sudo rbac-playbook groups/mbs.yml -t mbs == Koji The koji build system needs to have some tag/target work done to handle -builds from the new branch and to update where builds from rawhide go. +builds from the new branch and to update where builds from epel10 go. Run https://pagure.io/releng/blob/main/f/scripts/branching-epel/make-koji-release-tags[make-koji-release-tags] @@ -193,8 +195,6 @@ $ bodhi releases create \ --no-create-automatic-updates ---- -WARNING: TODO: Check if any of this applies for EPEL10 - You need to run the `bodhi openshift` playbook, so that UI will know about the new release. Then, you need to restart *fm-consumer@config.service* and *bodhi-celery.service* services on @@ -208,7 +208,7 @@ $ sudo systemctl restart fm-consumer@config.service bodhi-celery.service === Update block_retired.py script https://pagure.io/releng/blob/main/f/scripts/block_retired.py[block_retired.py] -script in releng repo should be updated with rawhide release and also +script in releng repo should be updated with EPEL10 release and also branched release should be added to the script. Please look at this @@ -224,11 +224,11 @@ Please follow the instructions in the https://pagure.io/fedora-infrastructure/issue/9239#comment-671446[fedora infra ticket] to update the database of mirrormanager. -=== Branch new rawhide in Koschei +=== Branch new EPEL10 minor in Koschei -WARNING: TODO: Check if any of this applies for EPEL10 +WARNING: TODO: Check if we want Koschei for minor -Branch new fedora rawhide in +Branch new fedora EPEL10 minor in https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/koschei/#_branching_a_new_fedora_release[koschei]. == Consider Before Running From 8ccfa23756bfb75cba19443d0c58f3b286745089 Mon Sep 17 00:00:00 2001 From: Diego Herrera Date: Mar 10 2025 17:21:08 +0000 Subject: [PATCH 7/23] Remove warning for Koschei Signed-off-by: Diego Herrera --- diff --git a/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc b/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc index 62caea5..8ab8f55 100644 --- a/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc +++ b/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc @@ -226,8 +226,6 @@ infra ticket] to update the database of mirrormanager. === Branch new EPEL10 minor in Koschei -WARNING: TODO: Check if we want Koschei for minor - Branch new fedora EPEL10 minor in https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/koschei/#_branching_a_new_fedora_release[koschei]. From 916e6b2634500cc17ce5737d19937f671c961522 Mon Sep 17 00:00:00 2001 From: Diego Herrera Date: Mar 10 2025 17:21:08 +0000 Subject: [PATCH 8/23] Remove iptable instructions Signed-off-by: Diego Herrera --- diff --git a/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc b/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc index 8ab8f55..19ca17b 100644 --- a/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc +++ b/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc @@ -11,28 +11,11 @@ Originally, the EPEL repository had packages that built against the latest RHEL One day before the mass branching, we send out announcemt because during mass branching, new koji builds for EPEL are disabled. This will imply a koji outage in Fedora too. -== Disable EPEL10 builds in koji - -The recommended method to achieve this is by adding firewall rules to both koji01 and koji02 servers, effectively blocking connections from proxy01 and proxy10. This can be achieved with the following iptables commands: - -[source,bash,subs="attributes"] ----- -iptables -I INPUT -m tcp -p tcp --dport 80 -s proxy01.iad2.fedoraproject.org -j REJECT -iptables -I INPUT -m tcp -p tcp --dport 80 -s proxy10.iad2.fedoraproject.org -j REJECT ----- - -These commands reject incoming traffic on port 80 from the specified proxies, preventing external submissions. Internal connections routed via proxy101 and proxy110 will continue to function as expected. - -To reverse the firewall changes and allow external submissions again, use: - -[source,bash,subs="attributes"] ----- -iptables -D INPUT -m tcp -p tcp --dport 80 -s proxy01.iad2.fedoraproject.org -j REJECT -iptables -D INPUT -m tcp -p tcp --dport 80 -s proxy10.iad2.fedoraproject.org -j REJECT - ----- - -This change should be implemented on both koji01 and koji02 machine. +[NOTE] +==== +In Fedora it is recommended to https://docs.fedoraproject.org/en-US/infra/release_guide/sop_mass_branching/#_disable_rawhide_builds_in_koji[stop connections to koji through firewall]. +But it was https://pagure.io/infra-docs-fpo/pull-request/357[decided not to require it] for this process. +==== === Cancel all running build for EPEL10 From 9fae8c31144c3d034888160637b6d78b5a1953fb Mon Sep 17 00:00:00 2001 From: Diego Herrera Date: Mar 10 2025 17:21:08 +0000 Subject: [PATCH 9/23] Clean up errors Signed-off-by: Diego Herrera --- diff --git a/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc b/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc index 19ca17b..dece206 100644 --- a/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc +++ b/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc @@ -38,17 +38,17 @@ Now we need to update dist-git in two steps: * Create the new branch in git -For both of these actions you will need the file generated by pdc above. +For both of these actions you will need the list of active repos in EPEL10. === Get current EPEL active packages To get the list of all the current active packages, you must run the `scripts/branching-epe/get_all_active_packages_branching.sh` script. ``` -sh scripts/branching-epel/get_all_active_packages_branching.sh 10.{next_minor} +sh scripts/branching-epel/get_all_active_packages_branching.sh 10.1 ``` -The results will be saved to the components_epel10.{next_minor}.txt +The results will be saved to the components_epel10.1.txt === Create the git branches @@ -59,7 +59,7 @@ $ sudo -u pagure python /usr/local/bin/mass-branching-git.py < ``` Where `` will be like `epel10.` and the -`` the path to the file generated by pdc above. +`` the path to the file generated in the previous step. == Ansible @@ -211,7 +211,3 @@ infra ticket] to update the database of mirrormanager. Branch new fedora EPEL10 minor in https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/koschei/#_branching_a_new_fedora_release[koschei]. - -== Consider Before Running - -NOTE: FIXME: Need some love here From e8665e2e70efade7d517fffecf384e163fa3b7ec Mon Sep 17 00:00:00 2001 From: Diego Herrera Date: Mar 10 2025 17:21:08 +0000 Subject: [PATCH 10/23] Add variables to the docs Signed-off-by: Diego Herrera --- diff --git a/modules/release_guide/pages/_partials/attributes.adoc b/modules/release_guide/pages/_partials/attributes.adoc index 01ed9bd..5465f1f 100644 --- a/modules/release_guide/pages/_partials/attributes.adoc +++ b/modules/release_guide/pages/_partials/attributes.adoc @@ -6,3 +6,6 @@ :branched_name: Forty Two :current: 41 :old_release: 40 +:epel_major: 10 +:epel_branched: 10.1 +:epel_current: 10.0 diff --git a/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc b/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc index dece206..e1db413 100644 --- a/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc +++ b/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc @@ -4,7 +4,7 @@ include::_partials/attributes.adoc[] == Description -Originally, the EPEL repository had packages that built against the latest RHEL minor, but since EPEL10, the EPEL repository now builds against the latest Centos Stream release. To keep consistency for RHEL releases, EPEL repository goes through a mass branching process to generate a new minor release branch. This process let's the developers continue to maintain their packages against Centos Stream without risking to break packages that already work on the current active RHEL minor target. +Originally, the EPEL repository had packages that built against the latest RHEL minor, but since EPEL{epel_major}, the EPEL repository now builds against the latest Centos Stream release. To keep consistency for RHEL releases, EPEL repository goes through a mass branching process to generate a new minor release branch. This process let's the developers continue to maintain their packages against Centos Stream without risking to break packages that already work on the current active RHEL minor target. == Send announcement @@ -17,13 +17,13 @@ In Fedora it is recommended to https://docs.fedoraproject.org/en-US/infra/releas But it was https://pagure.io/infra-docs-fpo/pull-request/357[decided not to require it] for this process. ==== -=== Cancel all running build for EPEL10 +=== Cancel all running build for EPEL{epel_major} List all running tasks and select only those relevant for current branching. [source,bash,subs="attributes"] ---- -$ koji list-builds --state=0 --type=rpm | grep epel10 | awk '{print $1}' +$ koji list-builds --state=0 --type=rpm | grep epel{epel_major} | awk '{print $1}' ---- Cancel each of those tasks @@ -38,17 +38,17 @@ Now we need to update dist-git in two steps: * Create the new branch in git -For both of these actions you will need the list of active repos in EPEL10. +For both of these actions you will need the list of active repos in EPEL{epel_major}. === Get current EPEL active packages To get the list of all the current active packages, you must run the `scripts/branching-epe/get_all_active_packages_branching.sh` script. ``` -sh scripts/branching-epel/get_all_active_packages_branching.sh 10.1 +sh scripts/branching-epel/get_all_active_packages_branching.sh {epel_branched} ``` -The results will be saved to the components_epel10.1.txt +The results will be saved to the components_epel{epel_branched}.txt === Create the git branches @@ -58,7 +58,7 @@ On `pkgs01.stg` (for testing) or `pkgs02` (for production), run: $ sudo -u pagure python /usr/local/bin/mass-branching-git.py ``` -Where `` will be like `epel10.` and the +Where `` will be like `epel{epel_branched}` and the `` the path to the file generated in the previous step. == Ansible @@ -93,7 +93,7 @@ Make sure that the SLAs are added, check https://pagure.io/fedora-infra/ansible/ === Koji hub -Update the koji hub config to allow side tags for new koji epel10 minor (currently epel10.1) tag. +Update the koji hub config to allow side tags for new koji epel{epel_major} minor (currently epel{epel_branched}) tag. Please check the file `roles/koji_hub/templates/hub.conf.j2` from https://pagure.io/fedora-infra/ansible/c/9860fa89b5a9e223fc7a42299fa30a8ffdbfbd77[this PR] for your reference. @@ -114,7 +114,7 @@ $ sudo rbac-playbook groups/mbs.yml -t mbs == Koji The koji build system needs to have some tag/target work done to handle -builds from the new branch and to update where builds from epel10 go. +builds from the new branch and to update where builds from epel{epel_major} go. Run https://pagure.io/releng/blob/main/f/scripts/branching-epel/make-koji-release-tags[make-koji-release-tags] @@ -122,8 +122,8 @@ script in https://pagure.io/releng[pagure releng] repo [NOTE] ==== -Make sure that things tagged on epel10.0-testing are tagged correctly. -https://bodhi.fedoraproject.org/updates/?search=&status=pending&status=testing&releases=EPEL-10.0 +Make sure that things tagged on epel{epel_current}-testing are tagged correctly. +https://bodhi.fedoraproject.org/updates/?search=&status=pending&status=testing&releases=EPEL-{epel_current} ==== == Bodhi @@ -136,8 +136,8 @@ missing repos. The following commands should be run on [source,bash,subs="attributes"] ---- -$ sudo ln -s /mnt/koji/compose/updates/empty-repo/ /mnt/koji/compose/updates/epel10.1-updates -$ sudo ln -s /mnt/koji/compose/updates/empty-repo/ /mnt/koji/compose/updates/epel10.1-updates-testing +$ sudo ln -s /mnt/koji/compose/updates/empty-repo/ /mnt/koji/compose/updates/epel{epel_branched}-updates +$ sudo ln -s /mnt/koji/compose/updates/empty-repo/ /mnt/koji/compose/updates/epel{epel_branched}-updates-testing ---- === Creating Empty Repos @@ -149,29 +149,29 @@ from https://pagure.io/releng[pagure releng] repo. This should be run on [source,bash,subs="attributes"] ---- -$ sudo -u ftpsync sh scripts/branching-epel/create_empty_repos.sh 10.1 +$ sudo -u ftpsync sh scripts/branching-epel/create_empty_repos.sh {epel_branched} ---- -=== Creating EPEL10.1 release +=== Creating EPEL{epel_branched} release To create a new minor release in bodhi, you need to run: [source,bash,subs="attributes"] ---- $ bodhi releases create \ - --name "EPEL-10.1" \ - --long-name "Fedora EPEL 10.1" \ - --version 10 \ - --branch epel10 \ + --name "EPEL-{epel_branched}" \ + --long-name "Fedora EPEL {epel_branched}" \ + --version {epel_major} \ + --branch epel{epel_major} \ --id-prefix FEDORA-EPEL \ - --dist-tag epel10.1 \ - --stable-tag epel10.1 \ - --testing-tag epel10.1-testing \ - --candidate-tag epel10.1-testing-candidate \ - --pending-signing-tag epel10.1-signing-pending \ - --pending-testing-tag epel10.1-testing-pending \ - --pending-stable-tag epel10.1-pending \ - --override-tag epel10.1-override \ + --dist-tag epel{epel_branched} \ + --stable-tag epel{epel_branched} \ + --testing-tag epel{epel_branched}-testing \ + --candidate-tag epel{epel_branched}-testing-candidate \ + --pending-signing-tag epel{epel_branched}-signing-pending \ + --pending-testing-tag epel{epel_branched}-testing-pending \ + --pending-stable-tag epel{epel_branched}-pending \ + --override-tag epel{epel_branched}-override \ --mail-template fedora_epel_legacy_errata_template \ --state current \ --composed-by-bodhi \ @@ -191,7 +191,7 @@ $ sudo systemctl restart fm-consumer@config.service bodhi-celery.service === Update block_retired.py script https://pagure.io/releng/blob/main/f/scripts/block_retired.py[block_retired.py] -script in releng repo should be updated with EPEL10 release and also +script in releng repo should be updated with EPEL{epel_major} release and also branched release should be added to the script. Please look at this @@ -200,14 +200,14 @@ commit] as an example. === Updating MirrorManager -We need to update the mirrormanager so that it will point EPEL10 to the -new EPEL10 minor. +We need to update the mirrormanager so that it will point EPEL{epel_major} to the +new EPEL{epel_major} minor. Please follow the instructions in the https://pagure.io/fedora-infrastructure/issue/9239#comment-671446[fedora infra ticket] to update the database of mirrormanager. -=== Branch new EPEL10 minor in Koschei +=== Branch new EPEL{epel_major} minor in Koschei -Branch new fedora EPEL10 minor in +Branch new fedora EPEL{epel_major} minor in https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/koschei/#_branching_a_new_fedora_release[koschei]. From bd17b501278a3258df3ad648b4aac388b67b6d41 Mon Sep 17 00:00:00 2001 From: Diego Herrera Date: Mar 10 2025 17:21:08 +0000 Subject: [PATCH 11/23] Add fixes Signed-off-by: Diego Herrera --- diff --git a/modules/release_guide/pages/_partials/attributes.adoc b/modules/release_guide/pages/_partials/attributes.adoc index 5465f1f..4af0877 100644 --- a/modules/release_guide/pages/_partials/attributes.adoc +++ b/modules/release_guide/pages/_partials/attributes.adoc @@ -7,5 +7,5 @@ :current: 41 :old_release: 40 :epel_major: 10 -:epel_branched: 10.1 +:epel_new: 10.1 :epel_current: 10.0 diff --git a/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc b/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc index e1db413..c085e1e 100644 --- a/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc +++ b/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc @@ -8,7 +8,7 @@ Originally, the EPEL repository had packages that built against the latest RHEL == Send announcement -One day before the mass branching, we send out announcemt because during mass branching, +One day before the mass branching, we send out announcement because during mass branching, new koji builds for EPEL are disabled. This will imply a koji outage in Fedora too. [NOTE] @@ -44,21 +44,23 @@ For both of these actions you will need the list of active repos in EPEL{epel_ma To get the list of all the current active packages, you must run the `scripts/branching-epe/get_all_active_packages_branching.sh` script. -``` -sh scripts/branching-epel/get_all_active_packages_branching.sh {epel_branched} -``` +[source,bash,subs="attributes"] +---- +sh scripts/branching-epel/get_all_active_packages_branching.sh {epel_new} +---- -The results will be saved to the components_epel{epel_branched}.txt +The results will be saved to the components_epel{epel_new}.txt === Create the git branches On `pkgs01.stg` (for testing) or `pkgs02` (for production), run: -``` -$ sudo -u pagure python /usr/local/bin/mass-branching-git.py -``` +[source,bash,subs="attributes"] +---- +$ sudo -u pagure python /usr/local/bin/mass-branching-git.py {epel_new} components_epel{epel_new} +---- -Where `` will be like `epel{epel_branched}` and the +Where `` will be like `epel{epel_new}` and the `` the path to the file generated in the previous step. == Ansible @@ -93,7 +95,7 @@ Make sure that the SLAs are added, check https://pagure.io/fedora-infra/ansible/ === Koji hub -Update the koji hub config to allow side tags for new koji epel{epel_major} minor (currently epel{epel_branched}) tag. +Update the koji hub config to allow side tags for new koji epel{epel_major} minor (currently epel{epel_new}) tag. Please check the file `roles/koji_hub/templates/hub.conf.j2` from https://pagure.io/fedora-infra/ansible/c/9860fa89b5a9e223fc7a42299fa30a8ffdbfbd77[this PR] for your reference. @@ -136,8 +138,8 @@ missing repos. The following commands should be run on [source,bash,subs="attributes"] ---- -$ sudo ln -s /mnt/koji/compose/updates/empty-repo/ /mnt/koji/compose/updates/epel{epel_branched}-updates -$ sudo ln -s /mnt/koji/compose/updates/empty-repo/ /mnt/koji/compose/updates/epel{epel_branched}-updates-testing +$ sudo ln -s /mnt/koji/compose/updates/empty-repo/ /mnt/koji/compose/updates/epel{epel_new}-updates +$ sudo ln -s /mnt/koji/compose/updates/empty-repo/ /mnt/koji/compose/updates/epel{epel_new}-updates-testing ---- === Creating Empty Repos @@ -149,29 +151,29 @@ from https://pagure.io/releng[pagure releng] repo. This should be run on [source,bash,subs="attributes"] ---- -$ sudo -u ftpsync sh scripts/branching-epel/create_empty_repos.sh {epel_branched} +$ sudo -u ftpsync sh scripts/branching-epel/create_empty_repos.sh {epel_new} ---- -=== Creating EPEL{epel_branched} release +=== Creating EPEL{epel_new} release To create a new minor release in bodhi, you need to run: [source,bash,subs="attributes"] ---- $ bodhi releases create \ - --name "EPEL-{epel_branched}" \ - --long-name "Fedora EPEL {epel_branched}" \ + --name "EPEL-{epel_new}" \ + --long-name "Fedora EPEL {epel_new}" \ --version {epel_major} \ --branch epel{epel_major} \ --id-prefix FEDORA-EPEL \ - --dist-tag epel{epel_branched} \ - --stable-tag epel{epel_branched} \ - --testing-tag epel{epel_branched}-testing \ - --candidate-tag epel{epel_branched}-testing-candidate \ - --pending-signing-tag epel{epel_branched}-signing-pending \ - --pending-testing-tag epel{epel_branched}-testing-pending \ - --pending-stable-tag epel{epel_branched}-pending \ - --override-tag epel{epel_branched}-override \ + --dist-tag epel{epel_new} \ + --stable-tag epel{epel_new} \ + --testing-tag epel{epel_new}-testing \ + --candidate-tag epel{epel_new}-testing-candidate \ + --pending-signing-tag epel{epel_new}-signing-pending \ + --pending-testing-tag epel{epel_new}-testing-pending \ + --pending-stable-tag epel{epel_new}-pending \ + --override-tag epel{epel_new}-override \ --mail-template fedora_epel_legacy_errata_template \ --state current \ --composed-by-bodhi \ From 6eae406dd16d561a00d50616e1895f4faae1f07c Mon Sep 17 00:00:00 2001 From: Diego Herrera Date: Mar 10 2025 17:21:08 +0000 Subject: [PATCH 12/23] Fix grep on koji list command Signed-off-by: Diego Herrera --- diff --git a/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc b/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc index c085e1e..1c74a8c 100644 --- a/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc +++ b/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc @@ -23,7 +23,7 @@ List all running tasks and select only those relevant for current branching. [source,bash,subs="attributes"] ---- -$ koji list-builds --state=0 --type=rpm | grep epel{epel_major} | awk '{print $1}' +$ koji list-builds --state=0 --type=rpm | grep el{epel_major}_0 | awk '{print $1}' ---- Cancel each of those tasks From 0cb62b2aef3ad7bec34bed5c2c9ffad93f369e16 Mon Sep 17 00:00:00 2001 From: Diego Herrera Date: Mar 10 2025 17:21:08 +0000 Subject: [PATCH 13/23] Fix explanation Signed-off-by: Diego Herrera --- diff --git a/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc b/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc index 1c74a8c..04c45fe 100644 --- a/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc +++ b/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc @@ -57,11 +57,11 @@ On `pkgs01.stg` (for testing) or `pkgs02` (for production), run: [source,bash,subs="attributes"] ---- -$ sudo -u pagure python /usr/local/bin/mass-branching-git.py {epel_new} components_epel{epel_new} +$ sudo -u pagure python /usr/local/bin/mass-branching-git.py {epel_new} components_epel{epel_new}.txt ---- -Where `` will be like `epel{epel_new}` and the -`` the path to the file generated in the previous step. +The first argument is the new epel version and the +second one is the path to the file generated in the previous step. == Ansible From b80b4a83bf7e095eb3e6a62ba47437002ca3eea6 Mon Sep 17 00:00:00 2001 From: Diego Herrera Date: Mar 10 2025 17:21:08 +0000 Subject: [PATCH 14/23] Change new parameter to work as a flag Signed-off-by: Diego Herrera --- diff --git a/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc b/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc index 04c45fe..1166cea 100644 --- a/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc +++ b/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc @@ -46,7 +46,7 @@ To get the list of all the current active packages, you must run the `scripts/br [source,bash,subs="attributes"] ---- -sh scripts/branching-epel/get_all_active_packages_branching.sh {epel_new} +sh scripts/branching-epel/get_all_active_packages_branching.sh {epel_major} ---- The results will be saved to the components_epel{epel_new}.txt @@ -57,7 +57,7 @@ On `pkgs01.stg` (for testing) or `pkgs02` (for production), run: [source,bash,subs="attributes"] ---- -$ sudo -u pagure python /usr/local/bin/mass-branching-git.py {epel_new} components_epel{epel_new}.txt +$ sudo -u pagure python /usr/local/bin/mass-branching-git.py epel{epel_current} components_epel{epel_major}.txt epel{epel_major} ---- The first argument is the new epel version and the From f901165fbe90435fa07999a950882be1de676c9a Mon Sep 17 00:00:00 2001 From: Diego Herrera Date: Mar 10 2025 17:21:08 +0000 Subject: [PATCH 15/23] Add manual koji steps to the SOP --- diff --git a/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc b/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc index 1166cea..6eb5d41 100644 --- a/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc +++ b/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc @@ -9,13 +9,20 @@ Originally, the EPEL repository had packages that built against the latest RHEL == Send announcement One day before the mass branching, we send out announcement because during mass branching, -new koji builds for EPEL are disabled. This will imply a koji outage in Fedora too. +new koji builds for EPEL are disabled. + +=== Remove Targets -[NOTE] -==== In Fedora it is recommended to https://docs.fedoraproject.org/en-US/infra/release_guide/sop_mass_branching/#_disable_rawhide_builds_in_koji[stop connections to koji through firewall]. But it was https://pagure.io/infra-docs-fpo/pull-request/357[decided not to require it] for this process. -==== + +In the case of EPEL, just removing the targets is enough to disable it's usage for the time being while the branching process it's being done. + +[source,bash,subs="attributes"] +---- +$ koji remove-target epel10-candidate +$ koji remove-target epel10 +---- === Cancel all running build for EPEL{epel_major} @@ -57,7 +64,7 @@ On `pkgs01.stg` (for testing) or `pkgs02` (for production), run: [source,bash,subs="attributes"] ---- -$ sudo -u pagure python /usr/local/bin/mass-branching-git.py epel{epel_current} components_epel{epel_major}.txt epel{epel_major} +$ sudo -u pagure python /usr/local/bin/mass-branching-git.py --branch-from epel{epel_major} epel{epel_current} components_epel{epel_major}.txt ---- The first argument is the new epel version and the @@ -190,6 +197,17 @@ $ sudo rbac-playbook openshift-apps/bodhi.yml $ sudo systemctl restart fm-consumer@config.service bodhi-celery.service ``` +=== Recreate targets + +Once mass-branching is complete and the new release exists in bodhi, +re-create these targets with the following commands: + +[source,bash,subs="attributes"] +---- +$ koji add-target epel${release_major} epel${release}-build epel${release}-testing-candidate +$ koji add-target epel${release_major}-candidate epel${release}-build epel${release}-testing-candidate +---- + === Update block_retired.py script https://pagure.io/releng/blob/main/f/scripts/block_retired.py[block_retired.py] From c491e5391146a01f687a239d910120797347a009 Mon Sep 17 00:00:00 2001 From: Diego Herrera Date: Mar 10 2025 17:21:08 +0000 Subject: [PATCH 16/23] Set the correct playbooks that are needed to run --- diff --git a/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc b/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc index 6eb5d41..860a9e8 100644 --- a/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc +++ b/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc @@ -111,14 +111,13 @@ Please check the file `roles/koji_hub/templates/hub.conf.j2` from https://pagure When done editing the files, commit, push and apply them via the corresponding ansible playbook: -``` +[source,bash,subs="attributes"] +---- +$ sudo rbac-playbook groups/batcave.yml $ sudo rbac-playbook groups/koji-hub.yml -$ sudo rbac-playbook groups/releng-compose.yml $ sudo rbac-playbook groups/bodhi-backend.yml -$ sudo rbac-playbook openshift-apps/greenwave.yml -$ sudo -i ansible-playbook /srv/web/infra/ansible/playbooks/$ groups/proxies.yml -t pkgdb2 -$ sudo rbac-playbook groups/mbs.yml -t mbs -``` +$ sudo rbac-playbook manual/autosign.yml +---- == Koji From 85e991b5d61ca8682a0f37db7e444060843e2702 Mon Sep 17 00:00:00 2001 From: Diego Herrera Date: Mar 10 2025 17:21:08 +0000 Subject: [PATCH 17/23] Fix the new epel link on koji --- diff --git a/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc b/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc index 860a9e8..9c8e464 100644 --- a/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc +++ b/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc @@ -144,8 +144,8 @@ missing repos. The following commands should be run on [source,bash,subs="attributes"] ---- -$ sudo ln -s /mnt/koji/compose/updates/empty-repo/ /mnt/koji/compose/updates/epel{epel_new}-updates -$ sudo ln -s /mnt/koji/compose/updates/empty-repo/ /mnt/koji/compose/updates/epel{epel_new}-updates-testing +$ sudo ln -s /mnt/koji/compose/updates/empty-repo/ /mnt/koji/compose/updates/epel{epel_new} +$ sudo ln -s /mnt/koji/compose/updates/empty-repo/ /mnt/koji/compose/updates/epel{epel_new}-testing ---- === Creating Empty Repos From 1f20d1434ac8610dec183e15120c536f5b1b9a36 Mon Sep 17 00:00:00 2001 From: Diego Herrera Date: Mar 10 2025 17:21:08 +0000 Subject: [PATCH 18/23] Fix version that needs new empty repos --- diff --git a/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc b/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc index 9c8e464..d1f2563 100644 --- a/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc +++ b/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc @@ -151,13 +151,13 @@ $ sudo ln -s /mnt/koji/compose/updates/empty-repo/ /mnt/koji/compose/updates/epe === Creating Empty Repos To create empty repos on the master mirror, run -https://pagure.io/releng/blob/main/f/scripts/branching-epel/create_empty_repos.sh[create_emtpy_repos.sh] +https://pagure.io/releng/blob/main/f/scripts/branching-epel/create_empty_repos.sh[create_empty_repos.sh] from https://pagure.io/releng[pagure releng] repo. This should be run on *bodhi-backend01.phx2.fedoraproject.org* [source,bash,subs="attributes"] ---- -$ sudo -u ftpsync sh scripts/branching-epel/create_empty_repos.sh {epel_new} +$ sudo -u ftpsync sh scripts/branching-epel/create_empty_repos.sh {epel_current} ---- === Creating EPEL{epel_new} release From d0f84ac8b87f413277b78fd87bf841ba0f329d0c Mon Sep 17 00:00:00 2001 From: Diego Herrera Date: Mar 10 2025 17:21:08 +0000 Subject: [PATCH 19/23] Remove deprecated steps --- diff --git a/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc b/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc index d1f2563..8d4d033 100644 --- a/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc +++ b/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc @@ -186,16 +186,6 @@ $ bodhi releases create \ --no-create-automatic-updates ---- -You need to run the `bodhi openshift` playbook, so that UI will know -about the new release. Then, you need to restart -*fm-consumer@config.service* and *bodhi-celery.service* services on -*bodhi-backend01.phx2.fedoraproject.org*: - -``` -$ sudo rbac-playbook openshift-apps/bodhi.yml -$ sudo systemctl restart fm-consumer@config.service bodhi-celery.service -``` - === Recreate targets Once mass-branching is complete and the new release exists in bodhi, @@ -207,16 +197,6 @@ $ koji add-target epel${release_major} epel${release}-build epel${rele $ koji add-target epel${release_major}-candidate epel${release}-build epel${release}-testing-candidate ---- -=== Update block_retired.py script - -https://pagure.io/releng/blob/main/f/scripts/block_retired.py[block_retired.py] -script in releng repo should be updated with EPEL{epel_major} release and also -branched release should be added to the script. - -Please look at this -https://pagure.io/releng/c/9eb97f491f7a767ab8b90498adfa3b34ee235247?branch=main[block_retired.py -commit] as an example. - === Updating MirrorManager We need to update the mirrormanager so that it will point EPEL{epel_major} to the From 08b30ef08e45b51bd1195ebc1dfedd94a2b6a07f Mon Sep 17 00:00:00 2001 From: Diego Herrera Date: Mar 10 2025 17:21:08 +0000 Subject: [PATCH 20/23] Setup compound variables for epel major and minor --- diff --git a/modules/release_guide/pages/_partials/attributes.adoc b/modules/release_guide/pages/_partials/attributes.adoc index 4af0877..a63d14e 100644 --- a/modules/release_guide/pages/_partials/attributes.adoc +++ b/modules/release_guide/pages/_partials/attributes.adoc @@ -7,5 +7,7 @@ :current: 41 :old_release: 40 :epel_major: 10 -:epel_new: 10.1 -:epel_current: 10.0 +:epel_minor: 0 +:epel_minor_new: 1 +:epel_current: {epel_major}.{epel_minor} +:epel_new: {epel_major}.{epel_minor_new} diff --git a/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc b/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc index 8d4d033..c84d2e9 100644 --- a/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc +++ b/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc @@ -20,8 +20,8 @@ In the case of EPEL, just removing the targets is enough to disable it's usage f [source,bash,subs="attributes"] ---- -$ koji remove-target epel10-candidate -$ koji remove-target epel10 +$ koji remove-target epel{epel_major}-candidate +$ koji remove-target epel{epel_major} ---- === Cancel all running build for EPEL{epel_major} @@ -30,7 +30,7 @@ List all running tasks and select only those relevant for current branching. [source,bash,subs="attributes"] ---- -$ koji list-builds --state=0 --type=rpm | grep el{epel_major}_0 | awk '{print $1}' +$ koji list-builds --state=0 --type=rpm | grep el{epel_major}_{epel_minor} | awk '{print $1}' ---- Cancel each of those tasks From 8a0fce0a3cb04defabf2c3578ccc07ac6dacf9c4 Mon Sep 17 00:00:00 2001 From: Diego Herrera Date: Mar 10 2025 17:21:08 +0000 Subject: [PATCH 21/23] EPEL minor branching - add files required to modify to make bodhi work --- diff --git a/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc b/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc index c84d2e9..e6572a5 100644 --- a/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc +++ b/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc @@ -84,6 +84,8 @@ infra ansible repo. This change includes, updating `koji-sync-listener.py`, * `roles/bodhi2/backend/files/new-updates-sync` * `roles/bodhi2/backend/tasks/main.yml` * `roles/bodhi2/backend/templates/koji_sync_listener.toml` +* `roles/koji_hub/templates/hub.conf.j2` +* `roles/robosignatory/templates/robosignatory.toml.j2` [NOTE] ==== From 24356224f084cf37200bf59759a7d3192a8b5db1 Mon Sep 17 00:00:00 2001 From: Diego Herrera Date: Mar 10 2025 17:21:08 +0000 Subject: [PATCH 22/23] EPEL minor branching - Update MirrorManager instructions --- diff --git a/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc b/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc index e6572a5..5e34ae9 100644 --- a/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc +++ b/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc @@ -201,12 +201,12 @@ $ koji add-target epel${release_major}-candidate epel${release}-build epel${rele === Updating MirrorManager -We need to update the mirrormanager so that it will point EPEL{epel_major} to the -new EPEL{epel_major} minor. +MirrorManager will load the changes automatically, you can check that everything is in order throught the following command: -Please follow the instructions in the -https://pagure.io/fedora-infrastructure/issue/9239#comment-671446[fedora -infra ticket] to update the database of mirrormanager. +[source,bash,subs="attributes"] +---- +$ curl 'https://mirrors.fedoraproject.org/metalink?arch=x86_64&repo=epel-{epel_current}' +---- === Branch new EPEL{epel_major} minor in Koschei From 6a10ef55233c5bc93130cf9e375ad809320c14e5 Mon Sep 17 00:00:00 2001 From: Diego Herrera Date: Mar 10 2025 17:21:08 +0000 Subject: [PATCH 23/23] EPEL minor branching - Reorganize ansible section --- diff --git a/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc b/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc index 5e34ae9..0041f37 100644 --- a/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc +++ b/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc @@ -72,7 +72,11 @@ second one is the path to the file generated in the previous step. == Ansible -Apps in https://pagure.io/fedora-infra/ansible[ansible] need to be updated to be aware of a new branch. +Apps in https://pagure.io/fedora-infra/ansible[ansible] need to be updated +to be aware of the new branch. + +The following sections describe the things that need to be updated, but it's recommended to use https://pagure.io/fedora-infra/ansible/pull-request/2473#request_diff[this PR] +for reference. === Bodhi @@ -84,8 +88,6 @@ infra ansible repo. This change includes, updating `koji-sync-listener.py`, * `roles/bodhi2/backend/files/new-updates-sync` * `roles/bodhi2/backend/tasks/main.yml` * `roles/bodhi2/backend/templates/koji_sync_listener.toml` -* `roles/koji_hub/templates/hub.conf.j2` -* `roles/robosignatory/templates/robosignatory.toml.j2` [NOTE] ==== @@ -94,24 +96,29 @@ would also need to be modified. But here it's not required since the signing key change between minor versions ==== -Please check these files from the https://pagure.io/fedora-infra/ansible/c/549e5d3ace41c04fdbef9d81f359f16c2fe0c2fa?branch=main[commit] for your reference. +=== Toddlers -== Toddlers +==== Add new SLA to the toddlers App -=== Add new SLA to the toddlers App - -Make sure that the SLAs are added, check https://pagure.io/fedora-infra/ansible/pull-request/2190#request_diff[this PR] for reference. +Make sure that the SLAs are added. Check https://pagure.io/fedora-infra/ansible/pull-request/2190#request_diff[this PR] +for reference on what needs to be available. === Koji hub Update the koji hub config to allow side tags for new koji epel{epel_major} minor (currently epel{epel_new}) tag. -Please check the file `roles/koji_hub/templates/hub.conf.j2` from https://pagure.io/fedora-infra/ansible/c/9860fa89b5a9e223fc7a42299fa30a8ffdbfbd77[this PR] for your reference. +* `roles/koji_hub/templates/hub.conf.j2` + +=== Robosignatory + +All minor branches use the same key, so it's not required to generate a new one, but it's important to update the configuration files so that the propper keys are used + +* `roles/robosignatory/templates/robosignatory.toml.j2` === Push the changes When done editing the files, commit, push and apply them via the corresponding -ansible playbook: +ansible playbooks: [source,bash,subs="attributes"] ----