From 3afbe349748458966f214b0fe86153c27c6e69e7 Mon Sep 17 00:00:00 2001 From: Aurélien Bompard Date: Aug 06 2024 15:43:43 +0000 Subject: [PATCH 1/4] Update Mirrormanager's URL Signed-off-by: Aurélien Bompard --- diff --git a/modules/ROOT/pages/map_critical_services.adoc b/modules/ROOT/pages/map_critical_services.adoc index 673dc6f..123d0e8 100644 --- a/modules/ROOT/pages/map_critical_services.adoc +++ b/modules/ROOT/pages/map_critical_services.adoc @@ -255,7 +255,7 @@ packages are included in the compose and the compose is available after finishin |Mirrormanager is used to manage all the mirrors that are providing fedora packages. |Without it Fedora infra wouldn't be able to manage all the mirrors of Fedora and DNF wouldn't be able to automatically provide the best mirror for users. -|`mm-frontend01.iad2.fedoraproject.org` `mm-backend01.iad2.fedoraproject.org` +|Hosted in OpenShift |https://pagure.io/odcs[ODCS] |ODCS (On Demand Compose Service) provides users Fedora composes on demand. It generates an RPM diff --git a/modules/ROOT/pages/services.adoc b/modules/ROOT/pages/services.adoc index 7ac1ab0..cac5caa 100644 --- a/modules/ROOT/pages/services.adoc +++ b/modules/ROOT/pages/services.adoc @@ -251,7 +251,7 @@ Small API exposing the metadata contained in different RPM repositories. + * https://pagure.io/mdapi[Sources] -MirrorManager https://admin.fedoraproject.org/mirrormanager/[admin.fp.o/mirrormanager]:: +MirrorManager https://mirrormanager.fedoraproject.org/[mirrormanager.fp.o]:: Fedora is distributed to millions of systems globally. This would not be possible without the donations of time, disk space, and bandwidth by hundreds of volunteer system administrators and their companies or institutions. Your diff --git a/modules/ROOT/pages/sle_services.adoc b/modules/ROOT/pages/sle_services.adoc index 89b6fb1..5185809 100644 --- a/modules/ROOT/pages/sle_services.adoc +++ b/modules/ROOT/pages/sle_services.adoc @@ -38,7 +38,7 @@ presented in our xref:sle.adoc[SLE Documentation]. * MDApi https://apps.fedoraproject.org/mdapi/ * Monitor gating * Messaging bridges -* MirrorManager https://admin.fedoraproject.org/mirrormanager +* MirrorManager https://mirrormanager.fedoraproject.org/ * MySQL databases * Nagios https://nagios.fedoraproject.org * ODCS diff --git a/modules/release_guide/pages/philosophy.adoc b/modules/release_guide/pages/philosophy.adoc index 42bec54..a4be561 100644 --- a/modules/release_guide/pages/philosophy.adoc +++ b/modules/release_guide/pages/philosophy.adoc @@ -115,7 +115,7 @@ blocker bugs and bodhi to track additions and changes. Official parts of Fedora are eligible to be delivered to `/pub/fedora/` or `/pub/alt/releases` on https://dl.fedoraproject.org/pub/[Fedora Download] and to get -https://admin.fedoraproject.org/mirrormanager[mirrorlists] in +https://mirrormanager.fedoraproject.org[mirrorlists] in https://github.com/fedora-infra/mirrormanager2[mirrormanager]. These Distribution Platforms are maintained by Fedora Infrastructure and releng. This is not a feature of the product content itself or how it diff --git a/modules/sysadmin_guide/pages/mirrorhiding.adoc b/modules/sysadmin_guide/pages/mirrorhiding.adoc index 3e62c87..b073afa 100644 --- a/modules/sysadmin_guide/pages/mirrorhiding.adoc +++ b/modules/sysadmin_guide/pages/mirrorhiding.adoc @@ -22,7 +22,7 @@ Purpose:: To hide a public mirror, so it doesn't appear on the publiclist or the mirrorlist, simply go into the MirrorManager administrative web user -interface, at https://admin.fedoraproject.org/mirrormanager. Fedora +interface, at https://mirrormanager.fedoraproject.org. Fedora sysadmins can see all Sites and Hosts. For each Site and Host, there is a checkbox marked "private", which if set, will hide that Site (and all its Hosts), or just that single Host, such that it won't appear on the diff --git a/modules/sysadmin_guide/pages/mirrormanager.adoc b/modules/sysadmin_guide/pages/mirrormanager.adoc index afa3da3..2717571 100644 --- a/modules/sysadmin_guide/pages/mirrormanager.adoc +++ b/modules/sysadmin_guide/pages/mirrormanager.adoc @@ -51,10 +51,10 @@ an overview of existing mirrors. Additionally the frontend servers provide::: * an overview of the mirror list usage - https://admin.fedoraproject.org/mirrormanager/statistics + https://mirrormanager.fedoraproject.org/statistics * a propagation overview - https://admin.fedoraproject.org/mirrormanager/propgation - * a mirror map https://admin.fedoraproject.org/mirrormanager/maps + https://mirrormanager.fedoraproject.org/propgation + * a mirror map https://mirrormanager.fedoraproject.org/maps The _mm-frontend-checkin01_ server is only used for _report_mirror_ check-ins. This is used by mirrors to report their status independent of @@ -65,7 +65,7 @@ the crawlers. MirrorManager should automatically detect the new release version, and will create a new `Version()` object in the database. This is visible on the Version page in the web UI, and on -https://admin.fedoraproject.org/mirrormanager/. +https://mirrormanager.fedoraproject.org/. If the versioning scheme changes, it's possible this will fail. If so, contact the Mirror Wrangler. From 0606efe590887d17c653c664f3185580c3d5cfa9 Mon Sep 17 00:00:00 2001 From: Aurélien Bompard Date: Aug 06 2024 15:43:43 +0000 Subject: [PATCH 2/4] MirrorManager: update the move-to-archive command Signed-off-by: Aurélien Bompard --- diff --git a/modules/howtos/pages/archive-old-fedora.adoc b/modules/howtos/pages/archive-old-fedora.adoc index 553f444..f91de23 100644 --- a/modules/howtos/pages/archive-old-fedora.adoc +++ b/modules/howtos/pages/archive-old-fedora.adoc @@ -45,7 +45,7 @@ copy a file. These need to be dealt with individually. + Another error which happens when I script things is that you find out that everything got copied into -/pub/archive/fedora/linux/releases/21/21 even though it shouldn't. +/pub/archive/fedora/linux/releases/21/21 even though it shouldn't. . If the target directory already exists, then we need to do a recursive rsync to update any changes in the trees since the previous copy. @@ -97,14 +97,22 @@ $ rsync -avSAXHP --delete 21/ /pub/archive/fedora-secondary/updates/testing/21/ . Announce to the mirror list this has been done and that in 2 weeks you will move the old trees to archives. -. In two weeks, log into mm-backend01 and run the archive script +. In two weeks, run the following playbook from batcave: + [source] ---- -$ sudo -i ssh root@mm-backend01.iad2.fedoraproject.org -$ sudo -u mirrormanager mm2_move-to-archive --originalCategory="Fedora Linux" --archiveCategory="Fedora Archive" --directoryRe='/21/Everything' +$ rbac-playbook -v /srv/web/infra/ansible/playbooks/manual/mirrormanager/move-to-archive.yml --extra-vars="product='Fedora' version='38'" ---- +. This will start a job named `move-to-archive-fedora-38` on Openshift. +The playbook will output two commands to view the job's logs and to cancel it if needed. +You can run those commands on os-control01 or on any host logged in to OpenShift. +Alternatively, you can follow the job's execution on the Openshift WebUI and cancel it from there. +The playbook will wait for the job to complete. + +. Note that the `Fedora Codecs` will not be archived, because we currently don't do that. +This category is explicitely excluded in the MirrorManager command itself. + . If there are problems, the postgres DB may have issues and so you need to get a DBA to update the backend to fix items. @@ -129,15 +137,15 @@ $ ln ../20/README . $ cd /pub/fedora-secondary/releases $ cd 21 -$ ls # make sure you have stuff here +$ ls # make sure you have stuff here $ rm -rf * $ ln ../20/README . $ cd ../../updates/21 -$ ls # make sure you have stuff here +$ ls # make sure you have stuff here $ rm -rf * $ ln ../20/README . $ cd ../testing/21 -$ ls # make sure you have stuff here +$ ls # make sure you have stuff here $ rm -rf * $ ln ../20/README . ---- diff --git a/modules/release_guide/pages/release_eol.adoc b/modules/release_guide/pages/release_eol.adoc index 266fbd3..a309374 100644 --- a/modules/release_guide/pages/release_eol.adoc +++ b/modules/release_guide/pages/release_eol.adoc @@ -191,12 +191,12 @@ $ rsync -avAXSHP {old_release}/ /pub/archive/fedora-secondary/updates/testing/{o ____ . Announce to the mirror list this has been done and that in 2 weeks you will move the old trees to archives. -. In two weeks, log into mm-backend01 and run the archive script +. In two weeks, run the following playbook from batcave: + ____ [source,subs="attributes+"] .... -$ sudo -u mirrormanager mm2_move-to-archive --originalCategory="Fedora Linux" --archiveCategory="Fedora Archive" --directoryRe='/{old_release}/Everything' +$ rbac-playbook -v /srv/web/infra/ansible/playbooks/manual/mirrormanager/move-to-archive.yml --extra-vars="product='Fedora' version='{old_release}'" .... ____ . If there are problems, the postgres DB may have issues and so you need diff --git a/modules/sysadmin_guide/pages/mirrormanager.adoc b/modules/sysadmin_guide/pages/mirrormanager.adoc index 2717571..803bf59 100644 --- a/modules/sysadmin_guide/pages/mirrormanager.adoc +++ b/modules/sysadmin_guide/pages/mirrormanager.adoc @@ -74,11 +74,11 @@ contact the Mirror Wrangler. Once the files of an EOL release have been copied to the archive directory tree and enough mirrors have picked the files up at the -archive location there is also a script to adapt those paths in +archive location there is also a playbook to adapt those paths in MirrorManager's database: .... -sudo -u mirrormanager mm2_move-to-archive --originalCategory='Fedora EPEL' --directoryRe='/4/' +$ rbac-playbook -v /srv/web/infra/ansible/playbooks/manual/mirrormanager/move-to-archive.yml --extra-vars="product='EPEL' version='7'" .... == mirrorlist containers and mirrorlist servers From 042c926241f5601526ae27666178821c9523f136 Mon Sep 17 00:00:00 2001 From: Aurélien Bompard Date: Aug 06 2024 15:43:43 +0000 Subject: [PATCH 3/4] MirrorManager: update the move-devel-to-release command Signed-off-by: Aurélien Bompard --- diff --git a/modules/release_guide/pages/final_release.adoc b/modules/release_guide/pages/final_release.adoc index a2a8a0c..3382ed7 100644 --- a/modules/release_guide/pages/final_release.adoc +++ b/modules/release_guide/pages/final_release.adoc @@ -141,12 +141,11 @@ In the first week after the release MirrorManager still uses the files at `fedora/linux/development/` and not at `fedora/linux/releases/` -Once enough mirrors have picked up the files in the release directory -following script (on _mm-backend01_) can be used to change the paths in +Once enough mirrors have picked up the files in the release directory, +run the following playbook from batcave to change the paths in MirrorManager: [source,subs="attributes+"] .... -sudo -u mirrormanager mm2_move-devel-to-release --version={branched} --category="Fedora Linux" -sudo -u mirrormanager mm2_move-devel-to-release --version={branched} --category="Fedora Secondary Arches" +$ rbac-playbook -v /srv/web/infra/ansible/playbooks/manual/mirrormanager/move-devel-to-release.yml --extra-vars="version='42'" .... diff --git a/modules/release_guide/pages/release_post_cleanup.adoc b/modules/release_guide/pages/release_post_cleanup.adoc index cb13476..e41a2ce 100644 --- a/modules/release_guide/pages/release_post_cleanup.adoc +++ b/modules/release_guide/pages/release_post_cleanup.adoc @@ -35,13 +35,11 @@ After the release, update MirrorManager to redirect requests from the developmen [discrete] === Steps -. SSH into `mm-backend01.iad2.fedoraproject.org`. -. Run the following commands to update the paths in MirrorManager: +. Run the following playbook on batcave to update the paths in MirrorManager: + [source,subs="attributes+"] .... -sudo -u mirrormanager mm2_move-devel-to-release --version={branched} --category="Fedora Linux" -sudo -u mirrormanager mm2_move-devel-to-release --version={branched} --category="Fedora Secondary Arches" +$ rbac-playbook -v /srv/web/infra/ansible/playbooks/manual/mirrormanager/move-devel-to-release.yml --extra-vars="version='{branched}'" .... == Cleanup of RCs and Unnecessary Composes diff --git a/modules/sysadmin_guide/pages/fedora-releases.adoc b/modules/sysadmin_guide/pages/fedora-releases.adoc index 6e5e1f1..5bb3166 100644 --- a/modules/sysadmin_guide/pages/fedora-releases.adoc +++ b/modules/sysadmin_guide/pages/fedora-releases.adoc @@ -360,5 +360,5 @@ After release: * post to infrastructure list that freeze is over. * Move MirrorManager repository tags from the `development/$version/` Directory objects, to the `releases/$version/` Directory objects. This is -done using the `move-devel-to-release --version=$version` command on -_bapp02_. This is usually done now a week or two after release. +done using the `playbooks/manual/mirrormanager/move-devel-to-release.yml` +playbook. This is usually done now a week or two after release. From e89773e48ba6b2ea5f2c36d5aa87070f47be6c7a Mon Sep 17 00:00:00 2001 From: Aurélien Bompard Date: Aug 06 2024 15:43:43 +0000 Subject: [PATCH 4/4] MirrorManager: update the rest of the docs Signed-off-by: Aurélien Bompard --- diff --git a/modules/sysadmin_guide/pages/fedorawebsites.adoc b/modules/sysadmin_guide/pages/fedorawebsites.adoc index 1ff611f..5759024 100644 --- a/modules/sysadmin_guide/pages/fedorawebsites.adoc +++ b/modules/sysadmin_guide/pages/fedorawebsites.adoc @@ -110,7 +110,7 @@ and checkout to the new branch: all sidebars, also in labs, spins, and arm. . Check all Download links and paths in `$\{PRODUCT}/prerelease/index.html`. You can find all paths in _bapp01_ -(`sudo su - mirrormanager first`) or you can look at the downlaod page +or you can look at the downlaod page http://dl.fedoraproject.org/pub/alt/stage . Add CHECKSUM files to `static/checksums` and verify that the paths are correct. The files should be in _sundries01_ and you can query them with: @@ -170,7 +170,7 @@ all sidebars, also in https://spins.fedoraproject.org, https://labs.fedoraproject.org/ and https://arm.fedoraproject.org. . Check all Download links and paths in `$\{PRODUCT}/prerelease/index.html` You can find all paths in _bapp01_ -(`sudo su - mirrormanager first`) or you can look at the downlaod page +or you can look at the downlaod page https://dl.fedoraproject.org/pub/alt/stage . Add CHECKSUM files to static/checksums and verify that the paths are correct. The files should be in sundries and you can query them with: @@ -220,8 +220,8 @@ and checkout to the new branch: `fedora-web git:(master) git checkout -t -b f22 origin/f22` . Update the global variables Change curr_state for all arches . Check all Download links and paths in `$\{PRODUCT}/download/index.html` -You can find all paths in _bapp01_ (`sudo su - mirrormanager first`) or you -can look at the downlaod page http://dl.fedoraproject.org/pub/alt/stage +You can find all paths in _bapp01_ or you can look at the downlaod page +http://dl.fedoraproject.org/pub/alt/stage . Add CHECKSUM files to `static/checksums` and verify that the paths are correct. The files should be in _sundries01_ and you can query them with: + diff --git a/modules/sysadmin_guide/pages/mirrormanager.adoc b/modules/sysadmin_guide/pages/mirrormanager.adoc index 803bf59..d60ee9c 100644 --- a/modules/sysadmin_guide/pages/mirrormanager.adoc +++ b/modules/sysadmin_guide/pages/mirrormanager.adoc @@ -9,8 +9,7 @@ Owner:: Contact:: #fedora-admin, sysadmin-main, sysadmin-web Servers:: - mm-frontend01, mm-frontend02, mm-frontend-checkin01, mm-backend01, - mm-crawler01, mm-crawler02 + Hosted in OpenShift Mirrorlist Servers:: Docker container on the proxy servers Purpose:: @@ -22,43 +21,42 @@ MirrorManager handles our mirroring system. It keeps track of lists of valid mirrors and handles handing out metalink URLs to end users to download packages from. -The backend server (_mm-backend01_) scans the master mirror (NFS mounted -at `/srv`) using the _mm2_update-master-directory-list_ script (_umdl_) -for changes. Changed directories are detected by comparing the ctime to -the value in the database. +Everything runs in OpenShift. There is a cron job to scan the master mirror +(NFS mounted at `/srv`) using the _mm2_update-master-directory-list_ script +(_umdl_) for changes. Changed directories are detected by comparing the ctime +to the value in the database. -The two crawlers (_mm-crawler01_ and _mm-crawler02_) compare the content -on the mirrors with the results from _umdl_ using RSYNC, HTTP, HTTPS. -The crawler process on _mm-crawler01_ starts at 0:00 and 12:00 and at -2:00 and 14:00 on _mm-crawler02_. +There are also jobs to compare the content on the mirrors with the results +from _umdl_ using RSYNC, HTTP, HTTPS. +The crawler schedule can be viewed in the `vars/apps/mirrormanager.yml` file +in Ansible. If the content on the mirrors is the same as on the master those mirrors are included in the dynamic metalink/mirrorlist. -Every hour the backend server generates a python pickle which contains -the information about the state of each mirror. This pickle file is used -by the mirrorlist containers on the proxy servers to dynamically -generate the metalink/mirrorlist for each client individually. +A hourly job generates a binary file which contains the information about the +state of each mirror. This file is used by the mirrorlist containers on the +proxy servers to dynamically generate the metalink/mirrorlist for each client +individually. -The frontend servers (_mm-frontend01_ and _mm-frontend02_) offer an -interface to manipulate the mirrors. Each mirror-admin can only change -the details of the associated mirror. Members of the FAS group -_sysadmin-web_ can seen and change all existing mirrors. +The `frontend` deployment runs the web interface to manipulate the mirrors. +Each mirror-admin can only change the details of the associated mirror. +Members of the FAS group _sysadmin-web_ can seen and change all existing +mirrors. -The mirrorlist provided by the frontend servers has no actively consumed +The mirrorlist provided by the frontend has no actively consumed content and is therefore heavily cached (12h). It is only used to give an overview of existing mirrors. -Additionally the frontend servers provide::: +Additionally the frontend provides::: * an overview of the mirror list usage https://mirrormanager.fedoraproject.org/statistics * a propagation overview https://mirrormanager.fedoraproject.org/propgation * a mirror map https://mirrormanager.fedoraproject.org/maps -The _mm-frontend-checkin01_ server is only used for _report_mirror_ -check-ins. This is used by mirrors to report their status independent of -the crawlers. +The frontend is also used for _report_mirror_ check-ins. This is used by +mirrors to report their status independent of the crawlers. == Release Preparation @@ -83,7 +81,7 @@ $ rbac-playbook -v /srv/web/infra/ansible/playbooks/manual/mirrormanager/move-to == mirrorlist containers and mirrorlist servers -Every hour at :55 after the hour, mm-backend01 generates a pkl file with +Every hour at :55 after the hour, a job generates a binary file with all the current mirrormanager information in it and syncs it to proxies and mirrorlist-servers. Each proxy accepts requests to _mirrors.fedoraproject.org_ on apache, then uses haproxy to determine what @@ -106,19 +104,30 @@ the host proxy server. === Regenerating the Publiclist -On _mm-backend01_: +On _os-control01_: .... -sudo -u mirrormanager /usr/bin/mm2_update-mirrorlist-server -sudo -u mirrormanager /usr/local/bin/sync_pkl_to_mirrorlists.sh +oc -n mirrormanager create job --from=cj/update-mirrorlist-cache update-mirrorlist-cache-manual .... -Those two commands generates a new mirrorlist pickle and transfers it to +This command generates a new mirrorlist file and transfers it to the proxies. The mirrorlist containers on the proxies are restarted 15 minutes after each full hour. -The mirrorlist generation can take up to 20 minutes. If a faster -solution is required the mirrorlist pickle from the previous run is +The mirrorlist generation can take up to 20 minutes. The logs can be viewed +with: + +.... +oc -n mirrormanager logs -f job/update-mirrorlist-cache-manual +.... + +Once done, the job should be deleted from openshift with: + +.... +oc -n mirrormanager delete job/update-mirrorlist-cache-manual +.... + +If a faster solution is required the mirrorlist file from the previous run is available at: ....