From 30b36c86e8a6dcf32e2b1e771a0d85d2078f8a59 Mon Sep 17 00:00:00 2001 From: James Antill Date: Sep 08 2025 19:14:14 +0000 Subject: [PATCH 1/2] Update the massupdate SOP. Signed-off-by: James Antill --- diff --git a/modules/sysadmin_guide/pages/massupgrade.adoc b/modules/sysadmin_guide/pages/massupgrade.adoc index 003972b..133ad63 100644 --- a/modules/sysadmin_guide/pages/massupgrade.adoc +++ b/modules/sysadmin_guide/pages/massupgrade.adoc @@ -27,8 +27,8 @@ various security and other upgrades. Owner::: Fedora Infrastructure Team Contact::: - #fedora-admin, sysadmin-main, infrastructure@lists.fedoraproject.org, - #fedora-noc + https://chat.fedoraproject.org/#/room/#admin:fedoraproject.org, sysadmin-main, https://docs.fedoraproject.org/en-US/infra/ + https://chat.fedoraproject.org/#/room/#noc:fedoraproject.org Location::: All over the world. Servers::: @@ -38,7 +38,7 @@ Purpose::: == Preparation -Mass updates are usually applied every few months or sooner if there's some +Mass updates are usually applied every few months, or sooner if there's some critical bugs fixed. Mass updates are done outside of freeze windows to avoid causing any problems for Fedora releases. @@ -46,7 +46,7 @@ The following items are all done before the actual mass update: * Plan a outage window or windows outside of a freeze. * File a outage ticket in the fedora-infrastructure tracker, using the outage -template. This should describe the exact time/date and what is included. +template. This should describe the exact date/time, the duration and what is included. * Get the outage ticket reviewed by someone else to confirm there's no mistakes in it. * Sent outage announcement to infrastructure and devel-announce lists (for @@ -56,11 +56,17 @@ and announce (for outages that affect all users). there for higher visibility. * Setup a hackmd or other shared document that lists all the virthosts and bare metal hosts that need rebooting and organize it per day. This is used -to track what admin is handling what server(s). +to coordinate which admin is handling which server(s). -Typically updates/reboots are done on all staging hosts on a monday, -then all non outage causing hosts on tuesday and then finally the outages -are on wednsday. +Typically updates/reboots are done in three seperate steps: + +* all staging hosts +* non outage causing production hosts +* outage causing production hosts + +It has been somewhat common to use Mon/Tues/Wed for these different steps, but as +long as the production outage is not on a Friday it is fine to fit it around +people's schedule. == Staging @@ -68,9 +74,12 @@ ____ Any updates that can be tested in staging or a pre-production environment should be tested there first. Including new kernels, updates to core database applications / libraries. Web applications, libraries, -etc. This is typically done a few days before the actual outage. +etc. This is typically done a couple of days before the actual outage, +or even the day before. Too far in advance and things may have changed again, so it's important to do this just before the production updates. +But for a significant update doing it the day before can make it difficult to +coordinate fixes for any non-trivial problems. ____ == Non outage causing hosts @@ -78,9 +87,9 @@ ____ Some hosts can be safely updated/rebooted without an outage because they either have multiple machines in a load balancer or are not visible to end users or other reasons. These updates are typically -done on tuesday of the outage week so they are done before the outage -on wed. These hosts include proxies and a number of virthosts that -have vm's that meet this criteria. +done outside the outage window, so they are done before the outage. +These hosts include proxies and a number of virthosts that +have VM's that meet this criteria. == Special Considerations @@ -121,6 +130,11 @@ to make sure all the clients are using one machine or the other, because if they are split routing between machines may be confused. +**NOTE**: Your fellow admins will likely be using bastion01 +to access batcave01 and run the update playbooks, so rebooting either +of these machines needs extreme coordination so that people aren't in +the middle of doing other things. + === batcave01 batcave01 is our ansible control host. It's where you run playbooks @@ -130,10 +144,15 @@ since it's rebooting it's own virthost. For this host you should go to the virthost and 'virsh shutdown' all the other vm's, then 'virsh shutdown' batcave01, then reboot the virthost manually. +**NOTE**: Your fellow admins will likely be using bastion01 +to access batcave01 and run the update playbooks, so rebooting either +of these machines needs extreme coordination so that people aren't in +the middle of doing other things. + === noc01 / dhcp server noc01 is our dhcp server. Unfortunately, when rebooting the vmhost that -contains noc01 vm, it means that that vmhost has no dhcp server to +contains noc01 VM, it means that that vmhost has no dhcp server to answer it when booting and trying to configure network to talk to the tang server. To work around this you can run a simple dhcpd on batcave01. Start it there and let the vmhost with noc01 come @@ -144,6 +163,11 @@ batcave01: 'systemctl start dhcpd' remember to stop it after the host comes back up. +=== COPR / OpenQA + +All of these hosts are generally updated outside of the "mass" update +so that the people who monitor those machines can be present. + === Special package management directives Sometimes we need to exclude something from being updated. @@ -164,29 +188,52 @@ come back up from reboot or aren't working right after reboot. It's important to avoid multiple people operating on a single machine in a read-write manner and interfering with changes. -Uusally for a mass update/reboot there will be a hackmd or similar +Usally for a mass update/reboot there will be a hackmd or similar document that tracks what machines have already been rebooted and who is working on which one. Please check with the leader for a link to this document. == Updates and Reboots via playbook -There's several playbooks related to this task: -vhost_update.yml applies updates to a vmhost and all it's guests -vhost_reboot.yml shuts down vm's and reboots a vmhost -vhost_update_reboot.yml does both of the above +People should mostly use the vhost_update_reboot.yml playbook which +runs both vhost_update.yml to apply updates and vhost_reboot.yml to +reboot the host and VM's. This can be called via. rbac-playbook so +that non-sysadmin-main people can help. For hosts out of outage you probably want to use these to make sure -updates are applied before reboots. Once updates are applied globally -before the outage you will want to just use the reboot playbook. +updates are applied before reboots (apply updates can take a lot of +time, esp. when not done in parallel). However once updates are +applied globally before the outage you will still want to use the +update_reboot playbook (but the update part should be very fast). + +By far the most common problem we have is that machines don't come +back after a reboot. This is usually a firmware booting problem, or +a luks problem. Both can be solved by logging into +the console for the vmhost and seeing what the error is and fixing +it manually. + +To monitor a machine you are update/rebooting in another window +(also from batcave01) you can run: `mtr --displaymode 1 -i 4 ` + +To get console access run (without the -stg if not in staging) from **noc01**: + +* `ipmitool -U admin -H -stg.mgmt.rdu3.fedoraproject.org -I lanplus shell` + +you'll need the passwds if you are doing this bit. (and then run `sol activate`). +If `ipmitool` doesn't work you can try: + +* `sshuttle 172.23.1.0/24 -r noc-cc01.rdu-cc.fedoraproject.org -v` + +== Checking hosts have updated + +Additionally you should use the updates-uptime-cmd.py python script +on batcave01 to see what machines have updates available and/or +need to be rebooted. You'll need to run the update sub-command, +before viewing the usual information, to get the latest data. -Additionally there are two more playbooks to check things: -check-for-nonvirt-updates.yml -check-for-updates.yml -See those playbooks for more information, but basically they allow -you to see how many updates are pending on all the virthosts/bare -metal machines and/or all machines. This is good to run at the end -of outages to confirm that everything was updated. +There are older playbooks check-for-nonvirt-updates.yml and +check-for-updates.yml, but the above script should be easier to +use and give clearer results. == Doing the upgrade @@ -195,7 +242,7 @@ If possible, system upgrades should be done in advance of the reboot make sure that the Infrastructure RHEL repo is updated as necessary to pull in the new packages (xref:infra-repo.adoc[Infrastructure Yum Repo SOP]) -Before outage, ansible can be use to just apply all updates to hosts or +Before outage, ansible can be used to just apply all updates to hosts or apply all updates to staging hosts before those are done. Something like: ansible -m shell 'yum clean all; yum update -y; rkhunter --propupd' hostlist @@ -209,7 +256,7 @@ ansible -m shell 'yum clean all; yum update -y; rkhunter --propupd' hostlist passphrases for encrypted volumes) . Consider running check-for-updates or check-for-nonvirt-updates to confirm that all hosts are updated. -. close fedorastatus outage +. Close fedorastatus outage . Close outage ticket. === Non virthost reboots From 79a7921ac3c290b5081319bb53f4ca3e950ba56d Mon Sep 17 00:00:00 2001 From: James Antill Date: Sep 09 2025 16:18:00 +0000 Subject: [PATCH 2/2] Fixes for massupdate SOP, from comments. Signed-off-by: James Antill --- diff --git a/modules/sysadmin_guide/pages/massupgrade.adoc b/modules/sysadmin_guide/pages/massupgrade.adoc index 133ad63..fcb9904 100644 --- a/modules/sysadmin_guide/pages/massupgrade.adoc +++ b/modules/sysadmin_guide/pages/massupgrade.adoc @@ -27,8 +27,9 @@ various security and other upgrades. Owner::: Fedora Infrastructure Team Contact::: - https://chat.fedoraproject.org/#/room/#admin:fedoraproject.org, sysadmin-main, https://docs.fedoraproject.org/en-US/infra/ - https://chat.fedoraproject.org/#/room/#noc:fedoraproject.org + https://matrix.to/#/#admin:fedoraproject.org?web-instance[element.io]=chat.fedoraproject.org[Fedora Infra. channel on Matrix] + https://matrix.to/#/#noc:fedoraproject.org?web-instance[element.io]=chat.fedoraproject.org[Fedora NOC channel on Matrix] + sysadmin-main, https://docs.fedoraproject.org/en-US/infra/ Location::: All over the world. Servers::: @@ -198,7 +199,7 @@ for a link to this document. People should mostly use the vhost_update_reboot.yml playbook which runs both vhost_update.yml to apply updates and vhost_reboot.yml to reboot the host and VM's. This can be called via. rbac-playbook so -that non-sysadmin-main people can help. +that non-sysadmin-main people can help (need sysadmin-updates). For hosts out of outage you probably want to use these to make sure updates are applied before reboots (apply updates can take a lot of @@ -215,14 +216,8 @@ it manually. To monitor a machine you are update/rebooting in another window (also from batcave01) you can run: `mtr --displaymode 1 -i 4 ` -To get console access run (without the -stg if not in staging) from **noc01**: - -* `ipmitool -U admin -H -stg.mgmt.rdu3.fedoraproject.org -I lanplus shell` - -you'll need the passwds if you are doing this bit. (and then run `sol activate`). -If `ipmitool` doesn't work you can try: - -* `sshuttle 172.23.1.0/24 -r noc-cc01.rdu-cc.fedoraproject.org -v` +Also read the https://docs.fedoraproject.org/en-US/infra/howtos/restart_datacenter_server/[howto restart a server] +docs. == Checking hosts have updated