#9094 Port pdc-updater to fedora-messaging
Closed: Fixed by pingou. Opened by mohanboddu.

Describe what you would like us to do:


pdc-updater is still using fedmsg and with the bridge not working since the move, pdc-updater is not doing its job. And one of them is updating pdc when there is a retirement commit. This caused the problem of those packages not being blocked in koji.

We should port it to fedora-messaging asap.

When do you need this to be done by? (YYYY/MM/DD)

ASAP


Where is the code base?

Metadata Update from @pingou:
- Issue tagged with: dev

https://github.com/fedora-infra/pdc-updater/

And based on quick search, the following files need to be updated (not including tests)
https://github.com/fedora-infra/pdc-updater/blob/master/pdcupdater/commands.py
https://github.com/fedora-infra/pdc-updater/blob/master/pdcupdater/handlers/init.py
https://github.com/fedora-infra/pdc-updater/blob/master/pdcupdater/consumer.py

Metadata Update from @smooge:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: groomed, high-gain, high-trouble

Metadata Update from @smooge:
- Issue assigned to pingou

@mohanboddu we're going to look at porting the logic of pdc-updater to toddlers.

Currently we see that different handlers are running on different pdc-backend hosts. Do you know the reason for this? (if there a load/permission issue?)

Thanks

Metadata Update from @pingou:
- Issue untagged with: groomed, high-gain, high-trouble
- Issue priority set to: Needs Review (was: Waiting on Assignee)

Metadata Update from @pingou:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: groomed, high-gain, high-trouble

@mohanboddu we're going to look at porting the logic of pdc-updater to toddlers.
Currently we see that different handlers are running on different pdc-backend hosts. Do you know the reason for this? (if there a load/permission issue?)
Thanks

I dont know this since it is setup by @ralph and I am just using it as is.

@churchyard pointed out that I have not updated this ticket with the latest info :)

So yesterday we deployed two toddlers with Mohan: pdc_retired_packages and packager_bugzilla_sync.

The first one is a port of the retirement handler from pdc-updater. It updates EOL dates in PDC upon packages having a dead.package file being added to their git repo. It is meant to be triggered by fedora-messaging messages from dist-git as well as by playtime*.

The second sync ACL from FAS to bugzilla on user accounts (basically syncs the fedorabugs groups from FAS to bugzilla). It is meant to be triggered by playtime* only.

  • playtime is a small scripts that is designed to be run by cron jobs and that will send simple fedora-message messages that toddlers listen and trigger on. So It's basically a way to trigger a toddler run at a specific day/time just as if we were running them in cron jobs (modulo the load of the queue at that time).

So all new packages that are retired should be (and I believe are) getting their EOL dates updated in PDC.

Not having deployed playtime yet, it has not run over the entire collection and thus it has likely not updated the packages that have been retired since pdc-updater stopped running.

playtime has been deployed and is set to trigger packager_bugzilla_sync every 6 hours and the pdc_retired_packages every month at 6:30 am on the 9th (so we can see how it behaves tomorrow).

There are a few more handlers from pdc-updater that we need to port so this ticket remains open.

So looking at the handlers set in pdc-updater, I see:

  • atomic.py which triggers of 'trac.git.receive'. Since trac is long gone, I think we don't have to port that handler

  • compose.py which triggers of 'pungi.compose.status.change'.

  • modules.py which triggers of mbs.module.state.change.

  • person.py which triggers of fas.user.create. Considering FAS is no longer sending fedmsg notification, we likely can skip this one.

  • pkgdb.py well pkgdb is long gone so we can skip this one.

  • retirement.py which was just ported as pdc_retired_packages.

  • rpms.py which trigers of buildsys.tag.

That means we still have 3 handlers to port to toddlers: compose.py, modules.py and rpms.py.

So looking at what is deployed and currently running: https://pagure.io/fedora-infra/ansible/blob/master/f/roles/pdc/backend/templates/pdcupdater.py

It looks like compose.py, modules.py, retirements.py (already a toddler) and depchain.containers.py are the ones running.

Which means we can ignore rpms.py until said otherwise.

compose.py is done in https://pagure.io/fedora-infra/toddlers/pull-request/18 (pending review)

@mohanboddu do you know if the depchain.containers.py handler is used by/for anything? It's a pretty complex one which won't be trivial to port :(

So I've been asking around on IRC who knew what this handler is doing and this is the closest I've come to:

cverna | pingou: the pdc-updater container thing might have been useful for freshmaker 
cverna | To automatically rebuild container based on the rpms update                   
cverna | I think it is safe to ignore                                                  

Which means, with the compose.py haven been ported and deployed, we only have modules.py on our plate, and @zlopez is working on this :)

modules.py is done in https://pagure.io/fedora-infra/toddlers/pull-request/22 (pending review)

https://pagure.io/fedora-infra/toddlers/pull-request/22 is now merged

The last toddler has been deployed.

I think we can close this ticket and we should also be able to retire some (all?) of the pdc-backend machines, I'll create the ticket for the pdc-backend machines.

Thanks for your help everyone!

Metadata Update from @pingou:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

Metadata