Hi releng folks!

So, I've been talking with @lruzicka lately about automating the checksum test case. He was looking at doing it in openQA, but I think that's not the best way - it doesn't really need a big full-fat test system, and we are supposed to check every single official image (not just release-blocking ones), and transferring all that data to the openQA server for testing is a waste (and might actually cause disk space issues).

IIRC, at some point we had a setup where @robatino had shell access to the compose host or some other VM which has the images available on local storage, and he would run a bash script to check this and then manually file the results. I think the best way to go would be to follow that basic process but automate it a bit more.

I'm thinking we could have a fedora-messaging consumer which listens out for pungi.compose.status.change messages, and when it sees a new compose become available, runs a script to validate that it has correct checksums for all images (this part isn't hard, we have the bones of it written in the test case). It could then use wikitcms to report the result to the wiki. This consumer would need to run on the compose host or another VM with the compose available on local storage, and that box would need to have a special permanent token for wiki login (like the openQA box does, for reporting the openQA results to the wiki).

There is a slight timing issue here, in that creation of the wiki validation pages is also done by a consumer that listens for pungi.compose.status.change. But I think the race would usually work out, because creating the pages will probably happen faster than all the checksum would run, so by the time the checks were done, the pages would be ready. If we have issues with that we could try something dumb like having the checksum consumer sleep a bit before reporting the result, or something smarter like I have to make the wiki event creator publish messages and we have the test consumer wait for that.

We don't create a wiki event for every compose, but this isn't really a problem - when there isn't an event, wiki reporting will just sensibly not do anything. This is a case we already handle all the time with openQA (openQA just tries to report every compose test result to the wiki, it doesn't bother checking if there's an event) and it doesn't cause any issues.