I think this is a reasonable request if it is how the results are stored now. At the start, I thought we would only have used one instance for storing all the results.

We also thought this at the start but realized that since the data will be coming from two data-centers, to lower the risks of losing data between the two it is better to have two instances of resultsdb.

We first thought that we could try to keep in sync these two instances but realized that the risk of inconsistent data between the two instances would make things more confusing than saying: these tests results are stored there and these ones there.

How hard do you think it would be to adjust GW for this? And how long do you think that would take?

Inside Red Hat we have a similar problem (many different systems all over the place producing test results) and we saw ResultsDB as a way to collect all the results in one place so that consumers don't have to worry about looking up results in multiple places. We have the resultsdb-updater service, which listens for incoming results on a message bus, so that the testing systems don't have to worry about synchronously pushing their results over the network (across data centers, etc).

So it feels like it would be a big step backward here, if we end up with Fedora ResultsDB and a totally separate one in CentOS.

Since we have the two different testing systems (Taskotron and CentOS Jenkins) I guess we do want something to aggregate the results from both. It sounds like you want the point of aggregation to be Greenwave. But there could well be other tools which want to consume these results as well, now they would have to do the aggregation themselves too? I think it would make more sense to make ResultsDB be the aggregation point instead -- maybe with a service to send the results from CentOS to Fedora, if you want to avoid synchronous calls across the data centers.

We could even look at repurposing resultsdb-updater to do that.

Another thing to consider here, we would like to eventually make Greenwave be message-bus-driven, so that it can update its decisions as soon as new results appear. How would that work in this scenario? Is there (or will there be) also a fedmsg hub in CentOS infrastructure? Or would Greenwave need to actually listen on two separate message buses, one in Fedora and one in CentOS?

The messages sent by both the CI pipeline and the resultsdb instance hosted in RDU are going to go to the Fedora message bus (but listening to more than one bus is actually trivial in fedmsg).
So being message-based is still possible.
(CI messages are currently being sent to the staging bus if you want to check them out)

The reason why we choose to deploy another resultsdb instance is to account for potential message loss. The closer to the source the less risk there is to loose a message. So having an instance in RDU made sense in this regards.
We then thought we could store the data in the RDU resultsdb instance as well as the PHX one, except that in the case of message loss how do you cope with it?
We could try syncing the two resultsdb instance but that still means there a time where the two instances are inconsistent and confusing.
Having partial information in one instance and full information in the other instance is going to make greenwave's job much harder.

It is crucial to try to minimize the risk of message loss as we are going to gate updates based on them. So if the connection between the two data-centers is dropped for an hour or more, all messages sent by the CI pipeline in the mean time will be lost forever and the corresponding updates will be stuck in bodhi, unless we have a local storage for these message. That's why the went for the RDU resultsdb instance.
We could of course re-trigger the tests and I think we will likely need a mechanism for this but having a storage of these messages is also helpful.

We are consolidating on resultsdb so greenwave has one interface to query for information even if it needs to query it at different places.

As discussed in #47, we're leaning towards dropping this issue after discussing it in a bluejeans session this morning.