So, this is what I think happened.

There was a last f34 push on the 6th as f34 went EOL on the 7th.

It failed.
Changes for EOL were pushed out and the sync script was updated to not sync f34 anymore.
It was resumed and finished, but never synced anywhere.

So, we could just say 'too bad'.
We could sync it to archives.
Something else?

I think there's something else going on here. The rsync update I linked to went stable on the 3rd and was first included in an update then:

https://kojipkgs.fedoraproject.org/compose/updates/Fedora-34-updates-20220603.0/compose/Everything/aarch64/os/Packages/r/rsync-3.2.4-1.fc34.aarch64.rpm

Going from the timestamp dates, the F34 stuff in archives was sync'd from /pub/fedora/linux/updates/34 with whatever was in the directories on May 12 2022

drwxr-xr-x. 30 263 263 4096 May 12  2022 /srv/pub/archive/fedora/linux/updates/34/Everything/x86_64/Packages

The build date for the rsync in updates is

Name        : rsync
Version     : 3.2.3
Release     : 6.fc34
Build Date  : Thu 14 Apr 2022 08:56:17 UTC

The latest update I find in the 34 archives is electron-cash with Build Date : Wed 11 May 2022 19:25:01 UTC

All I can find shows it synced those composes...

Date: Tue, 7 Jun 2022 02:50:52 +0000 (UTC)

...
Syncing f34 updates from /mnt/koji/compose/updates/Fedora-34-updates-20220606.0 -> /mnt/koji/compose/updates/Fedora-34-updates-20220607.0
Error sending fedora-messaging message: [Errno 13] Permission denied
Synced f34 updates to /mnt/koji/compose/updates/Fedora-34-updates-20220607.0

Hmm. But is that just syncing to the prod location (master mirror).. That location is no longer available right?

What copies them to https://archives.fedoraproject.org/pub/archive/fedora/linux/updates/34/ ?

Does that happen at EOL as a one off process or is it automated?

It happens usually months after EOL and is done by some infra or releng person.

It's a pretty simple sync, so I am not sure what could have gone wrong there.

I wonder if we were to repeat the EOL SOP steps that copy these files in place (maybe with a --dry-run) what would we see?

We cannot. The orig source dirs have been deleted.

it's rsyncing the existing content to the archive location and removing the old orig location. It's a destructive operation. We now have no way to know what used to be there (aside from that it should be an exact copy because it was rsynced).

I think it's much more llikely something failed syncing from compose to dest than dest to archive.

OK. IIUC you are saying that the only way something ends up in archive is if it was ever in $dest.. i.e. nothing ever gets copied from $compose to $archive.

That just doesn't add up with my experience (again, I'm focusing in on the version of rsync). I have a container build log from May 5th of this year where:

[2023-05-05T13:43:58.133Z]  rsync                                aarch64  3.2.4-1.fc34                         updates                393 k

If you are right then that would mean that newer version of rsync-3.2.4-1.fc34 would have never been available in updates, when clearly it was.

It would seem I am the cause of the missing data. In May I did a sync of various old releases which were EOL from /pub/fedora/linux to /pub/archive/fedora/linux. At the time I also copied over F34 directories.

Around this time I stopped doing this work, and the next person who did the 'move f34 to archive' saw that the updates directory was already there. This was arond 2023-01-17 when Fedora 35 was being archived. The updates were then removed and the only copies are now in koji. [I think I am doubly at fault, because I was asked about it and didn't do a check to see when the copy was done.]

This was a failing on my part and I will work with the releng team on whatever needs to be done to fix it.

Thanks @smooge!

I may be oversimplifying here, but maybe we can just copy the outputs that still exist at https://kojipkgs.fedoraproject.org/compose/updates/Fedora-34-updates-20220607.0/compose/Everything/ into place?

I do not know. I will work with @kevin and @humaton on what is needed next week. (They are both officially off til then).

Any update on this?

We have put in a workaround but are worried the repo location we are now using will get cleaned up at some point (which may mean the actual correct content may be lost when that happens).

Hi folks and @dustymabe

Thanks for your patience on this, today I synced the latest available for F34 from Koji for both update and update-testing to archive;

For reference, these should be the states for the update and update-testing

update
Everything => Fedora-34-updates-20220607.0
Modular => Fedora-Modular-34-updates-20220603.0

cf: https://archives.fedoraproject.org/pub/archive/fedora/linux/updates/34/

update-testing
Everything => Fedora-34-updates-testing-20220607.0
Modular => Fedora-Modular-34-updates-testing-20220527.0

cf: https://archives.fedoraproject.org/pub/archive/fedora/linux/updates/testing/34/

NOTE: These were the latest/last composes for 34 on koji

Please check if everything is according to the need, not sure if mirrormanager cron will take some time to pick it up but it should be no later than tomorrow. Once, I get a confirmation I'll mark this as Fixed. Thanks

I'm closing this ticket as "fixed" for now, if there is a discrepancy please feel free to ping us!

hey @jnsamyak - it looks like primary architectures are fixed now, but secondary architectures are not. Can you fix it for s390x and ppc64le?

For example with the current state:

• correct for x86_64 and aarch64:
  • https://archives.fedoraproject.org/pub/archive/fedora/linux/updates/34/Everything/x86_64/Packages/r/rsync-3.2.4-1.fc34.x86_64.rpm
  • https://archives.fedoraproject.org/pub/archive/fedora/linux/updates/34/Everything/aarch64/Packages/r/rsync-3.2.4-1.fc34.aarch64.rpm
• incorrect for ppc64le and s390x:
  • https://archives.fedoraproject.org/pub/archive/fedora-secondary/updates/34/Everything/ppc64le/Packages/r/rsync-3.2.3-6.fc34.ppc64le.rpm
  • https://archives.fedoraproject.org/pub/archive/fedora-secondary/updates/34/Everything/s390x/Packages/r/rsync-3.2.3-6.fc34.s390x.rpm

Thanks again for your patience on this, just now I synced the latest available for F34 (ppc64le and s390x) from Koji for both update and update-testing to archive;

For reference, these should be the states for the update and update-testing

update
Everything => Fedora-34-updates-20220607.0
Modular => Fedora-Modular-34-updates-20220603.0

cf: https://archives.fedoraproject.org/pub/archive/fedora-secondary/updates/34/

update-testing
Everything => Fedora-34-updates-testing-20220607.0
Modular => Fedora-Modular-34-updates-testing-20220527.0

cf: https://archives.fedoraproject.org/pub/archive/fedora-secondary/updates/testing/34/

Please let me know if you found any issues here, happy to help :D

I'm closing this ticket as "fixed" for now, if there is a discrepancy please feel free to ping us!

Thanks!