I blocked a lot of IPs on proxy01 and it still doesn't seem to help with it. I'm still trying to investigate what to block.

Thanks for taking care. I added an announcement in Discourse to make people aware and avoid further tickets to be opened

-> https://discussion.fedoraproject.org/t/for-your-information-ddos-affecting-most-of-the-fedoraproject-org-services/161568

Thank you for looking into this.

Ping if you need any extra eyes/hands to help on this.

So, things should all be back up now hopefully.

The issue was an odd one and I am still not 100% sure of the root cause yet.

Some things:

• both proxies were seeing packet loss/droppped packets/icmp

• requests to the registry would process normally for a bit then get 'stuck' sending back to the client. This would pile up connection slots in apache and fill it up so it couldn't process anything else.

I updated and rebooted both proxy01 and proxy10

and everything started to work much better.

There was a httpd update for CVE-2025-54090 but I looked at that last week and thought we were not affected by it (we don't use expr anywhere much, but perhaps the scope is wider).

I am also still seeing here a issue updating flatpaks, but I think it's some caching on my end or something. Others have seen updates/installs working.

Please let us know if you still see anything.

While performing flatpak update pulls from https://registry.fedoraproject.org/v2/f42 fail due to continual redirects in Silverblue 42. Is this related? Pulls from Flathub complete successfully.

:~$ flatpak update
Looking for updates…
        ID                                               Branch            Op           Remote           Download
 1. [✗] org.fedoraproject.Platform.GL.default            f42               u            fedora           < 260.3 MB
 2. [✗] org.fedoraproject.Platform                       f42               u            fedora           < 497.3 MB
Warning: While fetching https://registry.fedoraproject.org/v2/f42/mesa/blobs/sha256:99de218ae89fb6dcc7df0f6aeb290e1a68707562f4cb33fcc360c20f42d10704: [47] Number of redirects hit maximum amount
Error: While fetching https://registry.fedoraproject.org/v2/f42/flatpak-runtime/blobs/sha256:af95e496a19003f4288d988a6acc12cd6668b4a6405fc5cdf78711a4db2f6b02: [47] Number of redirects hit maximum amount
Updates complete.

EDIT: Disregard - it seems to be a caching issue on my end.

It was actually a caching issue on our end. ;) I just fixed it a few min ago...

I am going to go ahead and close this now as everything seems back to normal.

Still not fully sure if this was caused by a bug or a DDoS or a combo of the two.