fedora-infrastructure

#12948 Anubis on kojipkgs is breaking Incus/LXC image builds
Closed: Fixed by kevin. Opened by stgraber.

Closed: Fixed
Reopen Issue

Hello,

I'm the project leader of linuxcontainers.org.
One of the things we do are daily builds of various distributions, made available at https://images.linuxcontainers.org which are then picked up by LXC, Incus, TrueNAS and a variety of other platforms.

Starting around November 19th, we've been unable to make any more builds: https://jenkins.linuxcontainers.org/job/image-fedora/
Looking into it, it looks like this coincides with Anubis being enabled on that domain, effectively preventing our tooling from downloading the files it needs.

Is there an alternative location we should be using to retrieve those artifacts?
We're specifically looking at the daily builds of
https://kojipkgs.fedoraproject.org/packages/Fedora-Container-Base-Generic/

If not, is there a way for you to put an exception in your Anubis setup so that we don't get hit by it?

All our builders are currently covered by one of those (Zabbly AS399760):
- 2602:fc62:a::/48
- 2602:fc62:b::/48
- 45.45.148.0/24

Our system fully expires any image that's gone more than 10 days without a refresh.
Given the timeline above, we're looking at the Fedora images disappearing any time now :(

Thanks!

Stéphane

For rawhide you can get these from:

https://dl.fedoraproject.org/pub/fedora/linux/development/rawhide/Container/

(thats the master mirrors, but any fedora mirror that carries rawhide should have those)

For GA containers: https://dl.fedoraproject.org/pub/fedora/linux/releases/43/Container/ for example.

Or if you can pull from registries: https://quay.io/repository/fedora/fedora or registry.fedoraproject.org/fedora:43 (or whatever)

As for the anubis challenge... Can you make sure you are setting user-agent and accept headers? The latest anubis seems to challenge things that don't have those headers set at all. :(
If that doesn't work at all, we can look at allowlisting you... but it would be better if we didn't have to.

Thanks for the reply.

The GA artifacts are quite old so we'd definitely prefer being able to pull the dailies instead, but it's good to know that we have those around as a fallback at least.

The OCI images pulled from a registry could be a viable workaround since we actually had a community member fix our ability to pull base images from OCI registries just a couple of weeks ago.

I'll look at tweaking the headers we're using to access kojipkgs as I can see both Firefox and curl here not getting challenged, but our tooling (distrobuilder) getting the challenge. We're using the default Go HTTP client in this case, so the User-Agent is definitely being set, but it's set to something that may trigger Anubis. I'll try to make it look like curl and see if that helps :)

Let me know if that doesn't end up working out. We can allowlist, but I'd prefer not to have to maintain that longer term (then we would have to update anytime you changed, etc).

Looks like it worked!

We've landed this change in distrobuilder: https://github.com/lxc/distrobuilder/pull/954

After which we rebuilt all our worker images and kicked off new builds. They've managed to make it pass Anubis so we should be good to go.

Thanks!

Metadata Update from @kevin:
- Issue assigned to kevin
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: low-trouble, medium-gain

Excellent.

Sorry for the trouble... and I wish the ai scrapers wouldn't be making us do all this. ;(

let us know if there's anything more we can adjust from our end.

Metadata Update from @kevin:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)
Metadata

low-trouble medium-gain

None
None
Normal
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.