#8864 Request to become tier 1 mirror
Closed: Fixed by kevin. Opened by survient.

Describe what you would like us to do:


Hello. I am a systems engineer at Rackspace who helps maintain our mirror server infrastructure along with fobriste which includes hosting a tier 2 Fedora mirror. We have done some work to improve the resiliency of our mirror servers and we would now like to request to become a Tier 1 mirror. We have 6 environments globally that include multiple web heads per region all registered to MirrorManager with 1 gig links per environment. These locations include:

US
Chicago, Illinois
Dulles, Virginia
Dallas, Texas

UK
London

Hong Kong
Tsuen Wan

Australia
Sydney

We currently pull content from one Tier 1 upstream location and replicate the content globally on our own networks. This means that if we are set up as a Tier 1 mirror we would only be pulling from the Tier 0 master download server once and then replicating on our own networks to 6 different public mirror locations.

When do you need this to be done by? (YYYY/MM/DD)

If your team decides to move forward with the request we would be looking to get set up as soon as your team would be willing.


We don't have any objection to this I don't think. Did you need it before the F32 release (tomorrow), or can later in the week be fine?

Also, we want to make sure and list requirements for this and such on the wiki. I think @adrian was going to work on/do that?

Metadata Update from @kevin:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: groomed, low-trouble, medium-gain

Appreciated Kevin. Later in the week is fine for us.

Our information as listed in Mirror Manager should be accurate if you need a point of reference. If listed as a static mirror for rsync we generally recommend to be listed as just mirror.rackspace.com since it is globally load balanced. Our contact is mirroradmin at rackspace.com. We are currently mirroring fedora-enchilada and fedora-epel.

I see that some mirrors are using ACLs from the MirrorManager database, if there are some docs on that I'd be happy to look into whether we can set that up.

Also apologies for the giant text, I can't quite figure out how to change the formatting.

We are currently mirroring fedora-enchilada and fedora-epel.

The biggest change I would like to make for tier 1 mirrors is to require mirroring and providing of fedora-buffet to make sure quick-fedora-mirror can be user by tier 2 mirrors.

I see that some mirrors are using ACLs from the MirrorManager database, if there are some docs on that I'd be happy to look into whether we can set that up.

Ignore that. That does no longer exist and I will remove the comments from the wiki.

I'm am looking into setting us up to mirror all of fedora-buffet. Our current size with fedora-enchilada is 3.2 TB and fedora-epel is 329 GB and if buffet only brings us to 3.8 TB(using the mailing list reply as reference) that would be fine. That said a difference of 271 gigs to cover fedora-archive, fedora-secondary, and fedora-alt doesn't seem right, especially when pulling from the DIRECTORY_SIZES.txt:

5.7T /pub/alt/
7.9T /pub/archive/fedora
6.8T /pub/archive/fedora-secondary
3.1T /pub/fedora/
368G /pub/epel/

due to the format of DIRECTORY_SIZES.txt for fedora-secondary I had to do an rsync dry run against a t1 mirror to get the following size result:

rsync --info=stats2 -h -r rsync.kernel.org::fedora-buffet/fedora-secondary/ | tail -n 1
total size is 2.75T speedup is 38,329.09

Added up with the current content that is roughly 26.62 TB of data. Obviously hardlinks will take a decent chunk out of that but for it to condense the footprint down to only 3.8 TB doesn't feel right. I'm hoping I'm just missing something here.

adrian did you have any further thoughts on this request? It feels like the discussions on the mirror list are at a bit of an impasse. While we'd love to mirror all of fedora-buffet without a second thought at 38TB it is over twice as large as the total amount of data across all projects we're currently mirroring which would be a bit difficult to maintain.

Thanks for the continued assistance and patience.

Sorry this dropped off the radar.

I think requiring everything is too much... could we say that archive is optional there? I don't know how many archive mirrors we have, but it's not many.

Thoughts?

@adrian any thoughts here?

Sorry. Let's put them on the tier 1 ACL. Mirroring fedora-buffet for tier 1 mirrors sounded like a good idea, but the amount of data we have is just to big to required that of a tier 1 mirror.

Thanks folks. Once you've let us know that we've been added to the ACL I'll update our sync configuration accordingly. We are syncing from the following two IPs:

166.78.229.128 (new infrastructure)
72.4.120.222 (legacy infrastructure)

If only one IP can be allowed then please allow 166.78.229.128. We will only ever be pulling from one of these sources at a time however we are in the middle of migrating our automation from one management server to another so eventually we will cut over to the new infrastructure.

Thanks for the assistance and please feel free to reach out if any further information is needed.

we can add both, but do those have dns names? If we add them by dns names you can adjust the ip's yourself later if you need to...

we can also just do IP's if you prefer.

Unfortunately we do not have DNS entries for those IPs. We would prefer if you can just do the IPs for now. Once our legacy infrastructure is no longer in use I will open a new issue to have the ACL removed for posterity. Appreciate the help.

ok. Added!

Thanks again for mirroring, and let us know if you run into any problems!

Metadata Update from @kevin:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

Metadata