I guess disk size vars are in private repo right ???

I guess disk size vars are in private repo right ???

Well, it depends. They are not in private repo ever, but:

• For the / disk of the vm, they are in host_vars for the host as 'lvm_size'
  This is used when installing the vm/host. If you change it after the initial install it won't do anything until you re-install the host.
  This would apply to noc-cc01 and pagure02.
  (side note, I resized noc-cc01 manually, and removed some large files to make pagure02 use less space)

• For the download hosts, the /srv/ volumes were manually created because they are... gigantic.

download-ib01.fedoraproject.org | CHANGED | rc=0 >>
Filesystem Size Used Avail Use% Mounted on
/dev/vdb1 42T 40T 2.5T 95% /srv
download-cc-rdu01.fedoraproject.org | CHANGED | rc=0 >>
Filesystem Size Used Avail Use% Mounted on
/dev/vdb1 42T 40T 2.4T 95% /srv

We do have a small bit of space we could add to them...
vmhost-x86-cc03.rdu-cc.fedoraproject.org has 5.7TB free.
ibiblio02.fedoraproject.org only has 1.96TB free.

Not sure how much thats going to help longer term.
We could stop mirroring archives there... thats 28TB.
Not sure what else we could easily drop.

Sotrinator is another weird one. It's using space for copr backups. It was manually created.
There's ~10TB still available, so we could grow things some there... or perhaps there's some copr data that could be pruned?

CC: @frostyx @praiskup

Sotrinator is another weird one. It's using space for copr backups. It was manually created.
There's ~10TB still available, so we could grow things some there... or perhaps there's some copr data that could be pruned?

Ideally, we shouldn't prune. There are incremental backups, so the backup would soon start growing back, and removing the increments goes against the purpose.
It's likely we'll migrate out to PULP soon (which should have its own backups). https://github.com/fedora-copr/copr/issues/2533 Hence, it's pretty likely that, if we add the remaning 10TB to Copr, we would be OK for the time being (before we move to PULP).

Thanks for the updates, especially noc-cc01 which looks great - I can stage removing that override for after freeze. The rest will need to stay (for now at least)

I should note that I'm fine with keeping these overrides where they are if we can't increase storage, I just wanted to raise awareness of the situation. When we're talking TBs of data, even 1% is a lot, so having a threshold at 95% will still warn us before it fills up entirely.

Yeah. For the download machines, it's only going to grow over time... but I suppose we could just wait until it's really close to full, then drop archive.

Or perhaps we could look at splitting archive into two volumes... old-archive and newer-archive or something?

Not an immediate problem though.

@praiskup Feel free to add that last 10T in if you need it.

So... download-ib01 and download-cc-rdu01 are not 100% full.
They were pushed over by the beta content and RC's. :(

We could add a tiny bit of space, but it wouldn't be enough for long.

So, I think we need to decide between dropping archive there or fedora-secondary...

I was able to free up enough space to get them back to working for now.
But we will still likely need to drop something at some point.

I'm going to close this, now that we've started to get into the habit of looking at Zabbix things as well as nagios in the regular chat meetings.