Also, looks like the first build has technically "failed" but since the build itself succeeded and only the "tagBuild" subtask failed ... it's now stuck as "failed" but I can't resubmit it: https://koji.fedoraproject.org/koji/buildinfo?buildID=2201477

@decathorpe could you check wich items is taking more space on disk ??

du -ms /var/tmp/koji/tasks/* | sort -hr |more

The problem seemed to be that builders were not able to remove old epel7 buildroots (and since that errored out no build roots were removed).

For example:

May 08 20:02:51 buildvm-ppc64le-28.iad2.fedoraproject.org kojid[3007743]: 2023-05-08 20:02:49,651 [INFO] {3007743} koji.TaskManager:878 Removing buildroot: 42169432/epel7-build/ppc64le
May 08 20:02:51 buildvm-ppc64le-28.iad2.fedoraproject.org kojid[3007743]: 2023-05-08 20:02:49,652 [ERROR] {3007743} koji:517 Unable to remove directory lib: [Errno 39] Directory not empty: 'lib'
May 08 20:02:51 buildvm-ppc64le-28.iad2.fedoraproject.org kojid[3007743]: 2023-05-08 20:02:49,652 [ERROR] {3007743} koji:517 Unable to remove directory var: [Errno 39] Directory not empty: 'var'
May 08 20:02:51 buildvm-ppc64le-28.iad2.fedoraproject.org kojid[3007743]: 2023-05-08 20:02:49,652 [ERROR] {3007743} koji:517 Unable to remove directory root: [Errno 39] Directory not empty: 'root'
May 08 20:02:51 buildvm-ppc64le-28.iad2.fedoraproject.org kojid[3007743]: 2023-05-08 20:02:49,652 [WARNING] {3007743} koji.build:96 file removal failed for /var/lib/mock/epel7-build-42169432-5132696

So, I cleaned manually out all the /var/lib/mock/epel7-build* repos and things mostly got back to working.
However, there was one more machine that was saying it was out of space, even when it had a bunch of space available.
I rebooted that one and it didn't come back, so I am just going to reinstall it. It must have become corrupted somehow.

I went and retagged all the things that had failed tagbuilds.

I'll leave this open however until I can file an upstream koji ticket on why epel7 buildroots aren't getting cleaned up as expected (ie, the underlying cause).

https://pagure.io/koji/issue/3825