There is:

        fo = tempfile.TemporaryFile(dir=tempdir)
        try:
            resp = requests.get(url)
            for chunk in resp.iter_content(chunk_size=8192):
                fo.write(chunk)
        finally:
            resp.close()

in Koji code. I think it is intended that this code reads the file chunk by chunk instead of putting it all into memory, however there is a parameter missing. For this to work, IMHO requests.get(url) need to have stream=True, otherwise it actually tires to store the file in a single bytestring and that overflows.

        fo = tempfile.TemporaryFile(dir=tempdir)
        try:
            resp = requests.get(url, stream=True)   #  HERE: don't store resp.content, but allow to iterate
            for chunk in resp.iter_content(chunk_size=8192):
                fo.write(chunk)
        finally:
            resp.close()

Or (source):

        fo = tempfile.TemporaryFile(dir=tempdir)
        try:
            resp = requests.get(url, stream=True)
            shutil.copyfileobj(resp.raw, fo)
        finally:
            resp.close()

With context managers:

        with tempfile.TemporaryFile(dir=tempdir) as fo:
            with requests.get(url, stream=True) as resp:
                shutil.copyfileobj(resp.raw, fo)

Regression introduced in https://pagure.io/koji/pull-request/1542

Also, the context managers are deliberately not used.

Probable fix: https://pagure.io/koji/pull-request/1993

Commit 388bb070 fixes this issue

hmm, closing it in koji, closed also this ticket. Please reopen.

Please deploy the fix.

ok. I have deployed a package with that patch applied. (It's still rolling out to s390x builders, but hopefully done in a bit here).

Please re-try your build and see if that fixes things?

I get this errors on the armv7 build:
https://koji.fedoraproject.org/koji/taskinfo?taskID=41534300

Traceback (most recent call last):
File "/usr/lib/python3.7/site-packages/koji/tasks.py", line 482, in localPath
resp = requests.get(url, stream=True)
File "/usr/lib/python3.7/site-packages/requests/api.py", line 75, in get
return request('get', url, params=params, kwargs)
File "/usr/lib/python3.7/site-packages/requests/api.py", line 60, in request
return session.request(method=method, url=url, kwargs)
File "/usr/lib/python3.7/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python3.7/site-packages/requests/sessions.py", line 686, in send
r.content
File "/usr/lib/python3.7/site-packages/requests/models.py", line 828, in content
self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b''
MemoryError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.7/site-packages/koji/daemon.py", line 1294, in runTask
response = (handler.run(),)
File "/usr/lib/python3.7/site-packages/koji/tasks.py", line 313, in run
return koji.util.call_with_argcheck(self.handler, self.params, self.opts)
File "/usr/lib/python3.7/site-packages/koji/util.py", line 263, in call_with_argcheck
return func(args, *kwargs)
File "/usr/sbin/kojid", line 1339, in handler
fn = self.localPath("work/%s" % pkg)
File "/usr/lib/python3.7/site-packages/koji/tasks.py", line 489, in localPath
resp.close()
UnboundLocalError: local variable 'resp' referenced before assignment

Ouch, maybe underlying https://pagure.io/koji/pull-request/1854 is not there? @kevin what is the current build?

https://kojipkgs.fedoraproject.org//packages/koji/1.20.0/2.fc31.infra/src/koji-1.20.0-2.fc31.infra.src.rpm

I applied https://pagure.io/koji/c/6aa1b3e.patch is there more needed?

It should be ok for fixing the original problem, weird is the traceback. It says, that stream=True is used, while it is continuing inside the requests.sessions via if not stream: code path. I don't know, where it can get lots/turned to False. I'm not able to replicate it in my test env for now.

Second traceback is due to missing https://pagure.io/koji/pull-request/1958, but it is only cosmetical issue in handling the traceback for now.

Are you able to reproduce it, or is it fixed?

Some more reports in https://bugzilla.redhat.com/show_bug.cgi?id=1802077 and https://pagure.io/fedpkg/issue/383

This is still happening:


[spot@dhcp1-96 texlive]$ fedpkg build
Building texlive-2019-18.fc31 for f31-candidate
Created task: 42256942
Task info: https://koji.fedoraproject.org/koji/taskinfo?taskID=42256942
Watching tasks (this may be safely interrupted)...
42256942 build (f31-candidate, /rpms/texlive.git:aedfc91ed1c237715b55e8cc23b2b2c6b63c1e61): free
42256942 build (f31-candidate, /rpms/texlive.git:aedfc91ed1c237715b55e8cc23b2b2c6b63c1e61): free -> open (buildhw-11.phx2.fedoraproject.org)
42256943 buildSRPMFromSCM (/rpms/texlive.git:aedfc91ed1c237715b55e8cc23b2b2c6b63c1e61): open (buildvm-armv7-14.arm.fedoraproject.org)
42256943 buildSRPMFromSCM (/rpms/texlive.git:aedfc91ed1c237715b55e8cc23b2b2c6b63c1e61): open (buildvm-armv7-14.arm.fedoraproject.org) -> closed
0 free 1 open 1 done 0 failed
42257443 buildArch (texlive-2019-18.fc31.src.rpm, noarch): open (buildvm-armv7-07.arm.fedoraproject.org)
42256942 build (f31-candidate, /rpms/texlive.git:aedfc91ed1c237715b55e8cc23b2b2c6b63c1e61): open (buildhw-11.phx2.fedoraproject.org) -> FAILED: Fault:
0 free 1 open 1 done 1 failed
42257443 buildArch (texlive-2019-18.fc31.src.rpm, noarch): open (buildvm-armv7-07.arm.fedoraproject.org) -> FAILED: Fault:
0 free 0 open 1 done 2 failed

42256942 build (f31-candidate, /rpms/texlive.git:aedfc91ed1c237715b55e8cc23b2b2c6b63c1e61) failed


Can we try 1.20.1?

We have updated to 1.20.1... please try again and cross your fingers. ;)

MemoryError no longer happens.
Many thanks
Martin