FedoraReview

#323 FedoraReview scans sources of a large package for 12+ hours
Opened by churchyard. Modified

Open
Close Issue

I did:

$ python2 try-fedora-review -b 1662538 -m fedora-rawhide-x86_64

The SRPM is > 300 MB.

It run trough the night, doing apparently nothing but spinning my CPU. This is from the logs:

...
01-06 21:44 root         DEBUG    Running check: CheckNoConflicts
01-06 21:44 root         DEBUG        CheckNoConflicts completed: 0.000 seconds
01-06 21:44 root         DEBUG    Running check: CheckDirectoryRequire
01-06 21:44 root         DEBUG    Running: dnf repoquery -C -l filesystem
01-06 21:44 root         DEBUG        CheckDirectoryRequire completed: 1.906 seconds
01-06 21:44 root         DEBUG    Running check: CheckFullVerReqSub
01-06 21:44 root         DEBUG        CheckFullVerReqSub completed: 0.000 seconds
01-06 21:44 root         DEBUG    Running check: CheckBundledFonts
01-06 21:44 root         DEBUG        CheckBundledFonts completed: 0.002 seconds
01-06 21:44 root         DEBUG    Running check: CheckLicenseField
01-06 21:44 root         DEBUG    Scanning sources in /var/lib/mock/fedora-rawhide-x86_64/root/builddir/build/BUILD/tvb-data-7d2d05b805aeed0b4d3fe26d2ba97f13341ce65a
01-07 08:22 root         DEBUG    Exception down the road...
Traceback (most recent call last):
  File "/home/churchyard/rpmbuild/FedoraReview/src/FedoraReview/review_helper.py", line 239, in run
    self._do_run(outfile)
  File "/home/churchyard/rpmbuild/FedoraReview/src/FedoraReview/review_helper.py", line 229, in _do_run
    self._do_report(outfile)
  File "/home/churchyard/rpmbuild/FedoraReview/src/FedoraReview/review_helper.py", line 98, in _do_report
    self._run_checks(self.bug.spec_file, self.bug.srpm_file, outfile)
  File "/home/churchyard/rpmbuild/FedoraReview/src/FedoraReview/review_helper.py", line 117, in _run_checks
    writedown=not Settings.no_report)
  File "/home/churchyard/rpmbuild/FedoraReview/src/FedoraReview/checks.py", line 378, in run_checks
    run_check(name)
  File "/home/churchyard/rpmbuild/FedoraReview/src/FedoraReview/checks.py", line 352, in run_check
    check.run()
  File "/home/churchyard/rpmbuild/FedoraReview/src/FedoraReview/plugins/generic.py", line 657, in run
    out = check_output(cmd, shell=True)
  File "/usr/lib64/python2.7/subprocess.py", line 217, in check_output
    output, unused_err = process.communicate()
  File "/usr/lib64/python2.7/subprocess.py", line 475, in communicate
    stdout = _eintr_retry_call(self.stdout.read)
  File "/usr/lib64/python2.7/subprocess.py", line 125, in _eintr_retry_call
    return func(*args)
KeyboardInterrupt
01-07 08:22 root         ERROR    Exception down the road...(logs in /home/churchyard/.cache/fedora-review.log)
01-07 08:22 root         DEBUG    Report completed:  38999.453 seconds

It's probably running licensecheck...

After 18 hours waiting for it to check the grafana request, I gave up and killed it. The line of the backtrace is definitely pointing to licensecheck. There are some alternatives, but I did not do more than a cursory look:

  • https://github.com/codeauroraforum/lid (Python, though their install instructions are a bit weird)
  • https://github.com/licensee/licensee (Ruby)
  • https://github.com/nexB/scancode-toolkit (Python 2, in the middle of porting to 3)
Edited by qulogic

licensee does not scan source files, which is what licensecheck does. The other two may, though I'm honestly not sure how to use them.

For grafana, licensecheck gets stuck on this massive JS webpack file

$ ls -l
-rw-r--r--. 1 elliott family 14362016 Jan 31 20:46 app.8c5db00ef9d76e580862.js

For python-tvb-data as originally reported, there are many such large files.

Not really a solution, but you should be able to walk-around the problem using --exclude.

With lid, I can run it on grafana, and it automatically runs with 4 processes and takes 8.5 minutes. The output is a little verbose since it prints out the license for every file it could find one in. The format can be switched to csv, but unfortunately, it only lists ~400 files out of ~4000 since it doesn't seem to output those without a license, and there are a few minor bugs. The threshold for recognition can be tweaked, but setting it to 0 took longer than I was willing to wait.

With scancode, I ran it with 4 processes (explicitly) and it took about 8 minutes too. It also explicitly listed the app.8c5db00ef9d76e580862.js as unscannable (there's a timeout that maybe occurred.) It produces pretty nice progress information and a nice html table of results. Unfortunately, it also only lists the files for which it could find a license. However, if you output a .csv, you do appear to get all files. The biggest issue is that it's Python 2 and uses some complicated vendoring, though upstream is working on both of these.

I increased the timeout to 20 minutes and scancode was able to process the app.8c5db00ef9d76e580862.js file in 13 minutes. It found 190 licenses in that file, which makes sense since it's a bundling of a bunch of JS packages. I don't know if that's exactly the right number of licenses.
Metadata
None
None
None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.