Issue #2481: koji.util.rmtree() is not multi-process/thread safe - koji

koji

#2481 koji.util.rmtree() is not multi-process/thread safe

Closed: Fixed Oct 5, 2020 by tkopecek. Opened Sep 9, 2020 by julian8628.

from https://pagure.io/koji/issue/2361#comment-677216:

more race condition problem found in rmtree()

test script:

import os
from multiprocessing import Process
import koji
_DIR = 'brew-testing-dir'
def _mkdirs():
    os.mkdir(_DIR)
    for i in range(0, 100):
        dd = os.path.join(_DIR, 'subdir_' + str(i))
        os.mkdir(dd)
        for j in range(0, 100):
            ddd = os.path.join(dd, 'subdir_' + str(j))
            os.mkdir(ddd)
def test_koji_util_rmtree_parallel():
    _mkdirs()
    tasks = [Process(target=koji.util.rmtree, args=(_DIR,))
             for i in range(0, 10)]
    for task in tasks:
        task.start()
    for task in tasks:
        task.join()
if __name__ == '__main__':
    test_koji_util_rmtree_parallel()

errors:

Traceback (most recent call last):
  File "/usr/lib64/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib64/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/home/yzhu/git/redhat/fork/koji/koji/util.py", line 442, in rmtree
    _rmtree(dev)
  File "/home/yzhu/git/redhat/fork/koji/koji/util.py", line 451, in _rmtree
    dirs = _stripcwd(dev)
  File "/home/yzhu/git/redhat/fork/koji/koji/util.py", line 475, in _stripcwd
    for fn in os.listdir('.'):
OSError: [Errno 116] Stale file handle: '.'
Traceback (most recent call last):
  File "/usr/lib64/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib64/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/home/yzhu/git/redhat/fork/koji/koji/util.py", line 442, in rmtree
    _rmtree(dev)
Process Process-1:
  File "/home/yzhu/git/redhat/fork/koji/koji/util.py", line 469, in _rmtree
    os.chdir(subdir)
FileNotFoundError: [Errno 2] No such file or directory: 'subdir_60'

Metadata Update from @tkopecek:
- Custom field Size adjusted to None
- Issue assigned to julian8628

Sep 10, 2020

mikem commented Sep 29, 2020

In what situations do we legitimately have parallel rmtrees for the same path?

Do we have an actual reported problem due to this behavior, or it is just this script?

Edited Sep 29, 2020 by mikem

julian8628 commented Sep 30, 2020

AFAIK,
- in kojira, it is possible to call rmtree() for the same path (it appeared in the testing of PR #2443)
- in issue #2361 the race condition happened in _stripcwd(), but it is still possible to meet this problem in other places in rmtree()