Build succeeded.

• tox-f33 : SUCCESS in 11m 06s

Some ideas/thoughts (here because it'd become too scattered inline, in no particular order):

• The log files we process are on NFS, so we'd better pipe the output of grep into a local temporary file instead of a file right beside the unprocessed one.
• We can also just use that file object in the stdout parameter in subprocess.run() and get rid of the intermediate shell process.
• We should clean up after ourselves! Ideally, pre_process() would be a context manager which can then clean up the intermediate processed file when it's no longer needed.
• Preprocessing a log file in plain Python[1] is roughly as fast (about 1 minute for about 8GB) as using grep or fgrep to do the same with cold caches (which is what we'll most likely be dealing with). With that, I'd say that the bottleneck should be shoveling a couple of gigs of log data over the network, not filtering them… What do you think, why is processing all log lines so slow, why does preprocessing it with grep speed things up? Does ReadProgress add so much overhead?

[1]: like this:

rawlog_path = "/mnt/fedora_stats/combined-http/latest/mirrors.fedoraproject.org-access.log"
processedlog_path = "/tmp/countme.py.out"
with open(rawlog_path, "rt") as rawlog, open(processedlog_path, "wt") as processedlog:
    for l in rawlog:
        if "countme" in l:
            print(l, end="", file=processedlog)

Build succeeded.

• tox-f33 : SUCCESS in 10m 58s

I did change it to local temporary file with a context-manager to clean-up after itself.

What you suggest with python being as fast as grep, I am not sure the problem is with bandwidth, as when I was testing it localy, there was no network access to slow things down.

It could be the case of the speedup I have observed was due to caching? I.e. I did re-run it several times, it was on my local disk, e.t.c.

Build succeeded.

• tox-f33 : SUCCESS in 10m 31s

Build succeeded.

• tox-f33 : SUCCESS in 10m 53s

Build failed. More information on how to proceed and troubleshoot errors available at https://fedoraproject.org/wiki/Zuul-based-ci

• tox-f33 : FAILURE in 10m 42s

Hmm, TIL that type-annotating generators/iterables isn't that straight-forward:

mypy run-test: commands[0] | python -m mypy --config-file /workspace/src/pagure.io/mirrors-countme/mypy.cfg .
countme/progress.py:91: error: The return type of a generator function should be "Generator" or one of its supertypes
countme/progress.py:91: error: Argument 1 to "contextmanager" has incompatible type "Callable[[Union[str, Path]], str]"; expected "Callable[..., Iterator[<nothing>]]"
countme/parse.py:9: error: The return type of a generator function should be "Generator" or one of its supertypes
countme/parse.py:9: error: Argument 1 to "contextmanager" has incompatible type "Callable[[Union[str, Path]], str]"; expected "Callable[..., Iterator[<nothing>]]"
Found 4 errors in 2 files (checked 12 source files)
ERROR: InvocationError for command /workspace/src/pagure.io/mirrors-countme/.tox/mypy/bin/python -m mypy --config-file mypy.cfg . (exited with code 1)

Build succeeded.

• tox-f33 : SUCCESS in 10m 35s