TLDR; This PR split the container images into base and code Dockerfile as well as a separate entrypoint script. Testing of the local git repo and active branch by mounting the folder into the container as new default. REPO and BRANCH env vars can be set which allow testing of remote repositories as before if wanted. Image build only if they not exist in the local podman cache or when manually triggered, every test run use the existing container otherwise. This allows local testing of current changes within a few minutes or even just seconds.

To reduce the resources and time to test changes, I created some scripts a few months ago which I used a lot when I was working on fixing the unit tests. It was hacky and mainly for my personal use, so I brushed it up a and moved all the logic into the run-tests-container.py script.

General design changes:
The Dockerfiles of the old container image contained everything and also included the pagure source that should be tested. I found this very inflexible, every small change required a rebuild of the whole container, which is very time consuming. Therefore I split them in three parts: A base and code Dockerfile and a separate entrypoint script.
The base image contains all packages, rpm also everything required by pagure.spec, pip all tox environments pre-populated with the packages based on all requirements.txt files. The code image uses the base image and adds the entrypoint script.
The actual pagure code isn't part of any of the images and will either be downloaded from a external repo or (default), mounted from the current directory, when the code container get started.

./run-tests-container.py --centos "tests/test_pagure_flask_api_issue_change_status.py" > start c8s code container, local git top-folder mounted as /wrkdir, entrypoint script performs git clone from /wrkdir and given branch, default is the currently active, to /pagure inside the container. The rest is mostly unchanged to the original code.

Available container images:

==> dev/containers/base-centos-stream8-rpms-py3 <==
FROM quay.io/centos/centos:stream8
==> dev/containers/base-fedora-pip-py3 <==
FROM quay.io/fedora/fedora:36-x86_64
==> dev/containers/base-fedora-rpms-py3 <==
FROM quay.io/fedora/fedora:36-x86_64

I added additional tests to run-tests-container.py, if git or podman are not installed, it will exit. If the base or code container image not exist, they will be build. If the new parameter --rebuild or --rebuild-code is used, a new image build is forced by ignoring the local cache. As long no rebuild is triggered manually, the existing images will be used for every test run.

pagure/dev/results_<test-image-name> will still be mounted into the container as /results, but the logging is done my run-tests-container.py, I don't see any pagure test code that still writes into /results.
Log files are automatically created for build, test and also shell activities and saved in the related pagure/dev/results_<test-image-name> folder, with the current unix timestamp as prefix. This allows to go back and check logs for a previous test run. Example:

pagure/dev/results_pagure-tests-fedora-pip-py3/1681673866_base-build.log
pagure/dev/results_pagure-tests-fedora-pip-py3/1681673866_code-build.log
pagure/dev/results_pagure-tests-fedora-pip-py3/1681673866_tests.log

Specifying a test case was possible from a CLI perspective in the past but then only passed to the tests inside the rpm container, not pip. This is now possible with all images by adding the path to the test file as last argument. Example to run one test case and just a single test on c8s:

dev/run-tests-container.py --centos tests/test_pagure_flask_api_issue_change_status.py
dev/run-tests-container.py --centos tests/test_pagure_flask_ui_priorities.py:PagureFlaskPrioritiestests.test_ticket_with_no_priority

Testing changes quickly locally, without pushing to remote, was something I needed very urgent but was also requested for example here: https://pagure.io/pagure/issue/5186
As outlined in the above issue, if a PR is already open, pushing changes to perform a local test also triggered the Jenkins CI in the related PR, which is a waste of resources.

@ngompa @pingou sorry for the wall of text, I tried to explain this quite large change as good as possible. Feel free to give it a try and please let me know what you think. I'm happy to further improve it but for sure, you can also just merge it if you like the proposed approach.

Important:
This PR doesn't touch anything related to the CI pipeline, it only affects local testing. If parts of this approach would also be valuable to reduce the CI runtime, I can work an that at a later point.