This ticket had assigned some Differential requests:
D733

Couldn't we get away with using just rpm for checking if a package is installed or not?

I would like to support everything dnf can accept on the dnf install line, i.e. groups, environments, provides, etc. It might be handy for some SIGs (checking an entire group) or cross-distro/version requirement specifications (a package might be called differently on Fedora21/22/RHEL7, but they might share the same provides). We can't query rpm for this.

We could change the code of rm_utils.is_installed(). The function runs dnf with non-root privileges and downloads the metadata to user cache, we can force it to use system cache with arg -C or --cacheonly. It this case, the metadata would be downloaded only in rpm_utils.install(). Drawback of this solution is that we cannot use "vanilla" cloud images for disposable clients since they don't contain cached metadata.
However, if we decide to go for custom images, this won't be a problem, we can ship images with metadata already downloaded. I think we can assume that non-disposable/local environment has (old?) metadata already downloaded, at least from taskotron installation.

Other solution by @kparal is that we would store metadata and packages on host system in /var/lib/taskotron/cache/ and copy it to disposable client from host system (and back as well); timestamp (rewrite only older), os version check, etc. included.

As an update, the way we're hoping to solve this is to have dnf metadata embedded in the prebuilt images that we produce. That way we don't need to change much outside the image building process

OK, here are the issues:
1. User-mode dnf uses it's own cachedir and downloads all metadata, even if system dnf cachedir exists and is fully up-to-date. That causes us to download metadata needlessly. However, this might not be such a big issue, because this only affects non-root accounts, which means a local execution or perhaps a persistent minion - but in both these environments the metadata is not wiped, so it persists to another task run. It's a one time annoyance every few days. In the disposable minion environment, where everything gets wiped, we use root account exclusively (at least for now), which means we're not hit by this. We would get hit by this if we started using a non-root account in disposable minions.
2. The disposable minion environment is hit by a different problem - once the baked-in metadata expires, we will be downloading a full set of fresh metadata (there are no deltas for metadata) every time we start a new task. The download size is about 50MB for Fedora 23 (currently), and the processing takes about 30 seconds (plus download time). This will start happening in about 1-2 days after the disposable images are created in case of general audience (when metalinks are used to access mirrors), and in 6-24 hours (approx. 12 hours) in case of our infrastructure (direct baseurl used to access mirrors). I consider this a bigger issue, because it creates another point of failure for tasks (metadata downloading can fail or time out) and it adds 30+ seconds even to tasks which usually take just a few seconds themselves (rpmlint). It also adds to Fedora infra load, but I don't know whether it is a problem or not at all.

If we plan to rebuild disposable images every day (right after a repo push occurs, for best results), or if we don't mind the startup delay, this might not be such a problem (after all, we //are// running it like this in our current dev environment). But otherwise we should improve this. Here's a suggestion (trying to solve both issues above):

We will use dnf -C/--cacheonly by default.
* This solves the problem of downloading metadata again into dnf user cache, the system cache will be used instead (fixed to that point of time). However, this will fail if no system cache exists, so it that case we will have to resort to user cache. But this should happen very rarely.
* This solves the problem of downloading and processing newer metadata for every single task after 6-24 hours after image generation. We will never look for newer metadata. That has some issues:
** Downloading of task deps might fail, because a newer version replaced an older version of a package in the repo in the meantime. If this happens, we will run the same command again, this time without --cacheonly.
** During disposable minion setup phase, libtaskotron would not get automatically updated to a latest version from the repo - a new image would have to be generated that includes it. Here I'm not exactly sure whether it's a bad thing or a good thing, it depends on what we want. Either way this can be easily resolved by refreshing metadata just specifically for the taskotron repos (and no other repos) during the minion setup phase.
* This will not work on disposable images not containing any cached dnf metadata, e.g. cloud images. In that case, we will fall back to running it without --cacheonly.

I'd also add a bit more clever logic whether we can or can't use sudo for dnf purposes when running under a non-root account. That should further eliminate the likelihood of using the dnf user cache needlessly.

What do you think?

It sounds to me like we should really be shooting for nightly image builds once that can actually be automated. It also sounds like we need to figure out how to prepopulate the dnf cache @ image build time.

What kind of clever logic did you have in mind for using dnf as non-root yser?

It sounds like a good plan to me

! In #642#9118, @tflink wrote:
It sounds to me like we should really be shooting for nightly image builds once that can actually be automated.

OK, great.

It also sounds like we need to figure out how to prepopulate the dnf cache @ image build time.

That should be really easy, we already talked with @jskladan about it. We just need to run dnf makecache in %post section in the kickstart, that should be all.

What kind of clever logic did you have in mind for using dnf as non-root yser?

Currently, if we are non-root, we always use user cache for dnf operations (verifying whether a package is installed). I'd like to test sudo availability first, and if we can use it without a password, use that. This could be the case on some persistent minions (or either custom or our future disposable minions) and we would further reduce the use of user dnf caches.

It sounds like a good plan to me

OK, I'll try to implement that. Any thoughts whether we want libtaskotron to be updated automatically when a new version is in our repo, or whether we want to stick to the baked-in version? We should consider not just our production env, but also task developer env (who is probably not likely to download an updated image every day). If the two use cases benefited from a different behavior, I can add a configuration option for that (but I'd avoid additional options unless really necessary). I don't really have a formed opinion here. You?

Fixed in D733.