This ticket had assigned some Differential requests:
D778

I have an idea! But I only have been thinking about it for a couple of hours, so... You know how it is.

First of all, the way I think this works, we are only going to solve the environment selection for disposable minions. And I think it's fine for the start.

I suppose that we will have kinds of tests:
# environment agnostic tests - these will be run on 'standard environment' (something like 'latest release, standard flavor, 64 bit)
# picky tests - these would want to specify any of flavor, release, arch

If we consider KOJI_BUILD as an example item type, we can automagically devise arch and release. I do not think it's necessary to specifically put ${item} there, as it's available in the arg_data, and I'd rather not use it just as an place-holder, since we might want to add some smart stuff later (like a user-defined regexp that will parse the needed bits from item, or something like that).

So I propose this:
base_image:
flavor: foobar
release: FROM_ITEM
arch: FROM_ITEM

where FROM_ITEM is an actual keyword, that means "try and parse it automagically for me". I'm not sure if allowing pure "value" to be set for the release/arch is a good idea, but we could always use something like VALUE=x86_64 instead of FROM_ITEM, if we really wanted to. foobaris just a placeholder there.

One could specify any of the three (flavor, release, arch), those not set will fall-back to defaults. So real-life examples could be:

base_image:
  arch: FROM_ITEM
base_image:
  flavor: minimal
  release: FROM_ITEM

And if we go the "formula can specify release/arch on itself" way, then possibly also something in the likes of:

base_image:
release: FROM_ITEM
arch: VALUE=i386

Thoughts?

I'm not sure I understand what VALUE=i386 gives us that i386 wouldn't - can you elaborate a bit more on that?

I also wonder if the default could be FROM_ITEM which ends up spitting out some kind of default if we can't determin what FROM_ITEM should be from the inputs

Well... VALUE=i386 gives us a way to possibly check the input. We could, of course, just look for "modifiers we know" (like FROM_ITEM, RE, `FOOBAR), and handle anything else as user input. I just don't like it that much. But if you prefer it, I do not have a huge problem with it.

! In #731#10146, @tflink wrote:
I also wonder if the default could be FROM_ITEM which ends up spitting out some kind of default if we can't determin what FROM_ITEM should be from the inputs

We could, but then a developer would either have to use force_imageurl, or have at least one of each possible combinations of release/flavor/arch of base images, to be able to run the code on any possible item/type combination. I'm not sure whether that's really what we want, but once again, I don't have a problem either way.

We could also solve it by adding something like "do not guess environment" config option. But my initial reasoning was to explicitly separate the "use defaults", and "guesstimate the values" execution paths.

BTW, I just realized that FROM_ITEM is quite not right of a name for arch, since at least with koji_build type, arch is not a part of the item, but rather passed in --arch cmdline argument.

If we decide to keep the "you need to explictly say, that it's OK to guess the release/arch" way of doing things, then FROM_ITEM should probably be replaced with something like DEVISE (/me wanted to say GUESS but that sounds... not good :D)

! In #731#10152, @jskladan wrote:
Well... VALUE=i386 gives us a way to possibly check the input. We could, of course, just look for "modifiers we know" (like FROM_ITEM, RE, `FOOBAR), and handle anything else as user input. I just don't like it that much. But if you prefer it, I do not have a huge problem with it.

I still don't understand. For arch, in particular, there are a limited number of possible values - all of which we know about. If we needed to check type by verifying that the build was valid or something like that, I can see the utility to adding the VALUE= part but I can't think of a case here where we'd need to check something that isn't as easy to just match to a known list of values. Am I missing something?

! In #731#10146, @tflink wrote:
I also wonder if the default could be FROM_ITEM which ends up spitting out some kind of default if we can't determin what FROM_ITEM should be from the inputs

We could, but then a developer would either have to use force_imageurl, or have at least one of each possible combinations of release/flavor/arch of base images, to be able to run the code on any possible item/type combination. I'm not sure whether that's really what we want, but once again, I don't have a problem either way.

I'd like to make things easy for the folks running tasks outside our environments. What I was thinking is to have the "default" be to choose from the first valid image from the following sequence:
1. imageurl if force=imageurl is true
2. parse from item/arch if those were provided @ execution time
3. closest local match (could just be the default cloud image) if available locally. big warning if there's a mismatch but don't crash
4. use the url from config
5. fail, stating clearly that there's a problem with image specification

Does that make more sense? My intention is to make users' lives easier, not add complications.

We could also solve it by adding something like "do not guess environment" config option. But my initial reasoning was to explicitly separate the "use defaults", and "guesstimate the values" execution paths.

I think that specifying an image type/version/url is pretty clearly saying "don't guess the environment" and I'm not sure we need something more than force_imageurl.

! In #731#10157, @tflink wrote:
utility to adding the VALUE= part but I can't think of a case here where we'd need to check something that isn't as easy to just match to a known list of values. Am I missing something?

Explicit is better than implicit, and keeping a lists of arbitrary values on multiple places feels like bad design. But I'd be oOK with a bit of a different approach, with mostly the same result - have a known list of special keywords (like RE might be), and just treat anything else as the desired value. Thus the user is responsible for providing something that makes sense (should they decide to override the default behavior, whatever it will be) and we get the "explicit" quite cheap.

! In #731#10146, @tflink wrote:
I'd like to make things easy for the folks running tasks outside our environments. What I was thinking is to have the "default" be to choose from the first valid image from the following sequence:
1. imageurl if force=imageurl is true
2. parse from item/arch if those were provided @ execution time
3. closest local match (could just be the default cloud image) if available locally. big warning if there's a mismatch but don't crash
4. use the url from config
5. fail, stating clearly that there's a problem with image specification

Does that make more sense? My intention is to make users' lives easier, not add complications.

I see where you are going with this, and I like it. But at least item 3 does not make sense to me - what is a "closest local match"? Is x86_64 closest match to i368? Is Fedora 24 closest match to Fedora 23? Or would that be applied only on flavor? If so, do we define the "universal fallback", or do we just not care about it at all?

Also how would we ever get to item 5, if in #4 we used the url from config? Or is if for when the imageurl in config is unset?

What bugs me about this is that even though the users will generally be happy because "it ran", I can see a situation, where tasks are misbehaving, because different run-environment was selected "just so it runs". But most of it just comes from the #3 in the sequence.

I think that specifying an image type/version/url is pretty clearly saying "don't guess the environment" `.

Specifying where? If you specified it in the formula, and used the sequence you described, then you still could not be sure what environment was selected (because of #3). The only other way to force/specify env is by imageurl, but then all of your tasks are running in that env, and you need to change the imageurl each time you get a new base image.

My idea behind the "do not guesstimate env" option was to allow using imagesdir as a source of the images (so you just can copy new file in, without changing imageurl), and at the same time being able to just say "I don't care about the env, just use the latest default (defined either by the disposable_(release, flavor, arch) config options, if we decide to keep them, or just latest by timestamp).

But to get back to the important stuff - for me, this makes the most sense for the env selection:
# if force_imageurl => imageurl
# if do_not_infer_base_image => latest image from imagesdir, either based on disposable_(release, flavor, arch) config options (if we decide to keep them) or just on timestamp. Include a log message stating what env would be used, if the do_not_infer_base_image was not set
# infer from inputs (item, type, arch) and formula (for overrides)
# fail, stating clearly that there's a problem with image specification

The main difference, for me, being that we won't need to guesstimate the closest match, should infering from inputs "fail" (as in "base image of inferred type was not found"), and all the behaviour is explicit and simple to understand.

Or, if we really put value to "always run, no matter what", then we could:
# if force_imageurl => imageurl
# infer from inputs (item, type, arch) and formula (for overrides)
# if inferred env was not found
* if in development mode => use "sane default" from imagesdir either based on disposable_(release, flavor, arch) config options (if we decide to keep them) or just on timestamp. Include a big fat warning, that in production, this would fail. Include the required inferred env in the warning message.
* if in production mode => fail, stating clearly that there's a problem with image specification

! In #731#10145, @jskladan wrote:
First of all, the way I think this works, we are only going to solve the environment selection for disposable minions. And I think it's fine for the start.

Just a note - we actually look into and use environment key even in local execution mode - we verify the rpm deps. Of course we can skip looking into certain sections (like base_image) when running in non-disposable mode.

So I propose this:
base_image:

It might be naming bikesheding, but I'd call this rather system, minion or image. Because base_image seems like first step in something that's going to be layered afterwards.

  flavor: foobar

And this could be template or image_template to keep our terminology consistent. //Flavor// is often used to refer to Workstation/Server/Cloud, but templates are different. We can have several templates for workstation, for different purposes. We can have a template for rpmlint, if we find compelling reasons.

! In #731#10152, @jskladan wrote:
Well... VALUE=i386 gives us a way to possibly check the input. We could, of course, just look for "modifiers we know" (like FROM_ITEM, RE, `FOOBAR), and handle anything else as user input. I just don't like it that much. But if you prefer it, I do not have a huge problem with it.

I would not prefix custom values with VALUE=, it seems unnecessary.

But I'd be oOK with a bit of a different approach, with mostly the same result - have a known list of special keywords (like RE might be), and just treat anything else as the desired value.

That's exactly what I understood Tim asked for. And what I imagined as well.

! In #731#10153, @jskladan wrote:
BTW, I just realized that FROM_ITEM is quite not right of a name for arch, since at least with koji_build type, arch is not a part of the item, but rather passed in --arch cmdline argument.

If we decide to keep the "you need to explictly say, that it's OK to guess the release/arch" way of doing things, then FROM_ITEM should probably be replaced with something like DEVISE (/me wanted to say GUESS but that sounds... not good :D)

We already have input section in the formula, containing args section. So we could name it FROM_INPUT or FROM_ARGS. We would then look into args section and use the argument which makes sense, so for release it would be item and for arch it would be arch. However, I'm a bit afraid we'll hit limitations here soon. Some item types can have arch encoded in item (we can introduce something like arch: FROM_ARG=item). Some might specify multiple archs in the input arg (that's currently valid) - because arch to be tested doesn't have to be the same as arch to be executed on. I guess we'll need to start with something simple and work on the limitation as we continue to hit them.

! In #731#10160, @jskladan wrote:
I see where you are going with this, and I like it. But at least item 3 does not make sense to me - what is a "closest local match"? Is x86_64 closest match to i368? Is Fedora 24 closest match to Fedora 23? Or would that be applied only on flavor? If so, do we define the "universal fallback", or do we just not care about it at all?

I'd be very wary of "closest match" approach. That can introduce a lot of issues and uncertainty when running the task. I imagined it would be very simple - does the formula mandate some specific image parameters? If no, use default (something like latest minimal x86_64). If yes, find latest image matching those parameters. If it's not available, error out.

If the user insists on ignoring image requirements specific in the formula, let him use --libvirt-image argument to force override the image being used (that is not implemented yet, but was approved as part of #408). Print a warning in that case.

But to get back to the important stuff - for me, this makes the most sense for the env selection:
# if force_imageurl => imageurl

Honestly, I think we need to get rid of imageurl and force_imageurl completely, it just does not fit into the concept of heterogeneous tasks. We need intelligent image downloader instead, which will understand what image is needed, figure out if it is available, and if not, it finds that image on our hosting service and downloads it (possibly after confirmation). It is not a simple task, so initially we might just say //"sorry, this task has specific requirements and none of your images fit it, you need to look , find a matching image and download it into "//.

Users will still be able to override the image to be picked by using --libvirt-image. But I don't think imageurl fits the bill any longer. We can keep it around for some time to have an auto-download of a generic image for formulas which have no specific requirements. But for those they do, I would completely ignore imageurl and phase it out.

# if do_not_infer_base_image => latest image from imagesdir, either based on disposable_(release, flavor, arch) config options (if we decide to keep them) or just on timestamp. Include a log message stating what env would be used, if the do_not_infer_base_image was not set
# infer from inputs (item, type, arch) and formula (for overrides)

I'm hesitating whether we should automatically infer the image from input args, without any request for this from formula. There are areas where it makes much sense, yes, like dist-git testing. You always want that. But generic tests might prefer running on the latest distro version and the item under test doesn't have to be tied to the system. So some part of our user base will be forced to specify the image preferences in the formula in any case. And by auto-inferring the image type, we're creating uncertainty and require people to study how our "intelligent decision algorithm" works in detail. I'm not sure about this.

Or, if we really put value to "always run, no matter what", then we could:
# if force_imageurl => imageurl
# infer from inputs (item, type, arch) and formula (for overrides)
# if inferred env was not found
* if in development mode => use "sane default" from imagesdir either based on disposable_(release, flavor, arch) config options (if we decide to keep them) or just on timestamp. Include a big fat warning, that in production, this would fail. Include the required inferred env in the warning message.

Please don't. If the required environment was not found, error out, and let the user override it explicitly and consciously with --libvirt_image.

! In #731#10177, @kparal wrote:
It might be naming bikesheding, but I'd call this rather system, minion or image. Because base_image seems like first step in something that's going to be layered afterwards.

As I said. I'm not set on names. I'd stay away from both system and minion, though.

And this could be template or image_template to keep our terminology consistent. //Flavor// is often used to refer to Workstation/Server/Cloud, but templates are different. We can have several templates for workstation, for different purposes. We can have a template for rpmlint, if we find compelling reasons.

Once again, not set on naming, but the concept of having task-specific images feels a bit over the top, honestly.

... That's exactly what I understood Tim asked for. And what I imagined as well.

I'm glad we can agree on this being OK.

We already have input section in the formula, containing args section. So we could name it FROM_INPUT or FROM_ARGS. We would then look into args section and use the argument which makes sense, so for release it would be item and for arch it would be arch. However, I'm a bit afraid we'll hit limitations here soon. Some item types can have arch encoded in item (we can introduce something like arch: FROM_ARG=item). Some might specify multiple archs in the input arg (that's currently valid) - because arch to be tested doesn't have to be the same as arch to be executed on. I guess we'll need to start with something simple and work on the limitation as we continue to hit them.

Although we do support multiple arches, we do not use it in any way, and to be honest - especially with this functionality coming in - I really feel we should get rid of it, especially if there are no evident testcases.

The FROM_ARG=item is in the lines of the RE "functionality" I mentioned earlier, but I'd much rather use the --arch argument for that.

I'd be very wary of "closest match" approach. That can introduce a lot of issues and uncertainty when running the task. I imagined it would be very simple - does the formula mandate some specific image parameters? If no, use default (something like latest minimal x86_64). If yes, find latest image matching those parameters. If it's not available, error out.

Exactly what I was talking about.

If the user insists on ignoring image requirements specific in the formula, let him use --libvirt-image argument to force override the image being used (that is not implemented yet, but was approved as part of #408). Print a warning in that case.

Honestly, I think we need to get rid of imageurl and force_imageurl completely, it just does not fit into the concept of heterogeneous tasks. We need intelligent image downloader instead, which will understand what image is needed, figure out if it is available, and if not, it finds that image on our hosting service and downloads it (possibly after confirmation). It is not a simple task, so initially we might just say //"sorry, this task has specific requirements and none of your images fit it, you need to look , find a matching image and download it into "//.

Users will still be able to override the image to be picked by using --libvirt-image. But I don't think imageurl fits the bill any longer. We can keep it around for some time to have an auto-download of a generic image for formulas which have no specific requirements. But for those they do, I would completely ignore imageurl and phase it out.

Once again, this is terminology - you could easily replace the if force_imageulr -> imageurl step with if --libvirt-image -> libvirt-image with no consequence to the merit of the 'algorithm'. We can easily abstract that to if user-side-override -> use user-side-override. No matter how it's actually implemented.

Not that I don't agree with you - I'd really like for us to get rid of these. But that's technicality, and tangential to what I think is important to solve here - the desired behaviour/algorithm of the whole base-image selection.

I'm hesitating whether we should automatically infer the image from input args, without any request for this from formula. There are areas where it makes much sense, yes, like dist-git testing. You always want that. But generic tests might prefer running on the latest distro version and the item under test doesn't have to be tied to the system. So some part of our user base will be forced to specify the image preferences in the formula in any case. And by auto-inferring the image type, we're creating uncertainty and require people to study how our "intelligent decision algorithm" works in detail. I'm not sure about this.

Well, I think that any kind of sane default behaviour is OK - as long as it's documented. I do not see huge difference between inferring the environment by default, and having an option to say release: latest, or taking the latest release (but then again, what is the latest release? Current stable? Rawhide?), and making the task devs use release: infer; arch: infer.
But if we're targeting dist-git testing now, then I think inferring the env by default makes more sense than the opposite.

Or, if we really put value to "always run, no matter what", then we could:
# if force_imageurl => imageurl
# infer from inputs (item, type, arch) and formula (for overrides)
# if inferred env was not found
* if in development mode => use "sane default" from imagesdir either based on disposable_(release, flavor, arch) config options (if we decide to keep them) or just on timestamp. Include a big fat warning, that in production, this would fail. Include the required inferred env in the warning message.

Please don't. If the required environment was not found, error out, and let the user override it explicitly and consciously with --libvirt_image.

yeah, I'm not fond of that, you can say by the whole paragraph above the one you quoted. But from what Tim said, it felt like "running at least somehow" is something he'd like to see, and this version of doing it would IMHO cause the least harm.

After some in person discussion, we think it could look something like this:

1. Move the image specification directly into environment key in formula. We imagine further keys will be added there in the future, so eventually it could look like this:

environment:
    rpm:
        - rpmlint
    base_image: minimal
    arch: x86_64
    release: 24
    distro: fedora
    memory: 4G
    virt_accel: True

The base_image now only refers to the image template name, we can also call it flavor, template, image_template, etc.
2. All the new keywords base_image, arch, release (and possibly distro if we add it now) will only be used to override the default value, they are not mandatory and should be used only when needed.
3. By default, we will use a default base_image as defined in the config file, arch will be the same as --arch input arg if available otherwise taken from config file, and release will be automatically detected from item if possible otherwise taken from config file.
4. We don't need any special keywords for release or arch right now, because defaults should be good enough and they are just meant for override. We can add specific keywords when needed.
5. People should always try to list all their requirements in the rpm section and not rely on that tool being available in their base image (outside of @core, I guess). That means the task will work even when run on a different (e.g. minimal) image, just the installation will take longer.
6. Consequently, we're free to tweak the base image in time and add or remove some software, as we see how majority of tasks are run.
7. We will try to keep the number of base image templates available quite low. The default base image can be named "default". We will add new base images only when we really need them, initially we probably have just "default", which will probably be very similar to minimal.
8. When it comes to arch handling, there are way too many places where we define and use arch, and it's getting really complex. There are arches of the item to be tested (for example a koji build is usually available for all primary arches), there is arch of the minion we execute on, there is arch that we pass to runtask and instruct the task to test. And there is possible arch of buildbot slave for non-disposable tests. We would like to simplify this. Maybe we should stop considering buildbot slaves not running in disposable mode, or least forget about them now. Let's no longer support multiple values for runtask --arch. Also, let's also be a slightly less efficient when scheduling tasks for simplicity sake, and support only "arch" or "noarch" mode. In "arch" mode, we schedule one task per every arch available for item, always (even though there are tasks which could theoretically do stuff like test i386+x86_64 in a single run, it's not worth the complications). In "noarch" mode, we schedule the task only once, on a default arch minion (most probably x86_64) and expect the task to test everything in a single pass.
The "arch"/"noarch" mode can be specified e.g. in formula, or somewhere else, like this:

schedule:
    noarch: True

We can then expand this schedule section in the future, example:

schedule:
    on_type: koji_build
    noarch: False
    ignore_arch: armhfp

The combination of these pieces seems to create quite simple yet flexible configuration directly in control of the maintainer:

input:
    args:
        - koji_build
        - arch  
schedule:
    noarch: False
    ignore_arch: armhfp
environment:
#    arch: x86_64

In input section, I receive arch variable which can be used for my task to identify what I'm going to test. In schedule section, I say I want to run separately for every available architecture and omit some specific architecture. And in environment, if I don't include anything (that's why arch is commented out), I'll get a minion matching the input arch, i.e. we'll be running runtask --arch i386 on a i386 minion image, and runtask --arch x86_64 on a x86_64 image. I can, however, override the environment to arch: x86_64 and then all those executions will run on a x86_64 minion, even the --arch i386 one. Of course, I should have a very good reason to do this (e.g. memory limitations) and it should be used sparingly.

This was written in a hurry, sorry, if something is confusing, please ask, I'll try to explain :)

hit enter by mistake, re-editing

! In #731#10223, @kparal wrote:
After some in person discussion, we think it could look something like this:

1. Move the image specification directly into environment key in formula. We imagine further keys will be added there in the future, so eventually it could look like this:
   ```
   environment:
   rpm:
   - rpmlint

   base_image: minimal
   arch: x86_64
   release: 24
   distro: fedora
   memory: 4G
   virt_accel: True
   `` Thebase_imagenow only refers to the image template name, we can also call itflavor,template,image_template`, etc.

I'm not a huge fan of the base_image word choice here and would be very much in favor of changing that.

1. All the new keywords base_image, arch, release (and possibly distro if we add it now) will only be used to override the default value, they are not mandatory and should be used only when needed.
2. By default, we will use a default base_image as defined in the config file, arch will be the same as --arch input arg if available otherwise taken from config file, and release will be automatically detected from item if possible otherwise taken from config file.
3. We don't need any special keywords for release or arch right now, because defaults should be good enough and they are just meant for override. We can add specific keywords when needed.

I'm not sure that we can leave that out for now. One of the use cases that we'll want to support soon is running tasks from the f24 branch on a f24 minion and f23 tasks on a f23 minion. Don't we need something beyond the defaults in order to support this?

1. People should always try to list all their requirements in the rpm section and not rely on that tool being available in their base image (outside of @core, I guess). That means the task will work even when run on a different (e.g. minimal) image, just the installation will take longer.

This sounds good as a "best practice" and I think it's something we should push reasonably hard in docs, tutorials etc.

1. Consequently, we're free to tweak the base image in time and add or remove some software, as we see how majority of tasks are run.
2. We will try to keep the number of base image templates available quite low. The default base image can be named "default". We will add new base images only when we really need them, initially we probably have just "default", which will probably be very similar to minimal.

What would we want to change from "minimal" to "default"? Wouldn't this pretty much be the same image we're making now?

1. When it comes to arch handling, there are way too many places where we define and use arch, and it's getting really complex. There are arches of the item to be tested (for example a koji build is usually available for all primary arches), there is arch of the minion we execute on, there is arch that we pass to runtask and instruct the task to test. And there is possible arch of buildbot slave for non-disposable tests. We would like to simplify this. Maybe we should stop considering buildbot slaves not running in disposable mode, or least forget about them now. Let's no longer support multiple values for runtask --arch. Also, let's also be a slightly less efficient when scheduling tasks for simplicity sake, and support only "arch" or "noarch" mode. In "arch" mode, we schedule one task per every arch available for item, always (even though there are tasks which could theoretically do stuff like test i386+x86_64 in a single run, it's not worth the complications). In "noarch" mode, we schedule the task only once, on a default arch minion (most probably x86_64) and expect the task to test everything in a single pass.

There's a lot of stuff in here and I'm not sure I'm understanding it all, so I'm going to attempt splitting it up into a few sub-topics but I suspect that I've missed a few things.

=== arch handling for scheduling ===

Did we ever come to a conclusion on whether we wanted to run some of the tasks in non-desposable mode? I don't recall a strong reason to make sure that capability stays around for now.

It sounds like the main idea here is to treat arches kind of how koji does - arch-specific stuff is delegated to arch specific builders for each supported arch and noarch stuff is tossed to whatever's available (which would be an alternative to tossing it to the 'default' arch). Did I understand correctly?

How would we determine the "arch-ness" of the task to be scheduled? Would we assume that if a formula doesn't have something akin to arch: noarch that it needs to be run as something arch-specific?

=== arch handling for task execution ===
Depcheck was one of the main use cases behind supporting multiple arches so that we could be more efficient there with the multilib case so we're not downloading as much. How would that check change if we stopped supporting multiple arches from the command line?

The "arch"/"noarch" mode can be specified e.g. in formula, or somewhere else, like this:
schedule: noarch: True
We can then expand this schedule section in the future, example:
schedule: on_type: koji_build noarch: False ignore_arch: armhfp

@mkrizek, any thoughts on how well this would work for the dist-git stuff and any triggering changes we'll be making?

The combination of these pieces seems to create quite simple yet flexible configuration directly in control of the maintainer:
```
input:
args:
- koji_build
- arch

schedule:
noarch: False
ignore_arch: armhfp

environment:

arch: x86_64

`` Ininputsection, I receivearchvariable which can be used for my task to identify what I'm going to test. Inschedulesection, I say I want to run separately for every available architecture and omit some specific architecture. And in environment, if I don't include anything (that's whyarchis commented out), I'll get a minion matching the input arch, i.e. we'll be runningruntask --arch i386on a i386 minion image, andruntask --arch x86_64on a x86_64 image. I can, however, override the environment toarch: x86_64and then all those executions will run on a x86_64 minion, even the--arch i386` one. Of course, I should have a very good reason to do this (e.g. memory limitations) and it should be used sparingly.

I like most of this so far but there's a lot here to digest. My primary concern at the moment is how we'll integrate triggering with all of this if some of the required information is encoded in formula files. If we end up reading the formulas for the dist-git stuff, that's one thing but either way, it's something we need to figure out.

! In #731#10225, @tflink wrote:
I'm not a huge fan of the base_image word choice here and would be very much in favor of changing that.

Patches welcome ;) For real - if you have a better name for anything, I'll be a happy panda. I hate coming up with names...

1. We don't need any special keywords for release or arch right now, because defaults should be good enough and they are just meant for override. We can add specific keywords when needed.

I'm not sure that we can leave that out for now. One of the use cases that we'll want to support soon is running tasks from the f24 branch on a f24 minion and f23 tasks on a f23 minion. Don't we need something beyond the defaults in order to support this?

Maybe there is a slight misunderstanding - the idea is that the release/arch will get inferred from the input, or can be overriden by a value set inside the formula.

1. We will try to keep the number of base image templates available quite low. The default base image can be named "default". We will add new base images only when we really need them, initially we probably have just "default", which will probably be very similar to minimal.

What would we want to change from "minimal" to "default"? Wouldn't this pretty much be the same image we're making now?

The idea here is that we can do "whatever" with our "default", without being bound to it having just the @minimal packageset. Should we ever really need @minimal, we have the "minimal" name free. What is actually part of the "default" image is outside of this conversation, IMO. We need to go through that kickstart, and make it sane, but it's worth it's own ticket.

=== arch handling for scheduling ===

Did we ever come to a conclusion on whether we wanted to run some of the tasks in non-desposable mode? I don't recall a strong reason to make sure that capability stays around for now.

I'm all for running disposable only. Kamil is also, from what I know.

How would we determine the "arch-ness" of the task to be scheduled? Would we assume that if a formula doesn't have something akin to arch: noarch that it needs to be run as something arch-specific?

Precisely.

=== arch handling for task execution ===
Depcheck was one of the main use cases behind supporting multiple arches so that we could be more efficient there with the multilib case so we're not downloading as much. How would that check change if we stopped supporting multiple arches from the command line?

Depcheck can easily be marked as arch:noarch. It does not use the --arch at all, we just plain test both at the same time - which is exactly what arch: noarch is for.

I like most of this so far but there's a lot here to digest. My primary concern at the moment is how we'll integrate triggering with all of this if some of the required information is encoded in formula files. If we end up reading the formulas for the dist-git stuff, that's one thing but either way, it's something we need to figure out.

Yup, stuff is getting complicated, and even more complicated :D These are the fun parts of practically defining the use-cases needed for actual implementation ;)

! In #731#10262, @jskladan wrote:

! In #731#10225, @tflink wrote:
I'm not a huge fan of the base_image word choice here and would be very much in favor of changing that.

Patches welcome ;) For real - if you have a better name for anything, I'll be a happy panda. I hate coming up with names...

Personally, I'd prefer 'image' or 'flavor'

1. We don't need any special keywords for release or arch right now, because defaults should be good enough and they are just meant for override. We can add specific keywords when needed.

I'm not sure that we can leave that out for now. One of the use cases that we'll want to support soon is running tasks from the f24 branch on a f24 minion and f23 tasks on a f23 minion. Don't we need something beyond the defaults in order to support this?

Maybe there is a slight misunderstanding - the idea is that the release/arch will get inferred from the input, or can be overriden by a value set inside the formula.

OK, that sounds good to me.

1. We will try to keep the number of base image templates available quite low. The default base image can be named "default". We will add new base images only when we really need them, initially we probably have just "default", which will probably be very similar to minimal.

What would we want to change from "minimal" to "default"? Wouldn't this pretty much be the same image we're making now?

The idea here is that we can do "whatever" with our "default", without being bound to it having just the @minimal packageset. Should we ever really need @minimal, we have the "minimal" name free. What is actually part of the "default" image is outside of this conversation, IMO. We need to go through that kickstart, and make it sane, but it's worth it's own ticket.

That sounds good to me. I assume that the "default" image would replace the "cloud" image that we're producing?

=== arch handling for scheduling ===

Did we ever come to a conclusion on whether we wanted to run some of the tasks in non-desposable mode? I don't recall a strong reason to make sure that capability stays around for now.

I'm all for running disposable only. Kamil is also, from what I know.

I'm all for reducing complexity. That's something we can look into if it's needed in the future.

How would we determine the "arch-ness" of the task to be scheduled? Would we assume that if a formula doesn't have something akin to arch: noarch that it needs to be run as something arch-specific?

Precisely.

=== arch handling for task execution ===
Depcheck was one of the main use cases behind supporting multiple arches so that we could be more efficient there with the multilib case so we're not downloading as much. How would that check change if we stopped supporting multiple arches from the command line?

Depcheck can easily be marked as arch:noarch. It does not use the --arch at all, we just plain test both at the same time - which is exactly what arch: noarch is for.

How well will that work when we start supporting depcheck on ARM? I suppose that we can deal with that when it becomes an issue, though.

I like most of this so far but there's a lot here to digest. My primary concern at the moment is how we'll integrate triggering with all of this if some of the required information is encoded in formula files. If we end up reading the formulas for the dist-git stuff, that's one thing but either way, it's something we need to figure out.

Yup, stuff is getting complicated, and even more complicated :D These are the fun parts of practically defining the use-cases needed for actual implementation ;)

Hopefully we're getting closer to figuring this out. So long as the bike shed ends up being blue and not red ...