This ticket had assigned some Differential requests:
D39

Before getting into what, if anything to do, I'll cover what could be done.

What are your opinions on running functional tests by default? We would add a cmdline option to run only unit tests (e.g. py.test --unit), if technically possible

Yeah, it's possible and wouldn't take much doing. If we do go this route, I'd like to change our other projects to keep consistency.

If we decide to not do this, we should at least add a short cmdline option for --functional, because it's killing me to type it every time (TAB completion doesn't work). I suggest py.test -f.

py.test -f isn't possible due to restrictions in pytest - single letter args are reserved for use by the main executable.

I'd rather keep the default as unit tests only, but I'm not going to fight any changes. As long as we have an easy way to exclude the functional tests from running, I'll go along with the majority on this one.

Tim, why would you like to keep the default as unit tests only?

My idea is that before a commit is pushed or posted for review, the person should always run the full test suite (including the functional tests), even if it takes a few minutes (if it took more, like 15 minutes, it would be a different situation, and we would probably want to use CI for that). Is this how you see it as well? Do you have an experience with some functional tests requiring long minutes to run?

I'd like to have the preferred way of running the test suite really simple. So if we're OK with running just unit tests before committing, let's keep it as it is (maybe just shorten --functional to --func). If we want to require people to run functional tests as well, let's have //that// as default and add --unit.

So it really boils down to experiences from other projects, I guess, which I don't have.

But at the bare minimum, let's use something shorter than --functional, because I really hate to spell it every time without TAB completion :-)

Tim, why would you like to keep the default as unit tests only?

Speed and not worrying about touching the filesystem/db/etc (depending on the code, we obviously don't have a db in libtaskotron). I use a TDD-ish style when writing code and run tests at least every couple of minutes. In my mind, the unit tests are run regularly during dev and the functional tests are only run as integration tests before code review or push or when working on code that is directly affected by those functional tests. Since the unit tests are run more frequently, it makes sense to have them as a default.

My idea is that before a commit is pushed or posted for review, the person should always run the full test suite (including the functional tests), even if it takes a few minutes (if it took more, like 15 minutes, it would be a different situation, and we would probably want to use CI for that). Is this how you see it as well? Do you have an experience with some functional tests requiring long minutes to run?

They can, yes. The bigger problem I've had is with blockerbugs - if you run the functional tests without setting the env properly, they wipe your dev database and you have to resync from bodhi/bugzilla and that can take a while. This should probably be fixed in blockerbugs, though.

I'd like to have the preferred way of running the test suite really simple. So if we're OK with running just unit tests before committing, let's keep it as it is (maybe just shorten --functional to --func). If we want to require people to run functional tests as well, let's have that as default and add --unit.

I can see the logic in this. Like I said, I'm not going to fight it as long as they remain separable and we make the change consistent over all of our active projects.

But at the bare minimum, let's use something shorter than --functional, because I really hate to spell it every time without TAB completion :-)

I guess I don't spell it out as often and tend to use bash history so it hasn't bothered me much. Let's send a RFC out to qa-devel@ to see what other folks think.

Speed and not worrying about touching the filesystem/db/etc (depending on the code, we obviously don't have a db in libtaskotron). I use a TDD-ish style when writing code and run tests at least every couple of minutes. In my mind, the unit tests are run regularly during dev and the functional tests are only run as integration tests before code review or push or when working on code that is directly affected by those functional tests. Since the unit tests are run more frequently, it makes sense to have them as a default.

If that's your usual workflow, it really make sense to have just the unit tests run by default. In my case, I finish my changes and run the test suite only at the end, so I tend to use functional testing more than just unit testing. But your approach makes total sense and is probably more rigorous, so let's support that by default.

I guess I don't spell it out as often and tend to use bash history so it hasn't bothered me much. Let's send a RFC out to qa-devel@ to see what other folks think.

I have difficulties spelling this particular word so I would appreciate something shorter, like --func, but I'm not going to fight for it nor start discussions on qa-devel@ about this, it's too trivial in my view. Closing.

If that's your usual workflow, it really make sense to have just the unit tests run by default. In my case, I finish my changes and run the test suite only at the end, so I tend to use functional testing more than just unit testing. But your approach makes total sense and is probably more rigorous, so let's support that by default.

I have difficulties spelling this particular word so I would appreciate something shorter, like --func, but I'm not going to fight for it nor start discussions on qa-devel@ about this, it's too trivial in my view. Closing.

I wasn't trying to dismiss your concerns, just stating that I hadn't had a problem with it and adding a bit more information on why I had configured things the way that they're configured. That doesn't mean that it can't/shouldn't be changed, just an explanation of why it has remained as is.

I'm OK with any of the following for the functional tests
* --func
* --long

I'm also OK with changing the defaults if it makes sense to others. It won't affect my workflow as long as I can turn off the longer functional tests. I'm not the only person using those tests and if it's easier for more people to have functional tests turned on by default, we should set it up that way