From 63a05b285b8d01a584fba9e8ad861961f4b9c163 Mon Sep 17 00:00:00 2001 From: Petr Šplíchal Date: Oct 20 2021 12:26:03 +0000 Subject: Document gating on selected tmt plans Describe how separate plan reporting is enabled. Add a new 'Using Plans' section to gating. Add a new 'Multiple Plans' section to tmt. Plus fix an obsolete key 'repository'. --- diff --git a/modules/ROOT/pages/gating.adoc b/modules/ROOT/pages/gating.adoc index 5705b33..948d425 100644 --- a/modules/ROOT/pages/gating.adoc +++ b/modules/ROOT/pages/gating.adoc @@ -50,6 +50,20 @@ The following Fedora CI tests can be enabled for gating: See Greenwave's https://docs.pagure.org/greenwave/package-specific-policies.html[Package-specific policies] for more technical details about setting the policy. +== Using Plans == + +If you are using xref:tmt.adoc[tmt] plans it is also possible to enable gating for selected plans only. +Instead of the generic `tier0` type use the name of the desired plan in the resultsdb testcase name: + + !PassingTestCaseRule {test_case_name: fedora-ci.koji-build..functional} + +For example, rule used to enable gating for the `/plans/basic` plan would look like this: + + !PassingTestCaseRule {test_case_name: fedora-ci.koji-build./plans/basic.functional} + +Before the above-mentioned rules can be used, separate plan reporting has to be enabled. +See the xref:tmt.adoc#_multiple_plans[Multiple Plans] section for details. + == Waive == If the failed test result is irrelevant you can waive it using the https://bodhi.fedoraproject.org[Bodhi web interface] or directly from the command line: diff --git a/modules/ROOT/pages/tmt.adoc b/modules/ROOT/pages/tmt.adoc index ffd6150..d71235f 100644 --- a/modules/ROOT/pages/tmt.adoc +++ b/modules/ROOT/pages/tmt.adoc @@ -303,13 +303,45 @@ In this way you can for example easily execute both upstream and fedora tests as discover: - name: fedora how: fmf - repository: https://src.fedoraproject.org/tests/selinux.git + url: https://src.fedoraproject.org/tests/selinux.git - name: upstream how: fmf - repository: https://github.com/SELinuxProject/selinux-testsuite + url: https://github.com/SELinuxProject/selinux-testsuite See also https://github.com/psss/tmt/blob/master/examples/multiple/basic.fmf[multiple config] example in tmt repo to get a better idea. +=== Multiple Plans === + +It is possible to use multiple plans to group relevant tests together or to be able to easily run a subset of tests. +For example, let's have a `/plans/features` plan which covers all functionality tests from the local git repository: + + discover: + how: fmf + execute: + how: tmt + +And a separate `/plans/integration` plan to enable integration testing with another component: + + discover: + how: fmf + url: https://src.fedoraproject.org/rpms/ltrace.git + execute: + how: tmt + +Running all tests from given plan is then very easy: + + tmt run plan --name /plans/features + +When run in the CI, results from such plans are reported as a single resultsdb testcase and are shown in pull requests as a single flag. +In order to enable separate result for each plan, create a `ci.fmf` file in the git repository root with the following content: + + resultsdb-testcase: separate + +Once the separate reporting is enabled, you can turn on gating for selected plans only. +Plan name becomes part of the resultsdb testcase name which is used in the `gating.yaml` config. +See the gating documentation on xref:gating.adoc#_using_plans[Using Plans] for more details. + + === Minimal Path === Here is an example of a minimal test creation path: