From 0b957ac85525fb1151ce00a8c1cc39eb582dd21e Mon Sep 17 00:00:00 2001 From: Aleksandra Fedorova Date: Nov 29 2021 17:42:14 +0000 Subject: [PATCH 1/5] Add warning to pipeline page --- diff --git a/modules/ROOT/pages/pipeline.adoc b/modules/ROOT/pages/pipeline.adoc index 8d62d57..14aa615 100644 --- a/modules/ROOT/pages/pipeline.adoc +++ b/modules/ROOT/pages/pipeline.adoc @@ -1,11 +1,13 @@ +WARNING: This page is outdated. + :toc: -= Pipeline = +== Pipeline == The testing *Pipeline* detects tests for enabled packages, executes the test coverage and gathers the results. Currently version `1.1.0` of the xref:standard-test-interface.adoc[Standard Test Interface] specification is supported. -== Instances == +=== Instances === There are several CI https://jenkins-continuous-infra.apps.ci.centos.org/blue/pipelines/[pipelines] enabled in the CentoOS Jenkins: @@ -22,13 +24,13 @@ They are for Fedora Rawhide, current and pending releases as reported by https:/ ** Other releases example: *** https://jenkins-continuous-infra.apps.ci.centos.org/job/fedora-f32-image-test/lastSuccessfulBuild/artifact/Fedora-32.qcow2 -== Reschedule == +=== Reschedule === In order to manually create a new job in the pipeline (e.g. to execute the tests again because of an infrastructure error) add the following comment to the pull request: [citest] -== Links == +=== Links === To learn more about the pipeline visit following links: @@ -39,9 +41,9 @@ To learn more about the pipeline visit following links: * https://fedoraproject.org/wiki/Fedora_requirements_for_CI_and_CD[Fedora requirements for CI and CD] * https://jenkins-continuous-infra.apps.ci.centos.org[CI-Pipeline instance in Centos CI] -== Examples == +=== Examples === -=== Commit === +==== Commit ==== Testing results appear as green or red dots directly in the Pagure interface. Clicking on them will bring you to result details. @@ -50,7 +52,7 @@ Clicking on them will bring you to result details. image::Pipeline-results.png[] -=== Pull Request === +==== Pull Request ==== For pull requests you can find test results in the right tab of the pull request page, for example: https://src.fedoraproject.org/rpms/python-virtualenv/pull-request/3[python-virtualenv] From 96c039b57b971ee5d3c07b37eee0093dbe2d332c Mon Sep 17 00:00:00 2001 From: Aleksandra Fedorova Date: Nov 29 2021 17:42:46 +0000 Subject: [PATCH 2/5] Rename how to --- diff --git a/modules/ROOT/pages/how-to-add-dist-git-test.adoc b/modules/ROOT/pages/how-to-add-dist-git-test.adoc deleted file mode 100644 index 32c3de8..0000000 --- a/modules/ROOT/pages/how-to-add-dist-git-test.adoc +++ /dev/null @@ -1,214 +0,0 @@ -:toc: - -= How to add simple dist-git test for a package = - -== Test as a single command == -=== Task === - -There is a package *somepackage*, which contains a binary: -*/usr/bin/somebinary* - -The most simple and obvious way to test this binary is to run it: -*somebinary --help* and check the exit status of the result. - -How to add this test for the package? - -https://docs.fedoraproject.org/en-US/ci/standard-test-roles/[Standard -Test Roles] framework provides solution for the task. It is fully -supported in Fedora Rawhide. - -=== Solution === - -Create file `tests/tests.yml` in the dist-git of the package: - -.rpms/somepackage.git: ----- -. -├── 0001-something.patch -├── somepackage.spec -├── sources -└── tests - └── tests.yml ----- - -*tests.yml* is an ansible playbook, where you describe test environment -and steps to run your tests. - -There are many options and -https://fedoraproject.org/wiki/CI/Examples[examples available], but -let's focus on the task at hand: we want to run just one command. - -For this case the content of the file should look as follows: - -.rpms/somepackage.git:tests/tests.yml -[source,yaml] ----- -- hosts: localhost - roles: - - role: standard-test-basic // <1> - tags: - - classic - tests: - - simple: - dir: . - run: "somebinary --help" // <2> ----- -<1> this is a standard test role, it takes care of the test environment, logging, archiving results, etc -<2> this is your test command, its exit code defines the outcome of the test - -Submit your change as a pull request to see test results in Pagure -interface, or push it directly to dist-git and get new test results -every time you build package in Koji. - -NOTE: Test will be running in a *non-blocking* mode until you configure -gating for it. - -== Tests in a (sub)package == - -=== Task === - -There is a package `somepackage`, and there is an integration test suite -for it packaged in a separate `sometests` package, which provides the -`run_some_tests` binary in the system path. - -There goal is to trigger this binary as a test for a main package. - -=== Solution === - -Same as above, you need to create a `tests.yml` configuration file with -the following content: - -.rpms/somepackage.git:tests/tests.yml -[source,yaml] ----- ---- -- hosts: localhost - roles: - - role: standard-test-basic - tags: - - classic - required_packages: - - sometests # <1> - - tests: - - integration_tests: # <2> - dir: . - run: run_some_tests # <3> ----- -<1> additional package which needs to be installed in the test environment -<2> any string, will be used as identifier for artifacts and test results -<3> test execution command - -== Tests in the source tarball == - - -== Tests in external repository == - -=== Task === - -Let's look into slightly more complicated setup now. - -Suppose there is a package `somepackage` which we are going to -test. There is an integration test suite for it, which is (sadly) not -yet packaged and located in a separate git repository -`https://somewhere/sometests.git`. Test suite has a dependency on some -packaged tool `sometool`. And in test repository there is a -`run_some_tests` script which triggers test execution. - -There goal is to trigger the execution of the test suite for a package. - -=== Solution === - -We need to create a `tests.yml` configuration file with the following -content: - -.rpms/somepackage.git:tests/tests.yml -[source,yaml] ----- ---- -- hosts: localhost - roles: - - role: standard-test-basic # <1> - tags: - - classic - - required_packages: - - sometool # <2> - - repositories: - - repo: "https://somewhere/sometests.git" # <3> - dest: "sometests" # <4> - - tests: - - integration_tests: # <5> - dir: "sometests" # <6> - run: "run_some_tests --all" # <7> ----- -<1> same basic test role as usual -<2> additional package which needs to be installed in the test environment -<3> path to remote git repository -<4> local path where the repository will be checked out -<5> any string, will be used as identifier for artifacts and test results -<6> same folder as in `<4>`, contains the checked out external repository -<7> test execution command - -== Questions == - -=== What if I want to run not one but a sequence of commands? === - -Put a bash script in **tests/scripts/** folder and run it from the -playbook. - -.rpms/somepackage.git: ----- -. -├── 0001-something.patch -├── somepackage.spec -├── sources -└── tests - ├── scripts - │ └── run_tests.sh # <1> - └── tests.yml ----- -<1> your custom test scenario - -Configure dist-git test to run this script: -[source,yaml] -.... -- hosts: localhost - roles: - - role: standard-test-basic # <1> - tags: - - classic - tests: - - simple: - dir: scripts # <2> - run: ./run_tests.sh # <3> -.... -<1> same standard role -<2> switch to subfolder (path is relative to `tests/` folder) -<3> this is the test script, its exit code is the outcome of the test - -=== What is under the hood? === - -To test the build we: - -* checkout dist-git repo -* take latest qcow image of Fedora Rawhide -* install all packages from the koji build on it -* run ansible playbook defined in tests.yml - -=== How do I verify my configuration? === - -It is possible to run and debug standard test roles locally. But we -highly recommend to use the pull-request workflow for it: simply create -a pull-request and wait for CI to react on it. - -We trigger almost the same CI machinery for PR testing as it is used for -gating of new builds. - -And as soon as result of the test is ready, it will appear on the pull -request page in http://src.fedoraproject.org/[Fedora Pagure] - -To restart the test add a comment to PR in Pagure, with the following -content: `[citest]` diff --git a/modules/ROOT/pages/how-to-add-sti-test.adoc b/modules/ROOT/pages/how-to-add-sti-test.adoc new file mode 100644 index 0000000..192a81c --- /dev/null +++ b/modules/ROOT/pages/how-to-add-sti-test.adoc @@ -0,0 +1,214 @@ +:toc: + += How to add simple STI test for a package = + +== Test as a single command == +=== Task === + +There is a package *somepackage*, which contains a binary: +*/usr/bin/somebinary* + +The most simple and obvious way to test this binary is to run it: +*somebinary --help* and check the exit status of the result. + +How to add this test for the package? + +https://docs.fedoraproject.org/en-US/ci/standard-test-roles/[Standard +Test Roles] framework provides solution for the task. It is fully +supported in Fedora Rawhide. + +=== Solution === + +Create file `tests/tests.yml` in the dist-git of the package: + +.rpms/somepackage.git: +---- +. +├── 0001-something.patch +├── somepackage.spec +├── sources +└── tests + └── tests.yml +---- + +*tests.yml* is an ansible playbook, where you describe test environment +and steps to run your tests. + +There are many options and +https://fedoraproject.org/wiki/CI/Examples[examples available], but +let's focus on the task at hand: we want to run just one command. + +For this case the content of the file should look as follows: + +.rpms/somepackage.git:tests/tests.yml +[source,yaml] +---- +- hosts: localhost + roles: + - role: standard-test-basic // <1> + tags: + - classic + tests: + - simple: + dir: . + run: "somebinary --help" // <2> +---- +<1> this is a standard test role, it takes care of the test environment, logging, archiving results, etc +<2> this is your test command, its exit code defines the outcome of the test + +Submit your change as a pull request to see test results in Pagure +interface, or push it directly to dist-git and get new test results +every time you build package in Koji. + +NOTE: Test will be running in a *non-blocking* mode until you configure +gating for it. + +== Tests in a (sub)package == + +=== Task === + +There is a package `somepackage`, and there is an integration test suite +for it packaged in a separate `sometests` package, which provides the +`run_some_tests` binary in the system path. + +There goal is to trigger this binary as a test for a main package. + +=== Solution === + +Same as above, you need to create a `tests.yml` configuration file with +the following content: + +.rpms/somepackage.git:tests/tests.yml +[source,yaml] +---- +--- +- hosts: localhost + roles: + - role: standard-test-basic + tags: + - classic + required_packages: + - sometests # <1> + + tests: + - integration_tests: # <2> + dir: . + run: run_some_tests # <3> +---- +<1> additional package which needs to be installed in the test environment +<2> any string, will be used as identifier for artifacts and test results +<3> test execution command + +== Tests in the source tarball == + + +== Tests in external repository == + +=== Task === + +Let's look into slightly more complicated setup now. + +Suppose there is a package `somepackage` which we are going to +test. There is an integration test suite for it, which is (sadly) not +yet packaged and located in a separate git repository +`https://somewhere/sometests.git`. Test suite has a dependency on some +packaged tool `sometool`. And in test repository there is a +`run_some_tests` script which triggers test execution. + +There goal is to trigger the execution of the test suite for a package. + +=== Solution === + +We need to create a `tests.yml` configuration file with the following +content: + +.rpms/somepackage.git:tests/tests.yml +[source,yaml] +---- +--- +- hosts: localhost + roles: + - role: standard-test-basic # <1> + tags: + - classic + + required_packages: + - sometool # <2> + + repositories: + - repo: "https://somewhere/sometests.git" # <3> + dest: "sometests" # <4> + + tests: + - integration_tests: # <5> + dir: "sometests" # <6> + run: "run_some_tests --all" # <7> +---- +<1> same basic test role as usual +<2> additional package which needs to be installed in the test environment +<3> path to remote git repository +<4> local path where the repository will be checked out +<5> any string, will be used as identifier for artifacts and test results +<6> same folder as in `<4>`, contains the checked out external repository +<7> test execution command + +== Questions == + +=== What if I want to run not one but a sequence of commands? === + +Put a bash script in **tests/scripts/** folder and run it from the +playbook. + +.rpms/somepackage.git: +---- +. +├── 0001-something.patch +├── somepackage.spec +├── sources +└── tests + ├── scripts + │ └── run_tests.sh # <1> + └── tests.yml +---- +<1> your custom test scenario + +Configure dist-git test to run this script: +[source,yaml] +.... +- hosts: localhost + roles: + - role: standard-test-basic # <1> + tags: + - classic + tests: + - simple: + dir: scripts # <2> + run: ./run_tests.sh # <3> +.... +<1> same standard role +<2> switch to subfolder (path is relative to `tests/` folder) +<3> this is the test script, its exit code is the outcome of the test + +=== What is under the hood? === + +To test the build we: + +* checkout dist-git repo +* take latest qcow image of Fedora Rawhide +* install all packages from the koji build on it +* run ansible playbook defined in tests.yml + +=== How do I verify my configuration? === + +It is possible to run and debug standard test roles locally. But we +highly recommend to use the pull-request workflow for it: simply create +a pull-request and wait for CI to react on it. + +We trigger almost the same CI machinery for PR testing as it is used for +gating of new builds. + +And as soon as result of the test is ready, it will appear on the pull +request page in http://src.fedoraproject.org/[Fedora Pagure] + +To restart the test add a comment to PR in Pagure, with the following +content: `[citest]` From 477ff382b500596b34a306a871747391cc727b9f Mon Sep 17 00:00:00 2001 From: Aleksandra Fedorova Date: Nov 29 2021 19:18:56 +0000 Subject: [PATCH 3/5] Rearrange pages in the nav bar * Everything related to STI tests is now moved under the STI tests subtree. * TMT interface gets the same visibility as STI. * Architecture and Guides sections removed, the content is lifted one level up --- diff --git a/modules/ROOT/nav.adoc b/modules/ROOT/nav.adoc index 1e8a447..f0d8d59 100644 --- a/modules/ROOT/nav.adoc +++ b/modules/ROOT/nav.adoc @@ -1,24 +1,31 @@ -* Guides -** xref:how-to-add-dist-git-test.adoc[How to add simple dist-git test] -** xref:quick-start-guide.adoc[Quick Start Guide] -** xref:onboarding-of-a-ci-system.adoc[Onboarding of a CI System] -** xref:pull-requests.adoc[Pull Requests] -** xref:share-test-code.adoc[Share Test Code] +* xref:manifesto.adoc[Manifesto] +* xref:pull-requests.adoc[Pull Requests] +* xref:gating.adoc[Gating] + +* Tests + ** xref:generic_tests.adoc[Generic Tests] -** xref:examples.adoc[Examples] -* Architecture -** xref:manifesto.adoc[Manifesto] -** xref:standard-test-interface.adoc[Standard Test Interface] -** xref:standard-test-roles.adoc[Standard Test Roles] -** xref:tmt.adoc[Test Management Tool] -** xref:tests.adoc[Tests] -** xref:pipeline.adoc[Pipeline] -** xref:gating.adoc[Gating] + +** STI tests +*** xref:standard-test-interface.adoc[Standard Test Interface] +*** xref:standard-test-roles.adoc[Standard Test Roles] +*** xref:how-to-add-sti-test.adoc[How to add simple STI test] +*** xref:quick-start-guide.adoc[STI Quick Start Guide] +*** xref:tests.adoc[Writing STI Tests] +*** xref:examples.adoc[STI Examples] +*** xref:share-test-code.adoc[How to share Test Code] + +** tmt tests +*** xref:tmt.adoc[Test Management Tool] + * Infrastructure ** xref:jenkins.adoc[Jenkins] -* Maintenance ** xref:rebuild-container-image.adoc[Rebuild container images] +** xref:pipeline.adoc[Pipeline] +** xref:onboarding-of-a-ci-system.adoc[Onboarding of a CI System] + * More ** xref:test-case-relevancy.adoc[Test Case Relevancy] ** xref:source-git.adoc[Source Git] + * xref:faq.adoc[FAQ] From 73ff2edcbaf9368ea86c4a62992a9e82395c48c1 Mon Sep 17 00:00:00 2001 From: Aleksandra Fedorova Date: Nov 30 2021 13:13:34 +0000 Subject: [PATCH 4/5] Fix headings in pipeline.adoc The main level-0 header should be on top of the page. --- diff --git a/modules/ROOT/pages/pipeline.adoc b/modules/ROOT/pages/pipeline.adoc index 14aa615..c60b2a3 100644 --- a/modules/ROOT/pages/pipeline.adoc +++ b/modules/ROOT/pages/pipeline.adoc @@ -1,13 +1,13 @@ += Pipeline = + WARNING: This page is outdated. :toc: -== Pipeline == - The testing *Pipeline* detects tests for enabled packages, executes the test coverage and gathers the results. Currently version `1.1.0` of the xref:standard-test-interface.adoc[Standard Test Interface] specification is supported. -=== Instances === +== Instances == There are several CI https://jenkins-continuous-infra.apps.ci.centos.org/blue/pipelines/[pipelines] enabled in the CentoOS Jenkins: @@ -24,13 +24,13 @@ They are for Fedora Rawhide, current and pending releases as reported by https:/ ** Other releases example: *** https://jenkins-continuous-infra.apps.ci.centos.org/job/fedora-f32-image-test/lastSuccessfulBuild/artifact/Fedora-32.qcow2 -=== Reschedule === +== Reschedule == In order to manually create a new job in the pipeline (e.g. to execute the tests again because of an infrastructure error) add the following comment to the pull request: [citest] -=== Links === +== Links == To learn more about the pipeline visit following links: @@ -41,9 +41,9 @@ To learn more about the pipeline visit following links: * https://fedoraproject.org/wiki/Fedora_requirements_for_CI_and_CD[Fedora requirements for CI and CD] * https://jenkins-continuous-infra.apps.ci.centos.org[CI-Pipeline instance in Centos CI] -=== Examples === +== Examples == -==== Commit ==== +=== Commit === Testing results appear as green or red dots directly in the Pagure interface. Clicking on them will bring you to result details. @@ -52,7 +52,7 @@ Clicking on them will bring you to result details. image::Pipeline-results.png[] -==== Pull Request ==== +=== Pull Request === For pull requests you can find test results in the right tab of the pull request page, for example: https://src.fedoraproject.org/rpms/python-virtualenv/pull-request/3[python-virtualenv] From 436a3167e215bb4cc3a0b331bb1a6773dfdb67eb Mon Sep 17 00:00:00 2001 From: Aleksandra Fedorova Date: Nov 30 2021 13:35:31 +0000 Subject: [PATCH 5/5] Move tmt tests one level up --- diff --git a/modules/ROOT/nav.adoc b/modules/ROOT/nav.adoc index f0d8d59..648fd35 100644 --- a/modules/ROOT/nav.adoc +++ b/modules/ROOT/nav.adoc @@ -15,8 +15,7 @@ *** xref:examples.adoc[STI Examples] *** xref:share-test-code.adoc[How to share Test Code] -** tmt tests -*** xref:tmt.adoc[Test Management Tool] +** xref:tmt.adoc[tmt tests] * Infrastructure ** xref:jenkins.adoc[Jenkins]