From 0e846ff3cd9d399e20944b6a8cfb7656992fdc40 Mon Sep 17 00:00:00 2001 From: ph0zzy Date: Feb 28 2019 10:01:44 +0000 Subject: add pages with examples --- diff --git a/modules/ROOT/nav.adoc b/modules/ROOT/nav.adoc index e65e045..410fea9 100644 --- a/modules/ROOT/nav.adoc +++ b/modules/ROOT/nav.adoc @@ -1,3 +1,4 @@ * xref:standard-test-interface.adoc[Standard Test Interface] * xref:standard-test-roles.adoc[Standard Test Roles] +* xref:examples.adoc[Examples] * xref:faq.adoc[FAQ] diff --git a/modules/ROOT/pages/examples.adoc b/modules/ROOT/pages/examples.adoc new file mode 100644 index 0000000..c6d96d2 --- /dev/null +++ b/modules/ROOT/pages/examples.adoc @@ -0,0 +1,192 @@ +On this page you can find some inspiration from real-life examples of tests already enabled in the Fedora CI. + +== did == + +For each component it makes sense to enable even the most simple test such as running the binary with `--help` or using an internal smoke test. +Here's an example from the https://src.fedoraproject.org/rpms/did/pull-request/5[did] component: + +[source,ansible] +---- +- hosts: localhost + roles: + - role: standard-test-basic + tags: + - classic + tests: + - smoke: + dir: . + run: did this year --test + required_packages: + - did +---- + +That's it. +As you see above, executing a single command as a test is very easy with the help of the xref:standard-test-roles.adoc#_basic[Basic] role. + +== Python == + +There are multiple versions of Python programming language available in Fedora and a number of related subpackages. +As all of them should be tested (including their various combinatios) we https://fedoraproject.org/wiki/CI/Share_Test_Code[share] test coverage for them in the `tests` namespace: + +* https://src.fedoraproject.org/tests/python[Python tests] + +The test repo contains basic smoke test for virtualenv together with example test https://pagure.io/fedora-ci/metadata[Metadata] stored in the https://fedoraproject.org/wiki/Flexible_Metadata_Format[Flexible Metadata Format]: + +* https://src.fedoraproject.org/tests/python/blob/master/f/main.fmf[main.fmf] +* https://src.fedoraproject.org/tests/python/blob/master/f/smoke/venv.fmf[venv.fmf] + +Once the test is avaible in the share test repository it can be easily linked from supported Python versions: + +Python 2:: +* https://src.fedoraproject.org/rpms/python2/blob/master/f/tests/tests.yml[python2] +* https://src.fedoraproject.org/rpms/python26/blob/master/f/tests/tests.yml[python26] +Python 3:: +* https://src.fedoraproject.org/rpms/python3/blob/master/f/tests/tests.yml[python3] +* https://src.fedoraproject.org/rpms/python34/blob/master/f/tests/tests.yml[python34] +* https://src.fedoraproject.org/rpms/python35/blob/master/f/tests/tests.yml[python35] +* https://src.fedoraproject.org/rpms/python36/blob/master/f/tests/tests.yml[python36] + +We test additional Python implementations as well:: + +* https://src.fedoraproject.org/rpms/pypy/blob/master/f/tests/tests.yml[pypy] +* https://src.fedoraproject.org/rpms/pypy3/blob/master/f/tests/tests.yml[pypy3] + +Plus we ensure that essential tools for venv and virtualnv, such as `setuptools`, `pip` or `virtualenv` itself correctly work with all supported versions: + +* https://src.fedoraproject.org/rpms/python-pip/blob/master/f/tests/tests.yml[python-pip] +* https://src.fedoraproject.org/rpms/python-wheel/blob/master/f/tests/tests.yml[python-wheel] +* https://src.fedoraproject.org/rpms/python-setuptools/blob/master/f/tests/tests.yml[python-setuptools] +* https://src.fedoraproject.org/rpms/python-virtualenv/blob/master/f/tests/tests.yml[python-virtualenv] +* https://src.fedoraproject.org/rpms/python-tox/blob/master/f/tests/tests.yml[python-tox] + +Note that for the last set of examples we run the same test several times with modified environment. +For example: + +[source,ansible] +---- +- smoke36: + dir: python/smoke + run: VERSION=3.6 ./venv.sh +- smoke37: + dir: python/smoke + run: VERSION=3.7 ./venv.sh +- smoke26: + dir: python/smoke + run: VERSION=2.6 METHOD=virtualenv TOX=false ./venv.sh +- smoke27: + dir: python/smoke + run: VERSION=2.7 METHOD=virtualenv ./venv.sh +- smoke34_virtualenv: + dir: python/smoke + run: VERSION=3.4 METHOD=virtualenv ./venv.sh +---- + +In this way we create several virtual test cases from a single test code which prevents duplication and minimizes future maintenance. + +== Shell == + +There are several shells which implement the POSIX specification: bash, ksh, mksh, zsh, dash. +All of them share a significant amount of test coverage and it does not make sense to commit & maintain identical tests in five different repositories (+ possible branches). +Thus we store test code in the `tests` namespace: + +* https://src.fedoraproject.org/tests/shell[Shell tests] + +These tests are then linked from all relevant `tests.yml` files: + +* https://src.fedoraproject.org/rpms/bash/blob/master/f/tests/tests.yml[bash] +* https://src.fedoraproject.org/rpms/ksh/blob/master/f/tests/tests.yml[ksh] +* https://src.fedoraproject.org/rpms/mksh/blob/master/f/tests/tests.yml[mksh] +* https://src.fedoraproject.org/rpms/zsh/blob/master/f/tests/tests.yml[zsh] +* https://src.fedoraproject.org/rpms/dash/blob/master/f/tests/tests.yml[dash] + +https://fedoraproject.org/wiki/Flexible_Metadata_Format[Flexible Metadata Format] filter is used to select appropriate tests instead of listing individual tests manually. +Environment variables `PACKAGES` and `SH_BIN` are used to specify which shell implementation is being tested: + +[source,ansible] +---- +- hosts: localhost + roles: + - role: standard-test-beakerlib + tags: + - classic + repositories: + - repo: "https://src.fedoraproject.org/tests/shell.git" + dest: "shell" + fmf_filter: "tier: 1, 2 & tags: classic" + environment: + PACKAGES: ksh + SH_BIN: ksh + required_packages: + - ksh + - expect # login requires expect + - which # smoke requires which +---- + +Some of the tests might be relevant only for selected components. +This can be handled easily by additional `component` condition: + +[source,ansible] +---- +repositories: +- repo: "https://src.fedoraproject.org/tests/shell.git" + dest: "shell" + fmf_filter: "tier: 1, 2 & component: dash" +---- + +See the https://pagure.io/fedora-ci/metadata[Metadata] page for the full list of so far drafted attributes. + +== SELinux == + +There are several components related to SELinux. +They are tightly connected so change in one of them can cause problems in other. +That's why their tests are shared and executed together: + +* https://src.fedoraproject.org/tests/selinux[SELinux] + +Instead of listing relevant tests to be executed manually in each dist git rpms repository https://fedoraproject.org/wiki/Flexible_Metadata_Format[Flexible Metadata Format] is used: + +[source,ansible] +---- +- hosts: localhost + roles: + - role: standard-test-beakerlib + tags: + - classic + repositories: + - repo: "https://src.fedoraproject.org/tests/selinux.git" + dest: "selinux" + fmf_filter: "tier: 1 | component: selinux-policy" +---- + +Provided `fmf_filter` selects all tests relevant for the `selinux-policy` component plus all Tier 1 selinux tests: + + tier: 1 | component: selinux-policy + +The following six components are covered: + +* https://src.fedoraproject.org/rpms/checkpolicy/blob/master/f/tests/tests.yml[checkpolicy] +* https://src.fedoraproject.org/rpms/libselinux/blob/master/f/tests/tests.yml[libselinux] +* https://src.fedoraproject.org/rpms/libsemanage/blob/master/f/tests/tests.yml[libsemanage] +* https://src.fedoraproject.org/rpms/libsepol/blob/master/f/tests/tests.yml[libsepol] +* https://src.fedoraproject.org/rpms/policycoreutils/blob/master/f/tests/tests.yml[policycoreutils] +* https://src.fedoraproject.org/rpms/selinux-policy/blob/master/f/tests/tests.yml[selinux-policy] + +Use the `fmf` command line tool to quickly check which tests will be scheduled: + + # dnf install -y fmf + # fedpkg clone -a tests/selinux + # cd selinux + # fmf ls --filter "tier: 1 | component: checkpolicy" + /selinux-policy/policy-rpm-macros + /checkpolicy/sedispol + /checkpolicy/checkmodule + /checkpolicy/sedismod + /checkpolicy/checkpolicy + /checkpolicy/checkpolicy-docs + /libsepol/sepol_check_context + /libsemanage/verify-options-in-semanage-conf + /libselinux/getsebool + /policycoreutils/booleans + +See the Flexible Metadata Format documentation for other options how to https://fmf.readthedocs.io/en/latest/overview.html#install[install] fmf. +