From 7bd730bdf19862d14d17d8e4bbff12415b754c77 Mon Sep 17 00:00:00 2001 From: ph0zzy Date: Mar 16 2019 12:16:57 +0000 Subject: add share test code page --- diff --git a/modules/ROOT/nav.adoc b/modules/ROOT/nav.adoc index 649d128..6e5a5ff 100644 --- a/modules/ROOT/nav.adoc +++ b/modules/ROOT/nav.adoc @@ -6,4 +6,5 @@ * xref:standard-test-roles.adoc[Standard Test Roles] * xref:examples.adoc[Examples] * xref:tests.adoc[Tests] +* xref:share-test-code.adoc[Share Test Code] * xref:faq.adoc[FAQ] diff --git a/modules/ROOT/pages/share-test-code.adoc b/modules/ROOT/pages/share-test-code.adoc new file mode 100644 index 0000000..423509e --- /dev/null +++ b/modules/ROOT/pages/share-test-code.adoc @@ -0,0 +1,182 @@ +:toc: + += Share Test Code = + +== Motivation == + +In order to make the CI workflow reliable and efficient it is crucial to keep the test coverage in a good shape at all times. +Sharing test code between several packages (even within multiple branches of the same package) may significantly help to: + +* Prevent test code duplication +* Minimize test maintenance +* Catch incompatibilities early + +In general, tests define how the software works and the basic functionality of many packages doesn’t change that often. +We try hard to keep the backward compatibility where possible. +Thus it seems natural that, for such components, tests guarding the spec could change at a slower pace than the distribution branches. + +See the whole https://lists.fedoraproject.org/archives/list/ci@lists.fedoraproject.org/thread/55U6V6UHA54MJLD2F6JF46EOLMVRUAE7[ci-list discussion] for some more context. + +== Implementation == + +Store test code in your preferred repository and reference the tests from the dist-git yaml file. +There is also a special `tests` namespace dedicated for storing Fedora CI integration tests: + +* https://src.fedoraproject.org/projects/tests + +Use `fedpkg` to quickly clone repositories from the tests namespace: + + fedpkg clone tests/shell + +Some of the xref:standard-test-roles.adoc[Standard Test Roles] (currently basic and beakerlib) support fetching test code from remote repositories directly in their config in this way: + +[source,ansible] +---- +- role: standard-test-beakerlib + repositories: + - repo: "https://src.fedoraproject.org/tests/shell.git" + dest: "shell" +---- + +It is also possible to specify version (branch, commit hash) which should be fetched from the remote repository: + +[source,ansible] +---- +- role: standard-test-beakerlib + repositories: + - repo: "https://src.fedoraproject.org/tests/shell.git" + dest: "shell" + version: "devel" +---- + +== Testing Tests == + +It is a good idea to ensure that updating tests in the shared repository does not negatively impact packages which they are testing. +To enable pull request pipeline for tests stored in the Fedora dist git tests namespace simply include `tests.yml` file in the root of the test repository. + +[source,ansible] +---- +- hosts: localhost + roles: + - role: standard-test-basic + tags: + - classic + tests: + - smoke27: + dir: smoke + run: VERSION=2.7 METHOD=virtualenv ./venv.sh + - smoke36: + dir: smoke + run: VERSION=3.6 METHOD=virtualenv ./venv.sh + - smoke37: + dir: smoke + run: VERSION=3.7 METHOD=virtualenv ./venv.sh + required_packages: + - python27 + - python36 + - python37 +---- + +The example above is a simplified version of the https://src.fedoraproject.org/tests/python/blob/master/f/tests.yml[tests.yml] file from the Python shared test repo and shows how to enable `smoke` test to be executed against three versions of the Python interpreter. + +== Examples == + +Here are some real-life examples where sharing test code can increase long-term efficiency. + +=== Shell === + +There are several shells which implement the POSIX specification: + +* bash +* ksh +* mksh +* zsh +* dash + +All of them share a significant amount of test coverage and it does not make sense to commit & maintain identical tests in five different repositories (+ possible branches). + +Shell tests repository: + +* https://src.fedoraproject.org/tests/shell + +Bash https://src.fedoraproject.org/rpms/bash/blob/master/f/tests/tests.yml[tests.yml]: + +[source,ansible] +---- +- hosts: localhost + roles: + - role: standard-test-beakerlib + tags: + - classic + repositories: + - repo: "https://src.fedoraproject.org/tests/shell.git" + dest: "shell" + tests: + - shell/func + - shell/login + - shell/smoke + required_packages: + - expect # login requires expect + - which # smoke requires which +---- + +Ksh https://src.fedoraproject.org/rpms/ksh/blob/master/f/tests/tests.yml[tests.yml]: + +[source,ansible] +---- +- hosts: localhost + roles: + - role: standard-test-beakerlib + tags: + - classic + repositories: + - repo: "https://src.fedoraproject.org/tests/shell.git" + dest: "shell" + tests: + - shell/func + - shell/login + - shell/smoke + environment: + PACKAGES: ksh + SH_BIN: ksh + required_packages: + - ksh + - expect # login requires expect + - which # smoke requires which +---- + +=== Ruby === + +Another example is Ruby: With about 80 packages related to Ruby on Rails it would be useful and efficient to have a single place for integration tests which verify that the framework is correctly working after updating any of these packages. +Conversely, maintaining those tests in 80 repos would be a tedious task. + +Currently the shared https://src.fedoraproject.org/tests/ruby[tests/ruby] repository hosts these three ruby integration tests: + +Available integration tests: + +* systemtap-static-probes-in-ruby - exercising ruby's systemtap api +* bundler-unit-test - run bundler's unit tests +* run-basic-rails-application - run a simple rails application + +=== SELinux === + +Several SELinux user space components are sharing test coverage in a single https://src.fedoraproject.org/tests/selinux.git[selinux] test repository: + +* https://src.fedoraproject.org/rpms/libsepol/blob/master/f/tests/tests.yml[libsepol] +* https://src.fedoraproject.org/rpms/libselinux/blob/master/f/tests/tests.yml[libselinux] +* https://src.fedoraproject.org/rpms/libsemanage/blob/master/f/tests/tests.yml[libsemanage] +* https://src.fedoraproject.org/rpms/libsepol/blob/master/f/tests/tests.yml[policycoreutils] + +== Start == + +In order to create a new repository in the tests namespace use the fedpkg's request-tests-repo command. +For example to create a shared test repository with the name foo, which will be available at https://src.fedoraproject.org/tests/foo.git + +* Setup authentication to pagure according to the help in request-repo command + + fedpkg request-repo -h + +* Request a new repository with a sensible decription + + fedpkg --module-name foo request-tests-repo "Description of the foo repository" +