From ffd0dd8d476f9bee7b95d8d4367a135d0baf6936 Mon Sep 17 00:00:00 2001 From: Irina Gulina Date: Jun 14 2017 07:39:48 +0000 Subject: [PATCH 1/7] Simple README with what MTF is and a link to docs --- diff --git a/README.md b/README.md new file mode 100644 index 0000000..f9705f7 --- /dev/null +++ b/README.md @@ -0,0 +1,5 @@ +Modularity testing framework (MTF) is a tool to test [Fedora modules](https://github.com/modularity-modules). + +For more information, check out the documentation page: + +http://modularity-testing-framework.readthedocs.io diff --git a/docs/example-config-minimal.yaml b/docs/example-config-minimal.yaml deleted file mode 100644 index d079c36..0000000 --- a/docs/example-config-minimal.yaml +++ /dev/null @@ -1,15 +0,0 @@ -document: modularity-testing -version: 1 -name: bash -modulemd-url: https://src.fedoraproject.org/cgit/modules/memcached.git/plain/memcached.yaml -packages: - rpms: - - bash - - rpm -module: - docker: - container: docker.io/phracek/memcached - rpm: - repo: https://kojipkgs.stg.fedoraproject.org/compose/branched/jkaluza/latest-Fedora-Modular-26/compose/Server/x86_64/os/ - - diff --git a/docs/howtowriteyamlconf.md b/docs/howtowriteyamlconf.md deleted file mode 100644 index 9403361..0000000 --- a/docs/howtowriteyamlconf.md +++ /dev/null @@ -1,85 +0,0 @@ -## Document identification -There is nothing important, just parser inside check if config yaml is the proper one, that nobody for example does not used modulemd.yaml as config.yaml -```yaml -document: modularity-testing -version: 1 -``` - -## Module generic part -This part contain generic part of module -```yaml -name: memcached -modulemd-url: http://raw.githubusercontent.com/container-images/memcached/master/memcached.yaml -compose-url: url_to_compose_in done in fedora -service: - port: 11211 -packages: - rpms: - - memcached - - perl-Carp -testdependencies: - rpms: - - nc -``` - * `name:` name of module - * `modulemd-url:` link to modulemd file, now it is not used anyhow, just for installing packages for proper module - * `compose-url:` final compose build (done by pungi) it contains repositories + moduleMD infromations for tooling - * `service:` In case module is service like memcached, store there port number, can be then used in tests, to not hardcode port number *(Optional)* - * `packages:` Which packages will be installed inside module (docker container, guest, any type of module) - * `testdependencies:` Install dependencies on host, what are important for module testing, for example when you would like to use `nc`, you have to install it explicitly, it is not in cloud images. - -## Module types specification -It contains specification for each type of module, now for __rpm__ and __docker__ based modules -```yaml -module: - docker: - setup: echo Do magic with general config stored on host; - echo More magic - cleanup: echo Cleanup magic - start: "docker run -it -e CACHE_SIZE=128 -p 11211:11211" - labels: - description: "memcached is a high-performance, distributed memory" - io.k8s.description: "memcached is a high-performance, distributed memory" - source: https://github.com/container-images/memcached.git - container: docker.io/phracek/memcached - rpm: - setup: echo Do magic with general config stored on host; - echo More magic - cleanup: echo Cleanup magic - start: systemctl start memcached - stop: systemctl stop memcached - status: systemctl status memcached - repos: - - http://download.englab.brq.redhat.com/pub/fedora/releases/25/Everything/x86_64/os/ - - https://phracek.fedorapeople.org/memcached-module-repo/ -``` - * `setup:` run setup/cleanup commands on HOST, for example config manipulation, selinux boolean manipulation there could be used also variables in python style like: {ROOT}, {HOSTNAME} see trans_dict in file https://pagure.io/modularity-testing-framework/blob/master/f/moduleframework/module_framework.py - * `cleanup:` similar to setup but done in after test finished. - * `start:` how to start service in case it is service, in case of generic module it is *(Optional)* - * `stop:` how to service service in case it is service, in case of generic module it is *(Optional)* - * `status:` how to check service state, in case of generic module it is *(Optional)* - * `labels:` docker labels to check, specific just for *docker* container *(Docker specific)* - * `container:` where is link to container, now it support docker.io link or using locally tar.gz file specified *(Docker specific)* - * `repo:` if *compose* is not set then this is used and contains repo what has to be used for this module (dependent repos are searched in PDC via moduleMD definition) *(Rpm specific)* - * `repos:` if *compose* and *repo* not set: contains all repos what has to be used for this module (typically baseruntime + specific one) *(Rpm specific)* *(OBSOLOTE)* - -## Simple tests inside config - This part is little but __controversial__ , some of users are fans and some hates this. It allows you to specify bash style tests directly inside config file -```yaml -test: - processrunning: - - 'ls /proc/*/exe -alh | grep memcached' -testhost: - selfcheck: - - 'echo errr | nc localhost 11211' - - 'echo set AAA 0 4 2 | nc localhost 11211' - - 'echo get AAA | nc localhost 11211' - selcheckError: - - 'echo errr | nc localhost 11211 |grep ERROR' -``` - * `test:` tests what will run inside container - it means that there has to be all dependencies for these test *(Optional)* - * every command has to finish with __0 return code__ otherwise it will __fail__ - * next level like __processrunning__ is test name what will be visible on output of avocado run, then all lines will be run as commands for this test - * `testhost:` it is similar to *test*, just difference is that it runs commands on host machine so that there could be more dependencies than just are in module. I', not sure if this part is useful, will see after discussion *(Optional)* - * other specification is same as `test` - * you have to call `mtf-generator` binary to generate python files from that (because unittests does not allow to have dynamically created tests) diff --git a/docs/installation.rst b/docs/installation.rst new file mode 100644 index 0000000..7556872 --- /dev/null +++ b/docs/installation.rst @@ -0,0 +1,161 @@ +Modularity-testing-framework +============================ + +Recent documentation on: + http://modularity-testing-framework.readthedocs.io + +Modularity prototype testing +---------------------------- + +- General Structure + - **YAML Config file** + - Each module will need to have yaml config file + - Config file should cover non generic part - part focused on general module testing + - It could contain also simple test + - **how to write config file** https://pagure.io/modularity-testing-framework/blob/master/f/docs/howtowriteyamlconf.md + + - **Self generated tests** + - there is possible to write simple tests directly in yaml config file + - Bash style testing + - It has to have solved dependencies inside each module type + - Now it just expect to end with *0* return code of command (like: *ls / |grep sbin* directory sbin exists in root dir) + - It can contain multiple lines + - It generates python covered bash tests + - You has to call `mtf-generator` binary explicitly, it then create these pythonish tests with bash inside, *Unittest* doesn not allow to have dynamic tests. + + - **Avocado tests** + - There is wrapper class what helps you to tests modules not focusing on module type + - It uses avocado-framework + - This test could be primarily used for more complex testing, not previous one + - General test for modules: *./base/modulelint.py* + - Example tests in *memcached* module + + - **Simple bash tests** + - There is helper what you can use for writing *bash* like tests + - library is https://pagure.io/modularity-testing-framework/blob/master/f/moduleframework/bashhelper.py and it is installed as *moduleframework-cmd* command in */usr/bin* + - Test has to call setup and cleanup of module explicitly + - These tests are dependent on return code of commands in test, so in case you have more tests subtest, just count return codes + - see example test for https://pagure.io/modularity-testing-framework/blob/master/f/examples/memcached/sanity2.sh + + + - **WIP: Behave tests** + - You can write tests for you module also in behave style + - it is first prototype + - see example in https://pagure.io/modularity-testing-framework/blob/master/f/examples/memcached-behave + +Running using VAGRANT +--------------------- +- install vagrant *dnf -y install vagrant* +- just run *vagrant up* + +Installation +------------ +- base dependencies: **docker python-pip** +- python dependencies: **avocado-framework yaml json behave** + +Stable version +~~~~~~~~~~~~~~ +- It is built as package for fedora (>26) +- stable COPR repo + - https://copr.fedorainfracloud.org/coprs/phracek/Modularity-testing-framework/ + - `dnf copr enable phracek/Modularity-testing-framework` + - `dnf install -y modularity-testing-framework` + +Development version +~~~~~~~~~~~~~~ +- Automatically built packages + - https://copr.fedorainfracloud.org/coprs/jscotka/modularity-testing-framework/ + + + +Enviromental variables +---------------------- +- variables allows you to overwrite some values inside *config.yaml* +- *AVOCADO_LOG_DEBUG=yes* enables avocado debug output - in case you find some strange hard to debug issues +- *DEBUG* Enable debugging output to test output +- *CONFIG* file with MTF configuration default is *config.yaml* +- *MODULE* which module type to test (in case there is not set *default-module* in config, you **HAVE TO** set it) + - *=docker* uses section *docker* inside config file and will use docker containerisation + - *=nspawn* systemd nspawn, it is lightweight virtualization, it does something like **MOCK** but it is not just chroot, but has own systemd etc. + - *=rpm* testing of local RPM packages directly on HOST (it could be **DESTRUCTIVE**) + +- *URL* see example config. It overwrites value *module.docker.container* or *module.rpm.repo* to whatever you want. It has to be proper type what is set in *MODULE* +- *MODULEMDURL* overwrite location of moduleMD file +- *COMPOSEURL* overwrite location of compose repo location +- *PROFILE* overwrite *default* profile to whatever you want to install instead of that +- *MTF_SKIP_DISABLING_SELINUX* In nspawn type on fedora-25 we have to disable selinux, because it does not work well with selinux enabled, this option allows to not do that. +- *MTF_DO_NOT_CLEANUP* Do not cleanup modules between tests, in case there is no interference in your tests you can use it, and it will be **fast** +- *MTF_REMOTE_REPOS* It disables downloading of packages done by koji and creating local repo, it make tests **fast**. (There is issue that composes (repos) are sometimes bad in fedora, unable to use) + + + +Schedule Tests +-------------- +- Now it is expected to run this **under root** +- Install modularity-testing-framework from COPR repo like: + - **dnf copr enable phracek/Modularity-testing-framework** + - install it by command: **dnf install -y modularity-testing-framework** + - It installs packages to python site-packages and to /usr/share/moduleframework +- To include tests into your module, add to your Makefile section **test** +- **test** section runs another Makefile in directory **tests** +- Your **Makefile** should contain: + - **Docker based module testing:** `cd tests; MODULE=docker make all` + - **Repo based module testing:** `MODULE=nspawn make all ` + - **Host Rpm based module testing:** `MODULE=rpm make all` + +- Makefile in tests directory looks like: + + $ cat tests/Makefile + MODULE_LINT=/usr/share/moduleframework/tools/modulelint.py + CMD=python -m avocado run --filter-by-tags=-WIP $(MODULE_LINT) *.py + + # + all: $(CMD) + + - **Makefile in MTF** https://pagure.io/modularity-testing-framework/blob/master/f/examples/testing-module/Makefile + +- `make check` - runs tests in your module directory + + +How to write tests +------------------ +- **minimal path creation** +- Install modularity-testing-framework from COPR repo like: + - *dnf copr enable phracek/Modularity-testing-framework* + - install it by command: *dnf install -y modularity-testing-framework* +- CREATE your config.yaml (see example https://pagure.io/modularity-testing-framework/blob/master/f/docs/example-config.yaml) +- If you have tests in config file call: */usr/bin/mtf-generator* or simply *mtf-generator*. +- Call command for running all python tests: **MODULE=docker avocado run /usr/share/moduleframework/tools/modulelint.py ./*.py** +- **additional tests** - see tests in https://pagure.io/modularity-testing-framework/blob/master/f/examples/testing-module directory as an example for you + +License +------- +Framework is released under the GPL, version 2 or later, see LICENSE file in project + + +Debugging & How To +------------------ + +First test takes so long time +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +It is normal, because first test downloads all packages from koji and creates localrepo. +It is workaround because of missing composes for modules (on demand done by pungi) +- If you would like to make it faster use env variables: + - *MTF_REMOTE_REPOS=yes* - It heps in case there are repos in koji https://kojipkgs.fedoraproject.org/repos/ (they are there just temporary, deleted after 2 weeks and probably it will not be created in near future anyhow) + - *MTF_DO_NOT_CLEANUP=yes* - in case it is still slow for you, it disable test cleaup (between tests) could cause side effects + +Unable to debug avocado output errors +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +- If you see error like: *Avocado crashed: TestError: Process died before it pushed early test_status.* + - add env variables: + - *AVOCADO_LOG_DEBUG=yes* + - *DEBUG=yes* + - **This is preferred variant for submitting issues to pagure** + + +How it works +------------ +- Structure of MTF: + - https://pagure.io/modularity-testing-framework/blob/master/f/docs/howitworks.png +- Test types: + - https://pagure.io/modularity-testing-framework/blob/master/f/docs/TestTypes.png diff --git a/docs/overview.rst b/docs/overview.rst deleted file mode 100644 index 7556872..0000000 --- a/docs/overview.rst +++ /dev/null @@ -1,161 +0,0 @@ -Modularity-testing-framework -============================ - -Recent documentation on: - http://modularity-testing-framework.readthedocs.io - -Modularity prototype testing ----------------------------- - -- General Structure - - **YAML Config file** - - Each module will need to have yaml config file - - Config file should cover non generic part - part focused on general module testing - - It could contain also simple test - - **how to write config file** https://pagure.io/modularity-testing-framework/blob/master/f/docs/howtowriteyamlconf.md - - - **Self generated tests** - - there is possible to write simple tests directly in yaml config file - - Bash style testing - - It has to have solved dependencies inside each module type - - Now it just expect to end with *0* return code of command (like: *ls / |grep sbin* directory sbin exists in root dir) - - It can contain multiple lines - - It generates python covered bash tests - - You has to call `mtf-generator` binary explicitly, it then create these pythonish tests with bash inside, *Unittest* doesn not allow to have dynamic tests. - - - **Avocado tests** - - There is wrapper class what helps you to tests modules not focusing on module type - - It uses avocado-framework - - This test could be primarily used for more complex testing, not previous one - - General test for modules: *./base/modulelint.py* - - Example tests in *memcached* module - - - **Simple bash tests** - - There is helper what you can use for writing *bash* like tests - - library is https://pagure.io/modularity-testing-framework/blob/master/f/moduleframework/bashhelper.py and it is installed as *moduleframework-cmd* command in */usr/bin* - - Test has to call setup and cleanup of module explicitly - - These tests are dependent on return code of commands in test, so in case you have more tests subtest, just count return codes - - see example test for https://pagure.io/modularity-testing-framework/blob/master/f/examples/memcached/sanity2.sh - - - - **WIP: Behave tests** - - You can write tests for you module also in behave style - - it is first prototype - - see example in https://pagure.io/modularity-testing-framework/blob/master/f/examples/memcached-behave - -Running using VAGRANT ---------------------- -- install vagrant *dnf -y install vagrant* -- just run *vagrant up* - -Installation ------------- -- base dependencies: **docker python-pip** -- python dependencies: **avocado-framework yaml json behave** - -Stable version -~~~~~~~~~~~~~~ -- It is built as package for fedora (>26) -- stable COPR repo - - https://copr.fedorainfracloud.org/coprs/phracek/Modularity-testing-framework/ - - `dnf copr enable phracek/Modularity-testing-framework` - - `dnf install -y modularity-testing-framework` - -Development version -~~~~~~~~~~~~~~ -- Automatically built packages - - https://copr.fedorainfracloud.org/coprs/jscotka/modularity-testing-framework/ - - - -Enviromental variables ----------------------- -- variables allows you to overwrite some values inside *config.yaml* -- *AVOCADO_LOG_DEBUG=yes* enables avocado debug output - in case you find some strange hard to debug issues -- *DEBUG* Enable debugging output to test output -- *CONFIG* file with MTF configuration default is *config.yaml* -- *MODULE* which module type to test (in case there is not set *default-module* in config, you **HAVE TO** set it) - - *=docker* uses section *docker* inside config file and will use docker containerisation - - *=nspawn* systemd nspawn, it is lightweight virtualization, it does something like **MOCK** but it is not just chroot, but has own systemd etc. - - *=rpm* testing of local RPM packages directly on HOST (it could be **DESTRUCTIVE**) - -- *URL* see example config. It overwrites value *module.docker.container* or *module.rpm.repo* to whatever you want. It has to be proper type what is set in *MODULE* -- *MODULEMDURL* overwrite location of moduleMD file -- *COMPOSEURL* overwrite location of compose repo location -- *PROFILE* overwrite *default* profile to whatever you want to install instead of that -- *MTF_SKIP_DISABLING_SELINUX* In nspawn type on fedora-25 we have to disable selinux, because it does not work well with selinux enabled, this option allows to not do that. -- *MTF_DO_NOT_CLEANUP* Do not cleanup modules between tests, in case there is no interference in your tests you can use it, and it will be **fast** -- *MTF_REMOTE_REPOS* It disables downloading of packages done by koji and creating local repo, it make tests **fast**. (There is issue that composes (repos) are sometimes bad in fedora, unable to use) - - - -Schedule Tests --------------- -- Now it is expected to run this **under root** -- Install modularity-testing-framework from COPR repo like: - - **dnf copr enable phracek/Modularity-testing-framework** - - install it by command: **dnf install -y modularity-testing-framework** - - It installs packages to python site-packages and to /usr/share/moduleframework -- To include tests into your module, add to your Makefile section **test** -- **test** section runs another Makefile in directory **tests** -- Your **Makefile** should contain: - - **Docker based module testing:** `cd tests; MODULE=docker make all` - - **Repo based module testing:** `MODULE=nspawn make all ` - - **Host Rpm based module testing:** `MODULE=rpm make all` - -- Makefile in tests directory looks like: - - $ cat tests/Makefile - MODULE_LINT=/usr/share/moduleframework/tools/modulelint.py - CMD=python -m avocado run --filter-by-tags=-WIP $(MODULE_LINT) *.py - - # - all: $(CMD) - - - **Makefile in MTF** https://pagure.io/modularity-testing-framework/blob/master/f/examples/testing-module/Makefile - -- `make check` - runs tests in your module directory - - -How to write tests ------------------- -- **minimal path creation** -- Install modularity-testing-framework from COPR repo like: - - *dnf copr enable phracek/Modularity-testing-framework* - - install it by command: *dnf install -y modularity-testing-framework* -- CREATE your config.yaml (see example https://pagure.io/modularity-testing-framework/blob/master/f/docs/example-config.yaml) -- If you have tests in config file call: */usr/bin/mtf-generator* or simply *mtf-generator*. -- Call command for running all python tests: **MODULE=docker avocado run /usr/share/moduleframework/tools/modulelint.py ./*.py** -- **additional tests** - see tests in https://pagure.io/modularity-testing-framework/blob/master/f/examples/testing-module directory as an example for you - -License -------- -Framework is released under the GPL, version 2 or later, see LICENSE file in project - - -Debugging & How To ------------------- - -First test takes so long time -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -It is normal, because first test downloads all packages from koji and creates localrepo. -It is workaround because of missing composes for modules (on demand done by pungi) -- If you would like to make it faster use env variables: - - *MTF_REMOTE_REPOS=yes* - It heps in case there are repos in koji https://kojipkgs.fedoraproject.org/repos/ (they are there just temporary, deleted after 2 weeks and probably it will not be created in near future anyhow) - - *MTF_DO_NOT_CLEANUP=yes* - in case it is still slow for you, it disable test cleaup (between tests) could cause side effects - -Unable to debug avocado output errors -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -- If you see error like: *Avocado crashed: TestError: Process died before it pushed early test_status.* - - add env variables: - - *AVOCADO_LOG_DEBUG=yes* - - *DEBUG=yes* - - **This is preferred variant for submitting issues to pagure** - - -How it works ------------- -- Structure of MTF: - - https://pagure.io/modularity-testing-framework/blob/master/f/docs/howitworks.png -- Test types: - - https://pagure.io/modularity-testing-framework/blob/master/f/docs/TestTypes.png diff --git a/docs/user_guide/example_config_minimal.yaml b/docs/user_guide/example_config_minimal.yaml new file mode 100644 index 0000000..d079c36 --- /dev/null +++ b/docs/user_guide/example_config_minimal.yaml @@ -0,0 +1,15 @@ +document: modularity-testing +version: 1 +name: bash +modulemd-url: https://src.fedoraproject.org/cgit/modules/memcached.git/plain/memcached.yaml +packages: + rpms: + - bash + - rpm +module: + docker: + container: docker.io/phracek/memcached + rpm: + repo: https://kojipkgs.stg.fedoraproject.org/compose/branched/jkaluza/latest-Fedora-Modular-26/compose/Server/x86_64/os/ + + diff --git a/docs/user_guide/how_to_write_conf_file.rst b/docs/user_guide/how_to_write_conf_file.rst new file mode 100644 index 0000000..9403361 --- /dev/null +++ b/docs/user_guide/how_to_write_conf_file.rst @@ -0,0 +1,85 @@ +## Document identification +There is nothing important, just parser inside check if config yaml is the proper one, that nobody for example does not used modulemd.yaml as config.yaml +```yaml +document: modularity-testing +version: 1 +``` + +## Module generic part +This part contain generic part of module +```yaml +name: memcached +modulemd-url: http://raw.githubusercontent.com/container-images/memcached/master/memcached.yaml +compose-url: url_to_compose_in done in fedora +service: + port: 11211 +packages: + rpms: + - memcached + - perl-Carp +testdependencies: + rpms: + - nc +``` + * `name:` name of module + * `modulemd-url:` link to modulemd file, now it is not used anyhow, just for installing packages for proper module + * `compose-url:` final compose build (done by pungi) it contains repositories + moduleMD infromations for tooling + * `service:` In case module is service like memcached, store there port number, can be then used in tests, to not hardcode port number *(Optional)* + * `packages:` Which packages will be installed inside module (docker container, guest, any type of module) + * `testdependencies:` Install dependencies on host, what are important for module testing, for example when you would like to use `nc`, you have to install it explicitly, it is not in cloud images. + +## Module types specification +It contains specification for each type of module, now for __rpm__ and __docker__ based modules +```yaml +module: + docker: + setup: echo Do magic with general config stored on host; + echo More magic + cleanup: echo Cleanup magic + start: "docker run -it -e CACHE_SIZE=128 -p 11211:11211" + labels: + description: "memcached is a high-performance, distributed memory" + io.k8s.description: "memcached is a high-performance, distributed memory" + source: https://github.com/container-images/memcached.git + container: docker.io/phracek/memcached + rpm: + setup: echo Do magic with general config stored on host; + echo More magic + cleanup: echo Cleanup magic + start: systemctl start memcached + stop: systemctl stop memcached + status: systemctl status memcached + repos: + - http://download.englab.brq.redhat.com/pub/fedora/releases/25/Everything/x86_64/os/ + - https://phracek.fedorapeople.org/memcached-module-repo/ +``` + * `setup:` run setup/cleanup commands on HOST, for example config manipulation, selinux boolean manipulation there could be used also variables in python style like: {ROOT}, {HOSTNAME} see trans_dict in file https://pagure.io/modularity-testing-framework/blob/master/f/moduleframework/module_framework.py + * `cleanup:` similar to setup but done in after test finished. + * `start:` how to start service in case it is service, in case of generic module it is *(Optional)* + * `stop:` how to service service in case it is service, in case of generic module it is *(Optional)* + * `status:` how to check service state, in case of generic module it is *(Optional)* + * `labels:` docker labels to check, specific just for *docker* container *(Docker specific)* + * `container:` where is link to container, now it support docker.io link or using locally tar.gz file specified *(Docker specific)* + * `repo:` if *compose* is not set then this is used and contains repo what has to be used for this module (dependent repos are searched in PDC via moduleMD definition) *(Rpm specific)* + * `repos:` if *compose* and *repo* not set: contains all repos what has to be used for this module (typically baseruntime + specific one) *(Rpm specific)* *(OBSOLOTE)* + +## Simple tests inside config + This part is little but __controversial__ , some of users are fans and some hates this. It allows you to specify bash style tests directly inside config file +```yaml +test: + processrunning: + - 'ls /proc/*/exe -alh | grep memcached' +testhost: + selfcheck: + - 'echo errr | nc localhost 11211' + - 'echo set AAA 0 4 2 | nc localhost 11211' + - 'echo get AAA | nc localhost 11211' + selcheckError: + - 'echo errr | nc localhost 11211 |grep ERROR' +``` + * `test:` tests what will run inside container - it means that there has to be all dependencies for these test *(Optional)* + * every command has to finish with __0 return code__ otherwise it will __fail__ + * next level like __processrunning__ is test name what will be visible on output of avocado run, then all lines will be run as commands for this test + * `testhost:` it is similar to *test*, just difference is that it runs commands on host machine so that there could be more dependencies than just are in module. I', not sure if this part is useful, will see after discussion *(Optional)* + * other specification is same as `test` + * you have to call `mtf-generator` binary to generate python files from that (because unittests does not allow to have dynamically created tests) diff --git a/docs/user_guide/scheduling.rst b/docs/user_guide/scheduling.rst new file mode 100644 index 0000000..707fce0 --- /dev/null +++ b/docs/user_guide/scheduling.rst @@ -0,0 +1,48 @@ +Workflow integration +============================ + +Testsuite of project +---------------------------- +- Upstream **testsuite** for project located in `/usr/share/moduleframework/examples/testing-module/` + - You can use it as an **inspiration** for your tests + - It contains various types how to schedule tests + - **CI** It contains info how it is scheduled in internal-ci or in taskotron or how to do + - Examples of **Manual** running of tests on localhost. + - Example how to run general **multi-host** tests + - Every new feature should be covered here - by new Makefile target or by new test run inside every testing module + + +Taskotron Wokflow +---------------------------- +- Production instance: https://taskotron.fedoraproject.org/resultsdb/results?testcases=dist.modularity-testing-framework + - Trigerred fedmsg via **module-stream-version** string + - Trigerred by **Module Build system** done message, list of all: https://apps.fedoraproject.org/datagrepper/raw?topic=org.fedoraproject.prod.mbs.module.state.change + - There is general `runtask.yml` taskotron trigger: https://pagure.io/taskotron/task-modularity-testing-framework + - There is just one for every module and it contains whole logic where to find tests for module. + - Not needed to duplicate `runtash.yml` for each component. Scheduler is same (existing Makefile) + - It run `tools/run-them.sh` script. It contains whole logic where are tests and how to find them. + - **run-them.sh** script for taskotron + - Test Subject: rpm repositories (tagged koji builds of packages) via `systemd-nspawn` + - Located in: `/usr/share/moduleframework/tools/run-them.sh` + - Scheduled as: `./run-them.sh testmodule testmodule-master-20170407121558 pdc` + - Example targets: `check-run-them-pdc-testmodule, check-run-them-pdc-baseruntime` + - Internal logic + - Contact *PDC* (Product definition center) for info about module like `koji tags, moduleMD file` + - Try dowload package from `modules` namespace in `dist-git` via `fedpkg clone` + - checkout to proper version found by PDC (scmurl) + - Try to find tests there ( if exist `Makefile` in `tests` directory) + - If None: Try to find module dir in MTF project tests in `/usr/share/moduleframework/examples` directory + - If None: Run at least general ModuleLinter (`/usr/share/moduleframework/tools/modulelint`) with general minimal config.yaml located in `docs` directory + +Internal Jenkins Instance +---------------------------- +- Production instance: `hidden` + - Trigerred via `fedmsg file` + - Used **tools/run-them.sh** script, for same behaviour as Taskotron + - **run-them.sh** script for Jenkins based on whole fedmsg + - Test Subject: Same as *Taskotron Workflow* + - Located in: Same as *Taskotron Workflow* + - Scheduled as: `run-them.sh testmodule /usr/share/moduleframework/tools/example_message_module.yaml fedmsg` + - Example targets: `check-run-them-fedmsg-testmodule` + - Internal logic + - Same as *Taskotron Workflow* diff --git a/docs/workflow_integration.rst b/docs/workflow_integration.rst deleted file mode 100644 index 707fce0..0000000 --- a/docs/workflow_integration.rst +++ /dev/null @@ -1,48 +0,0 @@ -Workflow integration -============================ - -Testsuite of project ----------------------------- -- Upstream **testsuite** for project located in `/usr/share/moduleframework/examples/testing-module/` - - You can use it as an **inspiration** for your tests - - It contains various types how to schedule tests - - **CI** It contains info how it is scheduled in internal-ci or in taskotron or how to do - - Examples of **Manual** running of tests on localhost. - - Example how to run general **multi-host** tests - - Every new feature should be covered here - by new Makefile target or by new test run inside every testing module - - -Taskotron Wokflow ----------------------------- -- Production instance: https://taskotron.fedoraproject.org/resultsdb/results?testcases=dist.modularity-testing-framework - - Trigerred fedmsg via **module-stream-version** string - - Trigerred by **Module Build system** done message, list of all: https://apps.fedoraproject.org/datagrepper/raw?topic=org.fedoraproject.prod.mbs.module.state.change - - There is general `runtask.yml` taskotron trigger: https://pagure.io/taskotron/task-modularity-testing-framework - - There is just one for every module and it contains whole logic where to find tests for module. - - Not needed to duplicate `runtash.yml` for each component. Scheduler is same (existing Makefile) - - It run `tools/run-them.sh` script. It contains whole logic where are tests and how to find them. - - **run-them.sh** script for taskotron - - Test Subject: rpm repositories (tagged koji builds of packages) via `systemd-nspawn` - - Located in: `/usr/share/moduleframework/tools/run-them.sh` - - Scheduled as: `./run-them.sh testmodule testmodule-master-20170407121558 pdc` - - Example targets: `check-run-them-pdc-testmodule, check-run-them-pdc-baseruntime` - - Internal logic - - Contact *PDC* (Product definition center) for info about module like `koji tags, moduleMD file` - - Try dowload package from `modules` namespace in `dist-git` via `fedpkg clone` - - checkout to proper version found by PDC (scmurl) - - Try to find tests there ( if exist `Makefile` in `tests` directory) - - If None: Try to find module dir in MTF project tests in `/usr/share/moduleframework/examples` directory - - If None: Run at least general ModuleLinter (`/usr/share/moduleframework/tools/modulelint`) with general minimal config.yaml located in `docs` directory - -Internal Jenkins Instance ----------------------------- -- Production instance: `hidden` - - Trigerred via `fedmsg file` - - Used **tools/run-them.sh** script, for same behaviour as Taskotron - - **run-them.sh** script for Jenkins based on whole fedmsg - - Test Subject: Same as *Taskotron Workflow* - - Located in: Same as *Taskotron Workflow* - - Scheduled as: `run-them.sh testmodule /usr/share/moduleframework/tools/example_message_module.yaml fedmsg` - - Example targets: `check-run-them-fedmsg-testmodule` - - Internal logic - - Same as *Taskotron Workflow* From d656da9105377cd7e1c249fbb983f4f9b649b681 Mon Sep 17 00:00:00 2001 From: Irina Gulina Date: Jun 14 2017 07:45:13 +0000 Subject: [PATCH 2/7] docs intro page, license, installation --- diff --git a/docs/index.rst b/docs/index.rst index fc86bfa..f2acea6 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -1,20 +1,50 @@ -Welcome to modularity-testing-framework's documentation! -======================================================== +Modularity Testing Framework Documentation +========================================== -Contents: +Welcome to the Modularity Testing Framework documentation! + +About +===== + +Modularity Testing Framework (MTF) is a tool to test `Fedora modules`_. + +Using MTF you can: + +* write tests for RPMs, modules and Docker containers +* write multi like Bash snippet tests in YAML definition file +* write Bash tests +* write Python tests +* schedule tests with Jenkins and Taskotron +* work in Vagrant environment + +MTF has a presence on the following websites: + +* `Documentation`_ is available on ReadTheDocs. +* A `Package repository`_ is available on Fedora Copr. +* `MTF's code`_ and the issue tracker for sharing bugs and feature ideas are stored on Pagure. + +.. _Documentation: http://modularity-testing-framework.readthedocs.io +.. _Fedora modules: https://github.com/modularity-modules +.. _Package repository: https://copr.fedorainfracloud.org/coprs/phracek/Modularity-testing-framework/ +.. _MTF's code: https://pagure.io/modularity-testing-framework + +Content +======= .. toctree:: - :maxdepth: 4 + :maxdepth: 1 - overview - how_to_schedule - workflow_integration - levels - clsoverview + installation + user_guide/index + license -Indices and tables -================== +Index and Search +================ * :ref:`genindex` -* :ref:`modindex` * :ref:`search` + +.. seealso:: + + `webchat.freenode.net `_ + Questions? Help? Ideas? Stop by the #fedora-modularity on freenode IRC chat channel diff --git a/docs/installation.rst b/docs/installation.rst index 7556872..19fbd7e 100644 --- a/docs/installation.rst +++ b/docs/installation.rst @@ -1,161 +1,135 @@ -Modularity-testing-framework -============================ - -Recent documentation on: - http://modularity-testing-framework.readthedocs.io - -Modularity prototype testing ----------------------------- - -- General Structure - - **YAML Config file** - - Each module will need to have yaml config file - - Config file should cover non generic part - part focused on general module testing - - It could contain also simple test - - **how to write config file** https://pagure.io/modularity-testing-framework/blob/master/f/docs/howtowriteyamlconf.md - - - **Self generated tests** - - there is possible to write simple tests directly in yaml config file - - Bash style testing - - It has to have solved dependencies inside each module type - - Now it just expect to end with *0* return code of command (like: *ls / |grep sbin* directory sbin exists in root dir) - - It can contain multiple lines - - It generates python covered bash tests - - You has to call `mtf-generator` binary explicitly, it then create these pythonish tests with bash inside, *Unittest* doesn not allow to have dynamic tests. - - - **Avocado tests** - - There is wrapper class what helps you to tests modules not focusing on module type - - It uses avocado-framework - - This test could be primarily used for more complex testing, not previous one - - General test for modules: *./base/modulelint.py* - - Example tests in *memcached* module - - - **Simple bash tests** - - There is helper what you can use for writing *bash* like tests - - library is https://pagure.io/modularity-testing-framework/blob/master/f/moduleframework/bashhelper.py and it is installed as *moduleframework-cmd* command in */usr/bin* - - Test has to call setup and cleanup of module explicitly - - These tests are dependent on return code of commands in test, so in case you have more tests subtest, just count return codes - - see example test for https://pagure.io/modularity-testing-framework/blob/master/f/examples/memcached/sanity2.sh - - - - **WIP: Behave tests** - - You can write tests for you module also in behave style - - it is first prototype - - see example in https://pagure.io/modularity-testing-framework/blob/master/f/examples/memcached-behave - -Running using VAGRANT +Installation +============ + +There are two ways to install and use MTF: to set up it locally or alternatively on a virtual machine via the Vagrant tool. + +.. contents:: Topics + +.. _using_vagrant: +Vagrant --------------------- -- install vagrant *dnf -y install vagrant* -- just run *vagrant up* -Installation ------------- -- base dependencies: **docker python-pip** -- python dependencies: **avocado-framework yaml json behave** - -Stable version -~~~~~~~~~~~~~~ -- It is built as package for fedora (>26) -- stable COPR repo - - https://copr.fedorainfracloud.org/coprs/phracek/Modularity-testing-framework/ - - `dnf copr enable phracek/Modularity-testing-framework` - - `dnf install -y modularity-testing-framework` - -Development version -~~~~~~~~~~~~~~ -- Automatically built packages - - https://copr.fedorainfracloud.org/coprs/jscotka/modularity-testing-framework/ - - - -Enviromental variables ----------------------- -- variables allows you to overwrite some values inside *config.yaml* -- *AVOCADO_LOG_DEBUG=yes* enables avocado debug output - in case you find some strange hard to debug issues -- *DEBUG* Enable debugging output to test output -- *CONFIG* file with MTF configuration default is *config.yaml* -- *MODULE* which module type to test (in case there is not set *default-module* in config, you **HAVE TO** set it) - - *=docker* uses section *docker* inside config file and will use docker containerisation - - *=nspawn* systemd nspawn, it is lightweight virtualization, it does something like **MOCK** but it is not just chroot, but has own systemd etc. - - *=rpm* testing of local RPM packages directly on HOST (it could be **DESTRUCTIVE**) - -- *URL* see example config. It overwrites value *module.docker.container* or *module.rpm.repo* to whatever you want. It has to be proper type what is set in *MODULE* -- *MODULEMDURL* overwrite location of moduleMD file -- *COMPOSEURL* overwrite location of compose repo location -- *PROFILE* overwrite *default* profile to whatever you want to install instead of that -- *MTF_SKIP_DISABLING_SELINUX* In nspawn type on fedora-25 we have to disable selinux, because it does not work well with selinux enabled, this option allows to not do that. -- *MTF_DO_NOT_CLEANUP* Do not cleanup modules between tests, in case there is no interference in your tests you can use it, and it will be **fast** -- *MTF_REMOTE_REPOS* It disables downloading of packages done by koji and creating local repo, it make tests **fast**. (There is issue that composes (repos) are sometimes bad in fedora, unable to use) - - - -Schedule Tests --------------- -- Now it is expected to run this **under root** -- Install modularity-testing-framework from COPR repo like: - - **dnf copr enable phracek/Modularity-testing-framework** - - install it by command: **dnf install -y modularity-testing-framework** - - It installs packages to python site-packages and to /usr/share/moduleframework -- To include tests into your module, add to your Makefile section **test** -- **test** section runs another Makefile in directory **tests** -- Your **Makefile** should contain: - - **Docker based module testing:** `cd tests; MODULE=docker make all` - - **Repo based module testing:** `MODULE=nspawn make all ` - - **Host Rpm based module testing:** `MODULE=rpm make all` - -- Makefile in tests directory looks like: - - $ cat tests/Makefile - MODULE_LINT=/usr/share/moduleframework/tools/modulelint.py - CMD=python -m avocado run --filter-by-tags=-WIP $(MODULE_LINT) *.py - - # - all: $(CMD) - - - **Makefile in MTF** https://pagure.io/modularity-testing-framework/blob/master/f/examples/testing-module/Makefile - -- `make check` - runs tests in your module directory - - -How to write tests ------------------- -- **minimal path creation** -- Install modularity-testing-framework from COPR repo like: - - *dnf copr enable phracek/Modularity-testing-framework* - - install it by command: *dnf install -y modularity-testing-framework* -- CREATE your config.yaml (see example https://pagure.io/modularity-testing-framework/blob/master/f/docs/example-config.yaml) -- If you have tests in config file call: */usr/bin/mtf-generator* or simply *mtf-generator*. -- Call command for running all python tests: **MODULE=docker avocado run /usr/share/moduleframework/tools/modulelint.py ./*.py** -- **additional tests** - see tests in https://pagure.io/modularity-testing-framework/blob/master/f/examples/testing-module directory as an example for you +`Vagrant`_ is a tool to aid developers in quickly deploying development environments. There is a `Vagrantfile`_ in the `modularity-testing-framework`_ git repository on Pagure that can automatically deploy a virtual machine on your host with a MTF environment configured. + +.. _Vagrant: https://docs.vagrantup.com/ +.. _Vagrantfile: https://pagure.io/modularity-testing-framework/blob/master/f/Vagrantfile +.. _modularity-testing-framework: https://pagure.io/modularity-testing-framework + +The MTF tool has made available for use of two providers: ``libvirt`` (for Linux host only) and ``virtualbox`` (for MAC OS, Windows and Linux hosts), where ``libvirt`` is a default one. See more about Vagrant providers `here`_. + +.. _here: https://www.vagrantup.com/docs/providers/basic_usage.html#default-provider + +This document assumes that you are running a recent version of Fedora although these steps should be roughly the same on other distributions, just be aware that package managers and names can differ if you are not using Fedora as your host. Consult `Vagrant installation documentation`_ to set up Vagrant for a different platform and adjuest the steps of this document accordingly. + +.. _Vagrant installation documentation: https://www.vagrantup.com/docs/installation/ + +.. note:: + Before you start using Vagrant-libvirt, please make sure your libvirt and qemu installation is working correctly and you are able to create qemu or kvm type virtual machines with virsh or virt-manager. + +Prerequisites for Vagrant +~~~~~~~~~~~~~~~~~~~~~~~~~ + +1. Install Vagrant. Ensure that ``vagrant-libvirt`` is among pulled dependencies. + +.. code-block:: bash + + # install Vagrant + $ sudo dnf -y install vagrant + +2. Start ``libvirtd`` service + +.. code-block:: bash + + # start libvirtd service + $ sudo systemctl start libvirtd + +Creating the Vagrant environment +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +After preparing the libvirt prerequisites using the instructions above: + +1. You are now prepared to check out the MTF code into your preferred location. + +.. code-block:: bash + + # cd to your prefered location + $ cd $HOME/ # Season to taste. + $ git clone https://pagure.io/modularity-testing-framework.git + +2. Next, cd into the ``modularity-testing-framework`` directory. -License -------- -Framework is released under the GPL, version 2 or later, see LICENSE file in project +.. code-block:: bash + # cd in modularity-testing-framework + $ cd modularity-testing-framework -Debugging & How To +3. The MTF tool provides a configuration Vagrantfile that you can use to configure the Vagrant environment as given or open the Vagrantfile in your favorite editor and modify it to better fit your development preferences. This step is entirely optional as the default Vagrantfile should work for most users. + +.. code-block:: bash + + # vim Vagrantfile + $ vim Vagrantfile + +4. If you’ve happy with the Vagrantfile, you can begin provisioning your Vagrant environment. Finish by running ``vagrant reload`` to reboot machine after provisioning and apply the latest kernel updates. + +.. code-block:: bash + + # Provision the Vagrant environment: + $ sudo vagrant up --provider=libvirt # or just `sudo vagrant up` as libvirt is a default one + # The above will run for a while while it provisions your development environment. + $ sudo vagrant reload # Reboot the machine at the end to apply kernel updates, etc. + +5. Once you have followed the steps above, you should have a running deployed MTF development machine. ssh into your Vagrant environment:: + +.. code-block:: bash + + # ssh into the Vagrant environment + $ sudo vagrant ssh + +.. _local_requirements: + +Local installation ------------------ -First test takes so long time -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -It is normal, because first test downloads all packages from koji and creates localrepo. -It is workaround because of missing composes for modules (on demand done by pungi) -- If you would like to make it faster use env variables: - - *MTF_REMOTE_REPOS=yes* - It heps in case there are repos in koji https://kojipkgs.fedoraproject.org/repos/ (they are there just temporary, deleted after 2 weeks and probably it will not be created in near future anyhow) - - *MTF_DO_NOT_CLEANUP=yes* - in case it is still slow for you, it disable test cleaup (between tests) could cause side effects - -Unable to debug avocado output errors -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -- If you see error like: *Avocado crashed: TestError: Process died before it pushed early test_status.* - - add env variables: - - *AVOCADO_LOG_DEBUG=yes* - - *DEBUG=yes* - - **This is preferred variant for submitting issues to pagure** - - -How it works ------------- -- Structure of MTF: - - https://pagure.io/modularity-testing-framework/blob/master/f/docs/howitworks.png -- Test types: - - https://pagure.io/modularity-testing-framework/blob/master/f/docs/TestTypes.png +Requirements +~~~~~~~~~~~~ + +MTF installer pulls its latest dependencies: ``python-devel``, ``python-setuptools`` and ``python-netifcaes``, ``docker``, `avocado`_, ``yaml`` and ``json``. + +MTF supports Gherkin-based testing in Python. To write tests in a natural language style, backed up by Python code, install BBD tool `behave`_ . Execute the following command to install behave with pip: + +.. _avocado: https://avocado-framework.github.io/ +.. _behave: http://pythonhosted.org/behave/ + +.. code-block:: bash + + # install behave + $ sudo pip install behave + +.. _installing_mtg: +Installing MTF +~~~~~~~~~~~~~ +Install MTF rpm from `Fedora Copr repo`_. + +.. _Fedora Copr repo: https://copr.fedorainfracloud.org/coprs/phracek/Modularity-testing-framework/ + +.. code-block:: bash + + # add modularity-testing-framework yum repo + $ sudo dnf copr enable phracek/Modularity-testing-framework + $ sudo dnf install -y modularity-testing-framework + +.. _getting_mtf: +Source code +----------- + +You may also wish to follow the `Pagure MTF repo`_ if you have a Pagure account. This stores the source code and the issue tracker for sharing bugs and feature ideas. The repository should be forked into your personal Pagure account where all work will be done. Any changes should be submitted through the pull request process. + +.. _Pagure MTF repo: https://pagure.io/modularity-testing-framework + +.. seealso:: + + :doc:`user_guide/index` + User Guide + `webchat.freenode.net `_ + Questions? Help? Ideas? Stop by the #fedora-modularity on freenode IRC chat channel diff --git a/docs/license.rst b/docs/license.rst new file mode 100644 index 0000000..59d2cf1 --- /dev/null +++ b/docs/license.rst @@ -0,0 +1,6 @@ +License +======= + +MTF is released under the GPLv2+, see `LICENSE`_ file in the source code repository. + +.. _LICENSE: https://pagure.io/modularity-testing-framework/blob/master/f/LICENSE From c2d36264347f3474131acfd4fdee6e97b48e4a58 Mon Sep 17 00:00:00 2001 From: Irina Gulina Date: Jun 14 2017 07:45:15 +0000 Subject: [PATCH 3/7] docs user guide main steps, conf file,env variables, troubleshooting --- diff --git a/docs/installation.rst b/docs/installation.rst index 19fbd7e..1907278 100644 --- a/docs/installation.rst +++ b/docs/installation.rst @@ -119,6 +119,8 @@ Install MTF rpm from `Fedora Copr repo`_. $ sudo dnf copr enable phracek/Modularity-testing-framework $ sudo dnf install -y modularity-testing-framework +MTF scripts, examples and documentation will be installed into ``/usr/share/moduleframework`` + .. _getting_mtf: Source code ----------- @@ -131,5 +133,5 @@ You may also wish to follow the `Pagure MTF repo`_ if you have a Pagure account. :doc:`user_guide/index` User Guide - `webchat.freenode.net `_ + `webchat.freenode.net `_ Questions? Help? Ideas? Stop by the #fedora-modularity on freenode IRC chat channel diff --git a/docs/user_guide/environment_variables.rst b/docs/user_guide/environment_variables.rst new file mode 100644 index 0000000..e664927 --- /dev/null +++ b/docs/user_guide/environment_variables.rst @@ -0,0 +1,20 @@ +Environment variables +===================== + +Environment variables allow to overwrite some values of a module configuration file ``config.yaml``. + +- **AVOCADO_LOG_DEBUG=yes** enables avocado debug output + - **DEBUG=yes** enables debugging mode to test output +- **CONFIG** defines MTF configuration file. It defaultis to ``config.yaml`` +- **MODULE** defines tested module type, if **defailt-module** is not set in ``config.yanl`` + + - **=docker** uses **docker** section of ``config.yaml`` + - **=rpm** uses **rpm** section of ``config.yaml`` and tests RPMs directly on a host + - **=nspawn** tests RPMs in a virtual environment of lightweight virtualization with systemd-nspawn + +- **URL** overwrites value of **module.docker.container** or **module.rpm.repo**. It has to be proper type what is set in **MODULE** +- **MODULEMDURL** overwrites location of a moduleMD file +- **COMPOSEURL** overwrites location of a compose Pungi build +- **MTF_SKIP_DISABLING_SELINUX=yes** does not disable SELinux In nspawn type on Fedora25 SELinux should be diabled, because it does not work well with selinux enabled, this option allows to not do that. +- **MTF_DO_NOT_CLEANUP=yes** does not cleanup modules between tests. It speeds up test execution. Use only if there is no interference between tests +- **MTF_REMOTE_REPOS=yes** disables downloading of Koji packages and creating local repo, and speeds up test execution. diff --git a/docs/user_guide/how_to_write_conf_file.rst b/docs/user_guide/how_to_write_conf_file.rst index 9403361..20781f8 100644 --- a/docs/user_guide/how_to_write_conf_file.rst +++ b/docs/user_guide/how_to_write_conf_file.rst @@ -1,85 +1,92 @@ -## Document identification -There is nothing important, just parser inside check if config yaml is the proper one, that nobody for example does not used modulemd.yaml as config.yaml -```yaml -document: modularity-testing -version: 1 -``` - -## Module generic part -This part contain generic part of module -```yaml -name: memcached -modulemd-url: http://raw.githubusercontent.com/container-images/memcached/master/memcached.yaml -compose-url: url_to_compose_in done in fedora -service: - port: 11211 -packages: - rpms: - - memcached - - perl-Carp -testdependencies: - rpms: - - nc -``` - * `name:` name of module - * `modulemd-url:` link to modulemd file, now it is not used anyhow, just for installing packages for proper module - * `compose-url:` final compose build (done by pungi) it contains repositories + moduleMD infromations for tooling - * `service:` In case module is service like memcached, store there port number, can be then used in tests, to not hardcode port number *(Optional)* - * `packages:` Which packages will be installed inside module (docker container, guest, any type of module) - * `testdependencies:` Install dependencies on host, what are important for module testing, for example when you would like to use `nc`, you have to install it explicitly, it is not in cloud images. - -## Module types specification -It contains specification for each type of module, now for __rpm__ and __docker__ based modules -```yaml -module: - docker: - setup: echo Do magic with general config stored on host; - echo More magic - cleanup: echo Cleanup magic - start: "docker run -it -e CACHE_SIZE=128 -p 11211:11211" - labels: - description: "memcached is a high-performance, distributed memory" - io.k8s.description: "memcached is a high-performance, distributed memory" - source: https://github.com/container-images/memcached.git - container: docker.io/phracek/memcached - rpm: - setup: echo Do magic with general config stored on host; - echo More magic - cleanup: echo Cleanup magic - start: systemctl start memcached - stop: systemctl stop memcached - status: systemctl status memcached - repos: - - http://download.englab.brq.redhat.com/pub/fedora/releases/25/Everything/x86_64/os/ - - https://phracek.fedorapeople.org/memcached-module-repo/ -``` - * `setup:` run setup/cleanup commands on HOST, for example config manipulation, selinux boolean manipulation there could be used also variables in python style like: {ROOT}, {HOSTNAME} see trans_dict in file https://pagure.io/modularity-testing-framework/blob/master/f/moduleframework/module_framework.py - * `cleanup:` similar to setup but done in after test finished. - * `start:` how to start service in case it is service, in case of generic module it is *(Optional)* - * `stop:` how to service service in case it is service, in case of generic module it is *(Optional)* - * `status:` how to check service state, in case of generic module it is *(Optional)* - * `labels:` docker labels to check, specific just for *docker* container *(Docker specific)* - * `container:` where is link to container, now it support docker.io link or using locally tar.gz file specified *(Docker specific)* - * `repo:` if *compose* is not set then this is used and contains repo what has to be used for this module (dependent repos are searched in PDC via moduleMD definition) *(Rpm specific)* - * `repos:` if *compose* and *repo* not set: contains all repos what has to be used for this module (typically baseruntime + specific one) *(Rpm specific)* *(OBSOLOTE)* - -## Simple tests inside config - This part is little but __controversial__ , some of users are fans and some hates this. It allows you to specify bash style tests directly inside config file -```yaml -test: - processrunning: - - 'ls /proc/*/exe -alh | grep memcached' -testhost: - selfcheck: - - 'echo errr | nc localhost 11211' - - 'echo set AAA 0 4 2 | nc localhost 11211' - - 'echo get AAA | nc localhost 11211' - selcheckError: - - 'echo errr | nc localhost 11211 |grep ERROR' -``` - * `test:` tests what will run inside container - it means that there has to be all dependencies for these test *(Optional)* - * every command has to finish with __0 return code__ otherwise it will __fail__ - * next level like __processrunning__ is test name what will be visible on output of avocado run, then all lines will be run as commands for this test - * `testhost:` it is similar to *test*, just difference is that it runs commands on host machine so that there could be more dependencies than just are in module. I', not sure if this part is useful, will see after discussion *(Optional)* - * other specification is same as `test` - * you have to call `mtf-generator` binary to generate python files from that (because unittests does not allow to have dynamically created tests) +Configuration file +================== + +To test a module create its configuration file ``config.yaml`` similar to an `example configuration file`_ described further. If the tested module doesn't represent any service, the `minimal configuration file`_ structure can be used. + +.. _example configuration file: https://pagure.io/modularity-testing-framework/blob/master/f/examples/memcached/config.yaml +.. _minimal configuration file: https://pagure.io/modularity-testing-framework/blob/master/f/docs/example-config-minimal.yaml + +An example of ``config.yaml`` header: + +.. code-block:: yaml + + document: modularity-testing + version: 1 + +An example of module general description: + +.. code-block:: yaml + + name: memcached + modulemd-url: http://raw.githubusercontent.com/container-images/memcached/master/memcached.yaml + compose-url: https://kojipkgs.fedoraproject.org/compose/latest-Fedora-Modular-26/compose/Server/x86_64/os/Packages/m/memcached-1.4.36-1.module_b2e063be.x86_64.rpm + service: + port: 11211 + packages: + rpms: + - memcached + - perl-Carp + testdependencies: + rpms: + - nc + +* **name** defines module name +* **modulemd-url** contains a link to a moduleMD file +* **compose-url** links to a final compose Pungi build. **repo** or **repos** can be used instead, see further +* **service** stores a port if a module has any +* **packages** defines a module type (by the moment only `rpms` type is supported) +* **testdependencies** covers dependencies to be installed and used in tests + +An example of module types specification: + +.. code-block:: yaml + default_module: docker + module: + docker: + setup: "docker run -it -e CACHE_SIZE=128 -p 11211:11211" + cleanup:"echo Cleanup magic" + labels: + description: "memcached is a high-performance, distributed memory" + io.k8s.description: "memcached is a high-performance, distributed memory" + source: https://github.com/container-images/memcached.git + container: docker.io/phracek/memcached + rpm: + setup: /usr/bin/memcached -p 11211 + cleanup: echo Cleanup magic + start: systemctl start memcached + stop: systemctl stop memcached + status: systemctl status memcached + repo: + - http://download.englab.brq.redhat.com/pub/fedora/releases/25/Everything/x86_64/os/ + - https://phracek.fedorapeople.org/memcached-module-repo/ + +* **default_module**, if specified, sets the default tested module type +* **setup** runs setup commands on a host machine, not in container, and prepares the environemt for tests, for example changes selinux policy or hostname +* **cleanup**: similar to setup but done after test finished +* **start** defines how to start module service if there is any +* **stop** defines how to stop module service if there is any +* **status** defines how to check the status of module service if there is any +* **labels** contains docker labels to check if any +* **container** contains a link to a container (docker.io or local tar.gz file) +* **repo** is used when **compose-url** is not set and contains a repo to be used for rpm module type testing + +Multi like Bash snippet tests +----------------------------- +A ``config.yaml`` file may contain multi like Bash snippet tests directly. Every Bash command has to finish with 0 return code otherwise it returns fail: + +.. code-block:: yaml + + test: + processrunning: + - 'ls /proc/*/exe -alh | grep memcached' + testhost: + selfcheck: + - 'echo errr | nc localhost 11211' + - 'echo set AAA 0 4 2 | nc localhost 11211' + - 'echo get AAA | nc localhost 11211' + selcheckError: + - 'echo errr | nc localhost 11211 |grep ERROR' + +* **test** defines a section of multi like bash snippet tests +* **processrunning** contains commands to run as tests and displayed as avocado output +* **testhost** is optional and similar to **test**. The difference is that it runs commands on host machine so that there could be more dependencies than just are in a module. diff --git a/docs/user_guide/index.rst b/docs/user_guide/index.rst new file mode 100644 index 0000000..1c12144 --- /dev/null +++ b/docs/user_guide/index.rst @@ -0,0 +1,95 @@ +User Guide +========== + +1. In a module's root directory create a directory ``tests`` and place there a module configuration file ``config.yaml`` described in detail in section `Configuration file`_. + +.. _Configuration file: how_to_write_conf_file + +2. Optionally write multi like Bash snippet tests directly in ``tests/config.yaml`` file as described in section `Multi like Bash snippet tests`_. + +.. _Multi like Bash snippet tests: how_to_write_conf_file#multi-like-bash-snippet-tests + +3. Check the list of `Environment variables`_. + +.. _Environment variables: environment_variables. + +4. Write your tests, for example see `sanity tests`_ and various tests examples in ``/usr/share/moduleframework/examples/testing-module/``. All tests methods are listed in section `Method index`_. + +.. _sanity tests: https://pagure.io/modularity-testing-framework/blob/master/f/examples/template/sanity_template.py +.. _Method index: docs/module_framework#modularity-testing-framework + +5. In a directory ``tests`` create a ``Makefile`` as below. Line ``generator`` is optional and needed only if you have multi like Bash snippet tests. + + .. code-block:: makefile + + MODULE_LINT=/usr/share/moduleframework/tools/modulelint.py + CMD=python -m avocado run $(MODULE_LINT) *.py + + # + all: + generator + $(CMD) + +6. In a module's root directory create a ``Makefile``, which contains a secton **test**. For example: + + .. code-block:: makefile + + .PHONY: build run default + + IMAGE_NAME = debugging-tools + MODULEMDURL=file://debugging-tools.yaml + + all: run + default: run + + build: + docker build --tag=$(IMAGE_NAME) . + + run: build + docker run -it --name $(IMAGE_NAME) --privileged --ipc=host --net=host --pid=host -e HOST=/host -e NAME=$(IMAGE_NAME) -e IMAGE=$(IMAGE_NAME) -v /run:/run -v /var/log:/var/log -v /etc/machine-id:/etc/machine-id -v /etc/localtime:/etc/localtime -v /:/host $(IMAGE_NAME) + + test: build + cd tests; MODULE=docker MODULEMD=$(MODULEMDURL) URL="docker=$(IMAGE_NAME)" make all + cd tests; MODULE=rpm MODULEMD=$(MODULEMDURL) URL="docker=$(IMAGE_NAME)" make all + +7. Execute tests from the module root directory by running + + .. code-block:: shell + + #run tests from a module root directory + $ make test + + or from ``tests`` directory by running + + .. code-block:: shell + + #run Python tests from tests/ directory + $ sudo MODULE=docker avocado run ./*.py + + or + + .. code-block:: shell + + #run Bash tests from tests/ directory + $ sudo MODULE=docker avocado run ./*.sh + + +Contents: + +.. toctree:: + :maxdepth: 2 + + how_to_write_conf_file + environment_variables + #bash_tests + #python_tests + scheduling + #general-reference + #glossary + troubleshooting + +.. seealso:: + + `webchat.freenode.net `_ + Questions? Help? Ideas? Stop by the #fedora-modularity on freenode IRC chat channel + diff --git a/docs/user_guide/troubleshooting.rst b/docs/user_guide/troubleshooting.rst new file mode 100644 index 0000000..20efd0c --- /dev/null +++ b/docs/user_guide/troubleshooting.rst @@ -0,0 +1,19 @@ +Troubleshooting +=============== + +First test takes so long time +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +It is expected behavior, because the first test run downloads all packages from Koji and creates a local +repo. It is workaround because of missing composes for modules (on demand done by pungi). To make tests execute faster use environment variables: + + - **MTF_REMOTE_REPOS=yes** - It heps in case there are repos in koji https://kojipkgs.fedoraproject.org/repos/ (they are there just temporary, deleted after 2 weeks and probably it will not be created in near future anyhow) + - **MTF_DO_NOT_CLEANUP=yes** to disable cleaup between tests. Use only if there is no interference in between tests. + +Unable to debug avocado output errors +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you see an error: ``Avocado crashed: TestError: Process died before it pushed early test_status.``, add environment variables: + + - **AVOCADO_LOG_DEBUG=yes** + - **DEBUG=yes** From 43e3cfc8c2b1cecc357c359c9c307e8b001901ba Mon Sep 17 00:00:00 2001 From: Irina Gulina Date: Jun 14 2017 07:45:15 +0000 Subject: [PATCH 4/7] docs list of test methods --- diff --git a/docs/user_guide/index.rst b/docs/user_guide/index.rst index 1c12144..6aa21c1 100644 --- a/docs/user_guide/index.rst +++ b/docs/user_guide/index.rst @@ -13,10 +13,9 @@ User Guide .. _Environment variables: environment_variables. -4. Write your tests, for example see `sanity tests`_ and various tests examples in ``/usr/share/moduleframework/examples/testing-module/``. All tests methods are listed in section `Method index`_. +4. Write your tests, for example see `sanity tests`_ and various tests examples in ``/usr/share/moduleframework/examples/testing-module/``. All tests methods are listed in section :ref:`genindex`. .. _sanity tests: https://pagure.io/modularity-testing-framework/blob/master/f/examples/template/sanity_template.py -.. _Method index: docs/module_framework#modularity-testing-framework 5. In a directory ``tests`` create a ``Makefile`` as below. Line ``generator`` is optional and needed only if you have multi like Bash snippet tests. From 6c9212afb2b41d4df50455a7d05367ee76f13dad Mon Sep 17 00:00:00 2001 From: Irina Gulina Date: Jun 14 2017 07:45:15 +0000 Subject: [PATCH 5/7] docs introducing glossary page --- diff --git a/docs/user_guide/glossary.rst b/docs/user_guide/glossary.rst new file mode 100644 index 0000000..338f2d1 --- /dev/null +++ b/docs/user_guide/glossary.rst @@ -0,0 +1,4 @@ +Glossary +======== + +**Module** A set of packages tested and released together as a distinct unit, complete with the metadata needed to manage it as a unit. May depend on other modules. diff --git a/docs/user_guide/index.rst b/docs/user_guide/index.rst index 6aa21c1..e1702cd 100644 --- a/docs/user_guide/index.rst +++ b/docs/user_guide/index.rst @@ -84,7 +84,7 @@ Contents: #python_tests scheduling #general-reference - #glossary + glossary troubleshooting .. seealso:: From da2a43fcfc4eb7d88b678c2ded5de0cfc692e884 Mon Sep 17 00:00:00 2001 From: Irina Gulina Date: Jun 14 2017 14:23:06 +0000 Subject: [PATCH 6/7] docs typos, errors and whitespaces --- diff --git a/docs/index.rst b/docs/index.rst index f2acea6..3d8d35a 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -11,7 +11,7 @@ Modularity Testing Framework (MTF) is a tool to test `Fedora modules`_. Using MTF you can: * write tests for RPMs, modules and Docker containers -* write multi like Bash snippet tests in YAML definition file +* write multiline Bash snippet tests in YAML definition file * write Bash tests * write Python tests * schedule tests with Jenkins and Taskotron @@ -47,4 +47,4 @@ Index and Search .. seealso:: `webchat.freenode.net `_ - Questions? Help? Ideas? Stop by the #fedora-modularity on freenode IRC chat channel + Questions? Help? Ideas? Stop by the #fedora-modularity chat channelon freenode IRC. diff --git a/docs/installation.rst b/docs/installation.rst index 1907278..cbaabad 100644 --- a/docs/installation.rst +++ b/docs/installation.rst @@ -15,7 +15,7 @@ Vagrant .. _Vagrantfile: https://pagure.io/modularity-testing-framework/blob/master/f/Vagrantfile .. _modularity-testing-framework: https://pagure.io/modularity-testing-framework -The MTF tool has made available for use of two providers: ``libvirt`` (for Linux host only) and ``virtualbox`` (for MAC OS, Windows and Linux hosts), where ``libvirt`` is a default one. See more about Vagrant providers `here`_. +The MTF tool has been made available for use of two providers: ``libvirt`` (for Linux host only) and ``virtualbox`` (for MAC OS, Windows and Linux hosts), where ``libvirt`` is a default one. See more about Vagrant providers `here`_. .. _here: https://www.vagrantup.com/docs/providers/basic_usage.html#default-provider @@ -56,7 +56,7 @@ After preparing the libvirt prerequisites using the instructions above: $ cd $HOME/ # Season to taste. $ git clone https://pagure.io/modularity-testing-framework.git -2. Next, cd into the ``modularity-testing-framework`` directory. +2. Next, enter into the ``modularity-testing-framework`` directory. .. code-block:: bash @@ -79,7 +79,7 @@ After preparing the libvirt prerequisites using the instructions above: # The above will run for a while while it provisions your development environment. $ sudo vagrant reload # Reboot the machine at the end to apply kernel updates, etc. -5. Once you have followed the steps above, you should have a running deployed MTF development machine. ssh into your Vagrant environment:: +5. Once you have followed the steps above, you should have a running deployed MTF development machine. Log into your Vagrant environment:: .. code-block:: bash @@ -96,7 +96,7 @@ Requirements MTF installer pulls its latest dependencies: ``python-devel``, ``python-setuptools`` and ``python-netifcaes``, ``docker``, `avocado`_, ``yaml`` and ``json``. -MTF supports Gherkin-based testing in Python. To write tests in a natural language style, backed up by Python code, install BBD tool `behave`_ . Execute the following command to install behave with pip: +MTF supports Gherkin-based testing in Python. To write tests in a natural language style, backed up by Python code, install the BBD tool `behave`_ . Execute the following command to install behave with pip: .. _avocado: https://avocado-framework.github.io/ .. _behave: http://pythonhosted.org/behave/ @@ -134,4 +134,4 @@ You may also wish to follow the `Pagure MTF repo`_ if you have a Pagure account. :doc:`user_guide/index` User Guide `webchat.freenode.net `_ - Questions? Help? Ideas? Stop by the #fedora-modularity on freenode IRC chat channel + Questions? Help? Ideas? Stop by the #fedora-modularity chat channel on freenode IRC. diff --git a/docs/user_guide/environment_variables.rst b/docs/user_guide/environment_variables.rst index e664927..ae627ed 100644 --- a/docs/user_guide/environment_variables.rst +++ b/docs/user_guide/environment_variables.rst @@ -3,18 +3,29 @@ Environment variables Environment variables allow to overwrite some values of a module configuration file ``config.yaml``. -- **AVOCADO_LOG_DEBUG=yes** enables avocado debug output - - **DEBUG=yes** enables debugging mode to test output -- **CONFIG** defines MTF configuration file. It defaultis to ``config.yaml`` -- **MODULE** defines tested module type, if **defailt-module** is not set in ``config.yanl`` - - - **=docker** uses **docker** section of ``config.yaml`` - - **=rpm** uses **rpm** section of ``config.yaml`` and tests RPMs directly on a host - - **=nspawn** tests RPMs in a virtual environment of lightweight virtualization with systemd-nspawn - -- **URL** overwrites value of **module.docker.container** or **module.rpm.repo**. It has to be proper type what is set in **MODULE** -- **MODULEMDURL** overwrites location of a moduleMD file -- **COMPOSEURL** overwrites location of a compose Pungi build -- **MTF_SKIP_DISABLING_SELINUX=yes** does not disable SELinux In nspawn type on Fedora25 SELinux should be diabled, because it does not work well with selinux enabled, this option allows to not do that. -- **MTF_DO_NOT_CLEANUP=yes** does not cleanup modules between tests. It speeds up test execution. Use only if there is no interference between tests -- **MTF_REMOTE_REPOS=yes** disables downloading of Koji packages and creating local repo, and speeds up test execution. +- **AVOCADO_LOG_DEBUG=yes** enables avocado debug output. +- **DEBUG=yes** enables debugging mode to test output. +- **CONFIG** defines the module configuration file. It defaults to ``config.yaml``. +- **MODULE** defines tested module type, if **default-module** is not set in ``config.yaml``. + + - **=docker** uses the **docker** section of ``config.yaml``. + - **=rpm** uses the **rpm** section of ``config.yaml`` and tests RPMs directly on a host. + - **=nspawn** tests RPMs in a virtual environment of lightweight virtualization with systemd-nspawn. + +- **URL** overrides the value of **module.docker.container** or **module.rpm.repo**. The **URL** should correspond to the **MODULE** variable, for example + + - **URL=docker.io/modularitycontainers/haproxy** if **MODULE=docker** + - **URL=https://phracek.fedorapeople.org/haproxy-module-repo** if **MODULE=nspawn** or **MODULE=rpm** + +- **MODULEMDURL** overwrites the location of a moduleMD file. +- **COMPOSEURL** overwrites the location of a compose Pungi build. +- **MTF_SKIP_DISABLING_SELINUX=yes** does not disable SELinux. In nspawn type on Fedora 25 SELinux should be disabled, because it does not work well with SELinux enabled, this option allows to not do that. +- **MTF_DO_NOT_CLEANUP=yes** does not clean up modules between tests. It speeds up test execution. Use only if there is no interference between tests. +- **MTF_REMOTE_REPOS=yes** disables downloading of Koji packages and creating a local repo, and speeds up test execution. + +.. seealso:: + + :doc:`index` + User Guide + `webchat.freenode.net `_ + Questions? Help? Ideas? Stop by the #fedora-modularity chat channel on freenode IRC. diff --git a/docs/user_guide/how_to_write_conf_file.rst b/docs/user_guide/how_to_write_conf_file.rst index 20781f8..462f9d7 100644 --- a/docs/user_guide/how_to_write_conf_file.rst +++ b/docs/user_guide/how_to_write_conf_file.rst @@ -34,7 +34,7 @@ An example of module general description: * **modulemd-url** contains a link to a moduleMD file * **compose-url** links to a final compose Pungi build. **repo** or **repos** can be used instead, see further * **service** stores a port if a module has any -* **packages** defines a module type (by the moment only `rpms` type is supported) +* **packages** defines a module type (at the moment only `rpms` type is supported) * **testdependencies** covers dependencies to be installed and used in tests An example of module types specification: @@ -66,13 +66,13 @@ An example of module types specification: * **start** defines how to start module service if there is any * **stop** defines how to stop module service if there is any * **status** defines how to check the status of module service if there is any -* **labels** contains docker labels to check if any +* **labels** contains docker labels to check if there is any * **container** contains a link to a container (docker.io or local tar.gz file) * **repo** is used when **compose-url** is not set and contains a repo to be used for rpm module type testing -Multi like Bash snippet tests +Multiline Bash snippet tests ----------------------------- -A ``config.yaml`` file may contain multi like Bash snippet tests directly. Every Bash command has to finish with 0 return code otherwise it returns fail: +A ``config.yaml`` file may contain multiline Bash snippet tests directly. Every Bash command has to finish with 0 return code otherwise it returns fail: .. code-block:: yaml @@ -87,6 +87,13 @@ A ``config.yaml`` file may contain multi like Bash snippet tests directly. Every selcheckError: - 'echo errr | nc localhost 11211 |grep ERROR' -* **test** defines a section of multi like bash snippet tests +* **test** defines a section of multiline bash snippet tests * **processrunning** contains commands to run as tests and displayed as avocado output -* **testhost** is optional and similar to **test**. The difference is that it runs commands on host machine so that there could be more dependencies than just are in a module. +* **testhost** is optional and similar to **test**. The difference is that it runs commands on host machine so that there could be more dependencies than there are just in a module. + +.. seealso:: + + :doc:`index` + User Guide + `webchat.freenode.net `_ + Questions? Help? Ideas? Stop by the #fedora-modularity chat channel on freenode IRC. diff --git a/docs/user_guide/index.rst b/docs/user_guide/index.rst index e1702cd..8d8a871 100644 --- a/docs/user_guide/index.rst +++ b/docs/user_guide/index.rst @@ -5,25 +5,25 @@ User Guide .. _Configuration file: how_to_write_conf_file -2. Optionally write multi like Bash snippet tests directly in ``tests/config.yaml`` file as described in section `Multi like Bash snippet tests`_. +2. Optionally write multiline Bash snippet tests directly in the ``tests/config.yaml`` file as described in section `Multiline Bash snippet tests`_. -.. _Multi like Bash snippet tests: how_to_write_conf_file#multi-like-bash-snippet-tests +.. _Multiline Bash snippet tests: how_to_write_conf_file#multiline-bash-snippet-tests 3. Check the list of `Environment variables`_. .. _Environment variables: environment_variables. 4. Write your tests, for example see `sanity tests`_ and various tests examples in ``/usr/share/moduleframework/examples/testing-module/``. All tests methods are listed in section :ref:`genindex`. - + .. _sanity tests: https://pagure.io/modularity-testing-framework/blob/master/f/examples/template/sanity_template.py -5. In a directory ``tests`` create a ``Makefile`` as below. Line ``generator`` is optional and needed only if you have multi like Bash snippet tests. +5. In the directory ``tests`` create a ``Makefile`` as below. Line ``generator`` is optional and needed only if you have multiline Bash snippet tests. .. code-block:: makefile MODULE_LINT=/usr/share/moduleframework/tools/modulelint.py CMD=python -m avocado run $(MODULE_LINT) *.py - + # all: generator @@ -37,16 +37,16 @@ User Guide IMAGE_NAME = debugging-tools MODULEMDURL=file://debugging-tools.yaml - + all: run default: run - + build: docker build --tag=$(IMAGE_NAME) . - + run: build docker run -it --name $(IMAGE_NAME) --privileged --ipc=host --net=host --pid=host -e HOST=/host -e NAME=$(IMAGE_NAME) -e IMAGE=$(IMAGE_NAME) -v /run:/run -v /var/log:/var/log -v /etc/machine-id:/etc/machine-id -v /etc/localtime:/etc/localtime -v /:/host $(IMAGE_NAME) - + test: build cd tests; MODULE=docker MODULEMD=$(MODULEMDURL) URL="docker=$(IMAGE_NAME)" make all cd tests; MODULE=rpm MODULEMD=$(MODULEMDURL) URL="docker=$(IMAGE_NAME)" make all @@ -58,18 +58,18 @@ User Guide #run tests from a module root directory $ make test - or from ``tests`` directory by running + or from the ``tests`` directory by running .. code-block:: shell - - #run Python tests from tests/ directory + + #run Python tests from the tests/ directory $ sudo MODULE=docker avocado run ./*.py or .. code-block:: shell - #run Bash tests from tests/ directory + #run Bash tests from the tests/ directory $ sudo MODULE=docker avocado run ./*.sh @@ -90,5 +90,4 @@ Contents: .. seealso:: `webchat.freenode.net `_ - Questions? Help? Ideas? Stop by the #fedora-modularity on freenode IRC chat channel - + Questions? Help? Ideas? Stop by the #fedora-modularity chat channel on freenode IRC. From c063c7adaec3fee382674861915f28c077b1b568 Mon Sep 17 00:00:00 2001 From: Irina Gulina Date: Jun 15 2017 13:26:16 +0000 Subject: [PATCH 7/7] docs following PR review remarks --- diff --git a/README.md b/README.md index f9705f7..ade767a 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -Modularity testing framework (MTF) is a tool to test [Fedora modules](https://github.com/modularity-modules). +Modularity testing framework (MTF) is a tool to test components of [a modular Fedora](https://docs.pagure.org/modularity/). For more information, check out the documentation page: diff --git a/docs/index.rst b/docs/index.rst index 3d8d35a..72339e8 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -6,16 +6,17 @@ Welcome to the Modularity Testing Framework documentation! About ===== -Modularity Testing Framework (MTF) is a tool to test `Fedora modules`_. +Modularity testing framework (MTF) is a tool to test components of `a modular Fedora`_. Using MTF you can: * write tests for RPMs, modules and Docker containers * write multiline Bash snippet tests in YAML definition file +* write multihost tests * write Bash tests * write Python tests * schedule tests with Jenkins and Taskotron -* work in Vagrant environment +* run tests on a local host or in Vagrant environment MTF has a presence on the following websites: @@ -24,7 +25,7 @@ MTF has a presence on the following websites: * `MTF's code`_ and the issue tracker for sharing bugs and feature ideas are stored on Pagure. .. _Documentation: http://modularity-testing-framework.readthedocs.io -.. _Fedora modules: https://github.com/modularity-modules +.. _a modular Fedora: https://docs.pagure.org/modularity/ .. _Package repository: https://copr.fedorainfracloud.org/coprs/phracek/Modularity-testing-framework/ .. _MTF's code: https://pagure.io/modularity-testing-framework diff --git a/docs/user_guide/index.rst b/docs/user_guide/index.rst index 8d8a871..478788d 100644 --- a/docs/user_guide/index.rst +++ b/docs/user_guide/index.rst @@ -17,7 +17,7 @@ User Guide .. _sanity tests: https://pagure.io/modularity-testing-framework/blob/master/f/examples/template/sanity_template.py -5. In the directory ``tests`` create a ``Makefile`` as below. Line ``generator`` is optional and needed only if you have multiline Bash snippet tests. +5. In the directory ``tests`` create a ``Makefile`` as below. Mind to keep the ``generator`` line only if there are multiline Bash snippet tests in the ``tests/config.yaml`` file. .. code-block:: makefile