From 50fea9ef9553fbaff782400eb744043ad5155cc6 Mon Sep 17 00:00:00 2001 From: Michal Srb Date: Oct 12 2020 08:06:05 +0000 Subject: [PATCH 1/6] Add page about generic tests. --- diff --git a/modules/ROOT/nav.adoc b/modules/ROOT/nav.adoc index 00e8b1d..1e8a447 100644 --- a/modules/ROOT/nav.adoc +++ b/modules/ROOT/nav.adoc @@ -4,6 +4,7 @@ ** xref:onboarding-of-a-ci-system.adoc[Onboarding of a CI System] ** xref:pull-requests.adoc[Pull Requests] ** xref:share-test-code.adoc[Share Test Code] +** xref:generic_tests.adoc[Generic Tests] ** xref:examples.adoc[Examples] * Architecture ** xref:manifesto.adoc[Manifesto] diff --git a/modules/ROOT/pages/generic_tests.adoc b/modules/ROOT/pages/generic_tests.adoc new file mode 100644 index 0000000..0d90ed9 --- /dev/null +++ b/modules/ROOT/pages/generic_tests.adoc @@ -0,0 +1,77 @@ +:toc: + += Generic Tests = + + +Generic tests are tests that don't check only specific components (e.g. "dnf" or "kernel"), but can be typically applied to all artifacts of a certain type. An example of such a test could be a test that can be run on all RPM builds in a Bodhi update. + + +Fedora CI currently provides 3 generic tests. All of them run on packages that are being pushed to Rawhide via automatic Bodhi updates. + + +== rpmdeplint == + + +`rpmdeplint` is a generic test that tries to identify problems in RPM packages in context of their dependency graph. + +There are four different checks that the test performs: + + +check-sat:: +This checks if all runtime dependencies of the given RPM packages would be satisfied if the packages are pushed to the given repository (like Rawhide). + +check-repoclosure:: +This is similar to check check-sat test, but check-repoclosure checks that all packages in the given repository that don't have any runtime dependency problems before the new packages are added, won't have any dependency problems after the packages are added to the repository. + +Packages are only considered to be available for dependency resolution if they are the latest version and not obsoleted by any other package. Therefore this check can detect problems where a package under test is updating an existing package in the repository, but it no longer provides a requirement needed by some other package in the repository. + +Packages with pre-existing repoclosure problems are ignored. + +check-conflicts:: +This checks for undeclared file conflicts in the given RPM packages: that is, when one of the given package contains a file +which is also contained in some other package. + +This command will not report a file as conflicting between two packages if: + +* there is an explicit RPM Conflicts between the two packages; or + +* the file’s checksum, permissions, owner, and group are identical in both packages (RPM allows both packages to own +the file in this case); or + +* the file’s color is different between the two packages (RPM will silently resolve the conflict in favour of the +64-bit file). + +[NOTE] +==== +Sometimes files can be owned by literally thousands of different packages. In order to properly check that there are no file conflicts, `rpmdeplint` would need to download all other packages. This would be very slow, so only a single other package is downloaded and checked. +==== + +check-upgrade:: +Checks that there are no existing packages in the repositories which would upgrade or obsolete the given packages. + +If this check fails, it means that the package under test will never be installed (since the package manager will always pick the newer or obsoleting package from the repositories instead) which is not desirable, assuming the package is intended as an update. + + +== rpminspect == + +RPM build deviation analysis tools. `rpminspect` looks at the output of an RPM build (e.g., the output of a Koji build) and examines the contents of the build artifacts to report: + +* Policy compliance + +* Changes from a previous build to the current build + +* General correctness and best practices + +`rpminspect` performs more than 30 different checks on packages. To list all of them, with a nice description, please run `rpminspect -v -l`. + + +== installability == + +This is a generic test that tries to perform following operations on the given packages: + +* dnf install +* dnf remove +* dnf update +* dnf downgrade + +All problems are logged and reported. From ebc6a9acec1cdc561b1ff994f379c5b208875f48 Mon Sep 17 00:00:00 2001 From: Michal Srb Date: Oct 12 2020 10:27:14 +0000 Subject: [PATCH 2/6] [generic_tests] Have one sentence per line --- diff --git a/modules/ROOT/pages/generic_tests.adoc b/modules/ROOT/pages/generic_tests.adoc index 0d90ed9..2b3a995 100644 --- a/modules/ROOT/pages/generic_tests.adoc +++ b/modules/ROOT/pages/generic_tests.adoc @@ -3,10 +3,12 @@ = Generic Tests = -Generic tests are tests that don't check only specific components (e.g. "dnf" or "kernel"), but can be typically applied to all artifacts of a certain type. An example of such a test could be a test that can be run on all RPM builds in a Bodhi update. +Generic tests are tests that don't check only specific components (e.g. "dnf" or "kernel"), but can be typically applied to all artifacts of a certain type. +An example of such a test could be a test that can be run on all RPM builds in a Bodhi update. -Fedora CI currently provides 3 generic tests. All of them run on packages that are being pushed to Rawhide via automatic Bodhi updates. +Fedora CI currently provides 3 generic tests. +All of them run on packages that are being pushed to Rawhide via automatic Bodhi updates. == rpmdeplint == @@ -23,7 +25,8 @@ This checks if all runtime dependencies of the given RPM packages would be satis check-repoclosure:: This is similar to check check-sat test, but check-repoclosure checks that all packages in the given repository that don't have any runtime dependency problems before the new packages are added, won't have any dependency problems after the packages are added to the repository. -Packages are only considered to be available for dependency resolution if they are the latest version and not obsoleted by any other package. Therefore this check can detect problems where a package under test is updating an existing package in the repository, but it no longer provides a requirement needed by some other package in the repository. +Packages are only considered to be available for dependency resolution if they are the latest version and not obsoleted by any other package. +Therefore this check can detect problems where a package under test is updating an existing package in the repository, but it no longer provides a requirement needed by some other package in the repository. Packages with pre-existing repoclosure problems are ignored. @@ -43,18 +46,21 @@ the file in this case); or [NOTE] ==== -Sometimes files can be owned by literally thousands of different packages. In order to properly check that there are no file conflicts, `rpmdeplint` would need to download all other packages. This would be very slow, so only a single other package is downloaded and checked. +Sometimes files can be owned by literally thousands of different packages. +In order to properly check that there are no file conflicts, `rpmdeplint` would need to download all other packages. +This would be very slow, so only a single other package is downloaded and checked. ==== check-upgrade:: Checks that there are no existing packages in the repositories which would upgrade or obsolete the given packages. -If this check fails, it means that the package under test will never be installed (since the package manager will always pick the newer or obsoleting package from the repositories instead) which is not desirable, assuming the package is intended as an update. +If this check fails, it means that the package under test will never be installed (since the package manager will always pick the newer or obsoleting package from the repositories instead) which is not desirable, assuming the package is intended as an update. == rpminspect == -RPM build deviation analysis tools. `rpminspect` looks at the output of an RPM build (e.g., the output of a Koji build) and examines the contents of the build artifacts to report: +RPM build deviation analysis tools. +`rpminspect` looks at the output of an RPM build (e.g., the output of a Koji build) and examines the contents of the build artifacts to report: * Policy compliance @@ -62,7 +68,8 @@ RPM build deviation analysis tools. `rpminspect` looks at the output of an RPM b * General correctness and best practices -`rpminspect` performs more than 30 different checks on packages. To list all of them, with a nice description, please run `rpminspect -v -l`. +`rpminspect` performs more than 30 different checks on packages. +To list all of them, with a nice description, please run `rpminspect -v -l`. == installability == From c04920389805c811d665fa38c0bfe9b3fd55a4dd Mon Sep 17 00:00:00 2001 From: Michal Srb Date: Oct 12 2020 10:28:39 +0000 Subject: [PATCH 3/6] [generic_tests] Turn rpmdeplint checks into 3rd level headers --- diff --git a/modules/ROOT/pages/generic_tests.adoc b/modules/ROOT/pages/generic_tests.adoc index 2b3a995..9582533 100644 --- a/modules/ROOT/pages/generic_tests.adoc +++ b/modules/ROOT/pages/generic_tests.adoc @@ -19,10 +19,10 @@ All of them run on packages that are being pushed to Rawhide via automatic Bodhi There are four different checks that the test performs: -check-sat:: +=== check-sat === This checks if all runtime dependencies of the given RPM packages would be satisfied if the packages are pushed to the given repository (like Rawhide). -check-repoclosure:: +=== check-repoclosure === This is similar to check check-sat test, but check-repoclosure checks that all packages in the given repository that don't have any runtime dependency problems before the new packages are added, won't have any dependency problems after the packages are added to the repository. Packages are only considered to be available for dependency resolution if they are the latest version and not obsoleted by any other package. @@ -30,7 +30,7 @@ Therefore this check can detect problems where a package under test is updating Packages with pre-existing repoclosure problems are ignored. -check-conflicts:: +=== check-conflicts === This checks for undeclared file conflicts in the given RPM packages: that is, when one of the given package contains a file which is also contained in some other package. @@ -51,7 +51,7 @@ In order to properly check that there are no file conflicts, `rpmdeplint` would This would be very slow, so only a single other package is downloaded and checked. ==== -check-upgrade:: +=== check-upgrade === Checks that there are no existing packages in the repositories which would upgrade or obsolete the given packages. If this check fails, it means that the package under test will never be installed (since the package manager will always pick the newer or obsoleting package from the repositories instead) which is not desirable, assuming the package is intended as an update. From adb8c42f57f860172ea24c0bcafed61be79948f2 Mon Sep 17 00:00:00 2001 From: Michal Srb Date: Oct 12 2020 10:35:29 +0000 Subject: [PATCH 4/6] [generic_tests] perform *the* following operations --- diff --git a/modules/ROOT/pages/generic_tests.adoc b/modules/ROOT/pages/generic_tests.adoc index 9582533..c47a9c5 100644 --- a/modules/ROOT/pages/generic_tests.adoc +++ b/modules/ROOT/pages/generic_tests.adoc @@ -74,7 +74,7 @@ To list all of them, with a nice description, please run `rpminspect -v -l`. == installability == -This is a generic test that tries to perform following operations on the given packages: +This is a generic test that tries to perform the following operations on the given packages: * dnf install * dnf remove From fd47b3404e29c3caeac159b550bf38cc3e594464 Mon Sep 17 00:00:00 2001 From: Michal Srb Date: Oct 12 2020 16:29:22 +0000 Subject: [PATCH 5/6] [generic_tests] rpminspect: add note about the previous build --- diff --git a/modules/ROOT/pages/generic_tests.adoc b/modules/ROOT/pages/generic_tests.adoc index c47a9c5..a405e61 100644 --- a/modules/ROOT/pages/generic_tests.adoc +++ b/modules/ROOT/pages/generic_tests.adoc @@ -65,6 +65,7 @@ RPM build deviation analysis tools. * Policy compliance * Changes from a previous build to the current build +** the previous build is the latest build in the stable repository (in Rawhide, it is simply the previous build) * General correctness and best practices From 8581b388964de4db52d1118be0a41f0114e94490 Mon Sep 17 00:00:00 2001 From: Michal Srb Date: Oct 14 2020 10:42:32 +0000 Subject: [PATCH 6/6] [generic_tests] Address sturivny's PR comments Thanks Serhii ;) --- diff --git a/modules/ROOT/pages/generic_tests.adoc b/modules/ROOT/pages/generic_tests.adoc index a405e61..d87babf 100644 --- a/modules/ROOT/pages/generic_tests.adoc +++ b/modules/ROOT/pages/generic_tests.adoc @@ -3,7 +3,7 @@ = Generic Tests = -Generic tests are tests that don't check only specific components (e.g. "dnf" or "kernel"), but can be typically applied to all artifacts of a certain type. +Generic tests are tests that don't check only specific components (e.g. "dnf" or "kernel") but can be typically applied to all artifacts of a certain type. An example of such a test could be a test that can be run on all RPM builds in a Bodhi update. @@ -14,7 +14,7 @@ All of them run on packages that are being pushed to Rawhide via automatic Bodhi == rpmdeplint == -`rpmdeplint` is a generic test that tries to identify problems in RPM packages in context of their dependency graph. +`rpmdeplint` is a generic test that tries to identify problems in RPM packages in the context of their dependency graph. There are four different checks that the test performs: @@ -31,17 +31,17 @@ Therefore this check can detect problems where a package under test is updating Packages with pre-existing repoclosure problems are ignored. === check-conflicts === -This checks for undeclared file conflicts in the given RPM packages: that is, when one of the given package contains a file +This checks for undeclared file conflicts in the given RPM packages: that is when one of the given packages contains a file which is also contained in some other package. This command will not report a file as conflicting between two packages if: * there is an explicit RPM Conflicts between the two packages; or -* the file’s checksum, permissions, owner, and group are identical in both packages (RPM allows both packages to own +* the file’s checksum, permissions, owner, and group are identical in both packages (RPM allows both packages to own the file in this case); or -* the file’s color is different between the two packages (RPM will silently resolve the conflict in favour of the +* the file’s color is different between the two packages (RPM will silently resolve the conflict in favor of the 64-bit file). [NOTE] @@ -54,7 +54,7 @@ This would be very slow, so only a single other package is downloaded and checke === check-upgrade === Checks that there are no existing packages in the repositories which would upgrade or obsolete the given packages. -If this check fails, it means that the package under test will never be installed (since the package manager will always pick the newer or obsoleting package from the repositories instead) which is not desirable, assuming the package is intended as an update. +If this check fails, it means that the package under test will never be installed (since the package manager will always pick the newer or obsoleting package from the repositories instead) which is not desirable, assuming the package is intended as an update. == rpminspect ==