I've pushed a feature branch 'no-must' trying to solve this. An example of output from this is at https://bugzilla.rpmfusion.org/show_bug.cgi?id=2300

Well ok, so my look at the issue:

Yes the MUSTs and SHOULDs can be distracting. However the separation into different sections is normally done by type of check (i.e. Generic/Python/Java/etc). I'd like to keep that. So...how about we do something that would produce:

{{{
Generic
=======
MUST:
[x] ...

SHOULD:
[!] ...

Java

MUST:
[x] ...
...
SHOULD:
...
}}}

Does that look acceptable?

I've tried to think about this before implementing. My basic conclusion is that the separation into sections is quite important for implementors (where did my test go?), but not for reviewers. Frankly, I don't care from which section the test is when doing a review.

The proposed layout works, but adds a lot of overhead: heading, blanks. But since you somehow is the helmsman, it's of course acceptable :) Also, it's not really consistent: from review perspective, the MUST/SHOULD/EXTRA is the most important IMHO. So if all these headings should be used, I'd prefer:

{{{
MUST
Generic
[x]...
Java
[x]....

 SHOULD
 Generic
 [x]...
 Java
 [x]....
 EXTRA
 Generic 
 [x]....

}}}

Another solution might be to add debug output like today to help implementors while keeping proposed layout.

I like your layout better than mine truth be told. As long as checks are grouped together by type (Generic/Java/etc) I am OK. So I'd be up to using this for 0.2.x release as well

Fixed in 0e6f54b1d9c9aefcd