#2994 Handle implicit issue link at start of line (#2987)
Merged by pingou. Opened by adamwill.
adamwill/pagure implicit-issue-fix  into  master

Download 2994.patch

This treats a line starting with #123 - where 123 is a valid
issue or pull request ID - as an implicit link, rather than as
a header line.

Doing this is a bit complicated, as markdown does header line
processing early, in the block processors (that's before the
inline patterns we usually use). So we can't do this entirely
with inline patterns. We have to add a preprocessor that runs
before the block processors, and mungs the relevant strings so
the block processor doesn't turn them into header HTML. Then
we adjust the ImplicitIssuePattern to handle these modified
strings as well.

This also adds tests for implicit links, enabled by mocking out
enough bits of flask that _get_ns_repo_user is happy.

We also clean up IMPLICIT_ISSUE_RE, and the similar
IMPLICIT_PR_RE, a bit. The [^|\w] bit of these regexes was
both bizarre and unnecessary. That is an inverted set which
will match anything but a literal pipe or any character in the
\w class. The following bit, (?<!\w), is a negative lookbehind
assertion which means 'match unless the previous character is in
the \w class'. So these two actually apply to the same character
and are almost entirely redundant.

I think the weird set was meant to be something like (^|w),
and the approximate idea here was to match 'start of string
or any non-word character followed by a #'. If so, then in fact
just removing the wacky inverted set is all we need to do, as
negative lookbehind assertions are allowed to match at the start
of the string. (There is actually a whole hidden complication
here where markdown, behind the scenes, adds some more bits to
the pattern we feed it to form a complete regex, but it happens
that everything works OK with that). The tests should suffice to
demonstrate that these regexes still behave as we expect. These
regexes originally came from @ralph, who says he's OK with this
change.

Signed-off-by: Adam Williamson awilliam@redhat.com

Couple of notes here:

  • We could change the start of the regexes to something like (^|[^\w]), to avoid using a negative lookbehind, which I guess is slightly exotic. But it's not that much simpler, and would require tweaking _obj_anchor_tag. Fun BONUS note: I actually thought this would not work, because of the 'hidden complication' I mention in the commit message - behind the scenes, markdown takes the pattern we specify and stuffs it into this format string: r'^(.*?)%s(.*)$'. So with that change, our true regex would be r'^(.*?)(^|[^\w])(?:PREPROCIMPLLINK|#)([0-9]+)(.*)$', and I thought that wouldn't match because it needs to match both the first two ^ characters. But actually, that works! re.compile(r'^^foo').match('foo') works, if you try it. Regexes are fun.

  • Some of the lines added to the test are meant to illustrate some possibly-questionable behaviour (and make sure we don't change these unusual cases without intending to). Mainly that using "not \w" as these regexes do means we match things like .#123.. I think this is defensible, but you could take the other point of view (that we should only match implicit links surrounded by whitespace), and it's possible I guess that someone might change this without realizing it. Now they can't. :)

This change (and the one at line #91) is a consequence of dropping the bogus inverted set in the regex. It's all to do with some funky internal details in markdown. I mentioned elsewhere that markdown actually adds some extra groups to the pattern's regex, which is important. Now also take a look at what markdown does when actually applying the pattern.

What it ultimately spits back out after applying the modified regex is a string consisting of leftData, match.group(1), placeholder, and match.groups()[-1].

leftData is just any text that comes before the point where the regex is actually applied. Then it gets interesting.

match.group(1) is the contents of the first group in the regex - which is the contents of that (.*?) group it adds at the start of the regex text we give it.

placeholder is what ultimately becomes the text the pattern's handleMatch() method spits back out after operating on the match.

match.groups()[-1] is the last match group, which is the contents of the (.*) group markdown adds to the end of the regex.

So basically it's trying to put all the bits back together again. This, however, relies on the pattern's handleMatch method correctly handling all the text in the part of the regex that we specify. What this space character here is actually doing is replacing the character consumed by the funky [^|\w] inverted set. In the modified regexes, that character isn't consumed by any part of IMPLICIT_ISSUE_RE; instead that character winds up in the (.*?) group that markdown adds, and that it then includes in the output of __applyPattern()...so we no longer need to do that work.

Whew, that's a lot of text to explain why a space went away :) But that's the reason. Now, the fun part: there was actually a subtle bug in the old code here. It assumes (though I doubt whoever wrote this actually totally understood all of this) that the character consumed by the set is a space. However, that's not always true. And THAT is why if you write something like foobar.#23 in a comment, it gets turned into foobar (link to issue #23) - the . is turned into a space.

With this tweak, that bug is fixed. Neat, no? :)

Do we want to .strip() or is it no necessary?

I don't think it's necessary, as markdown doesn't seem to treat a line that starts with any number of spaces and then a # as a header line...only lines that really start with #. So we don't need to worry about leading spaces here as markdown will leave lines with leading spaces alone and the inlinepattern will handle them fine.

rebased onto 1c68407538c5bc1fb95d86c696a7509ceff70de9

2 new commits added

  • Handle implicit issue link at start of line (#2987)
  • Remove some funky bits from inline pattern regexes

rebased onto 75c9f6d4ae440df3c00625f79ea20079628a8d5f

Thanks!! :)

Pull-Request has been merged by pingou

Metadata