Make sure that we invoke all the intended regex patterns in ToktokTokenizer...

[alexrudnick]

and do it only once. Also add more tests.

This is for #3202 .

[tomaarsen]

Hello!

Looks great - though it's plausible (but unlikely) that the duplicate FINAL_PERIOD_2 was on purpose. What do you think @alexrudnick?

• Tom Aarsen

[alexrudnick]

Thanks for taking a look, @tomaarsen !

So I considered this possibility, that perhaps FINAL_PERIOD_2 was intended to be called twice, but I think that's unlikely for a few reasons:

• There's no mention of it in the code; if there was some subtle reason why that regex ought to be called twice, then that seems like it'd be worthy of a comment.
• It's not called twice in the original Perl! https://github.com/jonsafari/tok-tok/blob/master/tok-tok.pl
• The only reason why you might need to call it twice would be if some other rule had created a situation where FINAL_PERIOD_2 would need to trigger later on, and I don't see any rules that would create that kind of situation.

Peeking at blame, it looks like @purificant wrote that line, so maybe they could chime in if they like?

[tomaarsen]

Thanks for the analysis! I think that clears it right up, especially the second point. Though @purificant did last affect this line, that was with his #2163 PR that applied black over all code. The only person to meaningfully write/edit this code was @alvations.

This looks ready to merge as far as I'm concerned!

[alexrudnick]

Right on, I just noticed that right after I said it, good catch! Code reformatting patch.

Thanks, merging!