Avoid truncating a pattern in the middle of an UTF-8 character

[b4n]

Fixes #1275, fixes #1805.

[coveralls]

Coverage increased (+0.003%) to 84.415% when pulling 22b51fa on b4n:utf8-pattern-truncation into 3522685 on universal-ctags:master.

[b4n]

@masatake do we wanna change this test case to balance the previous off-by-one-ness? I'm not sure, because now it properly checks the behavior when the line is the exactly as long, short and longer than the truncation, which seems a good thing to do, but you might remember if you had another goal here maybe.

[masatake]

It seems that there a bug has been existed since Exuberant-crags about newline handling.
line returned from readLineFromBypassAnyway can end with or without a newline char.
Both can occurs. Though the code trims the line with line[len - 1] = '\0''.

To combine your fix, I think the diff should be:

diff --git a/main/writer-etags.c b/main/writer-etags.c
index 762c8378..dbd6470f 100644
--- a/main/writer-etags.c
+++ b/main/writer-etags.c
@@ -98,18 +98,18 @@ static int writeEtagsEntry (tagWriter *writer,
                long seekValue;
                char *const line =
                                readLineFromBypassAnyway (etags->vLine, tag, &seekValue);
-               if (line == NULL)
+               if (line == NULL || line[0] == '\0')
                        return 0;
 
                len = strlen (line);
 
                if (tag->truncateLineAfterTag)
                        truncateTagLineAfterTag (line, tag->name, true);
-               else
-                       line [len - 1] = '\0';
+               else if (line [len - 1] == '\n')
+                       line [--len] = '\0';
 
                if (Option.patternLengthLimit < len)
-                       line [Option.patternLengthLimit - 1] = '\0';
+                       line [Option.patternLengthLimit] = '\0';
 
                length = mio_printf (mio, "%s\177%s\001%lu,%ld\n", line,
                                tag->name, tag->lineNumber, seekValue);

[masatake]

@b4n, thank you very much. I will take a look at this weekend.

[masatake]

@b4n, thank you for working on this topic.
Can I ask you to add explanation about the length adjustment for utf-8 input to Truncating the pattern for long input lines section in docs/news.rst?

[b4n]

@masatake Thanks for the review! I made the changes, tell me if it was what you were expecting.