U+0000 pasted with subsequent cursor position problems

[h-h-h-h]

Does this issue occur when all extensions are disabled?: N/A

• VS Code Version: 1.65.2
• OS Version: Windows_NT x64 10.0.19042

Steps to Reproduce:

1. Open dev console in Firefox and run: console.log([ "foo", "bar" ])
2. Expand Array item and copy 2 or more lines from the Array [... to the <prototype>... line.
3. Pasting it in the hex editor HxD reveals that U+0000 was copied: ...0d 0a 00 0d 0a...
4. Paste it in a VS Code buffer.
5. Place cursor at start of a line with visible characters (not the first).
6. Press Backspace and ArrowUp.
7. Cursor is now to the right of the first character of a line (like this: a|b) -- wrong. It should be at the very beginning of the line. When moving the cursor across the U+0000 character, it halts.

I'm not sure whether pasting U+0000 should just be prevented. It definitely shouldn't be displayed with zero-width however.

[alexdima]

@h-h-h-h I see the null character rendered by the editor. Here is the file which I used: 146326.txt. I clicked "Open anyway" when VS Code warned that the file is binary and the character is rendered for me:

Did you by any chance turn off "Render Control Characters"?

[h-h-h-h]

The bug description is still valid.

My settings when searching for "Render Control Characters":

Your file after clicking "Open anyway" and pasting what I described above (the yellow thingy is new for me; I believe it also appears on German characters "äöüß"):

After step 5 and first part of 6 ("5. Place cursor at start of a [newly pasted] line with visible characters (not the first)." "6. Press Backspace and ArrowUp."):

After last part of step 6:

After inserting an additional character at the very start of every line:

BTW: The yellow-rectangle and cursor position problem is the same when I replace U+0000 with "U+200C ZERO WIDTH NON-JOINER [ZWNJ]". When navigating vertically -- not horizontally -- this behavior should be corrected so that the arrow keys behave as expected. Horizontally, the cursor must still halt on zero-width chars, should you decide to keep them displayed as such (which is also debatable in a monospace editor). (And, in that case, having the cursor to the logical right of a zero-width char and pressing Up, then Down, should IMO place the cursor to the logical left of the zero-width char.)

For ZWNJ, the yellow rectangle makes sense. But U+0000, when pasting, should be displayed the same as when opening a file with it.

[h-h-h-h]

BTW: Since the "NUL" block that is displayed is wider than a regular character, navigating between lines with arrow keys where one line has a "NUL" block is also off (lines with similar length, navigating after the "NUL" block). That is, alternating Up and Down leads to different displayed cursor positions where they should be identical. The "NUL" block should be the width of one character. Or the width of two in conjunction with cursor position correction?

[h-h-h-h]

Related: #2878

[alexdima]

@h-h-h-h In your original description, you mention U+0000 as the problematic character. But I could not reproduce when creating a file containing U+0000 (see attached file), it renders as NUL for me. Could you please attach here the file you have screenshots of such that I could investigate what is the problematic character and what might be going on?

[h-h-h-h]

More problems/inconsistencies:

• When I copy-paste your file's content "This is a string with <...> null" to duplicate it, it's truncated before the null character, so that only "This is a string with <" is pasted. It's the same when running copy("a\u0000b") from the JavaScript dev console in either Firefox or Brave and pasting in VS Code; only a is pasted.
• Whether or not I see a yellow rectangle is different from time to time with identical paste contents. I don't understand what it depends on. The rectangle sometimes doesn't appear, even though there is a zero-width character. (Pasting after your "This is a string with <...> null" and a line break shows the yellow rectangle; pasting it in its own buffer doesn't.)

---

My check with the hex editor HxD was flawed, because it only accepts code points up to U+00FF when pasting. In "A█B" with a high code point, the block is also made into byte "00".

I now used the clipboard inspection tool InsideClipboard (an OS-level app) to check what's really going on.

First, the behavior of Firefox is inconsistent. It copies different sets of formats when you copy via context menu vs. via Ctrl+C from the dev console. With context menu, it's just "CF_UNICODE_TEXT" and "CF_TEXT". I configured InsideClipboard so that is displayed a hex dump, and "U+200B ZERO WIDTH SPACE [ZWSP]" in UTF-16LE was revealed: "0D 00 0A 00 0B 20 0D 00 0A 00". This is also what VS Code saves: 1.txt, 2.txt.

So, this navigation problem isn't about U+0000. But the problem is still real and is related to differing character widths: It's present for the zero-width space, for special code point representations as null in your example file, and obviously more chars of deviating width.

[alexdima]

When I copy-paste your file's content "This is a string with <...> null" to duplicate it, it's truncated before the null character, so that only "This is a string with <" is pasted. It's the same when running copy("a\u0000b") from the JavaScript dev console in either Firefox or Brave and pasting in VS Code; only a is pasted.

This is how the Windows OS works, this is nothing something specific to VS Code. The plaintext clipboard in Windows cannot hold text containing NULL.

Whether or not I see a yellow rectangle is different from time to time with identical paste contents. I don't understand what it depends on. The rectangle sometimes doesn't appear, even though there is a zero-width character. (Pasting after your "This is a string with <...> null" and a line break shows the yellow rectangle; pasting it in its own buffer doesn't.)

This is the Unicode Highlighting feature that you're running into. It highlights special characters, including confusables and invisibles. This feature always works for me.

Based on the latest information, here is a text file that contains both the NULL character and the U+200B ZERO WIDTH SPACE character and here is how it is rendered in VS Code:

146326.txt

This looks correct to me. Please let me know if this doesn't look correct to you.

[h-h-h-h]

This is basically an understandable rendering choice. And it's good to see that there's an otherwise invisible character. But this rendering leads to the cursor position problems (as does the wide rendering of NUL; press Up, Down, etc.):

[alexdima]

Thank you for your feedback. This is a well known problem, the cursor up and down movements do not operate on effective horizontal character offsets. See #142738 (comment)