column number wrong when utf-8 double-wide character is substituted after conceal

[girishji]

Steps to reproduce

Use the following script. It conceals letter 'g' with a double-wide char. Do :call cursor(1, 10) and cursor is placed on ば (correct). cursor(1,11) will place cursor on i (correct). cursor(1,12) also places cursor on i (wrong). On the ruler/statusline column numbers are wrong also, after the ば character.

vim --clean -S <(cat <<EOF                                                                                                                   [*main]
    vim9script
    :set conceallevel=2 ruler ft=help
    setline(1, 'At the beginning ')
    matchadd('Conceal', 'g', 10, -1, {conceal: 'ば'})
EOF
)

Expected behaviour

Column numbers are byte-based, and should not change when character is concealed or substituted.

Because of this bug, you can place the cursor using cursor() on some character, but when you get screen based column number using wincol() it would be wrong.

Version of Vim

9.1

Environment

VIM - Vi IMproved 9.1 (2024 Jan 02, compiled Mar 24 2024 08:43:25)
macOS version - arm64
Included patches: 1-200
Compiled by Homebrew
Huge version without GUI.  Features included (+) or not (-):
+acl               +file_in_path      +mouse_urxvt       -tag_any_white
+arabic            +find_in_path      +mouse_xterm       -tcl
+autocmd           +float             +multi_byte        +termguicolors
+autochdir         +folding           +multi_lang        +terminal
-autoservername    -footer            -mzscheme          +terminfo
-balloon_eval      +fork()            +netbeans_intg     +termresponse
+balloon_eval_term +gettext           +num64             +textobjects
-browse            -hangul_input      +packages          +textprop
++builtin_terms    +iconv             +path_extra        +timers
+byte_offset       +insert_expand     +perl              +title
+channel           +ipv6              +persistent_undo   -toolbar
+cindent           +job               +popupwin          +user_commands
-clientserver      +jumplist          +postscript        +vartabs
+clipboard         +keymap            +printer           +vertsplit
+cmdline_compl     +lambda            +profile           +vim9script
+cmdline_hist      +langmap           -python            +viminfo
+cmdline_info      +libcall           +python3           +virtualedit
+comments          +linebreak         +quickfix          +visual
+conceal           +lispindent        +reltime           +visualextra
+cryptv            +listcmds          +rightleft         +vreplace
+cscope            +localmap          +ruby              +wildignore
+cursorbind        +lua               +scrollbind        +wildmenu
+cursorshape       +menu              +signs             +windows
+dialog_con        +mksession         +smartindent       +writebackup
+diff              +modify_fname      +sodium            -X11
+digraphs          +mouse             +sound             -xattr
-dnd               -mouseshape        +spell             -xfontset
-ebcdic            +mouse_dec         +startuptime       -xim
+emacs_tags        -mouse_gpm         +statusline        -xpm
+eval              -mouse_jsbterm     -sun_workshop      -xsmp
+ex_extra          +mouse_netterm     +syntax            -xterm_clipboard
+extra_search      +mouse_sgr         +tag_binary        -xterm_save
-farsi             -mouse_sysmouse    -tag_old_static
   system vimrc file: "$VIM/vimrc"
     user vimrc file: "$HOME/.vimrc"
 2nd user vimrc file: "~/.vim/vimrc"
      user exrc file: "$HOME/.exrc"
       defaults file: "$VIMRUNTIME/defaults.vim"
  fall-back for $VIM: "/opt/homebrew/share/vim"
Compilation: clang -c -I. -Iproto -DHAVE_CONFIG_H -DMACOS_X -DMACOS_X_DARWIN -g -O2 -D_REENTRANT -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1
Linking: clang -o vim -lm -lncurses -lsodium -liconv -lintl -framework AppKit -L/opt/homebrew/opt/lua/lib -llua5.4 -mmacosx-version-min=13.6 -fstack-protector-strong -L/opt/homebrew/opt/perl/lib/perl5/5.38/darwin-thread-multi-2level/CORE -lperl -L/opt/homebrew/opt/python@3.12/Frameworks/Python.framework/Versions/3.12/lib/python3.12/config-3.12-darwin -lpython3.12 -framework CoreFoundation -lruby.3.3 -L/opt/homebrew/Cellar/ruby/3.3.0/lib
~/.vim

Logs and stack traces

No response

[zeertzjq]

Related #4687

[girishji]

Related #4687

It appears that is a different issue, related to supporting string as substitute (instead of single character).

The issue I reported is a bug. Column numbers (based on byte count) should not change when some character is concealed or substituted. What happens on screen (how many cells a character occupies) should not affect byte position in a string/line.

[zeertzjq]

I don't think byte position is wrong. If you press ga you can see that the cursor is actually on the n after using cursor(1, 12), not the i.

[girishji]

I don't think byte position is wrong. If you press ga you can see that the cursor is actually on the n after using cursor(1, 12), not the i.

Good point, but the cursor (visual character on screen) is still on i. So there is a mismatch between where the cursor is visually on the screen vs what Vim thinks (according to ga). Now try to open a popup window at n (using what wincol() reports), it would be incorrectly shifted one cell to the left because of this mismatch.

You can also move the cursor to the right (across the hidden character) and watch column numbers increment on the ruler. They'd be wrong too.

[girishji]

Searching for a string also shows the bug. Say the cursor is on A, and If you search /nn, the cursor lands on in (of beg|in|ning) instead of nn. Line is set as At the beばinninば .

This bug is probably not of much interest since the scenario of substituting single-cell wide char with double-wide char is unlikely. Maybe of importance to non-English users.

[ronin49]

@girishji
i think we will not do it
conceal is just a graphic thing so all vim's mechanics don't care about it
those vim doesn't see what you see using conceal