Tab completion incorrectly completes the wrong file when starting with the end of the filename

[protist]

Please check the existing issues to make sure you're not duplicating a report.

For bug reports, please provide the following information:

In a terminal, run zmanage info and paste the output below:

$ zmanage info
zsh: command not found: zmanage

But I'm running the zim master via Arch Linux AUR's zsh-zim-git: 9eaba9a

• gist of .zimrc: zmodules=(completion)
• gist of .zshrc: source /usr/lib/zim/init.zsh

Description

Tab completion incorrectly fully completes the wrong file when starting with the end of the filename.

Steps to reproduce

Tab completion works fine if starting from the start of the filename.

$ touch foo-bar foo-baz-bar
$ touch f<tab>
$ touch foo-ba<tab>
 -- file --
foo-bar      foo-baz-bar

However, if I start typing from the back (or middle) of the filename, it fails.

$ touch bar<tab>
$ touch foo-bar<tab>
           ^ cursor here
$ touch foo-bar
                ^ cursor here

I would expect the second tab to bring up the completion-selection menu. However, it immediately completes to the shorter version instead.

Another slightly different case is if I type the entirety of the shared string.

$ touch -bar<tab>
$ touch foo-bar
                ^ cursor here

In this case, I'm not even given the cursor in the middle, it jumps straight to the end.

[Eriner]

I think the current behavior makes sense. It won't bring up the tab complete menu in line two of your second code block because the test of "is valid file under cursor, or can we complete to one" passes on the first condition. This applies to the third code block as well. For sake of completion, you're expecting it to behave as below:

$ touch foo-chem-bar && rm foo-bar
$ touch -bar<tab>
$ touch foo--bar
            ^ cursor here
 -- file --
foo-baz-bar   foo-chem-bar

Notice the removal of foo-bar and the addition of foo-chem-bar

Your example doesn't work because it completes that to the prefix of the possible options ("foo"), and then performs the "is this a valid entry" test when you hit tab the second time, which passes.

[protist]

Yes, your code block is what I'm expecting.

IMO merely passing "is this a valid entry" test shouldn't cause tab to move on, to the end of the line. Instead, I think it should be "is this the only valid entry". Otherwise, how would you ever get to select/complete foo-baz-bar in my second block?

In addition, the current behaviour is still different to when you start typing from the beginning of the string. In the following example, the two files have the same beginning. After the second <tab>, it would pass "is this a valid entry" test, but won't auto-select it here (presumably because it doesn't pass "is this the only valid entry").

$ touch foo-bar foo-bar-baz
$ touch f<tab>
$ touch foo-bar<tab>
               ^ cursor here
$ touch foo-bar
 -- file --
foo-bar      foo-bar-baz

[Eriner]

Hm. I'm not sure what would need to be changed to meet your expectations of behavior of the shell. This is one of those quirks where I'm not sure where the "fix" needs to be.

[protist]

I'm not sure if you mean you are not sure what part of the code to change, or what end behaviour to change? If the latter, then my primary expectation would be as follows.

1. My main confusion is that tab should not auto-complete if there are other alternatives available. I feel like this behaviour is implied especially if the first tab places the cursor in the first ambiguous location, as per the second code block in the original post.

2. I guess this should only apply to middle-of-the-string completion. If the user starts typing at the beginning of the string, I suppose the default option should be to auto-complete the full string, even if there is another file that contains the visible "typed-out" string.

Thinking about it a bit more, that's probably the problem with the current implementation. As per the second code block above:

$ touch bar<tab>      # this tab auto-completes from the back/middle, as per (1)
$ touch foo-bar<tab>  # this tab has no knowledge of the previous completion, and completes as (2)
           ^ cursor here
$ touch foo-bar
                ^ cursor here

I'm not sure what the resolution is programmatically. Is there no way to "record" the type of auto-completion in the first line? Or could auto-completion behave differently if the cursor is located in the middle of the string, at an ambiguous location?

[Eriner]

I meant what zstyle or other options need to be changed to provide the behavior you expect.

[protist]

Oh. Right. Apologies for the misinterpretation.

[ericbn]

@protist, the Completion and expansion title in Chapter 6 of the Z-Shell Guide has:

% ls
filename1
% ls filex

Move the cursor to the x and hit tab. With expand-or-complete nothing happens; it's trying to complete a file called 'filex' --- or, with the option COMPLETE_IN_WORD set, it's trying to find a file whose name starts with 'file' and ends with 'x'. If you do
bindkey '^i' expand-or-complete-prefix
and try the same experiment, you will find the whole thing is completed to 'filename1x', so that the 'x' was ignored, but not removed.

TL;DR: Typing ls filex and moving the cursor over the x yields different results depending on the zsh configuration used:

Configuration	Result
bindkey '^i' expand-or-complete	Tries to complete a file called 'filex'. This is currently how zim is configured to work
bindkey '^i' expand-or-complete && setopt COMPLETE_IN_WORD	Tries to find a file whose name starts with 'file' and ends with 'x'
bindkey '^i' expand-or-complete-prefix	Ignores the 'x' when completing, but doesn't remove it

@protist, it the behavior you're looking for between these 3 options here?

[ericbn]

@protist, using setopt COMPLETE_IN_WORD will give you the behavior you expect:

$ setopt COMPLETE_IN_WORD
$ touch foo-bar foo-baz-bar
$ touch bar<tab>
$ touch foo-bar<tab>
           ^ cursor here
$ touch foo-bar
              ^ cursor here
-- file --
foo-bar      foo-baz-bar

I think this is something good to have in zim as a sane default. @Eriner, thoughts?

[protist]

@ericbn That looks ideal from your example, but it doesn't seem to work for me. It still has the same behaviour as prior to changing that setting. i.e. it still fully completes the word.

$ setopt COMPLETE_IN_WORD
$ touch bar<tab>
$ touch foo-bar<tab>
           ^ cursor here
$ touch foo-bar
                ^ cursor here

EDIT: this is with the minimal config from the first post.

[protist]

Thank you @ericbn. That commit works perfectly! Now the behaviour is as follows.

$ touch bar<tab>
$ touch foo-bar<tab>
              ^ cursor here
$ touch foo-bar
              ^ cursor here
-- file --
foo-bar      foo-baz-bar

And a subsequent tab can select from the menu. This is great, and no longer automatically completes. Just for the record, the cursor position was slightly unintuitive at first. Previously, it was at the -, and now it's at the r. The latter is the first ambiguous character, so it makes sense. I guess previously zsh considered - to act more as a word break, but now I think about it, the current behaviour makes more sense to me. I think it's better like this, agnostic of what a "word" is.

Thanks again!

[ericbn]

@protist, I also think it behaves better now. But having just fuzzy matching is giving me too many matches, as I usually want to match from the beginning of the name.

So maybe having zsh first try to match from the beginning, and then (if there's no match) try to do the fuzzy match is a better configuration. All case insensitive, of course.

Agree? @Eriner, thoughts?

[protist]

I haven't really noticed anything annoying yet personally, but I think I probably agree with you conceptually. Presumably, if you have barty and foobar in a directory, then typing bar or even b should complete to the former. That does make sense to me.

[ericbn]

Done. Thanks again for your input, @protist.

[protist]

No worries @ericbn. Thank you for the quick fixes and responsiveness!

Unfortunately though, the behaviour of your latest commit is essentially back to where we started!

$ touch foo-bar foo-baz-bar
$ touch bar<tab>
$ touch foo-bar<tab>
              ^ cursor here
$ touch foo-bar
                ^ cursor here

The second tab gives no option to complete, unlike zim's previous behaviour here. I can see the internal logic; it's consistent with your comments to "first try to match from the beginning". I'm not really sure what the resolution should be. The behaviour after the first tab makes sense. In isolation (with the first tab's behaviour hidden), so does the second. Perhaps one way to differentiate our expected behaviour might be to use the position of the cursor before the second tab is pressed. Is that possible/reasonable?

[YonatanAhituv]

It works for me: (asciinema recording). I'm on the latest commit.

[protist]

@AtomicCoding I don't think you are testing the same thing, but it's hard to tell without knowing your directory's contents. In your example, create another file called mafoobarin.py first, then try the tab completion of in.py again, pressing tab twice. It will only auto-complete main.py. That's what this issue is referring to.

[YonatanAhituv]

@protist Sorry! Misunderstood!

[protist]

@AtomicCoding No worries. Thanks for the interest though!

[YonatanAhituv]

@protist It still seems to work (recording).

[protist]

@AtomicCoding Huh, nice. Yes, that's exactly how it should behave IMO. Which commit are you using? I just updated to master (13eb86d), but I'm still seeing the same behaviour as before.

[ericbn]

@AtomicCoding, I cannot reproduce your last video recording with touch mafoobarin.py main.py<enter>touch in.py<tab><tab>. What you have there is the desired behavior indeed, but that's not what I'm getting yet.

My zim is also updated to master (13eb86d), and I tried it with zsh 5.3 and zsh 5.5.

[YonatanAhituv]

@ericbn @protist
I'm on the latest version of zim and on zsh 5.5.
These are my zsh options:

alwaystoend
autocd
autopushd
autoresume
nobgnice
nocaseglob
nocheckjobs
noclobber
completeinword
correct
extendedglob
extendedhistory
globcomplete
histignorealldups
histignoredups
histignorespace
histsavenodups
histverify
nohup
incappendhistory
interactive
interactivecomments
nolistbeep
login
longlistjobs
menucomplete
monitor
pathdirs
promptsubst
pushdignoredups
pushdsilent
pushdtohome
sharehistory
shinstdin
zle

Maybe try running:

setopt alwaystoend autocd autopushd autoresume nobgnice nocaseglob nocheckjobs noclobber completeinword correct extendedglob extendedhistory globcomplete histignorealldups histignoredups histignorespace histsavenodups histverify nohup incappendhistory interactive interactivecomments nolistbeep login longlistjobs menucomplete pathdirs promptsubst pushdignoredups pushdsilent pushdtohome sharehistory shinstdin

And if that still isn't the culprit, maybe it's plugin loading order:

zmodules=(directory environment git git-info history input utility brobot \
           brew prompt completion colors bd \
          osx notify python directory 256color interactive-cd \
          autosuggestions syntax-highlighting history-substring-search)

Seems to be the plugins I have loaded, removing all my custom plugins seemed to change it back to the incorrect behavior. Maybe they change zim's completion config or rebind things?

[Eriner]

@AtomicCoding What happens if you move completion to be the last module loaded?

[YonatanAhituv]

@Eriner Still works. I did some testing and it seems to be my custom plugin brobot. The only thing in the plugin that could affect it is setopt EXTENDED_GLOB GLOB_COMPLETE MENU_COMPLETE. It seems to be setopt GLOB_COMPLETE. That made the difference for me.

[YonatanAhituv]

@Eriner @ericbn @protist Can you try setting setopt GLOB_COMPLETE and trying again?

GLOB_COMPLETE
When the current word has a glob pattern, do not insert all the words resulting from the expansion but generate matches as for completion and cycle through them like MENU_COMPLETE. The matches are generated as if a '*' was added to the end of the word, or inserted at the cursor when COMPLETE_IN_WORD is set. This actually uses pattern matching, not globbing, so it works not only for files but for any completion, such as options, user names, etc.
Note that when the pattern matcher is used, matching control (for example, case-insensitive or anchored matching) cannot be used. This limitation only applies when the current word contains a pattern; simply turning on the GLOB_COMPLETE option does not have this effect.

Opened PR, #264

[ericbn]

Doing setopt GLOB_COMPLETE works, but I consider that a side effect, as GLOB_COMPLETE is not what we want to do here.

[protist]

@ericbn Was this meant to be closed? The referenced commit was an old one, right? And has subsequently been modified anyway?

[ericbn]

I think this still needs to be fixed, and that the current configuration we have is not good enough.

[ericbn]

@Eriner, I'm divided here after the long discussion that happened around this issue.

I'm not happy with the current matcher-list config we have in our completion module. I've gathered some nice configurations that work. Can you help me choose?

• zstyle ':completion:*' matcher-list 'm:{a-zA-Z}={A-Za-z} r:|[.,_-]=** r:|=*'
  Completes f-b to foo-bar (or to foo-something-bar) or anything with the separators above.
• zstyle ':completion:*' matcher-list 'm:{a-zA-Z}={A-Za-z} l:|=* r:|=*'
  Completes with any substring match, in the beginning, middle or end of the trial completion strings. Has the weird behavior of completing bar to or-bar, when there are foo-bar and foo-baz-bar files.
• zstyle ':completion:*' matcher-list 'm:{a-zA-Z}={A-Za-z} r:|?=** r:|=*'
  Fuzzy match, so even fb can match foobar.

All above are case-insensitive. Only the first one matches from the beginning first. I wish I could make the last one to that, but I could not.

... Or maybe something else that works?

[edwardsmit]

Does there need to be one chosen? Can you maybe make this configurable? Along with this excellent explanation the end user can then make his own decision.

[protist]

Can you maybe make this configurable?

@edwardsmit You can just put the relevant zstyle line in your zshrc, after the zim initiation, of course. Only the last definition is used. (You can check this with zstyle -L | grep matcher.)

@ericbn I've actually lost track a bit of which configs we've tried before. I tested the suggestions in a new directory after touch foo-bar foo-something-bar, and basically tried to break each one. 😬 My bullet points correspond with each bullet point of code from your last comment, in the same order.

• touch bar<TAB> does nothing.
• touch f-b<TAB> does nothing. (Also, oddly enough touch bar<TAB> completed to -bar for me with my two files, whereas with the two files you mention, I get the same behaviour as you.)
• This one seemed the most robust, but it's liberalness can obviously be a little off-putting at time. Also, I could finally make it break with foo-bar and foo-baz-bar as follows:

$ touch bar<tab>
$ touch or-bar<tab>
        ^ cursor moves here
$ touch or-bar
              ^ cursor moves here
-- no matches found --

(Actually, this fails similarly to some other code previously in this thread.)

[ericbn]

@protist, here I am back with another "trial". :- )

I found an issue in unix.stackexchange like yours. I tried the first answer and got satisfactory results with it. It's not fuzzy matching, but completes from the beginning first, then from middle or end.

The original answer suggests:

zstyle ':completion:*' matcher-list 'r:|=*' 'l:|=* r:|=*'

which should be the equivalent of:

zstyle ':completion:*' matcher-list 'r:|=*' '+l:|=*'

And a case-insentivite variation should be:

zstyle ':completion:*' matcher-list 'm:{a-zA-Z}={A-Za-z} r:|=*' '+l:|=*'

I'm using the last one above and could not break it with any of the tests we've been making here. Can you please try too?

[YonatanAhituv]

@ericbn

Works for me!

[protist]

I found an issue in unix.stackexchange exactly like yours

Huh, what a startling coincidence! 😲

However, I tried all three variants, but for me they still fail as per the original post. Firstly, touch foo-bar foo-something-bar, then:

$ touch bar<tab>
$ touch foo-bar<tab>
           ^ cursor here
$ touch foo-bar
                ^ cursor here

[ericbn]

@protist, I get mixed results depending on if I try to complete from touch bar<tab> or vi bar<tab>, and if I'm on Mac or PC:

Given we're on an empty directory and then touch foo-bar foo-something-bar.

zstyle	touch bar<tab>	vi bar<tab>
'r:|=*' 'l:|=* r:|=*'	Good on Mac, Bad on PC	Good on Mac and PC
'r:|=*' '+l:|=*'	Good on Mac, Bad on PC	Good on Mac and PC
'm:{a-zA-Z}={A-Za-z} r:|=*' '+l:|=*'	Bad on Mac and PC	Good on Mac and PC

By "Good" I mean offering both files in the completion menu. By "Bad" I mean completing to foo-bar without showing the completion menu.

Tried with zsh 5.3 and zsh 5.5.1 running on MacOS 10.13.4 (Mac), and zsh 5.4.2 running on Ubuntu 16.04.3 on Linux Subsystem for Windows (PC).

[protist]

@ericbn Thanks for that extremely thorough troubleshooting! I'm on Linux with zsh 5.5.1. I have no idea why the same version of zsh would produce different results on Mac and Linux. Seems very odd.

I also tried the first zstyle with vi bar<tab>, and it worked too, kinda. I get partial completion to -bar (I expected completion to foo-bar?), then the completion menu. Do you see completion to -bar for both touch bar and vi bar, or does the former complete to foo-bar as per my posts above?

[ericbn]

@protist, the partial completion to -bar (instead of foo-bar with the cursor in the middle) is actually what I think solves the main issue -- that is not having the completion menu (because there's a file that is named foo-bar). I also think it's a simpler behavior than having the completion from both sides at the same time, so I like it. :- )

I get all kinds of different completions depending of the OS or zsh version I'm in. I just classified as "Bad" when I don't get the completion menu, and "Good" when I do get it (no matter how many tabs I have to press).

[protist]

@ericbn Sorry for the delay; I've been quite ill. Your explanation makes sense. I actually don't feel strongly about how much gets completed, as long as the completion menu appears after. The only reason I mentioned it was because I thought it might be diagnostic for why/when "Bad"ness and "Good"ness occurs. I also find it bizarre that this difference exists between Mac and Linux.

[protist]

Thanks @ericbn! Tested and works well. I went through the same testing above, and it all works as expected. Thank you for persisting and well done on fixing this!

[Eriner]

Wow, cheers @ericbn and @protist. What a trip.

[protist]

Oh no @ericbn! Sorry, it's still buggy.

From above, I think we agreed that zsh should attempt to match from the beginning of the word first, and only if that failed to use the "fuzzier" matching. However, this doesn't seem to be the case.

$ rm *
$ touch baritone fboaor
$ touch b<tab>
$ touch bar<tab>

The first tab only completes to bar, and the second tab will show the selection menu. I think our previous thoughts were that the first tab should immediately complete to baritone?

[ericbn]

Just reverted the commit. Back to the drawing board...

[protist]

I've been coming across weird ones every now and then. This was a particularly weird one today.

$ mkdir test; cd test
$ touch BagFrame.gcode Cats+ScorePad.gcode CatToken_Rumi.gcode
$ ls -1
BagFrame.gcode
Cats+ScorePad.gcode
CatToken_Rumi.gcode
$ ls <tab>
$ ls ca<tab>
$ ls CatT<tab>
$ ls CatToken_Rumi.gcode

The word starting with B isn't even an option.

[ericbn]

I've been coming across weird ones every now and then. This was a particularly weird one today.

Hi @protist. Looks like this is an issue with Zsh 5.9 when using m:{a-zA-Z}={A-Za-z} in the completion matcher-list, which our completion module does use. See https://www.zsh.org/mla/workers/2022/msg01229.html

[protist]

Thanks @ericbn. Nice find. I tested the patch, and it does indeed fix that latest issue.

[ericbn]

@protist, what about using setopt MENU_COMPLETE? This shows the completion menu right on when there's an ambiguous completion -- which seems friendlier in terms of showing the completion options as soon as possible --, and also seems to fix the issues reported here. Only possible drawback is that is completes with the first option right on too.

Another option would be using:

zstyle ':completion:*:*:*:*:*' menu yes select
zstyle ':completion:*' completer _list _complete _ignored

It first shows the completion options on the first tab and then completes on the second tab. The drawback of this option is that is lists even when there's only one completion option.

[protist]

Thanks @ericbn. Honestly, I think either option is not as good as the current version. I do like being able to tab-complete as much as possible, then to type the rest of the filename. This seems the quickest workflow. As an example, I might want to specify a number of files with a similar prefix. e.g.

touch foobar{0..9}baz something_else
touch f<tab>

which will complete to touch foobar and I can then append *, for example.

With setopt MENU_COMPLETE, touch f<tab> completes immediately to foobar0baz, so I have to delete the 0baz. Or if I want to pick something in the middle, e.g. foobar5baz, it's quite slow. With the second option suggested, it seems like there is no completion of oobar at all?

[ericbn]

@protist, thanks for the feedback. Good point, I agree the partial completions are a nice to have. Both options I proposed were getting rid of them.

Not having partial completions fixes some of issues here, since the completion system won't have to pick up from those partial completions. If only we could make them cleaner.