Tab completions for paths broken again in 7.2.0

[ivirshup]

Tab completion for paths within strings seems to be broken again (previously noted many places, like #10961, #10996, notebook issue #3333). Here's what I get in v7.2.0:

Compared to the behavior in v7.1.1:

Probably related to #11503, as disabling Jedi (setting c.IPCompleter.use_jedi = False in ipython_config.py) gives the correct behavior.

System info

{'commit_hash': '523ed2fe5',
 'commit_source': 'installation',
 'default_encoding': 'UTF-8',
 'ipython_path': '/usr/local/lib/python3.6/site-packages/IPython',
 'ipython_version': '7.2.0',
 'os_name': 'posix',
 'platform': 'Darwin-17.7.0-x86_64-i386-64bit',
 'sys_executable': '/usr/local/opt/python/bin/python3.6',
 'sys_platform': 'darwin',
 'sys_version': '3.6.5 (default, Jun 17 2018, 12:13:06) \n'
                '[GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.2)]'}

[mimakaev]

I have the same issue, ipython 7.2.0.

[jvanheugten]

Downgrading to ipython==7.1.1 seemed to work as a fix for me.

[vityas]

Could someone share if enabling jedi helped their workflow, and if it did, I would like to know how?

It certainly made my life much harder: no multi-level completion, no folder completion, nor can I use autocompletion before running the cell (df = pd.DataFrame(); df.val<TAB> fails)

[rainwoodman]

The bug certainly has made my life much harder as well.

auto-completion is probably one of the most important feature of an interactive shell. Is there a way jedi and path auto-completion can coexist?
Some very simple heuristics in deciding to use jedi or the legacy completion will help a long way: if one is in a string, then it is unlikely one wants names from the current namespace.

[michaelsilverstein]

For those who have run into issues with downgrading, I found @ivirshup's comment to be most helpful - to disable jedi autocomplete in the ipython config. If you haven't already, you can generate an ipython_config.py with:

# Generate config
ipython profile create
# Edit config file in vim
vim .ipython/profile_default/ipython_kernel_config.py 

And change c.IPCompleter.use_jedi = False in ipython_config.py

[flying-sheep]

[The jedi completer] certainly made my life much harder: no multi-level completion, no folder completion, […]

Nonono. It’s not the jedi completer that does that. There’s multiple completers and you’re describing that the wrong completer is selected because the completer selection code is broken. And touching the list of completers by enabling jedi uncovered that bug. @takluyver describes the problem in #10996 (comment).

Disabling jedi again is a temporary workaround, but actually fixing the completer selection code (#11064 I think) would of course be a much better solution.

[martinRenou]

My understanding is that all the completers are working at the same time. Completers are defined here, and they are all used at the same time. The matches list (the result of the auto-completion) will contain the concatenation of all the completers results, and the file_matches completer is still part of it.

So the path/file name completion is not "broken" when jedi is enabled, it is just that there is so many results (coming from jedi) that you can't see your file path matching anymore.

My first idea for a fix would be to disable jedi auto-completion when typing in a string? Would that work?
The only thing which would not work is auto-completion when executing code with eval or exec, but I think we cannot fix every single use case. And it looks like some people like the file path auto-completion.

EDIT: Reading the other issue you linked to @flying-sheep #11064, I understand that there is already a logic for not using jedi when typing in a string, but it is broken?

[martinRenou]

Just to illustrate what I'm saying. Both screenshots were taken using jedi, you see that the path auto-completion still works, you just need to type a bit more because there are more matches.

[mimakaev]

That is generally my problem: typing a bit helps, but it is not always an option if you don't know the first letter of the file. Even if you clearly started typing a path, after each "/" the completer "resets" to the default mode of showing me abs(), all() and other builtins.

It is a big problem if you have long paths with a few nested folders. Bash autocompletion would just add things on "tab". With current ipython autocomplete you have to remember the first letter or a few of each folder/file, type it, and then still go around a few builtins before you find your file/folder.

[mimakaev]

If there were a way to remove all builtins from autocompletion, and all ipython magic - it would be already much better!

I know how to type 'abs', and most of us can type it in several times faster than selecting it from autocompletion list. I'm not sure there is a good reason for it, and other 3-4 letter long builtins, to be there in a first place! Ideally, if there were a way to edit the list of builtins, I would just leave a few long names that I use, like DepercationWarning, and get rid of everything else.

[rainwoodman]

Just look at the use case in the image: Why would I want to type a Python object name inside of a string literal?

[martinRenou]

I can find a simple use case:

exec("print(abs(-3))", [globals()])

Maybe it is a less common use case, but it can happen. I would agree that path completion is more important than this use case for most of the users.

[mimakaev]

Yeah, it is a possibility, and some people rely on exec a lot.

Ideally, autocomplete should detect that it is in a filename mode, and from there on function like a normal bash tab completion. A good proxy is starting with "/", "./" or "C:" "D:" etc.

It is even less likely that string starting with these variables would contain valid python expressions.

I'm sure one can make an example of a = "C:print(abs(-3))"; exec(a[2:]) but that's very unlikely.

[martinRenou]

Yeah, actually the code is supposed to skip jedi completion when you are typing a string. See this line:

ipython/IPython/core/completer.py

Line 1375 in 65778ad

# if we are in a string jedi is likely not the right candidate for

And it relies on jedi internal parsing logic for trying to figure it out if we are in a string or not.
I tested this code a bit and it turns out that it is a bit broken, for example when typing path = " it will see that you are in a string, but not with path = "./. I guess we can come up with a better solution using tokenize.

[rainwoodman]

Is it because Jedi doesn't think it is an ErrorLeaf, or is it because going back by two nodes is insufficient?

[martinRenou]

I'm not sure, and I quickly switched my mind to tokenize because I didn't like the fact that it relies on internal jedi stuff (private API).

[rainwoodman]

tokenize has its problem too. What if the cell raises an IndentationError or SyntaxError before the string opening?

You may need to backtrack lines gradually, but then you are always expecting an error near the cursor due to the incomplete string declaration?

[martinRenou]

With tokenize I'm thinking of a simple logic like it is done in inputtransformer2.py. There is some logic for knowing if the user is typing a list or a dictionary in the terminal, so that when pressing enter you do not execute the code, you put a new line:


Maybe we can try something equivalent with the " and ' characters (I can try to come up with a PR).

Also another idea could be to use regexp, I would be useless in this case though :P

[martinRenou]

tokenize has its problem too. What if the cell raises an IndentationError or SyntaxError before the string opening

I guess it would just not complete... What is the current behavior? Does the autocompletion still work when you have a SyntaxError in your code? We should check

[YubinXie]

Same issue after updating to 7.3.0.

[rainwoodman]

Yes it does work when you have a SyntaxError.

[takluyver]

IIRC tokenize is fairly robust about syntax errors - the kind of snippet you show can still be tokenised fine, the error would only come when you try to make sense of the tokens.

I'm not sure what it would do with something like an unterminated string on the line before, which might be a syntax error at the tokenisation level.

[Nicholas-Autio-Mitchell]

Similar issues with the latest conda installation:

python                    3.7.3
ipykernel                 5.1.0              
ipython                   7.4.0           

I actually get no tab completion at all. Only within a function (shift+double-tab) do I get the docstring completion. No method completion on objects at all. Typing paths within strings does work as expected i.e. no global variables listed.

[khughitt]

Issue remains in IPython 7.7.0 installed via conda:

ipython                   7.7.0            py37h5ca1d4c_0    conda-forge
ipython_genutils          0.2.0                      py_1    conda-forge
jedi                      0.14.1                   py37_0    conda-forge
python                    3.7.3                h33d41f4_1    conda-forge

[flying-sheep]

And it will until this issue and #10926 are closed. @JeremySikes73 and me gave some explanation there of what’s wrong, and I’m pretty sure the issue is the same: IPython’s matchers all return results while we should either a) run only the relevant ones or b) add code to the matchers that makes them return nothing when their results would be irrelevant.

Thank you for trying to help, but I think it’s more productive to read #10926, dive into the code and find out how to improve the situation than saying “it’s still an issue”.

[augustogoulart]

@meeseeksdev tag help wanted

[augustogoulart]

@meeseeksdev tag tab-completion

[sleebapaul]

For those who have run into issues with downgrading, I found @ivirshup's comment to be most helpful - to disable jedi autocomplete in the ipython config. If you haven't already, you can generate an ipython_config.py with:

# Generate config
ipython profile create
# Edit config file in vim
vim .ipython/profile_default/ipython_kernel_config.py 

And change c.IPCompleter.use_jedi = False in ipython_config.py

Which file to edit? ipython_kernel_config.py or ipython_config.py?

[joelostblom]

A temporary workaround for this problem is to append a second slash after pressing tab.

After typing '/<TAB>

After typing /

Once an item is selected, there will only be a single slash as separator:

After pressing return

Note that adding the second slash before pressing tab does not change the pop up menu.

[shivariyer]

I upgraded to the latest version of ipython as of today (7.11.0), and the problem still persists.

[ecoon]

@flying-sheep @augustogoulart Are you open to pull requests on this? (I mean, the "helpwanted" tag suggests so, but I'm not an existing contributor so not sure of the developer norms here.) I have some minor changes to the logic in the detection of whether we are in a string or not, which, while not a perfect fix, at least does a better job of turning off jedi.

This doesn't completely fix #10926 or this, since it would be nice to turn off (in my opinion):

• file completion matches when in a dictionary key string
• magic completion matches when in any string

but it does at least turn off jedi when in a string, which goes a long way in making my own use patterns workable.

[flying-sheep]

I’m just a small-time contributor myself, but I bet they’d like help to fix this!

[beingsantosh]

Updating IPython - 7.20.0, resolved for me. Regards

[RuRo]

The issue isn't fully fixed in 7.20.0. Path completion kind of works, if you start with './<tab>' or '/<tab>', but otherwise the completions are still cluttered with %magic and builtin completions. For anyone, coming to this issue from Google, you can kind of improve the situation by doing something along the lines of

import IPython.core.completer

def _magic_matches(self, text):
    return []

IPython.core.completer.IPCompleter.magic_matches = _magic_matches

In your ~/.ipython/profile_default/startup scripts. This completely removes all magic matches.

Unfortunately, the builtins are still suggested outside of string completions, which is a huge pain. Even worse, for some reason the builtins completions are added somewhere later in the IPCompleter pipeline, so you can't remove them by overwriting IPython.core.completer.Completer.global_matches and you can't even sort them to the end of the completion list by overwriting IPython.core.completer.completions_sorting_key.