Cannot make multi-line code blocks in ipython

[treyhunner]

I just did a pip install ipython while teaching a class today and typed an if statement and then hit enter after the first line and the code executed.

This seems like a bug.

I did a pip install ipython==6.5.0 and hitting Enter in a code block properly showed me the next indented line to type out.

I'm on Ubuntu 18.04 running ipython within tmux, though I doubt tmux is the issue here.

[Carreau]

you can use ctrl-o to force a new line.

[treyhunner]

I never would have guessed that one. I'm not sure how to make this more clear, but I do think it warrants some clarification/hand-holding. I tried every shortcut I could think of and ended up forcing a downgrade to fix my issue.

I use IPython to make it easier to type out and edit multi-line blocks in the REPL while teaching live audiences. I suspect others in my situation may have similar issues.

Maybe a warning somewhere that indicates that Ctrl-O can be used to make a multi-line code block? I'm not sure whether there's an appropriate place for that.

[Carreau]

It is a bug with a recent refactor, I was just trying telling you that you can use Ctrl-O in the meantime if you wish to use 7.x

[treyhunner]

Ah great! I'm glad it was just a bug. Thanks @Carreau! 😄

[Carreau]

Would you mind giving me a few examples (working and not workig) of what you expect, in order to provide test for the fixes ?

I have some but just collecting a few and don't want to influence you.

[treyhunner]

with open('hello.txt', mode='wt') as my_file:
    my_file.write('hi')
    my_file.write('hi again')

Can't remember the second example, but a number of loops:

numbers = [2, 1, 3, 4, 7, 8, 11]

for n in numbers:
    if n > 0:
        print(n*2)
    else:
        print(n/2)

[Deborah-Digges]

@Carreau is someone working on this issue? I'd be happy to help out

[Carreau]

hi @Deborah-Digges, I've had a quick look but no too much so far.

I've wrote this small test case to see the difference between the old input_splitter and the new input_transformer:

from IPython.core import inputtransformer2 as ipt2 # new way
from IPython.core import inputsplitter #oldl way

occ =  inputsplitter.InputSplitter().check_complete
cc = ipt2.TransformerManager().check_complete

comp = lambda x : (cc(x), occ(x), x)
print(comp('if'))
print(comp('if\n\n'))
print(comp("""
def foo():
    print('Hello')"""
))
print(comp('if True:'))

Which result in

(('invalid', None), ('invalid', None), 'if')
(('complete', None), ('invalid', None), 'if\n\n')
(('complete', None), ('incomplete', 4), "\ndef foo():\n    print('Hello')")
(('incomplete', 4), ('incomplete', 4), 'if True:')

You see that the 3rd item is the one that interest us. The second was out of curiosity as one of the "feature" of IPython is to force-execute if there is more than 2 newlines.

Part of the relevant code is in shortcut.py:L109-L127.

I guess there is some kind of conflation between the role of defining whether a check of code is "complete", or whether we should "execute or add a new line".

I'm guessing that the heuristic that check whether the input is multiline and whether the last character is already a new line or not should be sufficient.

One of the remaining questions is where to put that fix ?

• in shortcut.py? if so it will only fix terminal IPython
• in input_transformer2 ? Ithink that might be the right solution as this but will likely also affect QtConsole.

Let me know if that's enough to at least get you started.

Thanks !

[Deborah-Digges]

Hi @Carreau ! Thanks so much for the detailed explanation and test cases.

I was able to reproduce it locally with the dev version of ipython and will now start looking into the code for shortcut.py and inputtransformer.

[bxsx]

Would you mind giving me a few examples (working and not workig) of what you expect, in order to provide test for the fixes ?

You don't need a few examples. The bug occurs in very reproducible way.
Multiline works only if the line ends with colon. If there is no colon at the end of line then iPython executes code. This looks a bit more trivial with this information but never looked into the iPython source code, so maybe I am just a bit too optimistic here ;)

[Carreau]

iPython

Upper case I please, we don't want trouble with apple.

You don't need a few examples

Well, no I don't need, but I want multiple example. I can reproduce and have an idea of how to fix it, but having multiple case help me be sure that I'm no hitting an edge case. I do have a biased view on how to use IPython, so example from other are helpful.

[bxsx]

iPython
Upper case I please, we don't want trouble with apple.

At least I found out why there was the the upper I in the name, thanks and sorry :)

As a apology, here is a pull request #11354 that fixes this (major) issue. IMHO this is a blocking bug for IPython, you should consider to make a release soon (see how many issues is created at github by users regarding it).

[Carreau]

At least I found out why there was the the upper I in the name, thanks and sorry :)

That's not the (only) reason, as IPython 0.1 was release before the first iProduct, but usually people remember

As a apology, here is a pull request #11354 that fixes this (major) issue.

thanks I'll have a look when time allows

you should consider to make a release soon

Yes, once a volunteer have time have we'll do that. There are other critical issue like getting a release of jupyter_console out, and I don't think anyone here can take a couple of hours on $DAYJOB to do so. So it may have to wait for this week-end.

[bxsx]

Actually, I got a regression with my fix. Fixing it now.

[bxsx]

Actually, I got a regression with my fix. Fixing it now.

The PR #11354 is updated now.

[tebeka]

Thanks for the fix, any idea when is this going to be released? I'm using vim keybindings to CTRL-O doesn't work for me (ESC + o does ...)

[Carreau]

Thanks for the fix, any idea when is this going to be released?

That's going to be one one of the volunteer on the project get a couple of free hours to triage the few remaining issues for 7.1 and does a release. I'm hoping to maybe have a couple hours this week-end, but that may not be sufficient.

Any help on triaging/reviewing/tagging the existing PRs/Issues would be helpful.

[hatarist]

Still reproducible with the async with blocks, tried both the current master branch and v7.1.1 from PyPI.

In [16]: async with aiofiles.open('/tmp/foobar', 'r') as f:
    ...:     content = await f.read()

In [17]: content
Out[17]: 'hello'

[Carreau]

Hum, that is likely a weird interaction with autoawait.

[hatarist]

Oh, I didn't think it'd attach to this issue. Anyway, the reason it fails is that when we have

async with aiohttp.ClientSession() as session:
    pass|   # < cursor is there

it runs check_complete on each line ending, which, in turn, executes compile_command, and the latter one raises a SyntaxError because 'async with' is used outside of the async function.
In my fork I just muted that SyntaxError but it's certainly not the brightest way to fix that, lol.

Possible solutions/ideas:

• should check if the autoawait is turned on. if it is, could ignore that specific SyntaxError case. I don't think it's a good solution, but maybe I'm also overcomplicating things.

• if the autoawait's on, feed the compile_command() with the code wrapped with the _asyncify(). i guess, this way it won't raise SyntaxError, but i'm not sure if the newline issue will be resolved because _asyncify() itself adds some levels of indentation and it easily can go messy.

• maybe _AsyncSyntaxErrorVisitor can help any? But I guess it's for the other way around

I'm sorry for the lack of dedication, I'd submit thw PR, but I hate writing tests and whatnot and also not sure of the better way to fix that. But I hope it's still useful for someone.

[Carreau]

We could try something like this as well.

[Carreau]

That's your point nº 2, and it is indeed ugly. The newline works. What would be good is get proper support in CPython.

[billtubbs]

Is this issue still unresolved? I think I may have caught this bug...

This come up today for the first time (during a demo with a client when I was trying to demonstrate what a generator is in Python!!!).

Am I doing something wrong or what am I supposed to do to write multi-line code blocks (other than CTRL-o workaround)?

Expected result as demonstrated in the standard Python REPL:

(tsa) BillsMacBookPro:develop billtubbs$ python
Python 3.5.5 | packaged by conda-forge | (default, Jul 23 2018, 23:45:11) 
[GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> for i in range(5):
...     x = i*2
...     print(x)
... 
0
2
4
6
8
>>> exit()

Result today when I type the same into an iPython REPL:

(tsa) BillsMacBookPro:develop billtubbs$ ipython
Python 3.5.5 | packaged by conda-forge | (default, Jul 23 2018, 23:45:11) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.0.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: for i in range(5): 
   ...:     x = i*2                                                                   

In [2]:      

The iPython REPL automatically indents the second line as expected. But when I type enter at the end of the second line it executes the two lines instead of providing an optional third line.

As described above, I can get the desired result by pressing CTRL-o instead of pressing enter on the second line:

In [2]: for i in range(5): 
   ...:     x = i*2 
   ...:     print(x)                                                                  
0
2
4
6
8

[Carreau]

IPython 7.0.1 -- ...

Please upgrade your IPython, this issue is solve, there is just still an edge case with async code.

[billtubbs]

Oh sorry. I thought I did. After conda update ipython I get # All requested packages already installed.

Sorry, I'm a bit confused. What is the latest version and how do I upgrade to it?

[Carreau]

It will depends on how you installed it, I would suggest to try with pip install to see if this works. But you may also be working in an environment.

When things like that happend, I try to aggressively uninstall until I can't launch IPython and then reinstall.

[Diogo-Rossi]

Is there a way to disable multiline commands?
Most time it is useful but sometimes it leads to problems when executing code in some enviroments.
e.g.: This issue in VS Code

[dhivyeshrk]

Is anyone working on this issue? I would be very happy to contribute.