Issue with ipython and ipykernel

[ThomVett]

Dear pdbpp team,

I've been working in jupyter notebooks that have pdbpp installed in the virtual environment. I'm a user of the %debug magic function which lets me launch a debugger when an exception is raised from one of the jupyter notebook cells.

I use the following test code in a cell:

def foo():
    raise Exception("test")
    
foo()

and then use %debug in the next cell to jump in the debugger.

With the latest ipython (7.28) and pdbpp (10.3), this causes an issue and we get in the following error:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/scratch/ipykernel_27400/3913957351.py in <module>
----> 1 get_ipython().run_line_magic('debug', '')

~/.virtualenvs/pdb-pp-test-FM1DiXFv-py3.8/lib/python3.8/site-packages/IPython/core/interactiveshell.py in run_line_magic(self, magic_name, line, _stack_depth)
   2349                 kwargs['local_ns'] = self.get_local_scope(stack_depth)
   2350             with self.builtin_trap:
-> 2351                 result = fn(*args, **kwargs)
   2352             return result
   2353 

~/.virtualenvs/pdb-pp-test-FM1DiXFv-py3.8/lib/python3.8/site-packages/decorator.py in fun(*args, **kw)
    230             if not kwsyntax:
    231                 args, kw = fix(args, kw, sig)
--> 232             return caller(func, *(extras + args), **kw)
    233     fun.__name__ = func.__name__
    234     fun.__doc__ = func.__doc__

~/.virtualenvs/pdb-pp-test-FM1DiXFv-py3.8/lib/python3.8/site-packages/IPython/core/magic.py in <lambda>(f, *a, **k)
    185     # but it's overkill for just that one bit of state.
    186     def magic_deco(arg):
--> 187         call = lambda f, *a, **k: f(*a, **k)
    188 
    189         if callable(arg):

~/.virtualenvs/pdb-pp-test-FM1DiXFv-py3.8/lib/python3.8/site-packages/IPython/core/magics/execution.py in debug(self, line, cell)
    467 
    468         if not (args.breakpoint or args.statement or cell):
--> 469             self._debug_post_mortem()
    470         elif not (args.breakpoint or cell):
    471             # If there is no breakpoints, the line is just code to execute

~/.virtualenvs/pdb-pp-test-FM1DiXFv-py3.8/lib/python3.8/site-packages/IPython/core/magics/execution.py in _debug_post_mortem(self)
    481 
    482     def _debug_post_mortem(self):
--> 483         self.shell.debugger(force=True)
    484 
    485     def _debug_exec(self, code, breakpoint):

~/.virtualenvs/pdb-pp-test-FM1DiXFv-py3.8/lib/python3.8/site-packages/IPython/core/interactiveshell.py in debugger(self, force)
   1191             return
   1192 
-> 1193         self.InteractiveTB.debugger(force=True)
   1194 
   1195     #-------------------------------------------------------------------------

~/.virtualenvs/pdb-pp-test-FM1DiXFv-py3.8/lib/python3.8/site-packages/IPython/core/ultratb.py in debugger(self, force)
   1180         if force or self.call_pdb:
   1181             if self.pdb is None:
-> 1182                 self.pdb = self.debugger_cls()
   1183             # the system displayhook may have changed, restore the original
   1184             # for pdb

~/.virtualenvs/pdb-pp-test-FM1DiXFv-py3.8/lib/python3.8/site-packages/IPython/core/debugger.py in __init__(self, color_scheme, completekey, stdin, stdout, context, **kwargs)
    244 
    245         # `kwargs` ensures full compatibility with stdlib's `pdb.Pdb`.
--> 246         OldPdb.__init__(self, completekey, stdin, stdout, **kwargs)
    247 
    248         # IPython changes...

~/.virtualenvs/pdb-pp-test-FM1DiXFv-py3.8/lib/python3.8/site-packages/_pdbpp_path_hack/pdb.py in __init__(self, *args, **kwds)
    167         kwargs = self.config.default_pdb_kwargs.copy()
    168         kwargs.update(**kwds)
--> 169         super(Pdb, self).__init__(*args, **kwargs)
    170         self.prompt = self.config.prompt
    171         self.display_list = {}  # frame --> (name --> last seen value)

TypeError: super(type, obj): obj must be an instance or subtype of type

Attached is the requirements.txt file of the test env where I reproduced this.
pdbpp_requirements.txt

Workaround
Uninstalling pdbpp immediately fixed the issue.

[blueyed]

Thanks for the report.
I could not reproduce it using Python 3.9 or 3.8.12 however.

Adding the following before the (for you) crashing code (https://github.com/pdbpp/pdbpp/blob/0.10.3/pdb.py#L169, installed as /tmp/venv38/lib/python3.8/site-packages/pdb.py for me) shows:

=== <IPython.terminal.debugger.TerminalPdb object at 0x7f4e778094f0> <class 'pdb.Pdb'>
=== <class 'IPython.terminal.debugger.TerminalPdb'> <class 'type'>

        print("===", self, Pdb)
        print("===", type(self), type(Pdb))

1. what does it print for you?
2. Does it happen with HOME=/dev/null /tmp/venv38/bin/ipython also? (i.e. ignoring any custom user config)
3. Also try it with pdbpp from the Git master branch, please.

[ThomVett]

So I double checked python version, i have 3.8.12 as you have.

1: Adding the print statements results in the following:

=== <IPython.core.debugger.Pdb object at 0x7fd6ec25c1c0> <class 'IPython.core.debugger.InterruptiblePdb'>
=== <class 'IPython.core.debugger.Pdb'> <class 'type'>

I added the print statements at the following (on line 169) in the follwing file for my local - /.virtualenvs/test_pdbpp/lib/python3.8/site-packages/pdb.py which corresponds to the one you pointed out I think.

3. installing from latest master branch seems to fix the issue, the debugger open correclty in a Pdb debugger - without pdbpp i get an ipdb debugger.

4. I'm not sure how to try that option?

Hope this info helps, let me know if you need more!

[zpincus]

I too have observed this issue.

@blueyed, when you failed to reproduce, were you testing in a command-line ipython shell, or via a jupyter notebook?

I have found that using an ipython shell with pdbpp works fine (and the print statements above produce the same output you showed), but doing the same thing in a jupyter notebook produces the error @ThomVett noted, along with the identical printed output to what he reported.

I am using python 3.9.7, with pdbpp 0.10.3, and miniconda managing the virtual environments.

Installing the latest master branch of pdbpp with pip install git+https://github.com/pdbpp/pdbpp does not remedy the issue in my hands. Inserting the same print lines before line 369 of ~/miniconda3/envs/my_env/lib/python3.9/site-packages/pdbpp.py produces:

=== <IPython.core.debugger.Pdb object at 0x7f0ac8152f70> <class 'IPython.core.debugger.InterruptiblePdb'>
=== <class 'IPython.core.debugger.Pdb'> <class 'pdb.PdbMeta'>

Regarding running without custom user config, unfortunately HOME=/dev/null jupyter notebook fails badly; I'm not sure how best to coax jupyter notebook to run in a plain-vanilla config.

[zpincus]

@blueyed Can I ask you to please try to repro this bug via jupyter notebook?

This issue reproduces trivially for me in a jupyter notebook, but not in the command-line ipython shell. I believe that @ThomVett was using the notebook in the original report.

Anyhow, this bug makes pdbpp unusable for anyone who ever uses jupyter notebooks; it would be good to get some traction on it if possible.

[blueyed]

@zpincus

Installing the latest master branch of pdbpp with pip install git+https://github.com/pdbpp/pdbpp does not remedy the issue in my hands.

So it might be fixed on the main/master branch then already apparently.
You could git-bisect it to find what fixes it, and then that could be backported.

In general I recommend using the master branch though anyway still.

[zpincus]

Sorry, I am not sure I follow. The text you quoted above, @blueyed, states pretty clearly that using the master branch does NOT fix the problem. So there is no fix and nothing to backport.

However, that was from some months ago, so I tried again with the latest master to see if some fix had magically appeared in the interim. This is not the case; postmortem debugging in a jupyter notebook is still broken when pdbpp is installed. However, the error is different now! It produces a maximum recursion depth error.

Complete reproduction code in a clean conda environment and the full stacktrace are below.

conda create --yes --name pdbpp-trouble python jupyter
conda activate pdbpp-trouble
pip install git+https://github.com/pdbpp/pdbpp
jupyter notebook

Then click "new" in the upper right of the browser window that is opened and choose a new python session. In the first cell, paste:

def foo():
    raise Exception("test")
    
foo()

then press shift-enter to run the cell. In the next cell, type

%debug

and again press shift-enter, wait a moment for the recursion depth to be exceeded, and observe the following stack trace:

---------------------------------------------------------------------------
RecursionError                            Traceback (most recent call last)
Input In [2], in <cell line: 1>()
----> 1 get_ipython().run_line_magic('debug', '')

File ~/miniconda/envs/pdbpp-trouble/lib/python3.10/site-packages/IPython/core/interactiveshell.py:2305, in InteractiveShell.run_line_magic(self, magic_name, line, _stack_depth)
   2303     kwargs['local_ns'] = self.get_local_scope(stack_depth)
   2304 with self.builtin_trap:
-> 2305     result = fn(*args, **kwargs)
   2306 return result

File ~/miniconda/envs/pdbpp-trouble/lib/python3.10/site-packages/IPython/core/magics/execution.py:451, in ExecutionMagics.debug(self, line, cell)
    448 args = magic_arguments.parse_argstring(self.debug, line)
    450 if not (args.breakpoint or args.statement or cell):
--> 451     self._debug_post_mortem()
    452 elif not (args.breakpoint or cell):
    453     # If there is no breakpoints, the line is just code to execute
    454     self._debug_exec(line, None)

File ~/miniconda/envs/pdbpp-trouble/lib/python3.10/site-packages/IPython/core/magics/execution.py:465, in ExecutionMagics._debug_post_mortem(self)
    464 def _debug_post_mortem(self):
--> 465     self.shell.debugger(force=True)

File ~/miniconda/envs/pdbpp-trouble/lib/python3.10/site-packages/IPython/core/interactiveshell.py:1107, in InteractiveShell.debugger(self, force)
   1104     error('No traceback has been produced, nothing to debug.')
   1105     return
-> 1107 self.InteractiveTB.debugger(force=True)

File ~/miniconda/envs/pdbpp-trouble/lib/python3.10/site-packages/IPython/core/ultratb.py:924, in VerboseTB.debugger(self, force)
    922 if force or self.call_pdb:
    923     if self.pdb is None:
--> 924         self.pdb = self.debugger_cls()
    925     # the system displayhook may have changed, restore the original
    926     # for pdb
    927     display_trap = DisplayTrap(hook=sys.__displayhook__)

File ~/miniconda/envs/pdbpp-trouble/lib/python3.10/site-packages/pdbpp.py:307, in PdbMeta.__call__(cls, *args, **kwargs)
    305 else:
    306     set_global_pdb = use_global_pdb
--> 307 obj.__init__(*args, **kwargs)
    308 if set_global_pdb:
    309     obj._env = {"HOME": os.environ.get("HOME")}

File ~/miniconda/envs/pdbpp-trouble/lib/python3.10/site-packages/IPython/core/debugger.py:228, in Pdb.__init__(self, completekey, stdin, stdout, context, **kwargs)
    225         raise ValueError("Context must be a positive integer") from e
    227 # `kwargs` ensures full compatibility with stdlib's `pdb.Pdb`.
--> 228 OldPdb.__init__(self, completekey, stdin, stdout, **kwargs)
    230 # IPython changes...
    231 self.shell = get_ipython()

File ~/miniconda/envs/pdbpp-trouble/lib/python3.10/site-packages/pdbpp.py:369, in Pdb.__init__(self, *args, **kwds)
    367 kwargs = self.config.default_pdb_kwargs.copy()
    368 kwargs.update(**kwds)
--> 369 super(Pdb, self).__init__(*args, **kwargs)
    370 self.prompt = self.config.prompt
    371 self.display_list = {}  # frame --> (name --> last seen value)

File ~/miniconda/envs/pdbpp-trouble/lib/python3.10/site-packages/IPython/core/debugger.py:228, in Pdb.__init__(self, completekey, stdin, stdout, context, **kwargs)
    225         raise ValueError("Context must be a positive integer") from e
    227 # `kwargs` ensures full compatibility with stdlib's `pdb.Pdb`.
--> 228 OldPdb.__init__(self, completekey, stdin, stdout, **kwargs)
    230 # IPython changes...
    231 self.shell = get_ipython()

File ~/miniconda/envs/pdbpp-trouble/lib/python3.10/site-packages/pdbpp.py:369, in Pdb.__init__(self, *args, **kwds)
    367 kwargs = self.config.default_pdb_kwargs.copy()
    368 kwargs.update(**kwds)
--> 369 super(Pdb, self).__init__(*args, **kwargs)
    370 self.prompt = self.config.prompt
    371 self.display_list = {}  # frame --> (name --> last seen value)

    [... skipping similar frames: Pdb.__init__ at line 228 (1478 times), Pdb.__init__ at line 369 (1477 times)]

File ~/miniconda/envs/pdbpp-trouble/lib/python3.10/site-packages/pdbpp.py:369, in Pdb.__init__(self, *args, **kwds)
    367 kwargs = self.config.default_pdb_kwargs.copy()
    368 kwargs.update(**kwds)
--> 369 super(Pdb, self).__init__(*args, **kwargs)
    370 self.prompt = self.config.prompt
    371 self.display_list = {}  # frame --> (name --> last seen value)

File ~/miniconda/envs/pdbpp-trouble/lib/python3.10/site-packages/IPython/core/debugger.py:228, in Pdb.__init__(self, completekey, stdin, stdout, context, **kwargs)
    225         raise ValueError("Context must be a positive integer") from e
    227 # `kwargs` ensures full compatibility with stdlib's `pdb.Pdb`.
--> 228 OldPdb.__init__(self, completekey, stdin, stdout, **kwargs)
    230 # IPython changes...
    231 self.shell = get_ipython()

File ~/miniconda/envs/pdbpp-trouble/lib/python3.10/site-packages/pdbpp.py:358, in Pdb.__init__(self, *args, **kwds)
    355 self.start_lineno = kwds.pop('start_lineno', None)
    356 self.start_filename = kwds.pop('start_filename', None)
--> 358 self.config = self.get_config(self.ConfigFactory)
    359 self.config.setup(self)
    361 if "PDBPP_COLORS" in os.environ:

File ~/miniconda/envs/pdbpp-trouble/lib/python3.10/site-packages/fancycompleter.py:197, in ConfigurableClass.get_config(self, Config)
    195 # try to load config from the ~/filename file
    196 filename = '~/' + self.config_filename
--> 197 rcfile = os.path.normpath(os.path.expanduser(filename))
    198 if not os.path.exists(rcfile):
    199     return self.DefaultConfig()

File ~/miniconda/envs/pdbpp-trouble/lib/python3.10/posixpath.py:243, in expanduser(path)
    241     i = len(path)
    242 if i == 1:
--> 243     if 'HOME' not in os.environ:
    244         import pwd
    245         try:

File ~/miniconda/envs/pdbpp-trouble/lib/python3.10/_collections_abc.py:825, in Mapping.__contains__(self, key)
    823 def __contains__(self, key):
    824     try:
--> 825         self[key]
    826     except KeyError:
    827         return False

File ~/miniconda/envs/pdbpp-trouble/lib/python3.10/os.py:676, in _Environ.__getitem__(self, key)
    674 def __getitem__(self, key):
    675     try:
--> 676         value = self._data[self.encodekey(key)]
    677     except KeyError:
    678         # raise KeyError with the original key value
    679         raise KeyError(key) from None

RecursionError: maximum recursion depth exceeded

[blueyed]

Sorry for the confusion due to my misunderstanding.

This is not the case; postmortem debugging in a jupyter notebook is still broken when pdbpp is installed. However, the error is different now! It produces a maximum recursion depth error.

That might be something interesting to git-bisect then, if it is different from the released version.
(It looks like the recursion happens when initializing the Pdb object.)