gh-111201: A new Python REPL (GH-111567)

Co-authored-by: Łukasz Langa <lukasz@langa.pl>
Co-authored-by: Marta Gómez Macías <mgmacias@google.com>
Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>

.github/workflows/mypy.yml

@@ -8,6 +8,7 @@ on:
   pull_request:
     paths:
       - ".github/workflows/mypy.yml"
+      - "Lib/_pyrepl/**"
       - "Lib/test/libregrtest/**"
       - "Tools/build/generate_sbom.py"
       - "Tools/cases_generator/**"
@@ -35,8 +36,9 @@ jobs:
     strategy:
       matrix:
         target: [
+          "Lib/_pyrepl",
           "Lib/test/libregrtest",
-          "Tools/build/",
+          "Tools/build",
           "Tools/cases_generator",
           "Tools/clinic",
           "Tools/jit",

.github/workflows/reusable-macos.yml

@@ -22,6 +22,7 @@ jobs:
       HOMEBREW_NO_INSTALL_CLEANUP: 1
       HOMEBREW_NO_INSTALLED_DEPENDENTS_CHECK: 1
       PYTHONSTRICTEXTENSIONBUILD: 1
+      TERM: linux
     strategy:
       fail-fast: false
       matrix:

.github/workflows/reusable-ubuntu.yml

@@ -17,6 +17,7 @@ jobs:
       FORCE_COLOR: 1
       OPENSSL_VER: 3.0.13
       PYTHONSTRICTEXTENSIONBUILD: 1
+      TERM: linux
     steps:
     - uses: actions/checkout@v4
     - name: Register gcc problem matcher

Doc/glossary.rst

@@ -9,13 +9,14 @@ Glossary
 .. glossary::
 
    ``>>>``
-      The default Python prompt of the interactive shell.  Often seen for code
-      examples which can be executed interactively in the interpreter.
+      The default Python prompt of the :term:`interactive` shell.  Often
+      seen for code examples which can be executed interactively in the
+      interpreter.
 
    ``...``
       Can refer to:
 
-      * The default Python prompt of the interactive shell when entering the
+      * The default Python prompt of the :term:`interactive` shell when entering the
         code for an indented code block, when within a pair of matching left and
         right delimiters (parentheses, square brackets, curly braces or triple
         quotes), or after specifying a decorator.
@@ -620,7 +621,8 @@ Glossary
       execute them and see their results.  Just launch ``python`` with no
       arguments (possibly by selecting it from your computer's main
       menu). It is a very powerful way to test out new ideas or inspect
-      modules and packages (remember ``help(x)``).
+      modules and packages (remember ``help(x)``). For more on interactive
+      mode, see :ref:`tut-interac`.
 
    interpreted
       Python is an interpreted language, as opposed to a compiled one,
@@ -1084,6 +1086,10 @@ Glossary
 
       See also :term:`namespace package`.
 
+   REPL
+      An acronym for the "read–eval–print loop", another name for the
+      :term:`interactive` interpreter shell.
+
    __slots__
       A declaration inside a class that saves memory by pre-declaring space for
       instance attributes and eliminating instance dictionaries.  Though

Doc/tutorial/appendix.rst

@@ -10,6 +10,28 @@ Appendix
 Interactive Mode
 ================
 
+There are two variants of the interactive :term:`REPL`.  The classic
+basic interpreter is supported on all platforms with minimal line
+control capabilities.
+
+On Unix-like systems (e.g. Linux or macOS) with :mod:`curses` and
+:mod:`readline` support, a new interactive shell is used by default.
+This one supports color, multiline editing, history browsing, and
+paste mode.  To disable color, see :ref:`using-on-controlling-color` for
+details.  Function keys provide some additional functionality.
+:kbd:`F1` enters the interactive help browser :mod:`pydoc`.
+:kbd:`F2` allows for browsing command-line history without output nor the
+:term:`>>>` and :term:`...` prompts. :kbd:`F3` enters "paste mode", which
+makes pasting larger blocks of code easier. Press :kbd:`F3` to return to
+the regular prompt.
+
+When using the new interactive shell, exit the shell by typing :kbd:`exit`
+or :kbd:`quit`. Adding call parentheses after those commands is not
+required.
+
+If the new interactive shell is not desired, it can be disabled via
+the :envvar:`PYTHON_BASIC_REPL` environment variable.
+
 .. _tut-error:
 
 Error Handling

Doc/using/cmdline.rst

@@ -42,6 +42,7 @@ additional methods of invocation:
 * When called with standard input connected to a tty device, it prompts for
   commands and executes them until an EOF (an end-of-file character, you can
   produce that with :kbd:`Ctrl-D` on UNIX or :kbd:`Ctrl-Z, Enter` on Windows) is read.
+  For more on interactive mode, see :ref:`tut-interac`.
 * When called with a file name argument or with a file as standard input, it
   reads and executes a script from that file.
 * When called with a directory name argument, it reads and executes an
@@ -1182,6 +1183,15 @@ conflict.
 
    .. versionadded:: 3.13
 
+.. envvar:: PYTHON_BASIC_REPL
+
+   If this variable is set to ``1``, the interpreter will not attempt to
+   load the Python-based :term:`REPL` that requires :mod:`curses` and
+   :mod:`readline`, and will instead use the traditional parser-based
+   :term:`REPL`.
+
+   .. versionadded:: 3.13
+
 .. envvar:: PYTHON_HISTORY
 
    This environment variable can be used to set the location of a

Doc/whatsnew/3.13.rst

@@ -102,6 +102,34 @@ New typing features:
 New Features
 ============
 
+A Better Interactive Interpreter
+--------------------------------
+
+On Unix-like systems like Linux or macOS, Python now uses a new
+:term:`interactive` shell.  When the user starts the :term:`REPL`
+from a tty, and both :mod:`curses` and :mod:`readline` are available,
+the interactive shell now supports the following new features:
+
+* colorized prompts;
+* multiline editing with history preservation;
+* interactive help browsing using :kbd:`F1` with a separate command
+  history;
+* history browsing using :kbd:`F2` that skips output as well as the
+  :term:`>>>` and :term:`...` prompts;
+* "paste mode" with :kbd:`F3` that makes pasting larger blocks of code
+  easier (press :kbd:`F3` again to return to the regular prompt);
+* ability to issue REPL-specific commands like :kbd:`help`, :kbd:`exit`,
+  and :kbd:`quit` without the need to use call parentheses after the
+  command name.
+
+If the new interactive shell is not desired, it can be disabled via
+the :envvar:`PYTHON_BASIC_REPL` environment variable.
+
+For more on interactive mode, see :ref:`tut-interac`.
+
+(Contributed by Pablo Galindo Salgado, Łukasz Langa, and
+Lysandros Nikolaou in :gh:`111201` based on code from the PyPy project.)
+
 Improved Error Messages
 -----------------------
 

Lib/_pyrepl/__init__.py

@@ -0,0 +1,19 @@
+#   Copyright 2000-2008 Michael Hudson-Doyle <micahel@gmail.com>
+#                       Armin Rigo
+#
+#                        All Rights Reserved
+#
+#
+# Permission to use, copy, modify, and distribute this software and
+# its documentation for any purpose is hereby granted without fee,
+# provided that the above copyright notice appear in all copies and
+# that both that copyright notice and this permission notice appear in
+# supporting documentation.
+#
+# THE AUTHOR MICHAEL HUDSON DISCLAIMS ALL WARRANTIES WITH REGARD TO
+# THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY
+# AND FITNESS, IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL,
+# INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER
+# RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF
+# CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN
+# CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

Lib/_pyrepl/__main__.py

@@ -0,0 +1,43 @@
+import os
+import sys
+
+CAN_USE_PYREPL = True
+
+def interactive_console(mainmodule=None, quiet=False, pythonstartup=False):
+    global CAN_USE_PYREPL
+    if not CAN_USE_PYREPL:
+        return sys._baserepl()
+
+    startup_path = os.getenv("PYTHONSTARTUP")
+    if pythonstartup and startup_path:
+        import tokenize
+        with tokenize.open(startup_path) as f:
+            startup_code = compile(f.read(), startup_path, "exec")
+            exec(startup_code)
+
+    # set sys.{ps1,ps2} just before invoking the interactive interpreter. This
+    # mimics what CPython does in pythonrun.c
+    if not hasattr(sys, "ps1"):
+        sys.ps1 = ">>> "
+    if not hasattr(sys, "ps2"):
+        sys.ps2 = "... "
+    #
+    run_interactive = None
+    try:
+        import errno
+        if not os.isatty(sys.stdin.fileno()):
+            raise OSError(errno.ENOTTY, "tty required", "stdin")
+        from .simple_interact import check
+        if err := check():
+            raise RuntimeError(err)
+        from .simple_interact import run_multiline_interactive_console
+        run_interactive = run_multiline_interactive_console
+    except Exception as e:
+        print(f"warning: can't use pyrepl: {e}", file=sys.stderr)
+        CAN_USE_PYREPL = False
+    if run_interactive is None:
+        return sys._baserepl()
+    return run_interactive(mainmodule)
+
+if __name__ == "__main__":
+    interactive_console()

Lib/_pyrepl/_minimal_curses.py

@@ -0,0 +1,68 @@
+"""Minimal '_curses' module, the low-level interface for curses module
+which is not meant to be used directly.
+
+Based on ctypes.  It's too incomplete to be really called '_curses', so
+to use it, you have to import it and stick it in sys.modules['_curses']
+manually.
+
+Note that there is also a built-in module _minimal_curses which will
+hide this one if compiled in.
+"""
+
+import ctypes
+import ctypes.util
+
+
+class error(Exception):
+    pass
+
+
+def _find_clib():
+    trylibs = ["ncursesw", "ncurses", "curses"]
+
+    for lib in trylibs:
+        path = ctypes.util.find_library(lib)
+        if path:
+            return path
+    raise ModuleNotFoundError("curses library not found", name="_pyrepl._minimal_curses")
+
+
+_clibpath = _find_clib()
+clib = ctypes.cdll.LoadLibrary(_clibpath)
+
+clib.setupterm.argtypes = [ctypes.c_char_p, ctypes.c_int, ctypes.POINTER(ctypes.c_int)]
+clib.setupterm.restype = ctypes.c_int
+
+clib.tigetstr.argtypes = [ctypes.c_char_p]
+clib.tigetstr.restype = ctypes.POINTER(ctypes.c_char)
+
+clib.tparm.argtypes = [ctypes.c_char_p] + 9 * [ctypes.c_int]  # type: ignore[operator]
+clib.tparm.restype = ctypes.c_char_p
+
+OK = 0
+ERR = -1
+
+# ____________________________________________________________
+
+
+def setupterm(termstr, fd):
+    err = ctypes.c_int(0)
+    result = clib.setupterm(termstr, fd, ctypes.byref(err))
+    if result == ERR:
+        raise error("setupterm() failed (err=%d)" % err.value)
+
+
+def tigetstr(cap):
+    if not isinstance(cap, bytes):
+        cap = cap.encode("ascii")
+    result = clib.tigetstr(cap)
+    if ctypes.cast(result, ctypes.c_void_p).value == ERR:
+        return None
+    return ctypes.cast(result, ctypes.c_char_p).value
+
+
+def tparm(str, i1=0, i2=0, i3=0, i4=0, i5=0, i6=0, i7=0, i8=0, i9=0):
+    result = clib.tparm(str, i1, i2, i3, i4, i5, i6, i7, i8, i9)
+    if result is None:
+        raise error("tparm() returned NULL")
+    return result