JSONDecodeError when determining list of installed Python packages

[boegel]

It seems like pip list ... --format json sometimes returns nothing at all, which isn't being handled well.

 == 2023-01-25 17:38:10,411 pythonpackage.py:455 INFO Running command '/apps/gent/RHEL8/cascadelake-ib/software/Python/3.10.4-GCCcore-11.3.0/bin/python -m pip list --isolated --disable-pip-version-check --format json'
 == 2023-01-25 17:38:11,482 pythonpackage.py:464 DEBUG Command '/apps/gent/RHEL8/cascadelake-ib/software/Python/3.10.4-GCCcore-11.3.0/bin/python -m pip list --isolated --disable-pip-version-check --format json' returned with 0: stdout: ; stderr:
...
 == 2023-01-25 17:38:11,669 build_log.py:171 ERROR EasyBuild crashed with an error (at easybuild/base/exceptions.py:126 in __init__): Traceback (most recent call last):
   File "/user/gent/400/vsc40003/easybuild_easyinstalled/lib/python3.6/site-packages/easybuild_framework-4.7.1.dev0-py3.6.egg/easybuild/main.py", line 132, in build_and_install_software
     (ec_res['success'], app_log, err) = build_and_install_one(ec, init_env)
   File "/user/gent/400/vsc40003/easybuild_easyinstalled/lib/python3.6/site-packages/easybuild_framework-4.7.1.dev0-py3.6.egg/easybuild/framework/easyblock.py", line 4187, in build_and_install_one
     result = app.run_all_steps(run_test_cases=run_test_cases)
   File "/user/gent/400/vsc40003/easybuild_easyinstalled/lib/python3.6/site-packages/easybuild_framework-4.7.1.dev0-py3.6.egg/easybuild/framework/easyblock.py", line 4070, in run_all_steps
     self.run_step(step_name, step_methods)
   File "/user/gent/400/vsc40003/easybuild_easyinstalled/lib/python3.6/site-packages/easybuild_framework-4.7.1.dev0-py3.6.egg/easybuild/framework/easyblock.py", line 3905, in run_step
     step_method(self)()
   File "/user/gent/400/vsc40003/easybuild_easyinstalled/lib/python3.6/site-packages/easybuild_easyblocks-4.7.1.dev0-py3.6.egg/easybuild/easyblocks/generic/pythonpackage.py", line 884, in sanity_check_step
     pkgs = self.get_installed_python_packages(names_only=False, python_cmd=python_cmd)
   File "/user/gent/400/vsc40003/easybuild_easyinstalled/lib/python3.6/site-packages/easybuild_easyblocks-4.7.1.dev0-py3.6.egg/easybuild/easyblocks/generic/pythonpackage.py", line 465, in get_installed_python_packages
     pkgs = json.loads(stdout.strip())
   File "/usr/lib64/python3.6/json/__init__.py", line 354, in loads
     return _default_decoder.decode(s)
   File "/usr/lib64/python3.6/json/decoder.py", line 339, in decode
     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
   File "/usr/lib64/python3.6/json/decoder.py", line 357, in raw_decode
     raise JSONDecodeError("Expecting value", s, err.value) from None
  json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

This happened during the installation of spglib-python-2.0.0-foss-2022a.eb, and it was a fluke because it didn't happen on 2nd iteration, and there definitely are Python packages installed (since there's a bunch included with Python).

[boegel]

I can reproduce this outside an EasyBuild run using the following Python script:

import os

print("COLUMNS: %s" % os.getenv('COLUMNS'))
print("LINES: %s" % os.getenv('LINES'))

from easybuild.tools.options import set_up_configuration
from easybuild.tools.modules import modules_tool
from easybuild.tools.run import run_cmd

set_up_configuration()

del os.environ['PYTHONPATH']

modtool = modules_tool()
modtool.load(['Python/3.10.4-GCCcore-11.3.0'])
print(os.getenv('PYTHONPATH'))

out, ec = run_cmd("/apps/gent/RHEL8/cascadelake-ib/software/Python/3.10.4-GCCcore-11.3.0/bin/python -m pip list --isolated --disable-pip-version-check --format json")
if out:
    print("Output of 'pip list': %s..." % out[:20])
else:
    print("empty output of 'pip list'!")

To trigger the problem, I need to be in an interactive shell session, in which the terminal width/height is not set correctly, as shown by the $COLUMNS and $LINES environment variables (which are not exported, and hence None in the Python session).

[vsc40003@node4015 ~]$ echo $COLUMNS
80
[vsc40003@node4015 ~]$ echo $LINES
24
[vsc40003@node4015 ~]$ python issue2866.py
COLUMNS: None
LINES: None
== Temporary log file in case of crash /tmp/eb-bgxvgdne/easybuild-xx9kazr4.log
/apps/gent/RHEL8/cascadelake-ib/software/Python/3.10.4-GCCcore-11.3.0/easybuild/python
  >> running command:
        [started at: 2023-01-26 09:08:04]
        [working dir: /kyukon/home/gent/400/vsc40003]
        [output logged in /tmp/eb-bgxvgdne/easybuild-run_cmd-f3bj9koq.log]
        /apps/gent/RHEL8/cascadelake-ib/software/Python/3.10.4-GCCcore-11.3.0/bin/python -m pip list --isolated --disable-pip-version-check --format json
  >> command completed: exit 0, ran in 00h00m11s
empty output of 'pip list'!

The pip list command does produce output as expected after running the resize command:

[vsc40003@node4015 ~]$ resize
COLUMNS=272;
LINES=58;
export COLUMNS LINES;
[vsc40003@node4015 ~]$ python issue2866.py
COLUMNS: None
LINES: None
== Temporary log file in case of crash /tmp/eb-tdqkfnve/easybuild-_obx9fwc.log
/apps/gent/RHEL8/cascadelake-ib/software/Python/3.10.4-GCCcore-11.3.0/easybuild/python
  >> running command:
        [started at: 2023-01-26 09:08:25]
        [working dir: /kyukon/home/gent/400/vsc40003]
        [output logged in /tmp/eb-tdqkfnve/easybuild-run_cmd-pfgsbnpv.log]
        /apps/gent/RHEL8/cascadelake-ib/software/Python/3.10.4-GCCcore-11.3.0/bin/python -m pip list --isolated --disable-pip-version-check --format json
  >> command completed: exit 0, ran in 00h00m10s
Output of 'pip list': [{"name": "alabaster...

[boegel]

This problem was sort of self-inflicted, it's caused by running EasyBuild in a strange shell environment, one that was started as follows:

$ echo $COLUMNS $LINES
272 58

$ python3
Python 3.6.8 (default, Jun 14 2022, 12:54:58)
[GCC 8.5.0 20210514 (Red Hat 8.5.0-10)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pty
>>> pty.spawn(['/usr/bin/srun', '--export=USER,HOME,TERM', '--pty', '/bin/bash'])
srun: job 53639925 queued and waiting for resources
srun: job 53639925 has been allocated resources
$ echo $COLUMNS $LINES
80 24

A shell session started with pty.spawn doesn't have the terminal window size set correctly, see https://bugs.python.org/issue41541.

So other than avoiding the nasty crash when pip list didn't produce any output, there's not much to do on the EasyBuild side to fix this.