Pytest and MPI import leading to large numbers of error messages

[alexhjames]

ARMI has an import statement for mpi4py inside of a try statement in the context.py file. See code snippet below. If pytest is run with something that imports ARMI (directly or indirectly) and the user doesn't have ms-mpi installed then pytest outputs a huge wall of windows exception error text. See example below. The try statement functions correctly in that pytest test collection continues fine despite the dump of errors. An ipython call to the same import statement results in the ImportError that the try statement is handling and doesn't result in the same wall of error messages. Is it possible to suppress the pytest wall of errors by either adding something to the pytest call or by adding something to this line in ARMI? Or is the solution to just shut up and install ms-mpi?

Lines of code affected:

armi/armi/context.py

Lines 111 to 129 in 827d223

	try:
	# Check for MPI
	# The mpi4py module uses cPickle to serialize python objects in preparation for
	# network transmission. Sometimes, when cPickle fails, it gives very cryptic error
	# messages that do not help much. If you uncomment th following line, you can trick
	# mpi4py into using the pure-python pickle module in place of cPickle and now you
	# will generally get much more meaningful and useful error messages Then comment it
	# back out because it's slow.
	# import sys, pickle; sys.modules['cPickle'] = pickle
	from mpi4py import MPI
	MPI_COMM = MPI.COMM_WORLD
	MPI_RANK = MPI_COMM.Get_rank()
	MPI_SIZE = MPI_COMM.Get_size()
	MPI_NODENAME = MPI.Get_processor_name()
	MPI_NODENAMES = MPI_COMM.allgather(MPI_NODENAME)
	except ImportError:
	# stick with defaults
	pass

Example wall of text:

Example ipython import:

[john-science]

Okay, after talking to Alex, the situation is that this try/except has existed for over a decade:

armi/armi/context.py

Lines 111 to 129 in 827d223

	try:
	# Check for MPI
	# The mpi4py module uses cPickle to serialize python objects in preparation for
	# network transmission. Sometimes, when cPickle fails, it gives very cryptic error
	# messages that do not help much. If you uncomment th following line, you can trick
	# mpi4py into using the pure-python pickle module in place of cPickle and now you
	# will generally get much more meaningful and useful error messages Then comment it
	# back out because it's slow.
	# import sys, pickle; sys.modules['cPickle'] = pickle
	from mpi4py import MPI
	MPI_COMM = MPI.COMM_WORLD
	MPI_RANK = MPI_COMM.Get_rank()
	MPI_SIZE = MPI_COMM.Get_size()
	MPI_NODENAME = MPI.Get_processor_name()
	MPI_NODENAMES = MPI_COMM.allgather(MPI_NODENAME)
	except ImportError:
	# stick with defaults
	pass

but at some point, recently, on Windows, it has started spraying the STDERR messages hundreds of times into STDOUT.

The code doesn't halt, so it's not technically a "bug". But that's certainly hideous and unwanted.

For now, Alex, I would say "install MPI" is a solution. But, long term, we will need to find a Python way to catch that STDERR message if it exists, and suppress it.