Use inline backend or Plots pane to display plots

[OverLordGoldDragon]

Matplotlib figures, if not closed, prevent testing (or any test code execution) from advancing. When running pytest via pytest.main([__file__, "-s"]) (F5, __main__), plots are inlined and there isn't a problem. Plots may be part of testing, so silencing isn't always an option.

Moving plots to the Plots pane, or inline, or really anywhere as long as they don't stall testing would work. Is this currently doable?

[jitseniesen]

What happens if you run pytest outside Spyder? In other words, is this problem due to how pytest and matplotlib works or is it due to something we do in Spyder and this plugin?

[OverLordGoldDragon]

@jitseniesen All tests pass, though no plots generated when ran via Anaconda Powershell Prompt. Tests also pass if the test .py file is ran from within Spyder via "Run" as __main__, and plots are generated inline. Only spyder-unittest generates plots in a new window and requires them to be closed before advancing the test.

[jitseniesen]

Thanks. It looks like it is up to us then. Can you please give a minimal (or at least small) example of a test?

[OverLordGoldDragon]

Sure:

import pytest
import matplotlib.pyplot as plt

def test_plot():
    plt.plot(list(range(10)))
    plt.show()  # no stall if commented, but no plot either

if __name__ == '__main__':
    pytest.main([__file__, "-s"])

[StefRe]

spyder-unittest runs the tests as a regular python process (i.e. not on a ipython kernel). I'm not sure if it's possible to use spyder's inline backend (the plots pane) for rendering matplotlib output from such a process.

However, in my opinion, unit tests are supposed to run automatically without user interaction. If the plot is (part of) the test result then the test should assert that it equals the expected result. You can use compare_images for this:

import pytest
import matplotlib as mpl
import matplotlib.pyplot as plt
from matplotlib.testing.compare import compare_images

def test_plot_compare(tmp_path):
    expected = 'baseline_test_plot.png'
    actual = str(tmp_path / 'test_plot.png')
    with mpl.rc_context(rc=mpl.rc_params()):
        plt.plot(list(range(10)))
        plt.savefig(actual)
    
    assert compare_images(expected, actual, 0) is None

if __name__ == '__main__':
    pytest.main([__file__, "-s"])

As a bonus you also get the difference image to see how exactly the actual result differs from the expected.

The context manager is used here to make the test run correctly both on the spyder console and from the unittest plugin as the matplotlib rc settings used by spyder differ from the default values (figsize, dip, subplot.bottom). So if you create your baseline image in spyder and then run the test from the plugin it will fail unless you enforce the same parameters.

[OverLordGoldDragon]

@StefRe Thanks for the suggestion. From what I can tell, compare_images tests (1) correctness of figure saving, (2) Spyder vs. unittest rc settings. I never found (1) a problem, and designed all configs in Spyder to begin with, so both are redundant in my use. It further introduces much additional boilerplate testing code for testing a dedicated visualization package, easily at least doubling code length - calls are made to functions within functions within classes, with dict-customization, so digging in each to reproduce for compare_images would be vastly counterproductive. (Also much of figs never save, and some close / delete before returning from function call, making this workaround outright impossible).

[StefRe]

I guess we talk about different things, maybe I couldn't make myself clear or I misunderstood your question in the first place.
Compare_images does not test anything (figure saving or settings etc.) It's just an instrument to automatically compare the actual image produced by the test with an expected image (created manually before the test). It just replaces the visual verification you do when looking at the image by an automatice assertion. So I was under the impression that the image is the result of your code under test and that the test should assert that the correct image was generated. If this is not your intention then you could just prevent any plotting by matplotlib.use('template') (but you wrote in your question that "silencing isn't always an option").

[OverLordGoldDragon]

@StefRe By "silencing isn't always an option" I meant I need to personally validate generated plots (can't code "check if annotation floats nicely near histogram"). True that I don't always need to do this across all files at once, and that a workaround is to silence everything most of the time to get coverage statistics - but that's just ignoring the problem, not fixing it. Admittedly I need the plots to also be generated inline, as sorting through dozens of generated windows manually is a no-no - so this complicates things further (for Spyder).

Regardless, it's better than nothing. The issue can then be considered a "feature request". As far as a bug status, I'll call the issue resolved.

[jitseniesen]

I misunderstood the issue. I thought that the plugin behaves differently from running pytest directly (from the command line), which I would consider a bug. However, the problem is that it behaves differently from running in a Spyder console. That is fine with me, the Spyder console is meant to be used interactively and tests (normally) not.

If you need to do manual validation, it is not automatic testing, so this seems to be a rather atypical use case. I don't think I'll be working on it anytime soon, but I'm happy to keep the issue open.

[OverLordGoldDragon]

@jitseniesen It's automatic once I validate all figures look as expected, e.g. changing any non-visual features don't require inspection - hence why I consider the issue part-solved. Glad it's kept open as an enhancement - but yes, nothing urgent, just a nice feature to look forward to. Thanks @StefRe for suggesting matplotlib.use('template').

[OverLordGoldDragon]

My handling of the suggestion, for others' reference; add below to __init__.py of test directory:

import os
import matplotlib

if not os.environ.get('IS_MAIN', '0') == '1':
    matplotlib.use('template')  # suppress figures for spyder unit testing

and, below to individual test files:

# test code ...

if __name__ == '__main__':
    os.environ['IS_MAIN'] = '1'
    pytest.main([__file__, "-s"])

The effect is to suppress plots unless files are ran as __main__. Any more convenient alternatives welcome; in particular, I wonder if there's a way to avoid the duplicative second code blob - e.g. by detecting that a non-main unit test is being ran.

[OverLordGoldDragon]

I somehow lucked with the aforementioned workaround, as imports do happen before setting the environment flag. import os and setting should thus happen before importing anything from __init__ - and if nothing imports from there, it might not work at all, in which case matplotlib.use should be done on per-file basis.

import os
if __name__ == '__main__':
    os.environ['IS_MAIN'] = '1'

# other imports & code

[StefRe]

In order to disable interactive plotting of a program only when run from the spyder-unittest plugin, you can place the following conftest.py in the rootdir of your test directory tree:

def pytest_configure(config):
    import traceback
    t = traceback.extract_stack()
    if 'pytestworker.py' in t[0][0]:
       import matplotlib as mpl
       mpl.use('template')  

It checks whether the program under test was called from pytestworker.py (which is the backend filename to run tests by pytest in the spyder-unittest plugin) and if so it just disables any matplotlib plot output by using the template backend. You need this file just in the root of your test directory tree and it will work for all tests in that tree.

Although it's more like a dirty trick than a clean solution (as it depends on spyder-unittest's internals) I think it still may be an acceptable workaround for your specific situation.

[OverLordGoldDragon]

@StefRe Indeed a better workaround - thanks.