nengo / pytest-plt Goto Github PK

It's pretty a pretty common annoyance to generate plots and then not know which ones are new and which ones you generated some other time. Also, it's common when iterating on a test you're writing to change the parametrizations and therefore leave a bunch of stale plots behind that are no longer valid. All to say that it would be nice to have a command line option that would remove all of the existing plots in the plot directory before generating the plots for the current run. Something like

pytest --plots-clear --plots

would clear all the plots first, then generate new ones.

Currently the docs folder only contains stub files. They should be filled in with real documentation.

Currently, the plots folder created by pytest-plt is based on the test function's nodeid. So, if the nodeid is something like nengo/tests/test_thing.py::test_simple_thing you end up with a nested folder like

plots
└── nengo
    └── tests
        └── test_thing.py::test_simple_thing.pdf

We should simplify / flatten this, though first we should decide the best way to simplify / flatten this.

Right now you can do

pytest --plots

or

pytest --plots my_cool_plotdir

to pass a directory in which to make plots. However, it's now also possible to set the plot directory in a config file. And it's quite annoying to do something like

pytest --plots nengo/tests/test_ensemble.py

and end up inadvertently making a nengo/tests/test_ensemble.py directory with a bunch of plots in it. So maybe we should just remove the ability to pass in a plot directory through the command line, and have --plots just be a flag.

We should add tests to make sure that plots are created and that their filenames are as we expect when passing --plots, and that nothing is created when not passing --plots.

Currently, if you do not pass a string to --plots, the default directory that plots will be placed in is called plots. In Nengo core we had some automated ways of setting this directory (using the Simulator fixture), but since this needs to work for any repository, we can't expect a Simulator fixture to exist.

A few ways we could think about automating the name of this directory:

1. Add a configuration option that you can set through the setup.cfg / pytest.ini file that sets the default plot directory name for a given project.
2. Somehow use the passed in args / configuration to generate a plot directory (so that if you pass in, e.g., --target loihi you get a different default plot directory name than if you pass in --target sim).

I haven't done any digging into why this is, just noting it down for future work. When you run tests through PyCharm, they get invoked with something like this command:

python.exe "C:\Program Files\JetBrains\PyCharm Community Edition 2019.1.2\helpers\pycharm\_jb_pytest_runner.py" --target test_simulator.py::test_train_recurrent -- --plots

Something about this confuses the way we parse the dirname:

...\lib\site-packages\pytest_plt\plugin.py:80: in __init__
    self.dirname = dirname
...\lib\site-packages\pytest_plt\plugin.py:96: in dirname
    os.makedirs(_dirname)

OSError: [WinError 123] The filename, directory name, or volume label syntax is incorrect: 'test_simulator.py::test_train_recurrent'

This isn't a new problem, the same issue was present with the plt fixture in nengo core.

Although it's kind of an edge case, it'd be nice to get this working since a) PyCharm is a not-uncommon IDE, and b) it's possible this same problem could be triggered in other ways when pytest is invoked in a "non-standard" way.

To reproduce:

1. Create file test.py

def test_plt(plt):
    plt.plot(range(100))

2. Run pytest test.py --plots

Gives:

  File "d:\miniconda3\envs\nengo-dl\lib\site-packages\pytest_plt\plugin.py", line 188, in _finalize
    plotter.__exit__(None, None, None)
  File "d:\miniconda3\envs\nengo-dl\lib\site-packages\pytest_plt\plugin.py", line 144, in __exit__
    self.save(os.path.join(self.dirname, self.plt.saveas))
  File "d:\miniconda3\envs\nengo-dl\lib\site-packages\pytest_plt\plugin.py", line 152, in save
    self.plt.savefig(path, **savefig_kw)
  File "d:\miniconda3\envs\nengo-dl\lib\site-packages\matplotlib\pyplot.py", line 689, in savefig
    res = fig.savefig(*args, **kwargs)
  File "d:\miniconda3\envs\nengo-dl\lib\site-packages\matplotlib\figure.py", line 2094, in savefig
    self.canvas.print_figure(fname, **kwargs)
  File "d:\miniconda3\envs\nengo-dl\lib\site-packages\matplotlib\backend_bases.py", line 2075, in print_figure
    **kwargs)
  File "d:\miniconda3\envs\nengo-dl\lib\site-packages\matplotlib\backends\backend_pdf.py", line 2558, in print_pdf
    file = PdfFile(filename, metadata=metadata)
  File "d:\miniconda3\envs\nengo-dl\lib\site-packages\matplotlib\backends\backend_pdf.py", line 433, in __init__
    fh, opened = cbook.to_filehandle(filename, "wb", return_opened=True)
  File "d:\miniconda3\envs\nengo-dl\lib\site-packages\matplotlib\cbook\__init__.py", line 392, in to_filehandle
    fh = open(fname, flag, encoding=encoding)
OSError: [Errno 22] Invalid argument: 'plots\\tmp.test.py::test_plt.pdf'

I suspect it's the :: (since colons are used to denote drives on Windows).

In Nengo SPA some tests are parametrized like this:

def pytest_generate_tests(metafunc):
    if 'algebra' in metafunc.fixturenames:
        metafunc.parametrize('algebra', [a for a in TestConfig.algebras])

It would be nice if there were an official way to include the algebra argument (or any other argument in the general case) in the dirname or filename/path of the output file without setting saveas in every test that makes use of this. (If this is not included in the filename, plots will be overwritten.)