Testing Infrastructure about dlux HOT 10 CLOSED

unittest is based around a programmatic interface:

class ApertureTest(unittest.TestCase):
    # Tests go here

Whereas pytest just uses python assert and has a command line interface.

from dlux.

I've made a new branch for this. I figured that we would be using the pytest infrastructure, but there is also the option to use unittest. We also have the option to set up a GitHub action to Automatically run the tests when we merge down branches.

from dlux.

Fantastic, not sure what the main difference between pytest and unittest are, maybe @benjaminpope has some thoughts?

from dlux.

So I have 500+ lines of tests implemented but have reached some of the more complex functionality. I was wondering how we wanted to go about testing correctness if at all. For now I have just left this blank, but I was wondering if we should consider places where there are simple analytic solutions or confirm against poppy models. @benjaminpope and I spoke briefly but I don't remember what we decided.

from dlux.

I would do pytest, simply because it is what I always use and it works well - if you have already implemented stuff in unittest so long as it works no harm done.

I would say we should test correctness only for 2 examples:

• perfect circular aperture, test that we get the airy disk right (analytically or vs poppy)
• simple Fresnel case, even from a circular aperture if we like, check that we get the same pattern as poppy

So long as we test a few examples that should be fine to make everything else just a simpler test for the moment.

from dlux.

We do not have full and complete tests, but we do have the testing infrastructure built out, so I am closing this issue.

from dlux.

Hi all,
I thought it was worth re-opening this issue since the code has undergone large changes since the tests were written. They were never that good to begin with but they are pretty poor quality now. I think we need to rethink our testing strategy and try and test some more of the code since a large amount of it has been tested only by eye on specific examples. Due to the complexity I think that this is in general the way that it will go and we should prune back the testing code to deal with the helper.py and coordinate.py code which is used everywhere. I don't know though, let me know what you think.

Regards

Jordan

from dlux.

Yeah I agree - Due to the large number of rapid changes in the pre 1.0 phase, I figured tests were more of a hindrance than a help. As we edge closer to full release I think we should definitely start thinking about this some more.

The way I see it, anything that is in the docs should be tested, plus a few end-to-end tests. I don't think we need to build anything yet, but once we have the Source & Scene structure finalised then we should build the tests.

Open to other thoughts though.

from dlux.

Yep. Everything in the docs, end to end tests.

There is a way to do this automatically: if we make the docs notebooks into parametrized notebooks, eg here. We can then execute the notebooks using jupyter nbconvert to run them as scripts in GitHub Actions (eg). We then simply do a couple of iterations in the optimization or MCMC loops, instead of tens or hundreds, and check that the results are finite.

I think also tbh that every Layer should be tested just that it instantiates and updates, but this is a lower priority.

from dlux.

Infrastructure built with 0.9.

from dlux.