Set up testing framework about critical-data-book HOT 4 OPEN

I agree. To answer these two questions, I would suggest that for Section 3:

where possible, all case studies should be reproducible using publicly available data and code.

i.e. we should be testing for reproducibility, and we should be using the original data. All the Section 3 chapters use either MIMIC (II or III), or simulated data. Therefore, the data used in each chapter are publicly available. I can't comment on whether or not this is the case for other Sections.

from critical-data-book.

What would we be testing for exactly?

I ask because it might be hard to test without the original data. Some chapters will have generic functions, but others will not.

from critical-data-book.

What would we be testing for exactly?

I was thinking mainly of unit tests. e.g. are the functions returning the results that we think they are returning? When we make changes to the code or when dependencies change, this will help to make sure things don't break. Pete also notes that his sections could do with testing in his pull request.

I ask because it might be hard to test without the original data.

Yep, this does add an extra level of complication for some tests. If data is a requirement for a test, we could run the tests on our local Jenkins server or we could make a demo dataset available. I agree that getting complete coverage will be tricky, and the range of languages doesn't help either. I'm only familiar with Python testing frameworks, so will need to investigate R and Matlab.

from critical-data-book.

Thanks Pete, I agree that it would be great to add some tests to check that the complete studies return expected results. We can try to do the same thing for section 2 too. It'll take us a while, but the nice thing about having the code here is that we can continue to tidy it up over time.

from critical-data-book.