Unit-test suite doesn't clear database about overview-server HOT 7 CLOSED

I thought the point of using transactions was so that unit tests would not leave any changes in the database. As far as I can tell, this is working. Why do we also need to truncate the tables?

from overview-server.

No, the point of using transactions is that unit tests don't interfere with each other. We can't assume the database is pristine when we start unit testing, for many reasons. The one you've found, in Bug #48, is that our integration tests do commit to the database, and must. So even if we fix Bug #48, integration tests that crash or fail may leave the database non-empty. And no matter how much software we tweak, there will always be the possibility that the power fails during an integration test, and the database will be non-empty.

That's a pragmatic reason. There's a philosophical reason, too: unit tests should be self-contained and shouldn't assume anything about the underlying system.

from overview-server.

Well, we're already breaking the philosophy of unit tests by accessing a real database. Another rule of unit tests is that they clean up after themselves, hence the presence of teardown methods, so the integration tests should undo any commits they make.
I think assuming that the test database is clean at the beginning of tests is reasonable, even if there is an occasional power outage we need to deal with. If it turns out the be a real problem we can handle it at that point.

I'd also be very reluctant to have to add code that has to go through every table and truncate it. If there's a quick way to wipe out the whole database, that might be ok. I'm pretty sure Play will automatically recreate the tables every time a FakeApplication() is started, though that may slow down the tests a bit.

from overview-server.

"I think assuming that the test database is clean at the beginning of tests is reasonable" -- disagree. You're assuming:

• The integration-test framework is flawless (and always cleans up after itself).
• The unit-test framework is flawless (and never commits).
• Developers haven't used the SQL database to do testing on the SQL console.
• The user hasn't interrupted integration tests (e.g., with Ctrl-C, or kill signals).
• Play hasn't failed (e.g., with one of those out-of-memory errors we keep getting).

We have already encountered all of these problems, just six weeks into development. There are boundless possibilities for more. In my previous Rails experience, there's been something in the database at the start of unit tests around 30 per cent of the time, even though unit and integration tests cleaned up after themselves.

Maybe it's possible for us to guarantee an empty test database around 90 or 95 per cent of the time, but we'll never hit 100 per cent, and we'll have to spend a lot of effort. Bug #47 alone is more difficult to fix then this one, and that won't come close to resolving this problem.

Yes, Bug #47 is a bug. But our unit-test suite needs to be resilient. In my opinion, this bug is more urgent.

from overview-server.

I guess since I have not seen any of the issues you mention cause the unit tests to fail, I don't share your sense of urgency (I'd recommend considering the testDB off-limits for manual testing).
Why is it harder to clear the database at the end of the integration tests than at the beginning of the unit tests?

from overview-server.

Because I have to tie into the "end integration tests" hook, which isn't easy. I haven't managed to hook into the "begin integration tests" hook either.

from overview-server.

I truncate the document_set table before the tests now. We'll have to remember to update the code if that is not enough to clear the db.

from overview-server.