Provide an efficient way to share across a set of tests about test HOT 20 CLOSED

As I mentioned in #17, without a compelling concrete use-case I don't think this is a feature that provides positive value. Sharing state across tests is an anti-pattern that we shouldn't encourage; it leads to brittle tests that can fail when run independently.

This will also essentially cease to work if, as we've discussed in the past, we start running each test case in an individual isolate for parallelism and isolation. It doesn't seem worth adding a feature that will likely be useless in the mid-term future.

from test.

Sometimes it's more important to have fast tests (during development). jUnit also has @before and @after.
When I have to start a server which takes 3sec for each test this is all but fast (and I'm sure there are servers which take much longer).
Starting and stopping a database server where I can create a new database for each test doesn't add much of a shared state.

from test.

Like @zoechi said, In some case (like mine : http://stackoverflow.com/questions/29102574/how-can-i-make-test-inside-a-then-statement-in-dart/), we may want to have a shared context for some situation. To add to the list of examples, if we want to test a similar hand made OAuth system, we may want to share the key of authentication to successive tests. Like, posting a content with the key, and test the accessibility of this content with the same key. It can be done with different expect in the same test but it cost the possibility to have a description for each expect.

from test.

@fandegw That's exactly the type of thing I'm worried about this behavior encouraging. What you're describing is brittle and likely to produce hidden cross-test dependencies, and doesn't sound like it would actually produce any efficiency gains.

from test.

Sometimes it's OK when it's not perfect.

I have the impression there are two schools colliding in Dart.
There it the forgiving language which thrives to still run as far as possible even when it contains errors and then there are tools which try to prevent everything as soon as it might not be advisable in some situations which contradicts the intentions of the Dart language designers.

I had a similar discussion where the Polymer transformer just stopped with an error message because a CSS import was invalid. The argument was that nobody would want to have an application with an invalid CSS import. I'm glad this behavior was changed. When I want to debug some specific behavior I might just not care about that CSS right now.

This is about software development where it's often very helpful to get something up and running and the tools should help or go out of your way but never get in your way even less intentionally.

from test.

I don't think there's as strong a philosophical dichotomy as you're drawing. Every feature needs to be weighed individually to see if the benefit it adds is worth the costs.

from test.

Nothing against that. My problem here is that there is no rasonable workaround except forking your package and adding the feature.

from test.

Using setUp is a workaround. It won't be as fast, but it will work.

from test.

I have a set of tests in my Datastore mock package and I want to run each test once with the mock server and once with the Datastore Local Development Server (DevServer) and they should have the same output.

Currently the Datastore Local Development Server is run twice, even though it is used by one of both tests.

  group('begin transaction', () {
    String datasetName;
    DatastoreLocalDevServer server;
    var testFunction;

    setUp(() {
      datasetName = 'begin_transaction';
      server = initDatastoreLocalDevServer(datasetName);

     // run this test body once with the mock server and once with DevServer
      testFunction = (Connection connection) async {
        final response =
            await connection.beginTransaction(new BeginTransactionRequest());
        expect(response.transaction, isNotEmpty);
      };
    });

    test('Datastore Local Development Server', 
        () => executeLocalDatastoreTest(server, datasetName, testFunction)));

    test('Mock',  () => executeV1serviceMockTest(datasetName, testFunction)));
  });

With a shared state this would work just fine.
Maybe there is a better way for this use case. Any suggestions?

from test.

It seems to me that having a separate database instance for each test is good—it reduces the surface area for bugs caused by hidden interactions between tests.

from test.

I don't see how this last comment is related to my last comment. Should it be?
Each of these two test uses a different server. One test uses the Datastore local development server and the other test uses a mock server.
When both tests are (or actually when the one that uses that server is) done I want to shut the Local Development server down.
(In the above example the Mock server is instantiated inside the test but I changed this in the meantime so that now both servers are started in setUp() as well)
Now the Local development server is started twice (once for the test that actually uses it and once where it isn't even used).
I have to move each test in its own group for that to work.
I saw recently that the unit test configuration (if I remember right) provided a callback when all tests are finished but this seems to be removed as well in 0.12.0.

from test.

Why not just instantiate the server only in the test where it's used? If it's only used by one test, only instantiate it in that test.

from test.

Because then shutdown isn't called when the test fails

from test.

tearDown isn't magical; it's just a try/catch under the covers. If you write your own try/catch around the test, you'll get the same behavior.

from test.

I can't get why this feature is discussed so long without any success.
To my opinion, it's not a thing that would break anything.
The ones who disagree with it are free not to use it without any additional cost, but for some rare situations with complex and slow setups it would help a lot not to wait forever until tests are done.
Yes, one of it's cones is that using it will put some responsibility to developer to avoid changes in shared state between tests, but the pros worth it: it will have much more performance.
And again, for those who don't need that, just shouldn't use it. Even more, we can have two copies of tests, one with shared state, used in active development state, and one without, to finally check if there was any hidden errors in tests.

from test.

Hmmm at the risk of flogging a dead horse I just ran into this when porting my rest Api tests. I used to start my server just once then run a set of tests against it.

Luckily for me I seem to be getting away with starting / stopping my server for each test. I only get away with that as dart starts up so fast and my tests run the server with an in memory version of my daos. I doubt that will always be the case.

from test.

Nice :)

from test.

@zoechi use it wisely 😄

from test.

@kevmoo nothing to worry. I started writing a mock for a server to be able to run my tests because starting and stopping the server for each test would just make it impractical to run the number of tests I need to be confident everything works as expected. I would probably had spent another 2 months on the mock alone before I even could start writing tests. I don't have the resources to run my tests on big server farms. Now I am glad I focused on more productive things instead. I still can build the mock eventually but first I need to focus on getting something shipped.

from test.

Awesome! Thank you for adding this 👍

from test.