Support for Lifecycle Hooks about jqwik HOT 23 CLOSED

I have been considering two approaches for offering easy to use against each other: the classic approach with @Before... and @After... annotations for methods, as we know it from JUnit.

The other approach is (slightly) more involved but IMO a bit more flexible. It's about annotating inner lifecycle classes, e.g.:

class MyApplicationTests {

  private Table table;

  @PerProperty
  class InMemoryDatabaseSetup extends PerPropertyLifecycle {
    InMemoryDatabase database;
    void beforeEachProperty() {
      database = createAndInitializeDatabase();
      table = database.getTable("table");
    }
    void afterEachProperty() {
      database.cleanup();
    }
  }

  @Property
  void inMemoryPropertyTest(@ForAll final Key key, @ForAll final Value value) {
    // do things with table, key and value...
  }
}

The interesting thing about these lifecycle classes is that you can have abstract superclasses or predefined classes that can be reused without having to introduce a hierarchy of test classes. Think delegation over inheritance. Moreover, there would be a lot of symmetry for the different lifespans: @PerContainer, @PerProperty and @PerTry.

I'd love to get feedback about this idea!

from jqwik.

Implementation is reasonably advanced that I declare this issue as closed (for the time being).
Since there is no documentation yet for lifecycle hooks I'll add a short description of how it works:

• The base type of all hooks is LifecycleHook with the following specialized interfaces:
  • SkipExecutionHook: Allows containers and properties to be skipped if necessary
  • ResolveParameterHook: Inject parameters into property methods
  • BeforeContainerHook: Perform some set up logic before starting tests
    of a container class or the test engine itself.
  • AfterContainerHook: Perform some clean up after finishing all child containers or properties
  • AroundPropertyHook: Embed the execution of a property - with all its tries -
    in your own logic. This also allows to change the result of the execution.
  • AroundTryHook: Embed the execution of a single try in your own logic.
    This allows to change parameters and the result of a single try.
• Hook implementations can be added to a container or a property with the annotation
  @AddLifecycleHook(MyHookImplementation.class). There is also the possibility to
  add global hooks using Java's service loader mechanism.
• Each hook implementation can decide if it will be propagated to its descendants
  and which types of descendants it can be applied to. This propagationMode can
  be overridden in through @AddLifecycleHook.propagateTo()
• Sharing values across lifecycle calls should be done through instances of type Store.
  Those instances can be created and accessed through (static) methods on Store.
  Using stores (instead of member variables) is important, because there is only a
  single instance of your hook implementation and the stores allow to specify the lifespan
  • Lifespan.RUN, Lifespan.PROPERTY, Lifespan.TRY - of each stored value.

Here is an example of a hook that creates and publishes a simple timing report for a property:

@Target({ElementType.METHOD})
@Retention(RetentionPolicy.RUNTIME)
@AddLifecycleHook(Timing.TimingHook.class)
public @interface Timing {

	class TimingHook implements AroundPropertyHook, AroundTryHook {

		Store<Long> sumOfTriesTiming = Store.create("sumOfTriesTiming", Lifespan.PROPERTY, () -> 0L);

		@Override
		public PropertyExecutionResult aroundProperty(PropertyLifecycleContext context, PropertyExecutor property) {
			long beforeProperty = System.currentTimeMillis();
			PropertyExecutionResult executionResult = property.execute();
			long afterProperty = System.currentTimeMillis();
			long overallTime = afterProperty - beforeProperty;
			createTimingReport(context.reporter(), context.extendedLabel(), overallTime, executionResult.countTries());
			return executionResult;
		}

		private void createTimingReport(Reporter reporter, String label, long overallTime, int tries) {
			long averageTryTime = sumOfTriesTiming.get() / tries;
			String key = String.format("timing of %s", label);
			String report = String.format("%n\toverall: %d ms%n\taverage try: %d ms", overallTime, averageTryTime);
			reporter.publish(key, report);
		}

		@Override
		public TryExecutionResult aroundTry(TryLifecycleContext context, TryExecutor aTry, List<Object> parameters) {
			long beforeTry = System.currentTimeMillis();
			try {
				return aTry.execute(parameters);
			} finally {
				long afterTry = System.currentTimeMillis();
				long timeForTry = afterTry - beforeTry;
				sumOfTriesTiming.update(time -> time + timeForTry);
			}
		}
	}
}

As you can see it's rather straightforward to combine a self-made annotation - @Timing -
with a hook implementation in the timing interface itself. Therefore this hook can be used like this:

@Property(tries = 50)
@Timing
void slowProperty(@ForAll @IntRange(min = 10, max = 100) int delay) throws InterruptedException {
	Thread.sleep(delay);
}

Running this property creates the following report on my machine:

timestamp = 2020-02-17T17:46:32.757, timing of TimingExtensionExamples:slowProperty = 
	overall: 2220 ms
	average try: 41 ms

from jqwik.

jqwik is not a Jupiter extension but a test engine of its own. As a consequence jqwik has its own lifecycle which differs in large parts from how Jupiter works. That's why Jupiter extensions won't be supported.

I have plans, though, to make the already existing lifecycle public and extensible. Some stuff, like @Before|AfterEach|All can be implemented - but I'll definitely use different annotations to make confusion less likely. Extensions will be completely different though.

What kind of lifecycle annotations and extensions would you need in property tests most urgently?

from jqwik.

In theory it would be nice to be able to support all of these extensions:

• BeforeEachCallback
• AfterEachCallback
• BeforeAllCallback
• AfterAllCallback
• ParameterResolver

Or at least the last 3. I can give a sketch of what ideally I would like to have, it's not urgent in any way though. It would just be an improvement of how I've written my tests right now.

// An in-memory database
@ExtendWith(InMemoryExtension.class)
public class Example {

  private static Table table;

  @BeforeAll
  static void setup(final InMemoryDatabase database) {
    // the BeforeAllCallback implementation does all the boiler plate needed to create a database
    table = database.getTable("table");
  }

  @Property
  void inMemoryPropertyTest(@ForAll final Key key, @ForAll final Value value) {
    // do things with table, key and value...
  }
}

from jqwik.

Since the lifecycle of jqwik is different, there will probably be hooks for

• Before/After engine
• Before/After container
• Before/After property
• Before/After try

As for parameter injection I'm not sure yet. Given that @ForAll parameters require special handling it's not obvious how to tackle that in a consisten way. Easiest is probably to require an explicit annotation like @Inject for everything provided by extensions. I'm also wondering if injection of member variables wouldn't suffice for most if not all purposes.

from jqwik.

Most of the bare necessities are there now. Sadly in a experimental and undocumented state:

• Annotation @AddLifecycleHook(Class<? extends LifecycleHook>) to add a hook implementation to any container class or property method.

• Hook interfaces that can already be implemented and hooked in:

  • AroundPropertyHook
  • AroundTryHook
  • BeforeContainerHook
  • SkipExecutionHook

• Interface Store with a few static methods to create stores for data with different lifecycle needs.

• A few static methods on class PropertyLifecycle to hook in clean up or assertion logic for a full property run:

  • PropertyLifecycle.onSuccess(..)
  • PropertyLifecycle.after(..)

The exact interfaces and methods are unstable but they are already being used internally, e.g. for all the Statistics coverage checking.

from jqwik.

@emmmile Parameter resolution for property methods would (currently) require too much change to jqwik's lifecycle. Could you live with member variable injection instead? S.t. like:

@AddLifecycleHook(InMemoryDatabaseHook.class)
public class Example {

  private static InMemoryDatabase database;
  private static Table table;

  @BeforeContainer
  static void setup() {
    // the BeforeContainerHook implementation does all the boiler plate needed to create a database
    table = database.getTable("table");
  }

  @Property
  void inMemoryPropertyTest(@ForAll final Key key, @ForAll final Value value) {
    // do things with table, key and value...
  }

  @AfterContainer
  static void setup() {
    database.drop("table");
  }
}

from jqwik.

Is there any possibility to mess with random seed? i.e. to have last failed seed saved somewhere else than .jqwik-database file? I would expect something like @BeforeSuite possible and @AfterSuite with required data passed to methods.

Feedback: delegation over inheritance is very welcome.

from jqwik.

Is there any possibility to mess with random seed?

Currently it can only be read. What scenario do you have in mind for changing it?

from jqwik.

Is there any possibility to mess with random seed?

Currently it can only be read. What scenario do you have in mind for changing it?

It is just hypothetical scenario, which have visited my mind when I have read about lifecycle hooks:
There are sometime build setups which are running on CI/CD services, which does not persist workspace folder (i.e. folder with project). For example temporal jenkins nodes, or utilizing docker for build (similar like hateoas jenkins file shows). In such environments, .jqwik-database would not be persisted between builds. There are also organizations, where they does not heard about DevOps and builds are done using single "weBuildAnythingRightTM" script, which cannot be customized. In such minor scenario, I would try to use lifecycle hooks to save last failing seed after suite to external storage (database, "ftp", or something like webdav for example) and than use it in BeforeSuite to insert into jqwik.

EDIT: Scenario 2:
Imagine that project with jqwik tests have multiple branches and in some branch some luck will get right seed to find bug. Now merging that branch into other (perhaps master) branch should also take knowledge of the seed, which have found bug. This can be managed through CI/CD system (now I am not reasoning about, where that responsibility belongs). Perhaps just before release one would like to run tests with all "once failed" seeds...

PS: I think that each found bug should get its own @Example test, or unit test using jupiter, but sometime team does not have enough discipline to convince manager that running tests with last few seeds is waste of time.

Have on mind, that these are just hypothetical scenarios and I have no need for it now.

from jqwik.

Have on mind, that these are just hypothetical scenarios and I have no need for it now.

I'll put it in the backlog then.

from jqwik.

Just some questions I had in mind:

• Using delegation/composition: How are you ensuring ordering? For example, if my tests not only require to ramp up a database and but also a web server?
• In your example you are using an interface and an annotation. Would one of these not be enough?
• Have you considered using mixins? Like having interfaces with default methods a container would mix in?

from jqwik.

@gtrefs Thanks for thinking about it. A few answers/thoughts:

Using delegation/composition: How are you ensuring ordering? For example,
if my tests not only require to ramp up a database and but also a web server?

The underlying "Hooks" have a proximity value to tell how close you are to the underlying property method. I could expose that for overriding, e.g.

@PerProperty
class InMemoryDatabaseSetup extends PerPropertyLifecycle {
  void beforeEachProperty() { createDatabase(); }
  void afterEachProperty() { removeDatabase(); }
  int proximity() { return 1; }
}
@PerProperty
class WebServerStartup extends PerPropertyLifecycle {
  void beforeEachProperty() { startUpWebserver(); }
  void afterEachProperty() { shutDownWebserver(); }
  int proximity() { return 5; }
}

would mean that InMemoryDatabaseSetup would wrap WebServerStartup, i.e. the database would be created before server start up but removed after server shut down.

In your example you are using an interface and an annotation.
Would one of these not be enough?

Can you give an example?

Have you considered using mixins? Like having interfaces with default methods
a container would mix in?

Like this: ?

class MyApplicationTests implements PerPropertyLifecycle {

  private Table table;
  private InMemoryDatabase database;
  
  @Override  
  void beforeEachProperty() {
      database = createAndInitializeDatabase();
      table = database.getTable("table");
    }

  @Override
  void afterEachProperty() {
      database.cleanup();
  }

  @Property
  void inMemoryPropertyTest(@ForAll final Key key, @ForAll final Value value) {
    // do things with table, key and value...
  }
}

Wouldn't that restrict me to a single lifecycle method per container class?
But maybe I didn't understand your suggestion correctly.

from jqwik.

@gtrefs Thanks for thinking about it. A few answers/thoughts:

Using delegation/composition: How are you ensuring ordering? For example,
if my tests not only require to ramp up a database and but also a web server?

The underlying "Hooks" have a proximity value to tell how close you are to the underlying property method. I could expose that for overriding, e.g.

@PerProperty
class InMemoryDatabaseSetup extends PerPropertyLifecycle {
  void beforeEachProperty() { createDatabase(); }
  void afterEachProperty() { removeDatabase(); }
  int proximity() { return 1; }
}
@PerProperty
class WebServerStartup extends PerPropertyLifecycle {
  void beforeEachProperty() { startUpWebserver(); }
  void afterEachProperty() { shutDownWebserver(); }
  int proximity() { return 5; }
}

would mean that InMemoryDatabaseSetup would wrap WebServerStartup, i.e. the database would be created before server start up but removed after server shut down.

What does closeness to the underlying property method mean? Is there a difference between an order?

In your example you are using an interface and an annotation.
Would one of these not be enough?

Can you give an example?

  class InMemoryDatabaseSetup extends PerPropertyLifecycle {
    InMemoryDatabase database;
    void beforeEachProperty() {
      database = createAndInitializeDatabase();
      table = database.getTable("table");
    }
    void afterEachProperty() {
      database.cleanup();
    }
  }

The engine detects an inner class of type PerPropertyLifecycle. No additional annotations needed. Or:

 @PerProperty
  class InMemoryDatabaseSetup {
    InMemoryDatabase database;
    void beforeEachProperty() {
      database = createAndInitializeDatabase();
      table = database.getTable("table");
    }
    void afterEachProperty() {
      database.cleanup();
    }
  }

The engine detects an inner class of annotated with PerProperty and tries to call the corresponding methods. Her you could also use annotations on method like @BeforeEach or @AfterEach:

 @PerProperty
  class InMemoryDatabaseSetup {
    InMemoryDatabase database;
    @BeforeEach
    void setupDatabase() {
      database = createAndInitializeDatabase();
      table = database.getTable("table");
    }
   @AfterEach
    void clearDatabase() {
      database.cleanup();
    }
  }

Have you considered using mixins? Like having interfaces with default methods
a container would mix in?

Like this: ?

class MyApplicationTests implements PerPropertyLifecycle {

  private Table table;
  private InMemoryDatabase database;
  
  @Override  
  void beforeEachProperty() {
      database = createAndInitializeDatabase();
      table = database.getTable("table");
    }

  @Override
  void afterEachProperty() {
      database.cleanup();
  }

  @Property
  void inMemoryPropertyTest(@ForAll final Key key, @ForAll final Value value) {
    // do things with table, key and value...
  }
}

Wouldn't that restrict me to a single lifecycle method per container class?
But maybe I didn't understand your suggestion correctly.

More like this:

public interface InMemoryDatabase extends PerPropertyLifecycle {
   @BeforeEach
   default Database createDatabase(){
     return createAndInitializeDatabase();
   }

  @AfterEach
  default void cleanDatabase(Database database){
     database.cleanUp();
  }
}

public class MyTest implements InMemoryDatabase, MyWebserver {
   // do the composition here
}

But yes,

Wouldn't that restrict me to a single lifecycle method per container class?

would be an issue.

from jqwik.

What does closeness to the underlying property method mean?
Is there a difference between an order?

The term is supposed to make the meaning clearer. When you "wrap around" as you do with before and after methods, what does "order" mean? An earlier or a later call? IMO "order" is ambiguous here. "proximity" might not be such a common term which forces you to think about what you mean. Higher proximity -> closer to the eventual invocation of the method.

Hopefully, most hooks and lifecycle methods won't have to care about that.

from jqwik.

The engine detects an inner class of annotated with PerProperty and tries to call the
corresponding methods. Her you could also use annotations on method like
@BeforeEach or @AfterEach

Then you run into the order problem again. My opinion about annotations is that I try to avoid them when types give me (roughly) the same capabilities. To make the programming model fairly convenient to use we often combine both. It's a hard task to get it right, though.

from jqwik.

More like this:

public interface InMemoryDatabase extends PerPropertyLifecycle {
  @BeforeEach
   default Database createDatabase(){
      return createAndInitializeDatabase();
   }
...
class MyTest implements InMemoryDatabase, MyWebserver {...}

Which is basically back to the annotated lifecycle methods from Jupiter where you can do exactly that.

I guess I'm trying to steer away from annotated methods wherever I can. But that's just my gut feeling. For example #91 would free jqwik users from the need to use @Provide methods.

And combination of lifecycles would also work with the class based approach:

class MyTests {

  @PerProperty
  class DatabaseAndServer extends InMemoryDatabase, MyWebserver {
  }
}

from jqwik.

More like this:

public interface InMemoryDatabase extends PerPropertyLifecycle {
  @BeforeEach
   default Database createDatabase(){
      return createAndInitializeDatabase();
   }
...
class MyTest implements InMemoryDatabase, MyWebserver {...}

Which is basically back to the annotated lifecycle methods from Jupiter where you can do exactly that.

I guess I'm trying to steer away from annotated methods wherever I can. But that's just my gut feeling. For example #91 would free jqwik users from the need to use @Provide methods.

I see, you can also make the lifecycle methods return a PerPropertyLifecycle:

class MyTest {
  public PerPropertyLifecycle composed = database().andThen(webserver());
 
  public PerPropertyLifecycle database(){
    return new PerPropertyLifecycle(){
            void before(){
              setupDatabase();
            }
            void after(){
              cleanupDatabase();
           }
    }
  }
}

And combination of lifecycles would also work with the class based approach:

class MyTests {

  @PerProperty
  class DatabaseAndServer extends InMemoryDatabase, MyWebserver {
  }
}

I think this approach won't work in Java. Multiple class inheritance is not allowed.

from jqwik.

I just started work on allowing Parameter resolution in lifecycle hooks, similar to what Jupiter does with ParameterResolver. I guess I'll call the hook ParameterInjectionHook to make confusion with Jupiter's functionality less likely.

from jqwik.

I've eventually decided to name it ResolveParameterHook. It does more or less the same thing as Jupiter's parameter resolver but it can deal with the fact that lots of tries might mean lots of calls:

public interface ResolveParameterHook extends LifecycleHook {
	/**
	 * This method will be called only once per property, whereas the returned
	 * supplier is usually called for each try - and potentially more often during shrinking.
	 *
	 * @param parameterContext
	 * @param propertyContext
	 * @return a supplier function that should always return an equivalent object,
	 * 			i.e. an object that behaves the same when used in the same way.
	 */
	Optional<Supplier<Object>> resolve(ParameterResolutionContext parameterContext, PropertyLifecycleContext propertyContext);
}

from jqwik.

With ResolveParameterHook in place annotated lifecycle methods become more attractive because they could also take resolved parameters:

@BeforeProperty
void initializeTable(Database database) {
   ...
}

I don't see how that could go together with fixed signatures of some lifecycle interface.

from jqwik.

I created a new issue to account for lifecycle methods: #92

Thus, I'll shortly be able to close this issue when all lifecycle hooks are done.

from jqwik.

Closing for now. Feel free to comment, ask and maybe reopen.

from jqwik.