junit-team / junit5 Goto Github PK
View Code? Open in Web Editor NEW✅ The 5th major version of the programmer-friendly testing framework for Java and the JVM
Home Page: https://junit.org
License: Other
✅ The 5th major version of the programmer-friendly testing framework for Java and the JVM
Home Page: https://junit.org
License: Other
Currently if we need to test some method is throwing an exception by following methods
@test(expected=BalhException.class)
OR
@test
public void meTest() {
int a = 10;
int b = 20;
Cat c = new Cat();
try{
c.talk()
fail("this is bad")
} catch(BalhException e) {
//this is good
}
}
I prefer the 2nd option since there i can really tell that the point of failure is c.talk() method.
So I suggest a annotation for a method execution level tests which would look like following
@test
public void meTest() {
int a = 10;
int b = 20;
Cat c = new Cat();
@MethodTest(expected=BalhException.class)
c.talk()
}
(not checked for package cycles etc., yet)
org.junit.gen5.engine.api
It would be nice to have some code conventions defined, to make it easier for non-core-committers to provide useful pull requests.
Ideally they would come with formatters/checkstyle configurations.
Add something like it was proposed in junit-team/junit4#1055.
prototype-1
.org.junit
.junit5-engine
) using JUnit Team <[email protected]>
as the Git authorjunit:junit:4.12
from all existing modules except junit4-engine, junit4-launcher-runner, and open-test-alliance.JUnit Team user configuration for ~/.gitconfig
:
[user]
name = JUnit Team
email = [email protected]
Until we have reimplemented the junit5 engine (see #39) we should not execute the tests of the sample projects, as they are built upon the new features.
TestDescriptor
: immutable API for users of Launcher (in junit-engine-api)MutableTestDescriptor
extends TestDescriptor
: additional methods to add children etc.For greater clarity:
junit4-launcher-runner
module to junit4-runner
.org.junit.gen5.junit4runner
package to org.junit.gen5.junit4.runner
.The current proposal is to introduce the following:
@ScenarioTest
: class-level annotation used to denote that a test class contains steps that make up a single scenario test.@Step
: method-level annotation used to denote that a test method is a single step within the scenario test.The @Step
annotation will need to provide an attribute that can be used to declare the next step within the scenario test. Steps will then be ordered according to the resulting dependency graph and executed in exactly that order.
For example, a scenario test could look similar to the following:
@ScenarioTest
class WebSecurityScenarioTest {
@Step(next = "login")
void visitPageRequiringAuthorizationWhileNotLoggedIn() {
// attempt to visit page which requires that a user is logged in
// assert user is redirected to login page
}
@Step(next = "visitSecondPageRequiringAuthorizationWhileLoggedIn")
void login() {
// submit login form with valid credentials
// assert user is redirected back to previous page requiring authorization
}
@Step(next = "logout")
void visitSecondPageRequiringAuthorizationWhileLoggedIn() {
// visit another page which requires that a user is logged in
// assert user can access page
}
@Step(next = END)
void logout() {
// visit logout URL
// assert user has been logged out
}
}
open-test-alliance
module from JUnit Lambda repositoryThere are some object mocking frameworks available such as Mockito. Should JUnit care and give functionality to junit users to mock variables?
If it is yes, then I would like to see some form of automatic mocking mechanism for JPA meta-model like classes. But again the mocking should should be scoped for a single test. If im explaining the problem further, now when the hibernate meta models are generated they define variables as "SingularAttribute blah". So each of those fields will be null on Junit testing level. So the verification will not work properly unless you first mock a value and set them.
SUGGESTION:
Introduce annotation for all interfaces and classes supposed to be used from outside:
@PublicAPI(Stable)
@PublicAPI(Maintained)
@PublicAPI(Experimental)
@PublicAPI(Deprecated)
@PublicAPI(Internal)
@PublicAPI
annotation.Everything not annotated would be considered for internal use only.
Annotation would allow to produce API change reports for all releases.
I started to make it testable in #37.
<p>
on a separate line or at the beginning of the new paragraph?There is some good advice here:
Alternative solution for pre
/post
callbacks.
Having a tree of TestDescriptor
s that we use for the execution of tests, we need to guarantee that the descriptors are always resolved in the same tree structure, i.e. a MethodTestDescriptor
should always have its enclosing ClassTestDescriptor
as a parent, no matter how the method was found during the resolve phase (by unique id, by class reference, by method reference, etc.).
If we guarantee the tree structure to be equal we can then deduce the path
of a descriptor by the tree node leading to the descriptor. As for Java test the unique id of a deeper node in the tree is always prefixed by the unique id of its parent node, we could reduce the path
such that each level only adds the corresponding part.
We should investigate some time on this idea and clarify / introduce a path concept for the JUnit5TestEngine
.
In JUnit 4, we use a specific test runner that runs the test code on a dedicated UI thread, such as the AWT/Swing thread (or the JavaFX thread). This avoids cluttering your code with EventQueue.invokeAndWait
all over the place. As far as I understood, the current extensions proposal does not allow to do such thing.
The below runner simply wraps all calls to before, after, and test methods in an EventQueue.invokeAndWait
call. Tests simply have to declare @RunWith(AWTThreadRunner.class)
to get this behavior.
public class AWTThreadRunner extends BlockJUnit4ClassRunner {
public AWTThreadRunner(Class<?> klass) throws InitializationError {
super(klass);
}
@Override
protected Statement classBlock(RunNotifier notifier) {
return onAWTThreadStatement(super.classBlock(notifier));
}
@Override
protected Statement methodBlock(FrameworkMethod method) {
return onAWTThreadStatement(super.methodBlock(method));
}
private static Statement onAWTThreadStatement(final Statement aStatement) {
return new Statement() {
@Override
public void evaluate() throws Throwable {
EventQueue.invokeAndWait(new Runnable() {
public void run() {
aStatement.evaluate();
// left out exception handling code improve readability
}
});
}
};
}
}
Note: the actual implementation is a bit more complex in that it does proper exception handling and refuses JUnit's time-outs as that doesn't work together with running on a dedicated thread.
@TestTemplate
and accompanying extension point.@InvocationIndex
annotation or TestTemplateInfo
type for injecting the current invocation index, etc. into @Test
, @BeforeEach
, and @AfterEach
methods.
junit-jupiter-params
module
@since
tags@TestTemplate
and accompanying extension point documentation
module and include in the User Guide.junit-jupiter-params
module in the Dependency Metadata section.junit-jupiter-params
module in the Dependency Diagram.documentation
module and include in the User Guide.At the time of writing, the https://github.com/junit-team/junit-lambda/wiki/Prototype-Test-Decorators page does not have all that much detail, so this might already be considered.
It would be good if the library shipped with ready-to-use simple parameter annotations that could be used inside a single test instance. It would in many ways look like JUnit 4's @Parameterized
Runner, but for use inside a single class. When running the test, each @Parameterized
method would appear many times, once for each data set.
class MyTest {
// A simple test with a specified data set
@Test @Parameterized("mod10")
void testMod10(String value) { ... }
@Parameterized("mod10")
Collection<Object[]> getMod10Values { ... }
// Also including a method naming convention
@Test @Parameterized("mod11", name="{0}")
void testMod11(String value) { ... }
@Parameterized("mod11")
Collection<Object[]> getMod11Values { ... }
// In the absense of a @Parameterized data provider, use reflection
@Test @Parameterized("getMod12Values", name="{0}")
void testMod12(String value) { ... }
Collection<Object[]> getMod12Values{ ... }
}
Related to #90.
Hello,
my use case is to execute the same testcase concurrently in different threads on the same test object to find race conditions. If I understand the api correctly this should be possible by using the annotation @testinstance PER_CLASS and the yet to be implemented but planned "Introduce support for registering parameterized tests #14". What is missing is an annotation for a custom scheduler which gets preferably the root EngineTestExecutionNode and is responsible for actually running the tests. Is something like this planned already?
Thank you very much
Thomas
It seems the InstancePostProcessor could be made in sync with all the Callbacks, and probably be named TestCallbacks, giving it both a pre and post method similar to the other Callbacks.
An example use case is measuring how long the execution of the test method took (without all before/after calls). We use this to measure the performance of our code, and fail the test if it runs significantly slower than earlier executions.
TestExecutionContext
is hierarchical, and currently there are "engine", "class", and "method" context types.
However, once context is created, there is no simple indication of which type it is.
I think it would be better that TestExecutionContext
would have some API that can easily distinguish context type.
Implementation may be:
TestDescriptor
which is passed at constructor of DescriptorBasedTestExecutionContext
, then user can do context.getDescriptor() instanceof ClassTestDescriptor
.getContextType()
method that returns simple enum such like:public enum TestExecutionContextType {
ENGINE, CLASS, METHOD
}
There is no standard for testing on the JVM: the only common building block we have is java.lang.AssertionError
.
AssertionError
is great for signaling that a test has failed, but it doesn't go far enough. Each testing framework is therefore forced to fill the gap with custom subclasses of AssertionError
or RuntimeException
to provide a richer feature set to end users. The downside is that each framework has its own set of custom errors and exceptions, and this makes it a challenge for frameworks to interoperate.
For example, JUnit has long supported the notion of a failed assumption via its AssumptionViolatedException
, but assertion frameworks like AssertJ cannot integrate that feature without a direct dependency on JUnit. Furthermore, the status quo makes the work of IDEs and build tools more difficult than it should be.
The only real solution to this problem is to create a foundation that we can all build on!
Based on recent discussions with IDE and build tool developers from Eclipse, Gradle, and IntelliJ, the JUnit Lambda team is working on a proposal for an open source project to provide a minimal common foundation for testing libraries on the JVM. The primary goal of the project is to enable testing frameworks like JUnit, TestNG, Spock, etc. and third-party assertion libraries like Hamcrest, AssertJ, etc. to use a common set of exceptions that IDEs and build tools can support in a consistent manner across all testing scenarios -- for example, for consistent handling of failed assertions and failed assumptions as well as visualization of test execution in IDEs and reports.
We have already begun with a small set of errors and exceptions that we consider to be common for all testing and assertion frameworks. In fact, we are already using these exceptions in the JUnit Lambda Prototype.
Please take a look at our current draft in the open-test-alliance project and let us know what you think.
The name Open Test Alliance is simply a working title for the project. As such, the name is up for discussion, and recommendations are welcome. We will of course need a name that maps to an available Internet domain name so that the code can live in an appropriate Java package.
The draft implementation is currently a part of the JUnit Lambda repository; however, as soon as additional parties join the collaboration, we will collectively work to find a new home for the project.
What types of errors and exceptions should such a project support?
What types of properties should such errors and exceptions have?
What should the project be named?
We've already reached out to and asked for feedback from the maintainers of the following projects.
TestReporter
junit5-api moduleMap<String, String>
with a custom class, e.g. ReportingEntry
ConsoleRunner
JUnit5EngineExecutionContext#getExecutionListener()
TestReporterParameterResolver
ExecutionListenerAdapter
reportingEntryPublished
method to the end of TestExecutionListener
, EngineExecutionListener
, and their implementations.junit-engine-api
. TestReporter uses Map (instead of ReportEntry)I think we need a naming convention for our test classes. At the moment we have test classes that end with Test
, Tests
, and TestCase
. Personally, I would prefer one of the first two variants.
In addition, we should agree on how to write assertions. At the moment we have tests that use JUnit 4's Assert
, some use our new Assertions
class, others use AssertJ's Assertions
. Since we plan to migrate our tests to use JUnit 5 at some point, I would propose that we do not use anything in JUnit 4, particularly not org.junit.Assert
. Instead, I think, we should use our own Assertions
were feasible and AssertJ, where we need its power.
We had a brief discussion on Skype regarding a mocking framework. I've added Mockito to the test classpath and used it in a couple of tests. Other tests use hand-written stubs or spys. I think we can go with Mockito where easier and write doubles ourselves when we think it clearer.
The handleException()
method in ReflectionUtils
currently extracts a target exception from an InvocationTargetException
, and if the target exception is a checked Exception
it will be wrapped in an IllegalStateException
with the message "Unhandled exception"
.
This behavior, however, is inappropriate for a testing framework that reflectively invokes test methods.
Specifically, any exception thrown by a test method must be reported in TestExecutionResult
. If such an exception is wrapped in an IllegalStateException
, that makes it very difficult to retrieve the original exception, since it is impossible to know that some IllegalStateExceptions
should be treated differently than others.
We do not want to have to reason about the contents of an IllegalStateException
. Instead, we need an explicit way to retrieve an exception thrown by a test method.
throws Throwable
to throws Exception
RuntimeException
.ReflectionUtils.handleException()
accordingly.ClassSpecification
elements
@Ignore
annotation@RunWith
annotationsuite
method@RunWith(JUnit5.class)
annotationAllTestsSpecification
MethodSpecification
ClassSpecification
of method's classRunner
ClassFilter
PackageSpecification
UniqueIdSpecification
InitializationErrors
for malformed tests (e.g. non-public test methods) correctly@BeforeClass
or @AfterClass
Descriptions
Description#fUniqueId
instead of displayName
TestSources
if possibleWrite at least one test to apply the gradle-plugin so that the most stupid mistakes don't break the build.
To be usable on the command-line, it would be nice if junit-console were a jar that includes all of its dependencies (fat-jar, uber-jar, ...).
Although it is typically discouraged, having a way to order tests is useful to some people. The JUnit 4 @FixMethodOrder
annotation specifically and deliberately makes it impossible to implement a simple ordering scheme, forcing users who want/need this to resort to method names like test245_ensureReasonableTestName()
.
MethodSorter
for @FixMethodOrder
@FixMethodOrder(ORDERED)
, and then @Order(integer)
to specify individual tests. Tests could be run from smallest number to highest, with non-annotated methods having a default of 0. Methods with the same order could be run in parallel; different order tiers could not.@Order
based ordering.@Test
, @TestFactory
and @TestTemplate
methods.@Nested
classes.junit.jupiter.execution.order.random.seed
constant and document it.@API
declarations for types with changed visibility, etc.In order to avoid code duplication and dependencies on third-party libraries such as Apache Commons, introduce a simple ToStringBuilder
that generates output similar to that which is currently generated by ConditionEvaluationResult#toString()
.
ExtensionContext
currently has a displayName
property; however, this is limited in terms of robust identification.
uniqueId
property in ExtensionContext
name
property in ExtensionContext
Related to #58.
See https://github.com/junit-team/junit.contrib/tree/master/assumes
I hope the new JUnit add this in the core. Or, at least makes it easy to built it as an extension.
TestSpecificationElement
(or better name) as marker interface, and make the collection of elements accessible to engines, as a group or by specific typejunit-testplan-dsl
filterWith()
from Predicate<TestDescriptor>
to Predicate<TestIdentfier>
TestPlanSpecification
must implement Serializable
TestPlanSpecification
must be immutableA declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.