retest / recheck Goto Github PK

#154 accidentially removed the JUnit 4 dependency. As a result, hundreds of tests haven't been executed anymore, which resulted in a massive coverage drop from ~55% (PR before: https://github.com/retest/recheck/runs/79978478) to ~37% (PR after: https://github.com/retest/recheck/runs/84153719).

Our SonarCloud Quality Gate should cause an error when the overall coverage is less than 40%, but it didn't. We have to further investigate this to avoid similar issues in the future. (For testing, one can remove the JUnit 4 dependency again and see what SonarCloud does.)

Before submission, please check that ...

• this is related specifically to recheck and not its extension (e.g. recheck-web).
• the documentation does not mention a similar solution (e.g. possible configurations).
• there are no open or closed issues that are related to this idea.
• this, to my best knowledge, will not break something else.

Problem

When introducing changes to the default project files (all files within src/main/resources which are placed into .retest), the user must either delete the respective file (and potentially loose any custom modifications) or adapt the changes themselves (and we therefore must provide concise steps to replicate the changes made [e.g point to a commit]).

However, if the user did not touch these files (i.e. their content is equal to the default project files), it is save to assume that they can be replaced directly. This would remove the need for the user to replace the file themselves if they never edited these.

Solution

Terminology:
1. The old file refers to the default file before the change.
2. The new file refers to the default file after the change.
3. The project file refers to the file within the project directory .retest.

Thus it would be beneficial to have some kind of merging strategy for the project files.

• Currently, the new file is only replaced if the corresponding project file does not exist.
• Checksum: Save the checksum of the old file, compare it with the corresponding project file and replace with the new file if equal.

Following things should be respected:

1. the implementation should remain computationally cheap as this might get called a few times during a test.
2. any files provided from the resources folder will be accessible in standard folder form during development, but will be packaged into a jar for a production release.

Implementation Outline

The method ProjectConfiguration#ensureProjectConfigurationInitialized checks if the project has been setup before. All checks are performed here and may copy any file specified from src/main/resources to the project directory.

Additional Context

This might also be useful when considering #297 and we decide to save the provided filters somewhere on the file system.

Instead of ignoring current date or build number, have a rule that specifies e.g. currentDate or matches(\d\.\d\.\d\d). This is better than ignoring...

Question: Could be achieved with regular asserts also. Does it make sense, to add that functionality to recheck?

Describe the Bug

IllegalArgumentException: Unnamed path parameter cannot be null (path parameter at index 0 is null) error thrown during test execution.

Here's the relevant stack trace:

java.lang.IllegalArgumentException: Unnamed path parameter cannot be null (path parameter at index 0 is null)

	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)

	at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:72)

	at org.codehaus.groovy.reflection.CachedConstructor.doConstructorInvoke(CachedConstructor.java:59)

	at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrap.callConstructor(ConstructorSite.java:84)

	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:59)

	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:263)

	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:277)

	at io.restassured.internal.RequestSpecificationImpl.applyPathParamsAndSendRequest(RequestSpecificationImpl.groovy:1612)

	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

	at java.base/java.lang.reflect.Method.invoke(Method.java:566)

	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)

	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)

	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1262)

	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1029)

	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:816)

	at groovy.lang.GroovyObject.invokeMethod(GroovyObject.java:39)

	at org.codehaus.groovy.runtime.callsite.PogoInterceptableSite.call(PogoInterceptableSite.java:45)

	at org.codehaus.groovy.runtime.callsite.PogoInterceptableSite.callCurrent(PogoInterceptableSite.java:55)

	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:203)

	at io.restassured.internal.RequestSpecificationImpl.applyPathParamsAndSendRequest(RequestSpecificationImpl.groovy:1661)

	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

	at java.base/java.lang.reflect.Method.invoke(Method.java:566)

	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)

	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)

	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1262)

	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1029)

	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:816)

	at groovy.lang.GroovyObject.invokeMethod(GroovyObject.java:39)

	at org.codehaus.groovy.runtime.callsite.PogoInterceptableSite.call(PogoInterceptableSite.java:45)

	at org.codehaus.groovy.runtime.callsite.PogoInterceptableSite.callCurrent(PogoInterceptableSite.java:55)

	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:51)

	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:171)

	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:203)

	at io.restassured.internal.RequestSpecificationImpl.post(RequestSpecificationImpl.groovy:175)

	at io.restassured.internal.RequestSpecificationImpl.post(RequestSpecificationImpl.groovy)

How to Reproduce?

This seemed to be a random failure. The weirdest part is that this was in an API only test so no browsers are involved.

Setup

• recheck: 4.3.0
• Selenium version: 3.141.59
• Testing framework: TestNG 7.0.0
• OS: Ubuntu
• Java version: Java 8

It's since kryo 5 it's for good reasons the default to register classes, for more details see
EsotericSoftware/kryo#398

In KryoPersistence.java#81 remove the line and add registration for all used classes.

Currently, we produce the following output:

java.lang.AssertionError: 'com.mycompany.MyFirstTest':
No recheck file found. First time test was run? Created recheck file now, don't forget to commit...
        at de.retest.recheck.RecheckImpl.capTest(RecheckImpl.java:137)
        at com.mycompany.MyFirstTest.google(MyFirstTest.java:30)
        at ... some more stack trace ...

This should indicate where the recheck files where created (i.e. path to C:/...myproject/src/test/resources).

If someone accidentially uses cap() instead of capTest() within a test, like so:

@Test
void someTest() throws Exception {
   re.startTest();

   // ...

   re.cap(); // instead of re.capTest()
}

Then a NullPointerException is thrown. Here, a better error message would be very helpful.

Taken from AttributesDifferenceFinder:

private AttributeDifference differenceFor( final IdentifyingAttributes identAttributes, final Serializable expected,
		final Serializable actual, final String key ) {
	if ( expected == null ) {
		return null;
	}
	if ( RecheckIgnore.getInstance().shouldIgnoreAttribute( identAttributes, key ) ) {
		return null;
	}
	// FIXME If default is persisted, then this is true and the default isn't removed.
	if ( Objects.equals( expected, actual ) ) {
		return null;
	}
	// FIXME If default is persisted, then this should yield a AttributeDifference that removes it.
	if ( defaultValueFinder.isDefaultValue( identAttributes, key, actual ) ) {
		return null;
	}
	if ( key.equals( Attributes.SCREENSHOT ) ) {
		final Screenshot expectedScreenshot = (Screenshot) expected;
		final Screenshot actualScreenshot = (Screenshot) actual;
		final ImageDifference imgDiff = imgDiffCalc.compare( expectedScreenshot, actualScreenshot );
		if ( imgDiff.isEqual() ) {
			return null;
		} else {
			return new ScreenshotAttributeDifference( expectedScreenshot, actualScreenshot,
					imgDiff.getStrategyName(), imgDiff.getMatch() );
		}
	}
	return new AttributeDifference( key, expected, actual );
}

According to our docs:

Elements are identified by one special attribute $key, so called identifying attributes. If you directly match an element with no additional expressions (e.g. attributes), all of its child elements are ignored too.

# Match the element where its attribute $key fully matches the $value
matcher: $key=$value

This works fine when using an XPath, for example:

matcher: xpath=html[1]/[body[1]/div[1]

But when using a different matcher (i.e. retestid, id, class or type), child elements aren't ignored anymore. To verify this issue:

1. Pick an arbitrary page.
2. Create a GM.
3. Change something (test fails).
4. Ignore the parent of the changed element via XPath.
5. Run again (test passes).
6. Change the ignore to something else as mentioned above.
7. Run again (test fails).

I assume this is because in ElementXPathMatcher we do:

@Override
public boolean test( final Element element ) {
	return element.getIdentifyingAttributes().getPath().startsWith( xpath );
}

This includes child elements because they share the same XPath prefix. However, this isn't the case for retestid, id, class or type. Here, we don't check if a parent has been ignored.

A possible solution: Let all matchers also be a ElementXPathMatcher (e.g. via inheritance). Whenever such a matcher actually matches—for instance, ElementIdMatcher finds an element with an ID that should be ignored—then we store the XPath too and incorporate this in subsequent invocations. Note that:

1. A matcher might matches multiple elements, consequently, we would have to store and compare multiple XPaths.
2. This approach requires ordering. Assume we want to ignore id=foo, where this element's XPath equals html[1]/[body[1]/div[1]/div[1]. I don't believe this is the case, but if e.g. we check one of the child elements before we store the element-to-be-ignored's XPath, it won't be ignored.

Other ideas?

Before submission, please check that ...

• this is related specifically to recheck and not its extension (e.g. recheck-web).
• the documentation does not mention a fix (e.g. wrong API usage, ...).
• the bug is not related to a customization (e.g. custom implementations).
• there are no open or closed issues that are related to this problem.

Describe the Bug

While I git clone the files to my folder, and open Eclipse then import the maven project, I run the project with mvn install
Then the project start installing and compiling the dependencies. I'd add the recheck dependencies to pom.xml:

However, several hours passed, and the installation is not terminated yet:

How to Reproduce?

Steps to reproduce the behavior:

1. git clone the whole recheck to local folder
2. Open the pom.xml and add recheck dependencies to the dependency part
3. Open Eclipse, import the maven project, then click Run as -> maven install

Setup

• recheck: 1.11.1(latest)
• Testing framework:
• OS: Windows 10
• Java version: Java 15

Additional Context

Since, apparently, we extend the RecheckOptionsBuilder by RecheckWebOptionsBuilder we need to make sure, that it can handle such. With the current implementation (as of v1.5.0) the usage of the builder is really restricted, such that one first has to set all specific Recheck*Options and then proceed to the common RecheckOptions.

Example from retest/recheck-web#349:

RecheckWebOptions.builder() //
			.checkNamingStrategy( new CustomCheckNamingStrategy()  ) // returns RecheckWebOptionsBuilder
			.fileNamerStrategy( new CustomNamingStrategy() ) // returns RecheckOptionsBuilder
			.build()

That means that the builder must use generics such that it returns a self and construct the Recheck*Options accordingly.

Generally that means to have something like an abstract builder:

public abstract class AbstractRecheckOptionsBuilder<SELF extends AbstractRecheckOptionsBuilder<SELF, OPTIONS>, OPTIONS extends RecheckOptions> {

	public SELF setOptionX( /* ... */ ) {
		// do stuff
		return self();
	}

	protected abstract SELF self();

	protected abstract OPTIONS construct( RecheckOptions options );

	public OPTIONS build() {
		return construct( new RecheckOptions( /* initialize */ ) );
	}
}

For RecheckOptions:

public final class RecheckOptionsBuilder extends AbstractRecheckOptionsBuilder<RecheckOptionsBuilder, RecheckOptions> {

	@Override
	protected RecheckOptionsBuilder self() {
		return this;
	}

	@Override
	protected RecheckOptions construct( final RecheckOptions options ) {
		return options;
	}
}

For RecheckWebOptions:

public final class RecheckWebOptionsBuilder extends AbstractRecheckOptionsBuilder<RecheckWebOptionsBuilder, RecheckWebOptions> {

	@Override
	protected RecheckWebOptionsBuilder self() {
		return this;
	}

	@Override
	protected RecheckWebOptions construct( final RecheckOptions options ) {
		return new RecheckWebOptions( options, /* initialize */ );
	}
}

As of v0.4.3 (which is essentially this commit), replay results are not properly persisted as they miss the suffix dot (e.g. replayresult instead of replay.result).

Maybe caused by this TODO from RecheckImpl (closed source):

public File getResultFile() {
	return fileNamerStrategy.createFileNamer( execSuiteName )
	// TODO Remove after next release of recheck
	.getResultFile( ReportFileUtils.REPORT_FILE_EXTENSION.substring( 1 ) );
}

We currently have implementation-specific filters (e.g. for recheck-web) in recheck (under src/main/resources/filter/), which is not optimal. Implementations should provide their own filters, so that recheck isn't bloated with filter files over time. Moreover, adapted filters shouldn't require PRs and releases in/of recheck, but of the corresponding implementation.

@roesslerj @diba1013 I know you had this discussion before and it wasn't straightforward to implement. Could you provide a quick summary, such that we can discuss possible alternative solutions?

Use JUnit 5 and JUnit 4 to get rid of necessary explicit calls to capTest() and cap().

Each time RecheckImpl#cap() is being invoked, we trigger CloudPersistence#save(...) and upload the aggregated tests.report to rehub. Suppose there are 10 tests classes, each consisting of 3 test methods. If cap() is called @AfterEach as usual, then we get 10 * 3 = 30 reports (all named tests.report) uploaded to rehub, although only the last one is interesting.

This is because we are not aware of when all tests have been finished. A simple solution could be to create some sort of ID (which is stable as long as the JVM lives, such as a static UUID) when the first test has been started and then send this as meta data.

Besides "golden master"—the term we agreed on—we use different words to describe the same thing. For instance:

• RecheckStateFileProviderImpl: "state file"
• NoRecheckFileActionReplayResult: "recheck file"
• …

I think it would be helpful for both the user and us to unify the terminology and consistently use the term "golden master".

Suppose the following recheck-web test:

@ParameterizedTest
@MethodSource( "com.github.beatngu13.visualtesting.util.DriverFactory#getAll" )
void test(WebDriver driver) throws Exception {
	re.startTest();
	driver.get("https://example.com/");
	re.check(driver, "example");
	re.capTest();
}

If the user doesn't provide a test name via startTest(String) and uses the default startTest(), we simply overwrite the previous GM as all parameterized test cases have the same method name.

We should try to recognize this and then at least append a suffix.

Hi,

I'm trying to use recheck within a groovy script in SoapUI, in order to integrate it in my selenium suite. I get the following error:

ERROR:java.lang.IllegalStateException: Couldn't identify test class in call stack. 
This is needed to dynamically name the Golden Master files. 
 Please instantiate RecheckImpl with RecheckOptions, and provide a different FileNamerStrategy.

by calling either new RecheckImpl() as well as RecheckWebImpl(). Can I pass a parameter of some sort?

Thanks

Quite often, there are more unique differences than "normal" differences.

These normal differences are counted via ActionReplayResult#getElementDifferences(), whereas the unique differences are counted via ActionReplayResult#getUniqueDifferences(). As can been seen, getElementDifferences() simply uses StateDifferences#getNonEmptyDifferences(), but getUniqueDifferences() calls getElementDifferences() and "unwraps" the contained differences (which can be different types).

If we assume that all possible differences of an element are wrapped inside an ElementDifference, then getUniqueDifferences() contains at least the number of differences as getElementDifferences(). But if a single element has multiple differences, e.g., two attribute changes, then getUniqueDifferences() leads to more differences than getElementDifferences().

This is confusing because one expects "unique" to be less than "normal". So, we should probably adapt the naming scheme. For instance:

• getElementDifferences() => getElementDifferences() since it returns the list of differences of all elements that have changed.
• getUniqueDifferences() => getAllDifferences() since it unwraps each ElementDifference into one or more AttributesDifference, IdentifyingAttributesDifference, DurationDifference or InsertedDeletedElementDifference.

From AttributesDifferenceFinder:

private AttributeDifference differenceFor( final IdentifyingAttributes identAttributes, final Serializable expected,
		final Serializable actual, final String key ) {
	if ( GloballyIgnoredAttributes.getInstance().shouldIgnoreAttribute( key ) ) {
		return null;
	}
	if ( Objects.equals( expected, actual ) ) {
		return null;
	}
	if ( defaultValueFinder.isDefaultValue( identAttributes, key, actual ) ) {
		return null;
	}
	if ( key.equals( Attributes.SCREENSHOT ) ) {
		final Screenshot expectedScreenshot = (Screenshot) expected;
		final Screenshot actualScreenshot = (Screenshot) actual;
		final ImageDifference imgDiff = imgDiffCalc.compare( expectedScreenshot, actualScreenshot );
		if ( imgDiff.isEqual() ) {
			return null;
		} else {
			return new ScreenshotAttributeDifference( expectedScreenshot, actualScreenshot,
					imgDiff.getStrategyName(), imgDiff.getMatch() );
		}
	}
	return new AttributeDifference( key, expected, actual );
}

Suppose an attribute has been deleted. As an example, we use the following parameters:

• identAttributes: not important for this example
• key: "someKey"
• expected: "someValue"
• actual: null (since it has been deleted)

Following the program flow:

• GloballyIgnoredAttributes.getInstance().shouldIgnoreAttribute( key ) evalutes to false, as we assume "someKey" is not a globally ignored attribute.
• Objects.equals( expected, actual ) evaluates to false, as "someValue" is not equal to null.

Now defaultValueFinder.isDefaultValue( identAttributes, key, actual ) is the problem. If "someKey" is normally missing (e.g. default is null) or if the DefaultValueFinder interprets null as default like recheck-web's implementation does (see DefaultWebValueFinder below), then the deleted attribute isn't shown as a difference.

Is this the desired behavior?

// From recheck-web's DefaultWebValueFinder.
@Override
public boolean isDefaultValue( final IdentifyingAttributes identifyingAttributes, final String attributeKey,
		final Serializable attributeValue ) {
	final String attributeValueString = attributeValue != null ? attributeValue.toString() : null;
	final String defaultValueString = getDefaultValue( identifyingAttributes, attributeKey );
	if ( defaultValueString != null ) {
		return defaultValueString.equalsIgnoreCase( attributeValueString );
	}
	if ( attributeValueString == null || attributeValueString.trim().isEmpty() ) {
		return true;
	}
	if ( commonDefaults.contains( attributeValueString ) ) {
		return true;
	}
	return false;
}

We currently use Lombok to create a builder for our RecheckOptions:

@SuperBuilder
@Getter
public class RecheckOptions {

	@Builder.Default
	private final FileNamerStrategy fileNamerStrategy = new MavenConformFileNamerStrategy();

	@Builder.Default
	private final String suiteName = new MavenConformFileNamerStrategy().getTestClassName();

	@Builder.Default
	private final boolean rehubEnabled = false;

}

This is all fine until a user has to provide a custom FileNamerStrategy if, for instance, we cannot determine the test class/method name in the call stack (see e.g. #301 and retest/recheck-web#268).

The key point here is that if the user passes a FileNamerStrategy, we should reuse it in suiteName. Otherwise, one has to do:

RecheckOptions.builder()
		.fileNamerStrategy( fileNamerStrategy )
		.suiteName( fileNamerStrategy.getTestClassName() )
		.build()

Which is somewhat redudant. I would expect the following default(s):

@Builder.Default
private final FileNamerStrategy fileNamerStrategy = new MavenConformFileNamerStrategy();

@Builder.Default
private final String suiteName = fileNamerStrategy.getTestClassName();

But that's not possible with Lombok's builder.

Consequently, I would suggest to delombok RecheckOptions and reimplement a simplified version without Lombok, which addresses this issue but is still compatible and extensible.

The report generated with Recheck using the following pom:
<dependency> <groupId>de.retest</groupId> <artifactId>recheck-web</artifactId> <version>1.1.0</version> </dependency>
Which pulls recheck 1.1.1.

The report generated has reTestVersion="1.2.0" making the CLI fail with:
The report was created with another, incompatible recheck version. Please, use the same recheck version to load a report with which it was generated.

when the UI changes happen i am expecting to see the retestId in the log, but it only shows the expected and actual as bellow:
_

> 4 check(s) in 'de.retest.quickstarts.LoginTest' found the following difference(s):
> Test 'login' has 6 difference(s) in 4 state(s):
> 00 resulted in:
> 	Metadata Differences:
> 	  Please note that these differences do not affect the result and are not included in the difference count.
> 		url:
> 		  expected="<url expected>",
> 		    actual="<url actual>"
input (username) at 'html[1]/body[1]/form[1]/div[1]/input[1]':
		name:
		  expected="username",
		    actual="user", breaks="LoginTest.java:84"

_

Consider the following change:

Adapted HTML (removes id attribute completely): https://github.com/retest/recheck-web/pull/297/files#diff-b980e03c12cee358e1072d97aedd8dedR165

Our GM after apply via review: https://github.com/retest/recheck-web/pull/297/files#diff-b980e03c12cee358e1072d97aedd8dedR165

As can be seen, the XML element for the id attribute is now empty, but it is still part of the XML—although it doesn't exist anymore.

We should ensure that stuff which has been deleted gets also deleted from our GMs, especially for space reasons. While doing so, we should also ensure that this is the default in similar situations, e.g., when an element, identifying attribute, etc. gets deleted.

I'm using JDK 11, so I assume this should need to be addressing soon.

Instead of having typed IdentifyingAttributes and untyped Attributes, we want only typed Attributes, some of which are identifying.

This resolves some other issues, such as special treatment for screenshot, text, backgroundcolor, font-family etc.

See:

https://travis-ci.com/retest/recheck/builds/109360311#L1939

I assume this is related to:

#165

While source compatability should be Java 8+, we should also add support running on Java 11 (the current LTS). Running mvn verify in recheck-web yields:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-javadoc-plugin:3.0.1:jar (bundle-javadoc) on project recheck-web: MavenReportException: Error while generating Javadoc: 
[ERROR] Exit code: 1 - warning: unknown enum constant XmlAccessType.FIELD
[ERROR]   reason: class file for javax.xml.bind.annotation.XmlAccessType not found

This is because JAXB has been removed as of Java 9 (I will soon provide a fix for that).

However, as already suggested by @modulo11, I think we should compile with Java 8 and run the tests with Java 11 for proper feedback.

Other suggestions?

Before submission, please check that ...

• this is related specifically to recheck and not its extension (e.g. recheck-web).
• the documentation does not mention a fix (e.g. wrong API usage, ...).
• the bug is not related to a customization (e.g. custom implementations).
• there are no open or closed issues that are related to this problem.

Describe the Bug

When running the integration tests for recheck-web, an assertion in recheck is triggered. (The problem is visible in the recheck-web integration tests, but stems from a problem in recheck)

[ERROR]   RecheckRemoteWebElementIT.complete_html_should_be_same_as_driver:38 bestMatch and previousMatch have a match of 100%? At least paths should differ! html[1]/body[1]/form[3] # br # 1 == html[1]/body[1]/form[3] # br # 1
[ERROR]   RecheckRemoteWebElementIT.complete_html_should_be_same_as_web_element:49 bestMatch and previousMatch have a match of 100%? At least paths should differ! html[1]/body[1]/form[3] # br # 1 == html[1]/body[1]/form[3] # br # 1

This has not been present on earlier runs on the CI system when using Ubuntu 14.04.

How to Reproduce?

Check out retest-web, run mvn clean verify on Fedora 31 or Ubuntu 18.04.

Setup

• recheck: 1.8.0
• recheck-web: 8ef03a8eb49a136623f0d4608284e1eb28806e7a
• mvn 3.6.1 (Fedora 31),
• OS: Fedora 31 / Ubuntu 18.04 (Bionic Beaver)
• Java version: Fedora 31: 13.0.1, Ubuntu: 11.0.2 (mvn compile) & 13.0.1 (rest of the steps)

Steps to resolving this issue

This involves work across both projects.

• Reduce form-page.html to the minimal HTML needed to reproduce this issue (recheck-web)
• Create a test that can reproduce this issue (recheck)
• Fix underlying problem such that the test will not fail anymore (recheck)
• Removed @Disabled from the failing tests in RecheckRemoteWebElementIT (recheck-web )

Illegal reflective access by de.retest.recheck.ui.image.ImageUtils (file:/Users/javierortiz/.m2/repository/de/retest/recheck/1.3.0/recheck-1.3.0.jar) to field sun.awt.CGraphicsDevice.scale

Before submission, please check that ...

• this is related specifically to recheck and not its extension (e.g. recheck-web).
• the documentation does not mention a fix (e.g. wrong API usage, ...).
• the bug is not related to a customization (e.g. custom implementations).
• there are no open or closed issues that are related to this problem.

Describe the Bug

How to Reproduce?

Steps to reproduce the behavior:

1. ...
2. ...
3. ...

Setup

• recheck: [e.g. 1.6.0]
• Testing framework: [e.g. JUnit5 5.5.2, TestNG 7.0.0]
• OS: [e.g. MacOS 14, Windows 10]
• Java version: [e.g. OpenJDK 11]

Additional Context

When we try to assign a step name to a GM or vice versa, three situations can occur:

1. We cannot find a GM for the given step name, so we fail the test case. This is the default behavior whenever a test is being executed for the first time.
2. The step name corresponds to a golden master, so we can proceed with the comparison of expected and actual.
3. We have a golden master but without a step name.

AFAIK we currently don't handle no. 3? This can happen if a check is removed or a step renamed. The developer has to decide whether the checkpoint is missing, the GM can be deleted, or something else.

In any case, we should probably track the executed tests and then compare them agains the contents of the recheck folder? Afterwards, we could at least print out a warning when there is a GM but no corresponding has been executed.

Testing recheck web 1.5.1 with Review 1.5.0 (since there seems not to be a 1.5.1) I get some differences on the report:

The changes are applied but no actual changes make it to the recheck.ignore file.

I wouldn't be surprised it's related to the check/test name containing spaces.