When a snapshot assertion that uses custom comparison rules fails, it will still output a unified diff of all the text changes. This diff doesn't know about the custom rules, so all the differences will be printed, including those, that did not lead to the test failure (or even no differences at all if the custom rule involved a regex).

This problem is probably impossible to solve, as custom rules depend on the structured format being used while the unified diff is only calculated on text basis. There is no way for the diff engine to know which changes where excluded from the structural comparison.

As a first step, failure message could show a warning hint in case that custom rules were used during comparison.

To solve this, we need to propagate the information about rule mismatches up to the core where we currently assemble the assertion failure. This is not that easy because currently custom rules only affect the outcome (true/false) of snapshot comparison.

We do throw a RuntimeException in the DSL when the passed in object is null. We should consider accpeting a null value here and throw a proper AssertionError when executing the test. The error should state that an unexpected null value was encoutered and it could also contain the actual expected result (if the snapshot file already exists)

There are currently a few places where the snapshot engine implementation is tightly coupled to JUnit. I don't think this is necessary and we could try to separate the concerns here and split into

• pure reusable snapshot engine w/o JUnit dependency
• JUnit5 integration of the engine

This would also allow to integrate with further testing frameworks or to use the engine standalone

Turns out that, as by the current implementation, the annotation isn't even made to be working on test methods. Making that work severely interfers with orphan detection. I guess its best to only allow the annotation on test class level.

If you want to change the snapshot directory within a single test method, you could use the DSL via snapshot.in(Path.of(...))...

Thus wie should update the documentation accordingly and consequently remove ElementType.METHOD from the annotation's @Target

We could use ANSI color codes to improve dispaly of the unified diff in console windows

When snapshots get big, its hard to spot the differences in the unified diff that we print. We should maybe reduce the diff to only print the lines in which something changed.
TextSnapshot class should get an option to configure the number of context lines that are to be printed around a certain change

Snapshot naming does not work with @ParameterizedTest because the created SnapshotTest instance derives the snapshot name only from the name of the test method.

You can work around this by specifying an explicit snapshot name using snapshot.named("name") (see #6)

Currently we do an extra assertThat(...) to create a proper diff-able failure message. However, this discards the stack trace of the original assertion error.
Not quiete sure how much of a problem this is actually

This is a follow up issue on #15 and #19. We can not do this changes right away because they would be breaking.

• In version 1.7.0, the type de.skuzzle.test.snapshots.EnableSnapshotTests will be deprecated

• In version 2.0.0 we can remove the deprecated type

• Delete deprecated @EnableSnapshotTests annotation

• Adjust restrict-import rules in core pom accordingly

• Make JUnit5SnapshotExtension package private

• Move JUnit5 extension into ...-junit5 module

They should have common naming scheme, for example

XmlSnapshot.xml().inferJaxbContext(true)...
JsonSnapshot.json().withObjectMapper(...)...

We can use the new header information to determine orphaned snapshots without relying on folder structure or snapshot filenames (See #6, #7)

Also, orphaned snapshots are not detected when a test class was renamed

JAXB seems to not handle the 'new' java date classes like LocalDate, LocalDateTime without extra configuration. Normally, serialization details shall be outsourced to our clients. But we do provide the ObjectMapper for JSON serialization with sensible defaults too.
Therefore I think we should do the same with the defaults for JAXB serialization.

See also: https://stackoverflow.com/questions/36156741/marshalling-localdate-using-jaxb

Note: Changing the serialization behavior like this might break existing tests that currently rely on the 'wrong' serialization. This might be an issue for a major version bump then

The usage of Path.of(...) in DirectoryResolver might lead to issues when the build isn't run directly from its directory or in a multi module project.

See: https://sormuras.github.io/blog/2018-09-11-testing-in-the-modular-world.html and https://github.com/sormuras/junit-platform-maven-plugin

Currently our framework is deeply integrated into the whole test execution life cycle. We rely on several information that are provided by JUnit:

• We use the test method for determining snapshot names from SnapshotNaming strategy
• We use the number of executed assertions within the same test method for automatic snapshot naming
• We use the test method and the test class to check for presence of @ForceUpdateSnapshots and @DeleteOrphanedSnapshots annotations
• We use the test class to determine the default snapshot directory.
• We collect information about failed and skipped tests to improve orphan detection

Those are the main reasons why we currently can not offer a simple static assertThat(actual).matchesSnapshotText() API.

We could try to find heuristics to determine the currently required information.

A simple soultion would be to try to determine the surrounding test class by iterating the current call stack and to rely on manual snapshot naming. That would allow to write assertions like this:

SnapshotAssert
    .assertThat(actual)
    .named("my-snapshot")
    .as(JsonSnapshot.json)
    .matchesSnapshotStrucure();

The ComparisonRuleBuilder API currently only allows to customize comparisons based on paths. It does not allow to customize overall aspects like for example ordering strictness

Up until and including version 1.3.0, whitespaces are being ignored when using matchesSnapshotText(). There is also no option to change this from a user perspective.

We have to figure out what would be the desired default behavior and whether there is the need to have this configurable

We should add a system property or something that can be temporarily set from the command line (e.g. maven build) to force update of all snapshots

By now, we rely on the snapshot's file name to encode, for example, the Class and the test method. These information are needed to find orphaned snapshot files. This does't work reliably anymore when using explicit snapshot names (See #6)

We should write a simple header to every snapshot file to add these information

Our rendered diffs contain line numbers that are based on the raw test output. Snapshot files however contain some extra header lines at the beginning of the file. Thus, when looking up changed lines in the actual snapshot file, the numbers reported in the diff are off by the number of header lines in the snapshot file.

I think we should adjust the reported line numbers by the amount of header lines so that it is easy to find the respective lines in the persisted snapshot file

Soft assertions can as well be simulated by AssertJ's assertSoftly. Also, exception messages of failed snapshot tests can get quite large. It might not even be desired to collect and merge multiple such messages into one big assertion failure

Currently one of the main types, the DSL entrypoint Snapshot, is an inner interface of the outer SnapshotDSL type. I think it would be better and clearer for users if it were a top level type.

currently it is not possible to use XML structure compare for strings because they will be always run through a marshaller. If we receive a String as actual input to snapshot.assertThat(...) then we should just skip the serialization part and pass the string through

There are currently two known places where we depend on the behavior of actual test framework being used:

• AssumptionExceptionDetector tries to detect the propert assumption exception
• StaticOrphanedSnapshotDetector tries to identifiy whether a method contains a snapshot assertionn which only works for JUnit5 tests with Snapshot parameter

Since version 1.8.0 we're shipping test-framework specific artifacts. So it would just be natural to move this logic into the respective artifacts to have the core fully framework-agnostic.

Updating snapshots with the latest actual result might be cumbersome. You either need to rerun the whole test suite again with @ForceUpdateSnapshots and then another time without the annotation. Or you need to manually update the snapshot files.
Since 1.7.0 you could enable @SnapshotTestOptions(alwaysPersistActualResult = true). This will always write the latest actual result as a sibling file next to the actual snapshot. Then you could just delete the snapshot and rename the created .snapshot_actual file to .snapshot. This would update the snapshot to the latest actual result without having to rerun the test suite.

If snapshot files are already checked into the SCM, we could consider the following auto update workflow:

1. If we find a difference to the current snapshot AND the snapshot is checked into SCM AND the snapshot is not marked modified (in terms of git status), then we can auto-update the persisted snapshot with the current actual result. The snapshot file will subsequently be marked modified in git status and the user needs to confirm these modifications by committing the file.
2. If a snapshot file is marked modified by the SCM we can assume that it has been auto-updated by a previous test run. In that case we always fail the test. The user will need to commit the file in order to clear its status and to confirm the modifications to the snapshot file.

As it seems the BOM artifact is not published properly because placeholders in dependency declarations are not being resolved. Strange enough I've seen gradle projects where importing the BOM seems to work anyway, but importing the BOM in a maven project doesn't seem to work at all.

We have several places where we deal with linebreaks:

• Text-diff creation
• Reading snapshot files
• Writing snapshot files
• Other assertion exception message text
• Our default snapshot serializers come with their own rules

There is currently no defined unique behavior about which kind of linebreaks we are using. The matter is complicated by the fact that snapshot files are supposed to be checked into the SCM and Git has its very own idea of dealing with line breaks among different seettings and operation systems.

• Naturally, snapshot files are to be persisted with the linebreaks generated by the respective SnapshotSerializer
• Naturally, snapshot files need to be read without losing any line break information (<- impossible due to GIT's beahvior)
• Text diffs are always to be rendered with system line breaks. Line break changes are either highlighted separately at the beginning of the diff or ignored depending on diff settings
• Likewise, all exception message text should be rendered with system line breaks

If a snapshot assertion fails, we could always persist the new actual result in a sibling file next to the actual .snapshot. Maybe with .snapshot_actual extension.
That would allow to update the snapshot by just removing the old file and removing the _actual suffix which does not require to run the test again

The .snapshot_actual files should not be checked into any SCM to avoid unncessary noise in the Git status overview.

We should support HTML comparison. That would be a perfect fit for when building server-side rendered apps with for example Spring-Boot + thymeleaf.

XmlUnit seems to support HTML comparison when you pass a org.w3c.dom.Document. So we can implement a StructuralDataProvider that parses both the actual and snapshot String into such a Document and then pass it to XmlUnit. We can copy/reuse certain XmlUnit configuration API from our current XML package.

Here is a quick proof on concept using Jsoup:

        <dependency>
            <groupId>org.jsoup</groupId>
            <artifactId>jsoup</artifactId>
            <version>1.15.3</version>
        </dependency>

public class HtmlStructuralAssertions implements StructuralAssertions {

    @Override
    public void assertEquals(String storedSnapshot, String serializedActual) throws AssertionError, SnapshotException {
        final W3CDom w3cDom = new W3CDom();
        final Document parsedSnapshot = Jsoup.parse(storedSnapshot);
        final Document parsedActual = Jsoup.parse(serializedActual);

        XmlAssert.assertThat(w3cDom.fromJsoup(parsedSnapshot)).and(w3cDom.fromJsoup(parsedActual)).areIdentical();
    }
}

@EnableSnapshotTests
public class HtmlSnapshotTests {

    @Test
    void testCompareHtml(Snapshot snapshot) throws Exception {
        snapshot.assertThat("<html><body><h1>Test</h2></body></html>").as(Object::toString)
                .matchesAccordingTo(new HtmlStructuralAssertions());

    }
}

Currently, we just use the full class name as directory name. That leads to directories named like ParentClass$StaticInnterClass_snapshots. We should deconstruct the class name and create proper sub directories for the inner classes.

When doing this, we should also provide an automatic migration of existing snapshot files. Otherwise we might break existing tests and a users might need to inconveniently migrate existing snapshot files manually.

We currently use Objects.requireNonnull broadly on our code to perform argument validation. Unfortunately, this method throws an NPE instead of a more appropriate IllegalArgumentException. We should add our own utility methods for validating arguments

1. We should filter out StackTraceElements of our internal code (like AssertJ does)
2. We should always include the unified text diff
3. We could also include the location of the snaphot file

We should move away from javax. and use jarkata. instead

JSONAssert brings some nice options to customize how two objects are being compared. However, the API is a bit awkward and doesn't fit into our fluent/builder style DSL. As such, we should remove the public visibility of the JSONComparator class in JsonSnapshot and provide a nice fluent wrapper around it.

This issue needs to be resolved before we can go on with #17

In an effort to clean our public API from all JUnit dependencies, we need to deprecate the current de.skuzzle.test.snapshots.EnableSnapshotTests annotation.

• Mark current annotation as deprecated
• Move a copy into the respective junit5 package at de.skuzzle.test.snapshots.junit5
• Adjust implementation so that both annotations are supported

Snapshot directory that is derived from class name is always created, even if all snapshot tests use a custom directory

Random data within the snapshot is a huge problem and might need a lot of mocking or manual post processing of the actual test result in order to be able to reliably use snapshot assertions. This is kind of in the way of our design goal of "making it super simple to get a lot of assertions for free".

Few points for the discussion:

• At what level should normalization be applied? Normalize the actual test result before serializing, normalize the snapshot after normalization (string based), do not normalize the persisted data but normalize "on the fly" during comparison?
• Adding to that, should we provide reflection based normalizing for actual objects?
• Should we provide serialized structure (json/xml) based normalization for serialized string data (XPath, Json Path)?

Either way, I still think that normalizing the data is an anti-pattern. As stated in the readme, you should design your code in way that you can provide mocks returning deterministic instead of reandom data

When updating all snapshots, the framework should also check whether there are snapshot files that are no longer needed and remove them

We should add a new terminal operation to the snapshot DSL that should behave like the other terminal operations in terms of advancing the Snapshot instance's internal state. However it

• should not create an initial snapshot if one doesn't yet exists
• it should not perform any comparison if a snapshot already exists
• it should still contribute to orphan detection, so that an existing snapshot is not detected as orphan

In case there is a snapshot with multiple snapshot assertions, this would allow to disable for example only the first one, without disrupting the second one (when relying on automatic snapshot naming):

@Test
void sampleTest(Snapshot snapshot) throws Exception {
    snapshot.assertThat(actual1).asText().diabled();
    snapshot.assertThat(actual2).asText().matchesSnapshotText();
}

This could be another convenience feature to save users from accidently overriding snapshots.

The LocalResultCollector knows the results of all assertions of a single test method. Should be quite easy to add another artificial assertion in there which tests that no two results pertain to the same snapshot file

If no terminal operation is invoked on a Snapshot DSL call, then effectivly, not test is executed. As each Snapshot object is already registered with the JUnit execution context, we should be able to detect such incomplete calls easily. Detection might become a little bit complicated due to the fact that a Snapshot instance can be reused multiple times within the same test method if it uses multiple snapshot assertions

The framework should shomehow return the serialized snapshot so that the user has the possibility to further work with that data.

Should snapshot.assertThat(null)... be allowed? Or should the framework decline null actual values right away? Not sure about this one

We can consider throwing an assumption failure to mark the respective test as skipped in the IDE. But only if there is no other failing snapshot assertion within the same test method.

Orphan detection currently doesn't clean up optional context files like the raw snapshot or the actual result file

I don't like the name of the annotation that enables the snapshot testing capabilities, but I'm not quite sure what would be better.

• @EnableSnapshotAssertions
• @EnableSnapshotTests
• ...