Add a configurable tolerance. about shot HOT 15 CLOSED

There actually is a configurable tolerance now:

You can configure shot in build.gradle:

shot {
    tolerance = 0.1 // 0,1% tolerance
}

Like pedrovgs said: use with care, but I'd say it's useful in a few cases. For example: the case when emulators in different platforms result in slightly different images #265

from shot.

I find it more useful to calculate the tolerance value by comparing colors more than number of different pixels. I'm thinking in a scenario where we have a green pixel in a black background, even if it's just one pixel, the difference between colors is too big to ignore it. There are algorithms to quantify the difference between two colors under human perception criteria (https://en.wikipedia.org/wiki/Color_difference#CIEDE2000 & https://en.wikipedia.org/wiki/CIELUV#Color_and_hue_difference).

I'd obviously go for your first approach and then, if we find it has problems, try with the more complicated process 😅

from shot.

I personally did not face any issue related to the screenshot comparison during my last year using this testing strategy for Android because we knew from the very beginning that animations and async image loading were going to be something we need to override. However, some devices emulated on different OS versions or different Android versions could rasterize fonts with just a difference of a bunch of pixels. This is something that got the Karumi iOS developers crazy during our last iOS project.

from shot.

I've just run into this issue. A screenshot taken locally is off by 5 pixels from a screenshot generated on Firebase Test Lab. The screenshots to the naked eye look identical. Same emulator device model + API level. So, while I don't like adding a tolerance level, it might be a pragmatic solution. Anyone has any experience with this?

from shot.

+1 for this feature. Any updates?

from shot.

Hi all. As you can see I personally created this issue in June 2018. However, keeping in mind this project is only a gradle plugin and we can only interact with the screenshots once these are recorded there are some things we will not be able to implement. Excluding some views from the screenshot is one of them. At least, in a transparent way to the user. To do this we would need to implement our own version of facebook's library. Anyway, this feature can be implemented in a different way from the production code or the test code by changing the visibility of these views to View.INVISIBLE or changing any property.

About the tolerance, after years using the library I can fully understand why some users would love to have this feature, but this is not a good idea at all. When your screenshot test starts failing then your first thought is to configure the screenshot tolerance but the real issue is you need to hide some details of the UI. If you reduce the tolerance to the 95%, tomorrow you'll change it to 90% and this means bugs with an impact in the 10% or the UI or less will be considered as a test passing.

However, I'm going to create some tooling to improve this scenario in a different way.

from shot.

This might be a stupid question but, from an end user of the lib perspective, what sort of situations would lead him to find a tolerance factor useful?. Doesn't any minimum UI change mean the screen is not showing the same way anymore?

In other words and to make it clear: Should you assume that a screen is still valid for you if it's passing just on a 95%?. What if the resting 5% is a SnackBar that completely changes the expected behavior?

IMO, In some way tolerance factor could lead to flakiness.

Just to give a similar example, I wouldn't add tolerance for unit tests. From my point of view a test passes or not, but not passes but just on a given %.

Is tolerance factor a usual feature on any type of tests or it is probably a more specific feature for other types like property based ones?

from shot.

Uhm, alright. That indeed makes sense. I guess you always have the power to run your ss tests just on stock roms to avoid that but whatever, people would still want to run them on their physical devices.

So overall, if it's an opt in feature that comes along with the proper warnings on the readme about the potential danger of using tolerance factors, it could become handy for some people!

Thx for answering.

from shot.

I've just run into this issue. A screenshot taken locally is off by 5 pixels from a screenshot generated on Firebase Test Lab. The screenshots to the naked eye look identical. Same emulator device model + API level. So, while I don't like adding a tolerance level, it might be a pragmatic solution. Anyone has any experience with this?

Same.
So far I haven't been able to find a local configuration that would match whatever they use in the test lab so that it doesn't fail on many occasions in shadows, gradients, some scrollbars, edit text underlines, etc.

Not sure about a solution though, as often it's beyond just 5 pixels while being visually indistinguishable.

from shot.

Coincidentally, today I got an answer (in the Firebase slack channel) specifically regarding comparing screenshots created locally against the ones obtained from a Firebase Test Lab run:

We are not using the emulators that you get through the Android SDK, rather special built VMs. That could definitely be the reason for these minor differences. This might change in the future, but right now we don't use any SDK emulators.

from shot.

I'm afraid I can't fix the usage of different devices on the cloud test lab 😢 I don't think adding some tolerance configuration will fix this issue, we've been on a similar situation building iOS apps for some years using this strategy and that's not the way to go. The same happens with the different browsers rendering methods. If we can't get deterministic screenshots, then I'd not use screenshot testing.

from shot.

I had that same comparison issue using screenshots from Firebase Test Lab but I found that it's only with virtual devices. At least I tested with Pixel 2 physical device on Test Lab and comparison with screenshot made from local emulator worked.

from shot.

we have been running into this issue more and more. Tolerance would be really helpful

from shot.

Wanted to chime in with 1 more use case: Auto generated colors for things like a background icon. The ideal would be to allow users to mark an exclusion area somehow

from shot.

I've updated tolerance documentation in case you are interested: https://github.com/pedrovgs/Shot/#tolerance

from shot.