Golden tests occassionally failing with 0.00% diff

[guy-plentific]

I am posting this in the hope that someone else has experienced this issue as has an idea of what could be causing it. For apparently no reason, sometimes one or two of our golden tests fail with the error

Pixel test failed, 0.00% diff

I have noticed a similar issue here: flutter/devtools#2935

We have ruled out obvious causes (different versions of Flutter, different operating systems etc.) but are hoping that someone else has experienced something similar and can help us.

[coreysprague]

@guy-plentific is there any pattern to the tests that are impacted by this? Our app has about 900 golden tests in total, and we haven't encountered this.

[guy-plentific]

@coreysprague sorry for the late reply Corey - no, unfortunately there is no particular pattern. I realise there is very little for you to go by on here, I just thought I would mention in case you had seen this before or had any ideas.

[johannessachse]

@guy-plentific I can confirm this issue. But I don't have any solution to this.

[phipps980316]

Hey @guy-plentific, we started seeing this too with golden tests with 0.00% difference, but it seemed as tho there was a very very small difference in the images for some reason > 0.001%.

We couldn't determine the reason for it after checking different flutter versions across our CI runners but we added a workaround to allow the very small difference as it is was very insignificant.

Details of the workaround can be found here: Workaround

[guy-plentific]

Hey @guy-plentific, we started seeing this too with golden tests with 0.00% difference, but it seemed as tho there was a very very small difference in the images for some reason > 0.001%.

We couldn't determine the reason for it after checking different flutter versions across our CI runners but we added a workaround to allow the very small difference as it is was very insignificant.

Details of the workaround can be found here: Workaround

Thanks a lot Jon! Full link address is https://rows.com/blog/post/writing-a-localfilecomparator-with-threshold-for-flutter-golden-tests

[w3ggy]

Same problem after upgrading Flutter from 3.3.10 to 3.7.0.

Any idea why? Temporarily we set the threshold to 1% but it is not a solution I guess.

We see this error only on Bitrise with the same MacOS version. We cannot reproduce it locally. Maybe some additional system settings are handled now by Flutter starting 3.7 like it was with AppleFontSmoothing system setting.

UPDATED:

Looks like the issue is in different render system on Intel and Mac chips: flutter/flutter#111739 (comment)

[befirst]

This: #160 (comment) doesn't look like the long-term solution to a problem. For example, our team uses different mac machines, and when we generate snapshots they are different for m1/2 and Intel chips. It's impossible for now to use tests even with the workaround described above because it looks like it works only for comparison (flutter test) and not for a generation (flutter test --update-goldens), so after each feature, we have a lot of newly regenerated snapshots