How to implement image diffing for visual regression testing

[noahnu]

Discussed in https://github.com/tophat/syrupy/discussions/536

^{Originally posted by rassie August 4, 2021}
If I understand the documentation correctly, it's possible to implement a snapshot extension which would compare images visually (e.g. with pixelmatch) instead of byte-wise. Is it a matter of writing a custom SnapshotReporter and mix it into an extension or would I need to do more?

[noahnu]

Same as with the other discussion, sorry about the delayed response.

---

The SnapshotReporter is only used for displaying diff results. You'd likely want to override it, specifically the diff_lines method to communicate the results from pixelmatch. The actual assertion unfortunately happens here and cannot be overridden at this time.

That being said, it'd be fairly trivial to update this logic to let a custom extension override the equality check. That'd also enable other sorts of "tolerance" assertions. If there is still interest in this, we can move this discussion to a feature request.

Originally posted by @noahnu in https://github.com/tophat/syrupy/discussions/536#discussioncomment-1248653

[drjasonharrison]

What is the status of being able to create a custom SnapshotReporter that uses a tool like pixelmatch?

[noahnu]

It should be doable now. Something along the lines of:

class PixelMatchSnapshotExtension(PNGImageSnapshotExtension):
    def matches(self, *, serialized_data, snapshot_data) -> bool:
        pass # run pixelmatch here

    def diff_snapshots(self, serialized_data, snapshot_data) -> "SerializableData":
        pass # re-run pixelmatch and return a diff image (can cache on the class instance)

@pytest.fixture
def visual_snapshot(snapshot):
    return snapshot.use_extension(PixelMatchSnapshotExtension)

def test_case(visual_snapshot):
    assert generate_image(....) == visual_snapshot

matches:

syrupy/src/syrupy/extensions/base.py

Line 404 in b831ee2

def matches(

If you end up implementing a visual regression testing extension based on syrupy, consider open sourcing it and I can link to it on our README under a community section.