Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incorrectly delete unused snapshot in pytest bdd when target run using -k filter #770

Open
mawaliya opened this issue Jul 14, 2023 · 1 comment

Comments

@mawaliya
Copy link

mawaliya commented Jul 14, 2023

Describe the bug

In the pytest using pytest_bdd. When we have several test using scenario outlines. When different scenario outlines have the same examples, then we run targeted test into only one example of a scenario outline using -k filter. Syrupy with JSONSnapshotExtension extension, will think other snapshots with same example from different scenario outlines are unused.

To reproduce

  1. Have tests in bdd format like this:
    my_feature.feature
    @test1
    Scenario Outline: Test1
        Then The result is <file_type>

        Examples:
            | file_type |
            | JSON      |
            | CSV       |

    @test2
    Scenario Outline: Test2
        Then The result is <file_type>

        Examples:
            | file_type |
            | JSON      |
            | CSV       |
  1. Implement the test and snapshot using JSONSnapshotExtension extension.
    test_feature.py
import pytest
from pytest_bdd import parsers, scenarios, then
from syrupy.assertion import SnapshotAssertion
from syrupy.extensions.json import JSONSnapshotExtension

scenarios("my_feature.feature")


@pytest.fixture
def custom_snapshot(snapshot: SnapshotAssertion):
    return snapshot.use_extension(JSONSnapshotExtension)


@then(parsers.cfparse("The result is {file_type}"))
def check_the_file_type(file_type: str, custom_snapshot: SnapshotAssertion):
    assert file_type == custom_snapshot
  1. Run the snapshot generation
    pytest --snapshot-update
  2. Modify Test1 JSON snapshot to fail the test
  3. Run the targeted test only for test 1 JSON example
    pytest -m 'test1' -k 'JSON'
  4. Observe that the run will suggest unused snapshots for Test2 JSON.

Expected behavior

Snapshot from other test with the same examples are not considered unused.

Screenshots

image

Environment (please complete the following information):
wsl Ubuntu, Windows 11
Description: Ubuntu 22.04.2 LTS
Release: 22.04
Codename: jammy

platform linux -- Python 3.10.6, pytest-7.3.2, pluggy-1.2.0
configfile: pytest.ini
plugins: metadata-3.0.0, bdd-6.1.1, allure-pytest-bdd-2.13.2, bdd-html-0.1.14a0, html-3.2.0, syrupy-4.0.6, check-2.1.5

Additional context

@BarrensZeppelin
Copy link

BarrensZeppelin commented Mar 6, 2024

I don't think this is a tool compatibility issue.

Consider a pytest file like:

import pytest

@pytest.mark.parametrize("param", ["one", "two", "three"])
def test_parameterized(param, snapshot):
    assert param == snapshot

If you run pytest -k one you get the correct output:

collected 3 items / 2 deselected / 1 selected                                                   

test_syrupy.py .                                                                          [100%]

------------------------------------ snapshot report summary ------------------------------------
1 snapshot passed.
================================ 1 passed, 2 deselected in 0.02s ================================

But if you run pytest -k 'not one' you get an error for unused snapshots:

collected 3 items / 1 deselected / 2 selected                                                   

test_syrupy.py ..                                                                         [100%]

------------------------------------ snapshot report summary ------------------------------------
2 snapshots passed. 1 snapshot unused.

Re-run pytest with --snapshot-update to delete unused snapshots.
================================ 2 passed, 1 deselected in 0.02s ================================

EDIT: This is a completely separate issue related to syrupy's re-implementation of test selection with -k.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants