Cache + xdist

[clintonb]

Has anyone had success running tests that rely on caching with the xdist plugin? I created a fixture, similar to the database fixture, to add a cache key prefix: https://github.com/edx/course-discovery/blob/ad1dca5623b765c6d85d83dcf7e5f75c7b8e1181/conftest.py#L18-L40. However, I still have a couple tests that fail when using memcached (but not when using local memory).

1. Is there anything obvious that I am missing with my fixture?
2. If you have had success running cache-related tests in parallel mode, how did you handle cache key collisions?
3. Are others interested in such a fixture being contributed to this project?

[TakenBrandi]

@clintonb: Your snippet worked almost flawlessly for one of my projects until one day it didn't. I think the prefix prevents collisions without any issues, but (on redis at least) cache.clear() is flushing all cached items, not just the ones with the correct prefix, so any code that relies on the cached value being in the cache may fail if another test starts or finishes between when the cache value is set and when it's read. A local memory cache would likely be immune since flushing it wouldn't affect other processes.

[asfaltboy]

I'm having a (somewhat) similar issue, running tests that use webtest and end up with CSRF error 🤷‍♂

[ricky-sb]

Any solution to be able to use Redis with this?

[TakenBrandi]

I can no longer reach the repo where I did have this working, but IIRC, redis-py provides a method for deleting all keys with a given prefix and I called that, or slightly reworked the cleanup so that it only deleted keys with the correct prefix.

[cutamar]

For anybody needing a workaround/solution here:

@pytest.fixture(autouse=True)
def isolated_cache(worker_id, request):
    """
    This autofixture makes sure that every test has isolated cache access.
    Even when using pytest-xdist for parallel calls, this will make sure
    that they use different cache prefixes and are isolated and cleaned-up.
    :param worker_id:
    :param request:
    :return:
    """
    skip_if_no_django()

    cache.key_prefix = worker_id

    def remove_cache(worker_id):
        cache.delete_pattern("*", prefix=worker_id)

    remove_cache(worker_id)

    # Called after a test has finished.
    request.addfinalizer(lambda: remove_cache(worker_id))

[syphar]

We're using another workaround for this issue which doesn't need to clear the cache, so is somewhat faster:

@pytest.fixture(autouse=True)
def isolated_cache(settings):
    cache_version = uuid.uuid4().hex

    for name in settings.CACHES.keys():
        settings.CACHES[name]["VERSION"] = cache_version

    from django.test.signals import clear_cache_handlers
    clear_cache_handlers(setting="CACHES")

[sshishov]

For Redis we are using different databases (thanks redis support multiple "logical" DBs on one server):

@pytest.fixture(scope='session', autouse=True)
def _redis_add_suffix(worker_id: str) -> None:
    suffix = '1' if worker_id == 'master' else str(int(worker_id.replace('gw', '')) + 2)
    django_settings.CACHES['default']['LOCATION'] += f'/{suffix}'
    default_cache.client.get_client().connection_pool.connection_kwargs['db'] = int(suffix)
    default_cache.key_prefix = worker_id  # add this line if you want just to have prefixes as your worker_id


@pytest.fixture(autouse=True)
def _isolated_cache() -> None:
    default_cache.clear()

We are adding different logical DB to every worker, also we clear the case before every execution to make sure that one test did not affect another one.