Cache test timing is unreliable

[davidism]

Raising the sleep time helps, but not consistently. And it makes the tests slower. I'm not even clear why they fail: if a cache key is set to timeout after a certain period, sleeping longer than that period should mean the key is no longer valid. We need to retry the test so that random unrelated failures don't keep popping up in new PRs.

[mitsuhiko]

I think this is fixed now. I hope.

[davidism]

It's not fixed. 😦 I just had to re-run Travis a couple times because the cache tests kept failing.

[davidism]

@mitsuhiko A commit you made (7a3a91a) seems to have undone changes to the test layout. It added back tests/contib/test_cache.py, when the file had been moved to contrib/cache/, which is what you added the fix to (8202649).

Going to try reverting the added file and see if the tests stay stable.

[davidism]

Tests for #1233 passed the first time, so I'm going to close this for now.

[blbu]

Hi, will this be closed under #1249 (deprecate werkzeug.contrib.cache)?

Otherwise, I've "reproduced" the Appveyor Windows failed test on 6 Jan 2018 at https://ci.appveyor.com/project/pallets/werkzeug/build/1/job/bnt1188d64tysdr5 (noting Travis was mentioned above).

TL;DR: test_generic_timeout rarely broken, test_filesystemcache_clear always broken?

Environment:
Windows PowerShell 4.0, Python 3.6.5 (In a venv).

PS > pip install -e ".[dev]"
PS > pip install python-dotenv greenlet blinker
PS > pytest
...

pytest: platform win32 -- Python 3.6.5, pytest-3.5.0, py-1.5.3, pluggy-0.6.0
Werkzeug 0.15.dev0 (commit 6bb8e49?)

Observations:
The test test_generic_timeout will rarely fail:

line 129: c.set('baz', 'qux', 1)
line 130: assert c.get('baz') == 'qux'
AssertionError: assert None == 'qux'

So the previous line 129 has failed to set?

(I had originally written that raising fast_sleep(3) to fast_sleep(10) decreased repro rate of this error. fast_sleep is called on line 131...)

More interesting to me, the test test_filesystemcache_clear seems to fail 100% of the time, even with a delay like fast_sleep(30). The AssertionError I have is the same as at the end of the Appveyor build above.

Line 186: assert len(cache_files) == 0
AssertionError: assert 1 == 0

The file created by the cache is left on-disk.
I'm running my terminal as Administrator, have checked for other processes which might be accessing that directory. I.e., closed my editor and so on.

I've confirmed the source on disk includes the changes in 'Merge pull request #1233' 8393ee8

Because of that commit, I'm expecting _guaranteed_deletes to be True. Not reaching it in the above test, safe to ignore?

Removing the file interactively works:

PS > python
import os
os.listdir("...") # path emitted by assertion error, file from test is listed
os.remove("...") # path-to-file emitted by assertion error
os.listdir("...") # path emitted by assertion error, file from test is not listed

(The implementation of clear() in FileSystemCache.)

There is no OSError caught in the implementation of clear(). I.e., os.remove() does not appear to throw.

I could try and look at this in more depth if it's not to be closed?

I'm new. I'll accept any advice. Thank you for your time.

[davidism]

Closing this since we're not going to be maintaining the cache contrib. I'm going to disable tests for now so they don't cause problems with other PRs.