Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

warn / document that keymangler is generally needed w/ the function decorators #182

Open
brycepg opened this issue Jul 9, 2020 · 5 comments
Labels
bug Something isn't working caching decorator things to do with cache on arguments, etc documentation region

Comments

@brycepg
Copy link

brycepg commented Jul 9, 2020

Problem

Hi I think dogpile.cache is really awesome, however I've been encountering an issue using the memcached backend:

It looks like dogpile.cache uses __repr__ or __str__ of the parameters to the cached function for the default key. When caching a sequence, this means that the key can be very large. memcached however has a fixed key limit of 250. The solution to the error is nonobvious (implementing one's own key_mangler)

Here is the error in question

ValueError: key length 63566 too long, max is 250
> /home/user/envs/mypackage/lib/python3.7/site-packages/dogpile/cache/region.py(1360)get_or_create_for_user_func()

In the interest of usability, I would recommend adding default key_mangler, for all or certain backends. Or at least, there should be caveats section in the documentation.

Workaround Solution

Since someone else will likely encounter this, I will post my solution using the xxhash library:

from xxhash import xxh64_hexdigest
region = make_region(key_mangler=xxh64_hexdigest)
@CaselIT
Copy link
Member

CaselIT commented Jul 9, 2020

Note that you can probably also use the python standard library hashlib

@zzzeek
Copy link
Member

zzzeek commented Jul 10, 2020

key_mangler has the sha1_mangle_key mangler that goes along with it and it's documented with key_mangler. but at the moment the paramlinks don't seem to be working so i can't link to the key mangler param, going to see what's up with that.

@brycepg
Copy link
Author

brycepg commented Jul 10, 2020

Although I (now) know that dogpile.cache has all the tools to make the memcached backend work, it still has to be configured. This use-case is broken by default -- with the fix requiring guessing the cause of the error and reading more docs. Shouldn't this library have as little gotchas as possible to facilitate new users and adoption? (still it's awesome that it exists at all)

@zzzeek
Copy link
Member

zzzeek commented Jul 10, 2020

I think you're referring specifically to the cache decorators. the keys stored by dogpile are the keys you give it directly if you are using the straight region API like get_or_create, get, etc., and certainly if I use a cache library and say, cache.set("my key", value), by default I'd expect it to use the key "my key" and not decide I want to hash it.

the function decorators are a special case and they also have their own function_key_generator parameter you can set.

This is all documented so I don't see the problem as beyond the CSS styling and perhaps some extra notes here and there. we certainly couldn't change this behavior by default right now even if we wanted.

@zzzeek
Copy link
Member

zzzeek commented Jul 11, 2020

another option is to have the function key decorators specifically emit a warning if key_mangler isn't set. does anyone have interest in doing a PR for this?

@zzzeek zzzeek added bug Something isn't working caching decorator things to do with cache on arguments, etc region labels Jul 11, 2020
@zzzeek zzzeek changed the title There should be a default key_mangler when using backends with key limits (notably the pylibmc/memcached backend) warn / document that keymangler is generally needed w/ the function decorators Jul 11, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working caching decorator things to do with cache on arguments, etc documentation region
Projects
None yet
Development

No branches or pull requests

3 participants