Caching in DRF?

[jacobsvante]

Awesome work on v2, I'm really liking what I've seen so far. Big thumbs up!

One problem arose when I was migrating from tastypie to DRF. I use Django's cache framework to cache everything on the site I'm developing (it's very read-heavy), but when I try to access the API I get TypeError: can't pickle _Input objects. It works just fine with caching disabled.

It should be noted that I use nginx + uwsgi instead of the internal dev server.

Here's the traceback:

Traceback (most recent call last):
  File ".../lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 241, in __call__
    response = self.get_response(request)
  File ".../lib/python2.7/site-packages/django/core/handlers/base.py", line 192, in get_response
    response = self.handle_uncaught_exception(request, resolver, sys.exc_info())
  File ".../lib/python2.7/site-packages/django/core/handlers/base.py", line 221, in handle_uncaught_exception
    return debug.technical_500_response(request, *exc_info)
  File ".../lib/python2.7/site-packages/django/core/handlers/base.py", line 188, in get_response
    response = middleware_method(request, response)
  File ".../lib/python2.7/site-packages/django/middleware/cache.py", line 112, in process_response
    lambda r: self.cache.set(cache_key, r, timeout)
  File ".../lib/python2.7/site-packages/django/template/response.py", line 91, in add_post_render_callback
    callback(self)
  File ".../lib/python2.7/site-packages/django/middleware/cache.py", line 112, in <lambda>
    lambda r: self.cache.set(cache_key, r, timeout)
  File ".../lib/python2.7/site-packages/redis_cache/cache.py", line 218, in set
    result = self._set(key, pickle.dumps(value), int(timeout), client, _add_only)
  File "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy_reg.py", line 70, in _reduce_ex
    raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle _Input objects

_Input object is of type uwsgi._Input

[tomchristie]

Wow, that's an interesting one.
So Django's caching layer adds a post_render_callback to the response which is used to cache the response once it's been rendered. So far so good. The problem is that it uses pickle to do so, which is blowing up for Response objects (where it would pass for HttpResponse objects).

I've not looked at the pickle API enough to know exactly how this needs to work, but seems like we need to implement __getstate__ on Response to do something other than the default behavior.

I reckon we can probably get this sorted, just needs a bit of digging to figure out how to make Response objects properly pickle-able.

First step would be to throw together some failing tests.

[tomchristie]

I believe the fix would be to ensure that Response.getstate calls the superclass, then pops off .accepted_renderer / .renderer_context / .data from the dict before returning, (since those bits of may not be pickable.)

That should work okay, because the restored, unpicked Response will include the required .content, it just won't include the intermediate state it used along the way to get.

There's probably a slightly nicer solution that does correctly restore all the state, but I think this should get the job done and cover all sensible use-cases.

[jacobsvante]

Sounds great. I was just about to write some tests for this, but maybe you're already on your way with a fix?

[tomchristie]

@jmagnusson No, not started on anything. Be mighty pleased if you could tackle it, or provide tests.

[jacobsvante]

Okay, I'll see what I can do. Hopefully I will get some time to do it later this week :-)

[devshark]

Hi @jmagnusson , I know this is already a very old post, but I just like to know what django caching were you using?

Or if @tomchristie already has built an API caching functionality in the DRF.

I need a per-user API caching for my project because the API call is very heavy and only need to update the result once a day.

My dev setup is
Django 1.5.4
DRF 2.4.2
DRF-jwt 1.0.1

Thanks for the replies!

Cheers

[chibisov]

Hi @devshark, you could try http://chibisov.github.io/drf-extensions/docs/#caching with the usage of http://chibisov.github.io/drf-extensions/docs/#key-constructor and UserKeyBit

[tomchristie]

You might also want to follow #32 for when we (eventually) make some progress there.

[devshark]

Thanks @chibisov ! I am using that package now and so far so good.

Thanks for the reference @tomchristie . I will be watching that thread and see if it will fit my requirements in the future.

[mbaragiola]

Hello, sorry for bringing this issue back. Is http://chibisov.github.io/drf-extensions/docs/#caching still necessary or as of today Django's caching framework is enough for DRF? Thanks in advance.