Add HEADERS_KEEP to settings and to HttpCompressionMiddleware (#1988) #4017
Conversation
|
Not working. Can ignore. |
| @@ -18,6 +18,9 @@ | |||
| class HttpCompressionMiddleware(object): | |||
| """This middleware allows compressed (gzip, deflate) traffic to be | |||
| sent/received from web sites""" | |||
|
|
|||
| HEADERS_KEEP = settings.getbool('HEADERS_KEEP') | |||
There was a problem hiding this comment.
The reason this is failing is because the settings variable is not defined here, hence the NameError exception.
What would be the motivation behind this change? Perhaps there is an alternative solution.
There was a problem hiding this comment.
Thank you for taking the time to respond on here. Yeah I had a feeling it wouldn't work but was just throwing it out there to kinda break the ice for me. But I was trying to work on issue #1988 where they thought about keeping the encoding-header instead of chopping it off right at the end of the process_repsonse function.
@Gallaecio commented on the few questions I possed on that thread and think I'm going to try and model it after the ajaxcrawl.py in the downloadermiddlewares that as an init that's passed setting and set a variable something like self.keep_headers.
I was hesitant to try that cause I thought that would create a bigger problem on top of the namespace error. Since I thought it would have to change the arguments that are passed to HttpCompressionMiddleware where ever else it is invoked but they said it was created depending on the user's settings but also that that might be thinking too far ahead is how I took it. In addition to having trouble seeing where crawler and other object were coming from to be used as arguments.
It has been some time since I've worked on classes and projects as spread out as this.
|
Makes sense. I see you opened #4021, let's continue the discussion there. |
Added setting and variable to HttpCompressionMiddleware.
Not 100% if thats good to go but figured I can give it a shot and learn from any mess-ups.