Python3 compatibility issue in get_content_type()

[miki725]

I use Django with django_extensions which uses werkzeug for debugging when using runserver_plus. Werkzeug seems to blow up in get_content_type() with following traceback when using Python3:

Traceback (most recent call last):
  File "/Volumes/Data/Users/miki725/.virtualenvs/test-py3/lib/python3.4/site-packages/werkzeug/serving.py", line 177, in run_wsgi
    execute(self.server.app)
  File "/Volumes/Data/Users/miki725/.virtualenvs/test-py3/lib/python3.4/site-packages/werkzeug/serving.py", line 165, in execute
    application_iter = app(environ, start_response)
  File "/Volumes/Data/Users/miki725/.virtualenvs/test-py3/lib/python3.4/site-packages/werkzeug/debug/__init__.py", line 173, in __call__
    response = self.get_resource(request, arg)
  File "/Volumes/Data/Users/miki725/.virtualenvs/test-py3/lib/python3.4/site-packages/werkzeug/debug/__init__.py", line 154, in get_resource
    return Response(f.read(), mimetype=mimetype)
  File "/Volumes/Data/Users/miki725/.virtualenvs/test-py3/lib/python3.4/site-packages/werkzeug/wrappers.py", line 749, in __init__
    mimetype = get_content_type(mimetype, self.charset)
  File "/Volumes/Data/Users/miki725/.virtualenvs/test-py3/lib/python3.4/site-packages/werkzeug/utils.py", line 224, in get_content_type
    if mimetype.startswith('text/') or \
TypeError: startswith first arg must be bytes or a tuple of bytes, not str

[miki725]

submitted PR which fixes the issue

[untitaker]

I am not sure whether this is Werkzeug's fault or Django uses wrong string types in its responses.

[miki725]

Here is full traceback in ipdb and I dont see Django being called anywhere. Complete stackstace seems to be only in werkzeug.

ipdb> w
  /Volumes/Data/Users/miki725/.virtualenvs/test-py3/lib/python3.4/site-packages/werkzeug/serving.py(692)inner()
    690         make_server(hostname, port, application, threaded,
    691                     processes, request_handler,
--> 692                     passthrough_errors, ssl_context).serve_forever()
    693 
    694     if os.environ.get('WERKZEUG_RUN_MAIN') != 'true':

  /Volumes/Data/Users/miki725/.virtualenvs/test-py3/lib/python3.4/site-packages/werkzeug/serving.py(436)serve_forever()
    434         self.shutdown_signal = False
    435         try:
--> 436             HTTPServer.serve_forever(self)
    437         except KeyboardInterrupt:
    438             pass

  /usr/local/Cellar/python3/3.4.2_1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/socketserver.py(238)serve_forever()
    236                                        poll_interval)
    237                 if self in r:
--> 238                     self._handle_request_noblock()
    239 
    240                 self.service_actions()

  /usr/local/Cellar/python3/3.4.2_1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/socketserver.py(305)_handle_request_noblock()
    303         if self.verify_request(request, client_address):
    304             try:
--> 305                 self.process_request(request, client_address)
    306             except:
    307                 self.handle_error(request, client_address)

  /usr/local/Cellar/python3/3.4.2_1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/socketserver.py(331)process_request()
    329 
    330         """
--> 331         self.finish_request(request, client_address)
    332         self.shutdown_request(request)
    333 

  /usr/local/Cellar/python3/3.4.2_1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/socketserver.py(344)finish_request()
    342     def finish_request(self, request, client_address):
    343         """Finish one request by instantiating RequestHandlerClass."""
--> 344         self.RequestHandlerClass(request, client_address, self)
    345 
    346     def shutdown_request(self, request):

  /usr/local/Cellar/python3/3.4.2_1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/socketserver.py(669)__init__()
    667         self.setup()
    668         try:
--> 669             self.handle()
    670         finally:
    671             self.finish()

  /Volumes/Data/Users/miki725/.virtualenvs/test-py3/lib/python3.4/site-packages/werkzeug/serving.py(200)handle()
    198         rv = None
    199         try:
--> 200             rv = BaseHTTPRequestHandler.handle(self)
    201         except (socket.error, socket.timeout) as e:
    202             self.connection_dropped(e)

  /usr/local/Cellar/python3/3.4.2_1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/server.py(398)handle()
    396         self.close_connection = 1
    397 
--> 398         self.handle_one_request()
    399         while not self.close_connection:
    400             self.handle_one_request()

  /Volumes/Data/Users/miki725/.virtualenvs/test-py3/lib/python3.4/site-packages/werkzeug/serving.py(235)handle_one_request()
    233             self.close_connection = 1
    234         elif self.parse_request():
--> 235             return self.run_wsgi()
    236 
    237     def send_response(self, code, message=None):

  /Volumes/Data/Users/miki725/.virtualenvs/test-py3/lib/python3.4/site-packages/werkzeug/serving.py(177)run_wsgi()
    175 
    176         try:
--> 177             execute(self.server.app)
    178         except (socket.error, socket.timeout) as e:
    179             self.connection_dropped(e, environ)

  /Volumes/Data/Users/miki725/.virtualenvs/test-py3/lib/python3.4/site-packages/werkzeug/serving.py(165)execute()
    163 
    164         def execute(app):
--> 165             application_iter = app(environ, start_response)
    166             try:
    167                 for data in application_iter:

  /Volumes/Data/Users/miki725/.virtualenvs/test-py3/lib/python3.4/site-packages/werkzeug/debug/__init__.py(173)__call__()
    171             frame = self.frames.get(request.args.get('frm', type=int))
    172             if cmd == 'resource' and arg:
--> 173                 response = self.get_resource(request, arg)
    174             elif cmd == 'paste' and traceback is not None and \
    175                  secret == self.secret:

  /Volumes/Data/Users/miki725/.virtualenvs/test-py3/lib/python3.4/site-packages/werkzeug/debug/__init__.py(154)get_resource()
    152             f = open(filename, 'rb')
    153             try:
--> 154                 return Response(f.read(), mimetype=mimetype)
    155             finally:
    156                 f.close()

  /Volumes/Data/Users/miki725/.virtualenvs/test-py3/lib/python3.4/site-packages/werkzeug/wrappers.py(749)__init__()
    747                 mimetype = self.default_mimetype
    748             if mimetype is not None:
--> 749                 mimetype = get_content_type(mimetype, self.charset)
    750             content_type = mimetype
    751         if content_type is not None:

> /Volumes/Data/Users/miki725/.virtualenvs/test-py3/lib/python3.4/site-packages/werkzeug/utils.py(224)get_content_type()
    222     #     charset = charset.decode('utf-8')
    223     import ipdb; ipdb.set_trace()
--> 224     if mimetype.startswith('text/') or \
    225        mimetype == 'application/xml' or \
    226        (mimetype.startswith('application/') and

ipdb> type(mimetype), type(charset)
(<class 'bytes'>, <class 'str'>)

[miki725]

The problem seems to be in 2 frames above:

ipdb> up
> /Volumes/Data/Users/miki725/.virtualenvs/test-py3/lib/python3.4/site-packages/werkzeug/wrappers.py(749)__init__()
    748             if mimetype is not None:
--> 749                 mimetype = get_content_type(mimetype, self.charset)
    750             content_type = mimetype

ipdb> up
> /Volumes/Data/Users/miki725/.virtualenvs/test-py3/lib/python3.4/site-packages/werkzeug/debug/__init__.py(154)get_resource()
    153             try:
--> 154                 return Response(f.read(), mimetype=mimetype)
    155             finally:

ipdb> l
    149         if isfile(filename):
    150             mimetype = mimetypes.guess_type(filename)[0] \
    151                 or 'application/octet-stream'
    152             f = open(filename, 'rb')
    153             try:
--> 154                 return Response(f.read(), mimetype=mimetype)
    155             finally:
    156                 f.close()
    157         return Response('Not Found', status=404)
    158 
    159     def __call__(self, environ, start_response):

ipdb> mimetypes.guess_type(filename)[0]
b'text/javascript'

[miki725]

Whats strange is that guess_type returns bytes vs string. When trying to guess type for the same file in shell I get string:

>>> mimetype = mimetypes.guess_type('/Volumes/Data/Users/miki725/.virtualenvs/test-py3/lib/python3.4/site-packages/werkzeug/debug/shared/jquery.js')[0]
>>> type(mimetype)
<class 'str'>

Should I adjust PR to convert mimetype here instead of in get_content_type()?

[untitaker]

Whats strange is that guess_type returns bytes vs string

Your PR does fix the problem, but this should never have happened, as you said.

What is filename in the frame where the guess_type call happens?

[miki725]

its string. I though that maybe guess_type returns same type as filename but apparently no the case.

[orf]

Hi, this happens often when using the django-extensions package as said above. This hasn't been touched in nearly 2 years, is there any issue with merging the linked MR?

[pleasedontbelong]

Recently I got the same problem, for me it happens on javascript files. After debugging I found my problem was django-pipeline: it modifies the types_map and set this https://github.com/jazzband/django-pipeline/blob/master/pipeline/conf.py#L81-L87 here https://github.com/jazzband/django-pipeline/blob/master/pipeline/utils.py#L42-L43 .. :S :S ... to solve I just had to correctly set MIMETYPES on my pipeline settings.

Maybe you should reconsider merging the PR.

Hope this helps someone... happy debugging!

[davidism]

Django-Pipelines should be fixed so that it does not put bytes into the mimetype type map. Django-Extensions doesn't appear to be doing anything, but that just means some other dependency is misbehaving. I'm closing this as I think it should be addressed in the libraries adding the bad data. If someone can post a good reason that bytes should be expected in the mimetypes type map, I'll reconsider.