AST parsing fails on non-utf8 setup.py files

[techalchemy]

The AST parser simply dies on non-utf8 setup.py files. One package I've encountered with this issue is azure-storage:

>>> ast_parse_setup_py("/tmp/pkgs/azure-storage-0.36.0/setup.py")                                                                                                             
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/techalchemy/.virtualenvs/tempenv-74f7279776e1c/lib/python3.8/site-packages/requirementslib/models/setup_info.py", line 827, in ast_parse_setup_py
    ast_analyzer = ast_parse_file(path)
  File "/home/techalchemy/.virtualenvs/tempenv-74f7279776e1c/lib/python3.8/site-packages/requirementslib/models/setup_info.py", line 819, in ast_parse_file
    tree = ast.parse(read_source(path))
  File "/home/techalchemy/.pyenv/versions/3.8.1/lib/python3.8/ast.py", line 47, in parse
    return compile(source, filename, mode, flags,
  File "<unknown>", line 1
    #!/usr/bin/env python
    ^
SyntaxError: invalid character in identifier

invalid character in identifier (<unknown>, line 1)

I checked the encoding of the given file, revealing:

>>> import chardet                                                                                                                                                            
>>> with open("/tmp/pkgs/azure-storage-0.36.0/setup.py", "rb") as fh: 
  2     encoding = chardet.detect(fh.read())                                                                                                                                  
>>> encoding                                                                                                                                                                  
{'encoding': 'UTF-8-SIG', 'confidence': 1.0, 'language': ''}

So it seems it's not a safe assumption that these are utf-8 encoded files and we should build in a fallback

This is a blocker on pypa/pipenv#3369