Python 3 support.

[nsoranzo]

Python 2.7 will not be maintained past 2020.

Moreover, some Galaxy dependencies dropped Python2 support: cachetools, cmd2, cwltool, numpy, schema-salad

Add support for Python >= 3.5 while maintaining support for Python 2.7.

xref.: https://trello.com/c/dZcCVf9I/2702-migrate-to-python-3

Useful tools and documentation

https://docs.python.org/3/howto/pyporting.html
https://docs.python.org/2/library/2to3.html
http://python-future.org/
https://six.readthedocs.io/
http://python3porting.com/

Dependencies which need to be ported, dropped or updated

• bx-python: Add support for Python 3 and drop Python < 2.6 bxlab/bx-python#7, Python 3 compatibility bxlab/bx-python#25, Python3: Update bx-python dependency to 0.8.1 #5492
• Cheetah: replaced with Cheetah3 fork in Various Python3 related fixes #5359
• Fabric: Python 3 support fabric/fabric#1424, replaced with Fabric3 fork in Various Python3 related fixes #5359
• feedparser: was copied inside https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/util/sanitize_html.py , removed in Python3: evict sgmllib and BeautifulSoup #5642
• mercurial: https://www.mercurial-scm.org/wiki/Python3, we should use the command line interface through subprocess instead of using the Mercurial API Update Mercurial to Py3-compatible v5.2, and all other deps #9065
• NoseHTML: PR #Add Python3 and Unicode support nosehtml#1
• paste
• pulsar: Cannot install pulsar-galaxy-lib under Python3 pulsar#152, galaxyproject/pulsar@5b663c3, Create universal wheels pulsar#156, Python3: Update pulsar-galaxy-lib dependency to 0.8.3 #5494
• pygithub3: used only in scripts/bootstrap_history.py, should be replaced with PyGithub
• sgmllib: removed in Python3: evict sgmllib and BeautifulSoup #5642
• SVGFig: issue Eliminate or Replace SVGFig #1717, PR Swapping from svgfig to svgwrite #1747
• twill: branch 2.0 of Cito's fork supports Python >= 3.3, then Import pyparsing in testing so twill's older version doesn't get loaded. #2656 can be reverted. Or better: replace with selenium tests
• WebError: unmaintained, even the latest 0.13.1 release doesn't support Python 3 and there is no interest in adding support, see e.g. Add Python3 compatibility via future package Pylons/weberror#18 . It could be replaced with Werkzeug, as suggested in Downgrade WebError.  #1017 (weberror removed in Eliminate interactive debugging middleware #8877)
• WebHelpers: unmaintained, removed in Various Python3 related fixes #5359

Tips

• Use

2to3 -w -n FILE

to port a file to Python3, then revert unnecessary changes and use six for anything that breaks Python2.7

• If 2to3 suggests to enclose a call to the keys(), values() or items() methods of a dict inside list( ), it may be unnecessary, e.g. if the dict is just iterated over a loop. But if items are added or deleted from the dict (changing its size) inside the loop, then making a copy of it with list( ) is necessary.
• Add

from __future__ import print_function

at the top of a file when changing print to print() in it.

• Substitute UserDict.DictMixin with collections.(Mutable)Mapping and add missing methods to the class inheriting from it
• Substitute string.letters, string.lowercase and string.uppercase with string.ascii_letters, string.ascii_lowercase and string.ascii_uppercase respectively
• Change calls to string.maketrans to:

try:
    maketrans = str.maketrans
except AttributeError:
    from string import maketrans

• Change calls to pipes.quote() to six.moves.shlex_quote()
• Change calls to filter() and map(lambda ...) to list comprehensions or generator expressions
• If d is a dict, change:

d.keys()[0]
d.values()[0]
d.items()[0]

to:

next(iter(d.keys()))
next(iter(d.values()))
next(iter(d.items()))

• If a class overrides the __nonzero__() method, this should be renamed to __bool__() and after its definition, the following alias should be added:

    __nonzero__ = __bool__

• if the long type is needed, add the following line after the imports:

if sys.version_info > (3,):
    long = int

• Since Python 3.7 async and await are keywords, variables with these names need to be renamed.
• In Python 3 the return value from subprocess.check_output() is a bytestring. If this value is going to be processed as a string, it needs to be decoded.

[jmchilton]

wchartype is just one file with 200 lines it seems, we should probably just bring that into util and update from Python 3

[mvdbeek]

sanitizie_html.py is mostly copied from feedparser and uses sgmllib that is not part of python 3 anymore.
We could externalize this dependency and make sure it works well in python3.
Currently there are some hickups with sgmllib import, that should be easy to fix.

[natefoo]

@dannon is getting rid of the SVGFig dependency in #1747.

[natefoo]

Mercurial is used in Galaxy for tool installations and non-installable-revision updates.

[nsoranzo]

@natefoo Feel free to update the issue description for anything missing/wrong!

Mercurial is used in Galaxy for tool installations and non-installable-revision updates.

Do you think we can get rid of it or use python-hglib?

[natefoo]

Or hgapi, possibly. Some parts will be easier than others. There is some pretty low level stuff going on for example here, that would need to be reimplemented.

[hexylena]

Mercurial is used in Galaxy for tool installations and non-installable-revision updates.

Given that the hg ts api was deprecated in 15.10, can we please get rid of it instead of re-implementing it?

[natefoo]

hg push to a TS was deprecated. TS repo versioning and backend format remains hg, as is the install method (clone) and Galaxy-side on-disk updating method (unless I am mistaken).

[natefoo]

That said, I am in favor of removing any Galaxy-side hg installation stuff and not opposed to changing the TS repos to another format (but that would be a lot of work).

[hexylena]

@natefoo sorry for nitpicking, but

The direct access to Tool Shed repositories through the Mercurial API is deprecated

sounds like there should be zero access to the TS HG api, whether push or pull.

TS repo versioning and backend format remains hg, as is the install method (clone) and Galaxy-side on-disk updating method (unless I am mistaken).

yeah, we aren't going to change the backend format any time soon for TS1. If there's any way we can look at fixing the clone + on-disk update method with tarballs or similar as an interim fix...

[natefoo]

Perhaps a poorly worded notification, unless I'm wrong and that was really the intent. But if it was, there's no replacement for installs at this time. @martenson ?

[hexylena]

@natefoo I'm sure you're right, I'm just reading it too hopefully closely

[yhoogstrate]

There is a small problem when hg push will be entirely disabled.

When you create a .tar.gz archive of a file with chmod 444 and you upload it in the tool shed, galaxy can not overwrite the read only file anymore (in an upcoming revision). The only way to overwrite or remove such file is by using mercurial.

[natefoo]

@yhoogstrate That's a bug. =)

[nsoranzo]

Indeed, @yhoogstrate can you open a separate issue with all the details to reproduce?

[wm75]

I've seen it, thanks a lot! Got my first Python3.7-powered Galaxy up and running just seconds ago 👍 🥇

[wm75]

Think I'll raise an issue over at pysam though that it would be great if they could provide a whl for 0.15.1 so that you don't have to have devel libs installed to build it yourself.

[natefoo]

Should we open individual issues for Python 3 bugs? I got this when attempting to install a tool from the TS:

Traceback (most recent call last):
  File "lib/galaxy/web/framework/middleware/error.py", line 154, in __call__
    app_iter = self.application(environ, sr_checker)
  File "/home/nate/work/galaxy2/.venv/lib/python3.5/site-packages/paste/recursive.py", line 85, in __call__
    return self.application(environ, start_response)
  File "/home/nate/work/galaxy2/.venv/lib/python3.5/site-packages/paste/httpexceptions.py", line 640, in __call__
    return self.application(environ, start_response)
  File "lib/galaxy/web/framework/base.py", line 143, in __call__
    return self.handle_request(environ, start_response)
  File "lib/galaxy/web/framework/base.py", line 222, in handle_request
    body = method(trans, **kwargs)
  File "lib/galaxy/web/framework/decorators.py", line 101, in decorator
    return func(self, trans, *args, **kwargs)
  File "lib/galaxy/webapps/galaxy/controllers/admin_toolshed.py", line 892, in prepare_for_install
    repo_information_dict = json.loads(raw_text)
  File "/usr/lib/python3.5/json/__init__.py", line 312, in loads
    s.__class__.__name__))
TypeError: the JSON object must be str, not 'bytes'

[martenson]

I think so, maybe even with own github label.

…

On Wed, Apr 17, 2019, 16:18 Nate Coraor ***@***.***> wrote: Should we open individual issues for Python 3 bugs? I got this when attempting to install a tool from the TS: Traceback (most recent call last): File "lib/galaxy/web/framework/middleware/error.py", line 154, in __call__ app_iter = self.application(environ, sr_checker) File "/home/nate/work/galaxy2/.venv/lib/python3.5/site-packages/paste/recursive.py", line 85, in __call__ return self.application(environ, start_response) File "/home/nate/work/galaxy2/.venv/lib/python3.5/site-packages/paste/httpexceptions.py", line 640, in __call__ return self.application(environ, start_response) File "lib/galaxy/web/framework/base.py", line 143, in __call__ return self.handle_request(environ, start_response) File "lib/galaxy/web/framework/base.py", line 222, in handle_request body = method(trans, **kwargs) File "lib/galaxy/web/framework/decorators.py", line 101, in decorator return func(self, trans, *args, **kwargs) File "lib/galaxy/webapps/galaxy/controllers/admin_toolshed.py", line 892, in prepare_for_install repo_information_dict = json.loads(raw_text) File "/usr/lib/python3.5/json/__init__.py", line 312, in loads s.__class__.__name__))TypeError: the JSON object must be str, not 'bytes' — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1715 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAN3DKVPNJ4FPFMTFWJMH3DPQ6A2RANCNFSM4B3A7SKA> .

[nsoranzo]

We are now using the area/python3 label to tag Python3-related issues/PRs instead of cross-referencing this issue.

[stain]

If I am making a new library for potential future use by Galaxy (say for exporting workspace as RO-Crate) - am I OK to do this in Python 3 or do I need to make it backwards compatible to Python 2? From the progress in here it sound like I can dare to be modern..! :)

[mvdbeek]

IMO go for python 3. We will have a deprecation notice in the upcoming release and 19.09 should be the last release that officially supports python 2.

[martenson]

Congrats @nsoranzo, this is an accomplishment spearheaded by you that many people will benefit from. Thank you!

[bgruening]

Indeed, what a huge effort. I remember when this was started at GCC in Norwich and then later at ISMB in Dublin.

Thanks a lot, @nsoranzo and everyone involved!