Upgrade to Celery 4 or higher

[hackdna]

Celery version used by Refinery (v1.6.9) is 3.1.20 (released on 2016-01-22).
Celery 4.3.0 is the latest stable version currently (released on 2019-03-31).

From Celery docs: "Please use Celery 3.1 for versions older than Django 1.8."

https://docs.celeryproject.org/en/v4.0.0/whatsnew-4.0.html

Requires:

• replacing django-celery (Replace django-celery #872) with django-celery-results or memcached
• switching the codebase to the new (object-oriented) Celery API (Switch to the new Celery API #3319)
• addressing other deprecations (Address all current Celery code deprecations #2933, Address all current Celery configuration deprecations #3326)

See also: #2416 and #2614

There are known issues with django-celery, so it should probably be removed:
https://stackoverflow.com/questions/31504591/interfaceerror-connection-already-closed-using-django-celery-scrapy
https://tryolabs.com/blog/2014/02/12/long-running-process-and-django-orm/

[ilan-gold]

@hackdna I have brought us up to celery 4 (https://github.com/refinery-platform/refinery-platform/tree/hackdna/celery4-upgrade) but there appears to be some serious issues with the database backend and GroupResult (which is necessary, as things stand):

celery/django-celery-results#40
celery/django-celery-results#52

We can try Django's cache, but I marked/tested the point at which we were still in celery3 with a full API overhaul (it might not be perfect, but I was able to run analyses/import files): https://github.com/refinery-platform/refinery-platform/tree/c5273ac5ae63ebf9052e91aff42d6e764139b919

We should discuss this offline, especially if you have some experience with this that I do not. An option could potentially be just using the AnsyncResult directly

I was not planning on merging this anyway until after I refactored Celery (and certainly after it was working) but with the open pull requests, didn't want to leap ahead to refactoring without those merged and so worked on this. I think looking at this with fresh eyes might help when I come back to this after the refactoring.

From Travis:

=====================================================================

ERROR: test__run_galaxy_file_import_failure (tool_manager.tests.WorkflowToolLaunchTests)

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/home/travis/virtualenv/python2.7.15/lib/python2.7/site-packages/mock/mock.py", line 1305, in patched

    return func(*args, **keywargs)

  File "/home/travis/build/refinery-platform/refinery-platform/refinery/tool_manager/tests.py", line 1910, in test__run_galaxy_file_import_failure

    _run_galaxy_file_import(self.tool.analysis.uuid)

  File "/home/travis/build/refinery-platform/refinery-platform/refinery/analysis_manager/tasks.py", line 352, in _run_galaxy_file_import

    str(analysis_status.refinery_import_task_group_id)

  File "/home/travis/virtualenv/python2.7.15/lib/python2.7/site-packages/celery/result.py", line 839, in delete

    (backend or self.app.backend).delete_group(self.id)

  File "/home/travis/virtualenv/python2.7.15/lib/python2.7/site-packages/celery/backends/base.py", line 382, in delete_group

    return self._delete_group(group_id)

AttributeError: 'DatabaseBackend' object has no attribute '_delete_group'

The cache backend is not much better but appears to have less of a deep-seeded problem:

Traceback (most recent call last):
  File "/home/vagrant/.virtualenvs/refinery-platform/local/lib/python2.7/site-packages/celery/app/trace.py", line 375, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/vagrant/.virtualenvs/refinery-platform/local/lib/python2.7/site-packages/celery/app/trace.py", line 632, in __protected_call__
    return self.run(*args, **kwargs)
  File "/vagrant/refinery/analysis_manager/tasks.py", line 303, in run_analysis
    _refinery_file_import(analysis_uuid)
  File "/vagrant/refinery/analysis_manager/tasks.py", line 271, in _refinery_file_import
    str(analysis_status.refinery_import_task_group_id)
  File "/vagrant/refinery/analysis_manager/tasks.py", line 135, in get_group_result
    return GroupResult.restore(task_group_id)
  File "/home/vagrant/.virtualenvs/refinery-platform/local/lib/python2.7/site-packages/celery/result.py", line 875, in restore
    backend = backend or (app.backend if app else current_app.backend)
AttributeError: 'property' object has no attribute 'backend'