Restart kills preprocessing jobs

[cat-bro]

I restarted pulsar while it was staging in a large file (80GB+) over multiple preprocessing retries. The job has now failed because every subsequent postprocessing attempt resulted in a 416 error.

Mar 25 00:17:10 qld-pulsar-himem-0 pulsar[1821739]: Traceback (most recent call last):
Mar 25 00:17:10 qld-pulsar-himem-0 pulsar[1821739]:   File "/mnt/pulsar/venv/lib/python3.8/site-packages/pulsar/managers/util/retry.py", line 93, in _retry_over_time
Mar 25 00:17:10 qld-pulsar-himem-0 pulsar[1821739]:     return fun(*args, **kwargs)
Mar 25 00:17:10 qld-pulsar-himem-0 pulsar[1821739]:   File "/mnt/pulsar/venv/lib/python3.8/site-packages/pulsar/managers/staging/pre.py", line 20, in <lambda>
Mar 25 00:17:10 qld-pulsar-himem-0 pulsar[1821739]:     action_executor.execute(lambda: action.write_to_path(path), "action[%s]" % description)
Mar 25 00:17:10 qld-pulsar-himem-0 pulsar[1821739]:   File "/mnt/pulsar/venv/lib/python3.8/site-packages/pulsar/client/action_mapper.py", line 479, in write_to_path
Mar 25 00:17:10 qld-pulsar-himem-0 pulsar[1821739]:     get_file(self.url, path)
Mar 25 00:17:10 qld-pulsar-himem-0 pulsar[1821739]:   File "/mnt/pulsar/venv/lib/python3.8/site-packages/pulsar/client/transport/curl.py", line 99, in get_file
Mar 25 00:17:10 qld-pulsar-himem-0 pulsar[1821739]:     raise Exception(message)
Mar 25 00:17:10 qld-pulsar-himem-0 pulsar[1821739]: Exception: Failed to get_file properly for url https://galaxy.usegalaxy.org.au/_job_files?job_id=ad33365a835c74b0&job_key=6fe5ee7de1cd708b&path=%2Fmnt%2Ftmp%2Fjob_working_directory%2F008%2F466%2F8466449%2Ftool_script.sh&file_type=jobdir, remote server returned status code of 416

[mvdbeek]

The line numbers point at a pulsar version prior to #348, which should have fixed that. 0.15.6 would be the pulsar version you want to run.