Do not write to /tmp when --tmpdir-prefix is set

[arunisaac]

Even when I set --tmpdir-prefix to a directory other than /tmp, toil-cwl-runner writes to /tmp. This can be quite bad on a HPC machine where /tmp is part of the root filesystem and writing to /tmp is frowned upon. For reference, here's my full toil command:

$ toil-cwl-runner --batchSystem single_machine --workDir work --tmpdir-prefix tmp-toil --tmp-outdir-prefix tmp-toil-out --outdir outputs/hornbeam-assemble/ --jobStore jobstore/hornbeam-assemble assemble.cwl hornbeam-assemble-inputs.yml

and here are the contents of the file toil-cwl-runner wrote to /tmp.

$ cat /tmp/tmp5gs4l2gp
toil.bus.JobUpdatedMessage	kind-CWLWorkflow/instance-gzk2lelm	0
toil.bus.JobIssuedMessage	assemble.cwl	kind-CWLWorkflow/instance-gzk2lelm	1
toil.bus.QueueSizeMessage	1
toil.bus.JobCompletedMessage	assemble.cwl	kind-CWLWorkflow/instance-gzk2lelm	0
toil.bus.ExternalBatchIdMessage	1	4925	SingleMachineBatchSystem
toil.bus.JobUpdatedMessage	kind-CWLWorkflow/instance-gzk2lelm	0
toil.bus.JobIssuedMessage	assemble.cwl.bwa-index._:249e87ff-7b2e-4209-8e3f-60f56bef1135	kind-CWLJob/instance-krz6hyqg	2
toil.bus.QueueSizeMessage	1
toil.bus.ExternalBatchIdMessage	2	4927	SingleMachineBatchSystem

Running TMPDIR=/some/other/path toil-cwl-runner ... works around this. But, this should not be necessary given the existence of the --tmpdir-prefix argument.

┆Issue is synchronized with this Jira Story
┆Issue Number: TOIL-1559

[adamnovak]

That looks like the message bus log file that the leader writes. It's part of the Toil core, while --tmpdir-prefix is part of only toil-cwl-runner.

It looks like toil-cwl-runner doesn't take --tmpdir-prefix and use it to change the temporary directory for the toil-cwl-runner Python process itself; it only uses it in the CWL "runtime context" and to pick a Toil work directory.

[arunisaac]

I see. That makes sense. Do we close this issue as a wontfix?