temp dir not respected with http retrieval

[jjkoehorst]

When on a Mac I was having an issue with running a cwl file and workflow. Using the command cwltool --tmpdir-prefix HERE tests/longread_quality/no_kraken.yaml the tmpdir-prefix was not really respected. After setting the TMPDIR=$(pwd) the workflow executed as expected.

INFO [job prepare_fasta_db] /Volumes/Git/m-unlock/cwl/HEREw4j3gf4y$ docker \
    run \
    -i \
    --mount=type=bind,source=/Volumes/Git/m-unlock/cwl/HEREw4j3gf4y,target=/VdqFTV \
    --mount=type=bind,source=/Volumes/Git/m-unlock/cwl/HEREvh_b36xh,target=/tmp \
    --mount=type=bind,source=/var/folders/jt/8knvb4rn6zzb5pn7247hc8gh0000gn/T/tmpo46ugyzl,target=/var/lib/cwl/stgb6cb00fb-2b24-4d95-8157-86b1142b1f73/human_small.fa.gz,readonly \
    --workdir=/VdqFTV \
    --read-only=true \
    --net=none \
    --user=501:20 \
    --rm \
    --cidfile=/Volumes/Git/m-unlock/cwl/HERE_2j7v0_i/20230517161845-734875.cid \
    --env=TMPDIR=/tmp \
    --env=HOME=/VdqFTV \
    docker-registry.wur.nl/m-unlock/docker/python:3.10.6 \
    bash \
    script.sh \
    TEST_filter-reference_uniq.fa.gz \
    /var/lib/cwl/stgb6cb00fb-2b24-4d95-8157-86b1142b1f73/human_small.fa.gz
docker: Error response from daemon: invalid mount config for type "bind": bind source path does not exist: /var/folders/jt/8knvb4rn6zzb5pn7247hc8gh0000gn/T/tmpo46ugyzl.

With setting the TMPDIR:

INFO [job prepare_fasta_db] /Volumes/Git/m-unlock/cwl/HEREbl3rmhat$ docker \
    run \
    -i \
    --mount=type=bind,source=/Volumes/Git/m-unlock/cwl/HEREbl3rmhat,target=/IYEHYT \
    --mount=type=bind,source=/Volumes/Git/m-unlock/cwl/HEREwtdc_zny,target=/tmp \
    --mount=type=bind,source=/Volumes/Git/m-unlock/cwl/tmpcwvmc_0k,target=/var/lib/cwl/stg85eb3423-a1e0-4673-85e9-2b4cba9c4643/human_small.fa.gz,readonly \
    --workdir=/IYEHYT \
    --read-only=true \
    --net=none \
    --user=501:20 \
    --rm \
    --cidfile=/Volumes/Git/m-unlock/cwl/HERE5kbw8bqn/20230517162126-434296.cid \
    --env=TMPDIR=/tmp \
    --env=HOME=/IYEHYT \
    docker-registry.wur.nl/m-unlock/docker/python:3.10.6 \
    bash \
    script.sh \
    TEST_filter-reference_uniq.fa.gz \
    /var/lib/cwl/stg85eb3423-a1e0-4673-85e9-2b4cba9c4643/human_small.fa.gz
/var/lib/cwl/stg85eb3423-a1e0-4673-85e9-2b4cba9c4643/human_small.fa.gz: gzip compressed data, was "KI270707.1_TEST.fa", last modified: Fri May 21 12:29:32 2021, from Unix, original size 32533
INFO [job prepare_fasta_db] Max memory used: 0MiB
INFO [job prepare_fasta_db] completed success

Expected Behavior

When using the --tmpdir-prefix HERE it should respect the tmpdir prefix.

Actual Behavior

Not all elements in cwltool respect the tmpdir prefix

Workflow Code

cwl:tool: ../../workflows/workflow_longread_quality.cwl
identifier: TEST
readtype: Nanopore
fastq_rich: true
threads: 6
memory: 4000
longreads:
   - class: File
     location: http://download.systemsbiology.nl/unlock/cwl/test_data/ont_guppy_reads.fq.gz
filter_references:
   - class: File
     location: http://download.systemsbiology.nl/unlock/cwl/test_data/human_small.fa.gz

Your Environment

• cwltool version: /Users/jasperk/mambaforge/bin/cwltool 3.1.20230513155734