Job outputs from working directory seems to result in no outputs found

[hexylena]

We've set outputs_to_working_directory: true on our server, but it seems this results in different command lines when passed to pulsar, either /outputs or /working are used, but results are only searched in the outputs directory.

Is this a galaxy bug? Should we patch galaxy to not change command lines? Or is this a pulsar bug and pulsar should be aware of checking the working directories? (Would that require an update to all of our destinations?)

@gmauro could you please paste some of the command lines we saw? cc @bgruening for why it isn't working for today.

[gmauro]

The launch_config file from a Galaxy test instance (that works):

{"command_line": "set -o | grep -q pipefail && set -o pipefail;  ln -s '/data/share/staging//158/inputs/dataset_164.dat' 'localref.fa' && bwa index 'localref.fa' &&    bwa mem -t \"${GALAXY_SLOTS:-1}\" -v 1                 'localref.fa' '/data/share/staging//158/inputs/dataset_169.dat' '/data/share/staging//158/inputs/dataset_170.dat'  | samtools sort -@${GALAXY_SLOTS:-2} -O bam -o '/data/share/staging//158/outputs/dataset_203.dat'; return_code=$?; sh -c \"exit $return_code\"", "remote_staging": {"client_outputs": {"metadata_directory": "/srv/galaxy/jobs/000/158/metadata", "dynamic_outputs": ["primary_.*|galaxy.json|metadata_.*|dataset_\\d+\\.dat|__instrument_.*|dataset_\\d+_files.+"], "version_file": "/srv/galaxy/jobs/000/158/COMMAND_VERSION", "work_dir_outputs": [], "working_directory": "/srv/galaxy/jobs/000/158/working", "output_files": ["/data/000/dataset_203.dat"]}, "setup": [{"action": {"url": "http://galaxy.ufr.garinci.com/api/jobs/1343954bc98f8001/files?job_key=f0744a54b49ca61a&path=/data/000/dataset_164.dat&file_type=input", "path": "/data/000/dataset_164.dat", "action_type": "remote_transfer"}, "type": "input", "name": "dataset_164.dat"}, {"action": {"url": "http://galaxy.ufr.garinci.com/api/jobs/1343954bc98f8001/files?job_key=f0744a54b49ca61a&path=/data/000/dataset_169.dat&file_type=input", "path": "/data/000/dataset_169.dat", "action_type": "remote_transfer"}, "type": "input", "name": "dataset_169.dat"}, {"action": {"url": "http://galaxy.ufr.garinci.com/api/jobs/1343954bc98f8001/files?job_key=f0744a54b49ca61a&path=/data/000/dataset_170.dat&file_type=input", "path": "/data/000/dataset_170.dat", "action_type": "remote_transfer"}, "type": "input", "name": "dataset_170.dat"}], "ssh_key": null, "action_mapper": {"paths": [], "ssh_port": null, "ssh_key": null, "ssh_host": null, "ssh_user": null, "files_endpoint": "http://galaxy.ufr.garinci.com/api/jobs/1343954bc98f8001/files?job_key=f0744a54b49ca61a", "default_action": "remote_transfer"}}, "dependencies_description": {"requirements": [{"version": "0.7.17", "type": "package", "specs": [], "name": "bwa"}, {"version": "1.6", "type": "package", "specs": [], "name": "samtools"}], "installed_tool_dependencies": []}, "env": [{"raw": false, "execute": null, "value": "$(dirname ${BASH_SOURCE[0]})/_job_tmp", "file": null, "name": "TEMP"}, {"raw": false, "execute": null, "value": "$TEMP", "file": null, "name": "TMPDIR"}, {"raw": false, "execute": null, "value": "-Djava.io.tmpdir=$TEMP", "file": null, "name": "_JAVA_OPTIONS"}, {"raw": false, "execute": "mkdir -p $TEMP", "value": null, "file": null, "name": null}, {"raw": false, "execute": null, "value": "C", "file": null, "name": "LC_ALL"}], "setup_params": {"preserve_galaxy_python_environment": null, "tool_id": "toolshed.g2.bx.psu.edu/repos/devteam/bwa/bwa_mem/0.7.17.1", "tool_version": "0.7.17.1", "job_id": "158", "use_metadata": true}, "submit_params": {}}

the launch_config file from usegalaxy.eu for the same job:

{"command_line": "set -o | grep -q pipefail && set -o pipefail;  ln -s '/data/share/staging//5423825/inputs/dataset_9713686.dat' 'localref.fa' && bwa index 'localref.fa' &&    bwa mem -t \"${GALAXY_SLOTS:-1}\" -v 1                 'localref.fa' '/data/share/staging//5423825/inputs/dataset_9713679.dat' '/data/share/staging//5423825/inputs/dataset_9713681.dat'  | samtools sort -@${GALAXY_SLOTS:-2} -O bam -o '/data/share/staging//5423825/working/galaxy_dataset_10295811.dat'; return_code=$?; sh -c \"exit $return_code\"", "remote_staging": {"client_outputs": {"metadata_directory": "/data/dnb02/galaxy_db/job_working_directory/005/423/5423825", "dynamic_outputs": ["primary_.*|galaxy.json|metadata_.*|dataset_\\d+\\.dat|__instrument_.*|dataset_\\d+_files.+"], "version_file": "/data/dnb02/galaxy_db/job_working_directory/005/423/5423825/COMMAND_VERSION", "work_dir_outputs": [], "working_directory": "/data/dnb02/galaxy_db/job_working_directory/005/423/5423825/working", "output_files": ["/data/dnb02/galaxy_db/job_working_directory/005/423/5423825/galaxy_dataset_10295811.dat"]}, "setup": [{"action": {"url": "https://usegalaxy.eu/_job_files?job_id=a31222b5e4cbaa3a&job_key=d65efb2fec13d707&path=/data/dnb02/galaxy_db/files/009/713/dataset_9713681.dat&file_type=input", "path": "/data/dnb02/galaxy_db/files/009/713/dataset_9713681.dat", "action_type": "remote_transfer"}, "type": "input", "name": "dataset_9713681.dat"}, {"action": {"url": "https://usegalaxy.eu/_job_files?job_id=a31222b5e4cbaa3a&job_key=d65efb2fec13d707&path=/data/dnb02/galaxy_db/files/009/713/dataset_9713679.dat&file_type=input", "path": "/data/dnb02/galaxy_db/files/009/713/dataset_9713679.dat", "action_type": "remote_transfer"}, "type": "input", "name": "dataset_9713679.dat"}, {"action": {"url": "https://usegalaxy.eu/_job_files?job_id=a31222b5e4cbaa3a&job_key=d65efb2fec13d707&path=/data/dnb02/galaxy_db/files/009/713/dataset_9713686.dat&file_type=input", "path": "/data/dnb02/galaxy_db/files/009/713/dataset_9713686.dat", "action_type": "remote_transfer"}, "type": "input", "name": "dataset_9713686.dat"}], "ssh_key": null, "action_mapper": {"paths": [], "ssh_port": null, "ssh_key": null, "ssh_host": null, "ssh_user": null, "files_endpoint": "https://usegalaxy.eu/_job_files?job_id=a31222b5e4cbaa3a&job_key=d65efb2fec13d707", "default_action": "remote_transfer"}}, "dependencies_description": {"requirements": [{"version": "0.7.17", "type": "package", "specs": [], "name": "bwa"}, {"version": "1.6", "type": "package", "specs": [], "name": "samtools"}], "installed_tool_dependencies": []}, "env": [], "setup_params": {"preserve_galaxy_python_environment": null, "tool_id": "toolshed.g2.bx.psu.edu/repos/devteam/bwa/bwa_mem/0.7.17.1", "tool_version": "0.7.17.1", "job_id": "5423825", "use_metadata": true}, "submit_params": {}}

The key differences are:

<   "command_line": "set -o | \
    grep -q pipefail && set -o pipefail; 
    ln -s '/data/share/staging//158/inputs/dataset_164.dat' 'localref.fa' && \
    bwa index 'localref.fa' &&    \
    bwa mem -t \"${GALAXY_SLOTS:-1}\" -v 1                 'localref.fa' '/data/share/staging//158/inputs/dataset_169.dat' '/data/share/staging//158/inputs/dataset_170.dat'  | \
    samtools sort -@${GALAXY_SLOTS:-2} -O bam -o '/data/share/staging//158/outputs/dataset_203.dat'; 
return_code=$?; sh -c \"exit $return_code\"",
---
>   "command_line": "set -o | \
    grep -q pipefail && set -o pipefail;  
    ln -s '/data/share/staging//5423825/inputs/dataset_9713686.dat' 'localref.fa' && \
    bwa index 'localref.fa' &&    \
    bwa mem -t \"${GALAXY_SLOTS:-1}\" -v 1                 'localref.fa' '/data/share/staging//5423825/inputs/dataset_9713679.dat' '/data/share/staging//5423825/inputs/dataset_9713681.dat'  | \
    samtools sort -@${GALAXY_SLOTS:-2} -O bam -o '/data/share/staging//5423825/working/galaxy_dataset_10295811.dat'; 
return_code=$?; sh -c \"exit $return_code\"",

[hexylena]

@Slugger70 @natefoo guessing neither of you set outputs_to_working_directory? Any ideas on this @jmchilton? (sorry know you're super busy.)

[Slugger70]

Hey @erasche, no we don't set outputs_to_working_directory. Soz...

[natefoo]

This needs to get fixed but as a temporary workaround you can just set (job_conf.yml):

execution:
  environments:
    pulsar:
      runner: pulsar_mq
      outputs_to_working_directory: false
      ... other options

or (job_conf.xml):

    <destinations>
        <destination id="pulsar" runner="pulsar_mq">
            <param id="outputs_to_working_directory">False</param>
            ... other params
        </destination>
    </destinations>

[martenson]

After trying to debug this again with @gmauro I'd like to capture the following here:

On usegalaxy.eu using the CZ pulsar node with outputs_to_working_directory: true defined in this playbookI am seeing certain tools to fail with message like:

Job 45770833's output dataset(s) could not be read

E.g. every run of toolshed.g2.bx.psu.edu/repos/devteam/samtools_sort/samtools_sort/2.0.4 fails, but every run of toolshed.g2.bx.psu.edu/repos/iuc/samtools_view/samtools_view/1.9+galaxy3 is fine.

The output file is there, in the working directory, but pulsar/galaxy never finds it for samtools_sort.

/pulsar/files/staging/45770833$ ll working/
total 8.0K
drwxr-xr-x 2 meta  109 Apr 22 09:30 ./
drwxr-xr-x 9 meta 4.0K Apr 22 09:57 ../
-rw------- 1 meta 1003 Apr 22 09:30 galaxy_dataset_d8c84413-7b1d-4738-9da4-33be8d0a5c67.dat

When the settings wasoutputs_to_working_directory: false I get the following from samtools_view run:

Exception: Failed to remove None from ['/data/dnb05/galaxy_db/files/d/1/a/dataset_d1afe842-887e-4431-849e-ff0f3ac6db12.dat']

So at the moment, I am not aware of configuration/setup where we can have one Pulsar destination execute both "kinds" of tools -- the ones that require false and ones that require true.

p.s. is this related to #285 ?

[mvdbeek]

So, outputs_to_working directory doesn't make sense in pulsar, and we've discussed disabling that automatically, so that option I don't think we should investigate. What commit of Galaxy and what pulsar version is this ?

[mvdbeek]

And what context is

Exception: Failed to remove None from ['/data/dnb05/galaxy_db/files/d/1/a/dataset_d1afe842-887e-4431-849e-ff0f3ac6db12.dat']

in ?

[martenson]

This is 0.14.11 Pulsar and galaxy.eu runs {"version_major": "21.09", "version_minor": "1.dev0"} @gmauro can you give the commit please? I am not sure where to look for it.

Failed to remove None from... is from Pulsar log when outputs_to_working_directory: false (after switching to true, the same tool worked afaik)

[mvdbeek]

Alright, to answer my question, that's in pulsar. The output definition for samtools view is this:

    <outputs>
        <data name="outputsam" format_source="input" from_work_dir="outfile" label="${tool.name} on ${on_string}: filtered alignments">
            <filter>
                mode['outtype'] == 'header' or mode['output_options']['reads_report_type'] != 'count'
            </filter>
            <change_format>
                <when input="mode.output_options.output_format.oformat" value="sam" format="sam"/>
                <when input="mode.output_options.output_format.oformat" value="bam" format="bam"/>
                <when input="mode.output_options.output_format.oformat" value="cram" format="cram"/>
            </change_format>
        </data>
        <data name="invoutputsam" format_source="input" from_work_dir="inv_outfile" label="${tool.name} on ${on_string}: unfiltered alignments">
            <filter>
                mode['outtype'] == 'selected_reads' and mode['output_options']['reads_report_type'] != 'count' and mode['output_options']['complementary_output']
            </filter>
            <change_format>
                <when input="mode.output_options.output_format.oformat" value="sam" format="sam"/>
                <when input="mode.output_options.output_format.oformat" value="bam" format="bam"/>
                <when input="mode.output_options.output_format.oformat" value="cram" format="cram"/>
            </change_format>
        </data>
        <data name="outputcnt" format="tabular" from_work_dir="outfile" label="${tool.name} on ${on_string}: Counts">
            <filter>
                mode['outtype'] != 'header' and mode['output_options']['reads_report_type'] == 'count'
            </filter>
        </data>
    </outputs>

I wouldn't be surprised if this happens when the output is filtered and not going to be produced.

[mvdbeek]

IOW I think that's probably a different issue, but this shouldn't be hard to fix

[gmauro]

eu is at this commit galaxyproject/galaxy@7aae40d

[mvdbeek]

Just to keep this updated, there was a bug in Galaxy that would prevent outputs_to_working_directory: false from being correctly parsed that caused the Exception: Failed to remove None from ['/data/dnb05/galaxy_db/files/d/1/a/dataset_d1afe842-887e-4431-849e-ff0f3ac6db12.dat'], that is fixed in galaxyproject/galaxy#15927.