Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

No Output when using snakemake on slurm #2

Open
linkmonica opened this issue Nov 7, 2019 · 7 comments
Open

No Output when using snakemake on slurm #2

linkmonica opened this issue Nov 7, 2019 · 7 comments

Comments

@linkmonica
Copy link

Hi,
I am trying to run scan-snv for whole chromosome in slurm cluster, I don't get any error or output. The config.yaml seems to be correct and coreXXX binary file is generated, and I don't see any error in ./output_dir/.snakemake./log.

Can you please let me know what can be wrong that no output is generated, demo works fine but as soon as I want to do run it in parallel.

Command:

module load anaconda3
conda activate scansnv
module load slurm-drmaa/1.2.0-dev.deca826

scansnv \
    --ref ./refs/b37_human_g1k_v37_decoy.fasta \
    --dbsnp ./refs/b37_dbsnp_138.b37.vcf \
    --shapeit-panel ./refs/1000GP_Phase3 \
    --output-dir test \
    --regions-file regions.bed \
    --bam Bulk Bulk.bam \
    --bam sc1 sc1.bam \
    --bam sc2 sc2.bam \
    --bam sc3 sc3.bam \
    --sc-sample sc1 \
    --sc-sample sc2 \
    --sc-sample sc3 \
    --bulk-sample Bulk \
    --abmodel-chunks 1 \
    --abmodel-samples-per-chunk 10000 \
    --abmodel-hsnp-chunk-size 50 \
    --hsnp-spikein-replicates 5 \
    --joblimit 10 --resume \
    --drmaa ' -p thin-shared -t 01:00:00 --cpus-per-task 1 --mem 5G'

Thanks in advance!
Monica.

@jluquette
Copy link
Member

Hi @linkmonica,

  1. Are you able to run the demo files on your system without using SLURM?
  2. Assuming (1) is true, do the demo files then also work with SLURM?

If (1) is true but (2) is not, my guess is the failure is happening in DRMAA. (This is consistent with the core dump file.) To use Snakemake on the SLURM cluster at my institution, I had to use a non-standard library for the SLURM-DRMAA interface (https://anaconda.org/jluquette/natefoo-slurm-drmaa). I believe this should be installed by default with SCAN-SNV, although I have no idea if it is portable. Could you try running without using module load slurm-drmaa/1.2.0-dev.deca826?

@linkmonica
Copy link
Author

Hi @jluquette
I tried by commenting module load slurm-drmaa/1.2.0-dev.deca826 and now I get different error:

Traceback (most recent call last):
File "path/to/anaconda3/2019.07/envs/scansnv/bin/scansnv", line 332, in
print(' '.join(snakemake_command))
TypeError: sequence item 6: expected str instance, int found

I am also trying to install SLURM-DRMAA interface (https://anaconda.org/jluquette/natefoo-slurm-drmaa) but no luck!
Any help will be appreciated, Thanks

@jluquette
Copy link
Member

Ok, thanks for the bug report. I just updated the script to fix that error. Could you clone the newest version of the private branch to see if it works for you?

Re: SLURM-DRMAA: I think this should be installed by default. Can you post the output of
conda list -n scansnv
where "scansnv" is whatever you named your conda environment?

@linkmonica
Copy link
Author

Output shows python-drmaa installed but not slurm-drmaa and I tried installing it from SLURM-DRMAA interface (https://anaconda.org/jluquette/natefoo-slurm-drmaa) but no luck!

conda list -n scansnv

# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
_r-mutex                  1.0.0               anacondar_1  
aioeasywebdav             2.2.0                    py36_0    conda-forge/label/cf201901
aiohttp                   3.4.4            py36h470a237_0    conda-forge/label/cf201901
anaconda-client           1.7.1                      py_0    conda-forge/label/cf201901
appdirs                   1.4.3                      py_1    conda-forge/label/cf201901
asn1crypto                0.24.0                py36_1003    conda-forge/label/cf201901
async-timeout             3.0.1                   py_1000    conda-forge/label/cf201901
attrs                     18.2.0                     py_0    conda-forge/label/cf201901
bcrypt                    3.1.4            py36h470a237_0    conda-forge/label/cf201901
beautifulsoup4            4.6.3                 py36_1000    conda-forge/label/cf201901
bedtools                  2.27.1               he941832_2    bioconda
blas                      1.0                         mkl  
boost                     1.66.0                   py36_1    conda-forge/label/cf201901
boost-cpp                 1.66.0                        1    conda-forge/label/cf201901
boto3                     1.7.84                     py_0    conda-forge/label/cf201901
botocore                  1.10.84                    py_0    conda-forge/label/cf201901
bwidget                   1.9.11                        1  
bzip2                     1.0.6                h470a237_2    conda-forge/label/cf201901
ca-certificates           2018.11.29           ha4d7672_0    conda-forge/label/cf201901
cachetools                2.1.0                      py_0    conda-forge/label/cf201901
cairo                     1.14.12              h276e583_5    conda-forge/label/cf201901
certifi                   2018.11.29            py36_1000    conda-forge/label/cf201901
cffi                      1.11.5           py36h5e8e0c9_1    conda-forge/label/cf201901
chardet                   3.0.4                 py36_1003    conda-forge/label/cf201901
clyent                    1.2.2                      py_1    conda-forge/label/cf201901
conda                     4.5.12                py36_1000    conda-forge/label/cf201901
conda-build               3.17.5                   py36_0    conda-forge/label/cf201901
conda-env                 2.6.0                         1    conda-forge/label/cf201901
configargparse            0.13.0                     py_1    conda-forge/label/cf201901
cryptography              2.3.1            py36hdffb7b8_0    conda-forge/label/cf201901
cryptography-vectors      2.3.1                 py36_1000    conda-forge/label/cf201901
curl                      7.63.0               h74213dd_0    conda-forge/label/cf201901
datrie                    0.7.1            py36h7b6447c_1  
decorator                 4.3.0                      py_0    conda-forge/label/cf201901
docutils                  0.14                  py36_1001    conda-forge/label/cf201901
dropbox                   7.3.1                    py36_0    conda-forge/label/cf201901
expat                     2.2.5                hfc679d8_2    conda-forge/label/cf201901
filechunkio               1.6                      py36_0    bioconda
filelock                  3.0.10                     py_0    conda-forge/label/cf201901
fontconfig                2.13.1               h65d0f4c_0    conda-forge/label/cf201901
freetype                  2.9.1                h6debe1e_4    conda-forge/label/cf201901
ftputil                   3.2                      py36_0    bioconda
gatk                      3.8                      py36_0    bioconda
gettext                   0.19.8.1             h5e8e0c9_1    conda-forge/label/cf201901
gitdb2                    2.0.5                      py_0    conda-forge/label/cf201901
gitpython                 2.1.11                     py_0    conda-forge/label/cf201901
glib                      2.56.2               h464dc38_1    conda-forge/label/cf201901
glob2                     0.6                        py_0    conda-forge/label/cf201901
google-auth               1.2.1                      py_0    conda-forge/label/cf201901
google-auth-httplib2      0.0.3                      py_2    conda-forge/label/cf201901
google-cloud-core         0.24.1                   py36_0    conda-forge/label/cf201901
google-cloud-storage      1.1.1                    py36_0    conda-forge/label/cf201901
google-resumable-media    0.0.2                    py36_0    conda-forge/label/cf201901
googleapis-common-protos  1.5.5                      py_0    conda-forge/label/cf201901
graphite2                 1.3.12               hfc679d8_1    conda-forge/label/cf201901
graphviz                  2.38.0               h08bfae6_9    conda-forge/label/cf201901
gsl                       2.2.1                h0c605f7_3  
harfbuzz                  1.9.0                h04dbb29_1    conda-forge/label/cf201901
httplib2                  0.12.0                py36_1000    conda-forge/label/cf201901
icu                       58.2                 hfc679d8_0    conda-forge/label/cf201901
idna                      2.8                   py36_1000    conda-forge/label/cf201901
idna_ssl                  1.0.0                         0    conda-forge/label/cf201901
intel-openmp              2019.1                      144  
ipython_genutils          0.2.0                      py_1    conda-forge/label/cf201901
jinja2                    2.10                       py_1    conda-forge/label/cf201901
jmespath                  0.9.3                      py_1    conda-forge/label/cf201901
jpeg                      9c                   h470a237_1    conda-forge/label/cf201901
jsonschema                3.0.0a3               py36_1000    conda-forge/label/cf201901
jupyter_core              4.4.0                      py_0    conda-forge/label/cf201901
krb5                      1.16.2               hbb41f41_0    conda-forge/label/cf201901
libarchive                3.3.3                h823be47_0    conda-forge/label/cf201901
libcurl                   7.63.0               hbdb9355_0    conda-forge/label/cf201901
libdeflate                1.0                  h470a237_0    bioconda
libedit                   3.1.20170329         h6b74fdf_2  
libffi                    3.2.1                hd88cf55_4  
libgcc-ng                 8.2.0                hdf63c60_1  
libgfortran               3.0.0                         1    conda-forge/label/cf201901
libgfortran-ng            7.2.0                hdf63c60_3    conda-forge/label/cf201901
libiconv                  1.15                 h470a237_3    conda-forge/label/cf201901
liblief                   0.9.0                h7725739_1  
libpng                    1.6.36               ha92aebf_0    conda-forge/label/cf201901
libprotobuf               3.6.1                hd28b015_0    conda-forge/label/cf201901
libssh2                   1.8.0                h5b517e9_3    conda-forge/label/cf201901
libstdcxx-ng              8.2.0                hdf63c60_1  
libtiff                   4.0.10               he6b73bb_1    conda-forge/label/cf201901
libtool                   2.4.6                h470a237_2    conda-forge/label/cf201901
libuuid                   2.32.1               h470a237_2    conda-forge/label/cf201901
libxcb                    1.13                 h470a237_2    conda-forge/label/cf201901
libxml2                   2.9.8                h422b904_5    conda-forge/label/cf201901
make                      4.2.1             h470a237_1002    conda-forge/label/cf201901
markupsafe                1.1.0            py36h470a237_0    conda-forge/label/cf201901
mkl                       2019.1                      144  
mkl_fft                   1.0.10                   py36_0    conda-forge/label/cf201901
mkl_random                1.0.2                    py36_0    conda-forge/label/cf201901
multidict                 4.5.1            py36h470a237_0    conda-forge/label/cf201901
natefoo-slurm-drmaa       1.2.0_dev_deca826               0    jluquette
nbformat                  4.4.0                      py_1    conda-forge/label/cf201901
ncurses                   6.1                  hf484d3e_0  
networkx                  2.2                        py_1    conda-forge/label/cf201901
numpy                     1.15.4           py36h7e9f1db_0  
numpy-base                1.15.4           py36hde5b4d6_0  
openblas                  0.3.4                ha44fe06_0    conda-forge/label/cf201901
openjdk                   8.0.192              h470a237_2    conda-forge/label/cf201901
openssl                   1.0.2p               h470a237_1    conda-forge/label/cf201901
pandas                    0.23.4           py36hf8a1672_0    conda-forge/label/cf201901
pango                     1.40.14              he752989_2    conda-forge/label/cf201901
paramiko                  2.4.2                 py36_1000    conda-forge/label/cf201901
patchelf                  0.9                  hfc679d8_2    conda-forge/label/cf201901
pcre                      8.41                 hfc679d8_3    conda-forge/label/cf201901
pip                       18.1                  py36_1000    conda-forge/label/cf201901
pixman                    0.34.0               h470a237_3    conda-forge/label/cf201901
pkginfo                   1.4.2                      py_1    conda-forge/label/cf201901
prettytable               0.7.2                      py_2    conda-forge/label/cf201901
protobuf                  3.6.1            py36hfc679d8_1    conda-forge/label/cf201901
psutil                    5.4.8            py36h470a237_0    conda-forge/label/cf201901
pthread-stubs             0.4                  h470a237_1    conda-forge/label/cf201901
py-lief                   0.9.0            py36h7725739_1  
pyasn1                    0.4.4                      py_1    conda-forge/label/cf201901
pyasn1-modules            0.0.5                    py36_0    bioconda
pycosat                   0.6.3            py36h470a237_1    conda-forge/label/cf201901
pycparser                 2.19                       py_0    conda-forge/label/cf201901
pygraphviz                1.3.1                    py36_0    bioconda
pynacl                    1.3.0            py36h470a237_0    conda-forge/label/cf201901
pyopenssl                 18.0.0                py36_1000    conda-forge/label/cf201901
pyrsistent                0.14.7           py36h470a237_0    conda-forge/label/cf201901
pysftp                    0.2.9                    py36_0    bioconda
pysocks                   1.6.8                 py36_1002    conda-forge/label/cf201901
python                    3.6.6                h5001a0f_3    conda-forge/label/cf201901
python-dateutil           2.7.5                      py_0    conda-forge/label/cf201901
python-drmaa              0.7.6                    py36_0    conda-forge/label/cf201901
python-irodsclient        0.7.0                      py_0    conda-forge/label/cf201901
python-libarchive-c       2.8                   py36_1004    conda-forge/label/cf201901
pytz                      2018.7                     py_0    conda-forge/label/cf201901
pyyaml                    3.13             py36h470a237_1    conda-forge/label/cf201901
r-assertthat              0.2.0           r351h6115d3f_1001    conda-forge/label/cf201901
r-base                    3.5.1                h391c2eb_4    conda-forge/label/cf201901
r-bh                      1.66.0_1              r351_2001    conda-forge/label/cf201901
r-bindr                   0.1.1           r351h6115d3f_1001    conda-forge/label/cf201901
r-bindrcpp                0.2.2            r351h9d2a408_1    conda-forge/label/cf201901
r-bit                     1.1_12           r351h470a237_2    conda-forge/label/cf201901
r-bit64                   0.9_7            r351hc070d10_0    conda-forge/label/cf201901
r-blob                    1.1.1                 r351_1001    conda-forge/label/cf201901
r-cli                     1.0.1           r351h6115d3f_1000    conda-forge/label/cf201901
r-crayon                  1.3.4           r351h6115d3f_1001    conda-forge/label/cf201901
r-dbi                     1.0.0           r351h6115d3f_1001    conda-forge/label/cf201901
r-dbplyr                  1.2.2           r351h6115d3f_1001    conda-forge/label/cf201901
r-digest                  0.6.18           r351hc070d10_0    conda-forge/label/cf201901
r-dplyr                   0.7.8            r351h9d2a408_0    conda-forge/label/cf201901
r-fansi                   0.3.0            r351hc070d10_0    conda-forge/label/cf201901
r-fastghquad              1.0              r351h29659fb_0    jluquette
r-glue                    1.3.0            r351h470a237_2    conda-forge/label/cf201901
r-magrittr                1.5             r351h6115d3f_1001    conda-forge/label/cf201901
r-memoise                 1.1.0           r351h6115d3f_1001    conda-forge/label/cf201901
r-pillar                  1.3.0           r351h6115d3f_1000    conda-forge/label/cf201901
r-pkgconfig               2.0.2           r351h6115d3f_1001    conda-forge/label/cf201901
r-plogr                   0.2.0           r351h6115d3f_1001    conda-forge/label/cf201901
r-prettyunits             1.0.2           r351h6115d3f_1001    conda-forge/label/cf201901
r-purrr                   0.2.5            r351hc070d10_2    conda-forge/label/cf201901
r-r6                      2.2.2           r351h6115d3f_1001    conda-forge/label/cf201901
r-rcpp                    1.0.0            r351h9d2a408_0    conda-forge/label/cf201901
r-rlang                   0.3.0.1          r351h470a237_0    conda-forge/label/cf201901
r-rsqlite                 2.1.1            r351h9d2a408_0    conda-forge/label/cf201901
r-scansnv                 0.1              r351hf484d3e_0    jluquette
r-tibble                  1.4.2            r351hc070d10_2    conda-forge/label/cf201901
r-tidyselect              0.2.5            r351h9d2a408_0    conda-forge/label/cf201901
r-utf8                    1.1.4            r351hc070d10_0    conda-forge/label/cf201901
ratelimiter               1.2.0                 py36_1000    conda-forge/label/cf201901
readline                  7.0                  h7b6447c_5  
requests                  2.13.0                   py36_0    conda-forge/label/cf201901
rpy2                      2.9.4           py36r351h941a26a_1    conda-forge/label/cf201901
rsa                       3.1.4                    py36_0    bioconda
ruamel_yaml               0.15.71          py36h470a237_0    conda-forge/label/cf201901
s3transfer                0.1.13                py36_1001    conda-forge/label/cf201901
samtools                  1.9                  h8ee4bcc_1    bioconda
scansnv                   0.9                           0    jluquette
setuptools                40.6.3                   py36_0    conda-forge/label/cf201901
shapeit                   2.r837               h09b0a5c_1    bioconda
six                       1.12.0                py36_1000    conda-forge/label/cf201901
smmap2                    2.0.5                      py_0    conda-forge/label/cf201901
snakemake                 5.3.1                         0    bioconda
snakemake-minimal         5.3.1                      py_0    bioconda
sqlite                    3.25.3               h7b6447c_0  
tk                        8.6.8                hbc83047_0  
tktable                   2.10                 h14c3975_0  
tqdm                      4.28.1                     py_0    conda-forge/label/cf201901
traitlets                 4.3.2                 py36_1000    conda-forge/label/cf201901
tzlocal                   1.5.1                      py_0    conda-forge/label/cf201901
urllib3                   1.12                     py36_0    bioconda
wheel                     0.32.3                   py36_0    conda-forge/label/cf201901
wrapt                     1.10.11          py36h470a237_1    conda-forge/label/cf201901
xmlrunner                 1.7.7                      py_0    conda-forge/label/cf201901
xorg-kbproto              1.0.7                h470a237_2    conda-forge/label/cf201901
xorg-libice               1.0.9                h470a237_4    conda-forge/label/cf201901
xorg-libsm                1.2.3                h8c8a85c_0    conda-forge/label/cf201901
xorg-libx11               1.6.6                h470a237_0    conda-forge/label/cf201901
xorg-libxau               1.0.8                h470a237_6    conda-forge/label/cf201901
xorg-libxdmcp             1.1.2                h470a237_7    conda-forge/label/cf201901
xorg-libxext              1.3.3                h470a237_4    conda-forge/label/cf201901
xorg-libxpm               3.5.12               h470a237_2    conda-forge/label/cf201901
xorg-libxrender           0.9.10               h470a237_2    conda-forge/label/cf201901
xorg-libxt                1.1.5                h470a237_2    conda-forge/label/cf201901
xorg-renderproto          0.11.1               h470a237_2    conda-forge/label/cf201901
xorg-xextproto            7.3.0                h470a237_2    conda-forge/label/cf201901
xorg-xproto               7.0.31               h470a237_7    conda-forge/label/cf201901
xz                        5.2.4                h14c3975_4  
yaml                      0.1.7                had09818_2  
yarl                      1.3.0            py36h470a237_0    conda-forge/label/cf201901
zlib                      1.2.11               ha838bed_2  

@jluquette
Copy link
Member

Hi @linkmonica,

It looks like you do have the correct library installed. From your output:
natefoo-slurm-drmaa 1.2.0_dev_deca826 0 jluquette

What issue are you running into now? Are you still getting the str instance, int found error after downloading the newest commit? Are you now getting a core dump?

@linkmonica
Copy link
Author

linkmonica commented Nov 20, 2019 via email

@jluquette
Copy link
Member

@linkmonica sorry for the late reply.

In my experience, DRMAA segfaults have only happened in relationship to specifying a -o parameter in the --drmaa string. This is a shot in the dark, but could you try just adding something like -o %logdir/slurm-%A.log to your --drmaa string? E.g., change

--drmaa ' -p thin-shared -t 01:00:00 --cpus-per-task 1 --mem 5G'

to

--drmaa ' -p thin-shared -t 01:00:00 --cpus-per-task 1 --mem 5G -o %logdir/slurm-%A.log'

Be sure to test this WITHOUT loading your system wide slurm-drmaa (i.e., do not run module load slurm-drmaa/1.2.0-dev.deca826).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants