Skip to content

Commit

Permalink
fix: auto memory calculation (#1210)
Browse files Browse the repository at this point in the history
<!-- Ensure that the PR title follows conventional commit style (<type>:
<description>)-->
<!-- Possible types are here:
https://github.com/commitizen/conventional-commit-types/blob/master/index.json
-->

### Description
Added better memory calculation

<!-- Add a description of your PR here-->

### QC
<!-- Make sure that you can tick the boxes below. -->

* [x] I confirm that:

For all wrappers added by this PR, 

* there is a test case which covers any introduced changes,
* `input:` and `output:` file paths in the resulting rule can be changed
arbitrarily,
* either the wrapper can only use a single core, or the example rule
contains a `threads: x` statement with `x` being a reasonable default,
* rule names in the test case are in
[snake_case](https://en.wikipedia.org/wiki/Snake_case) and somehow tell
what the rule is about or match the tools purpose or name (e.g.,
`map_reads` for a step that maps reads),
* all `environment.yaml` specifications follow [the respective best
practices](https://stackoverflow.com/a/64594513/2352071),
* wherever possible, command line arguments are inferred and set
automatically (e.g. based on file extensions in `input:` or `output:`),
* all fields of the example rules in the `Snakefile`s and their entries
are explained via comments (`input:`/`output:`/`params:` etc.),
* `stderr` and/or `stdout` are logged correctly (`log:`), depending on
the wrapped tool,
* temporary files are either written to a unique hidden folder in the
working directory, or (better) stored where the Python function
`tempfile.gettempdir()` points to (see
[here](https://docs.python.org/3/library/tempfile.html#tempfile.gettempdir);
this also means that using any Python `tempfile` default behavior
works),
* the `meta.yaml` contains a link to the documentation of the respective
tool or command,
* `Snakefile`s pass the linting (`snakemake --lint`),
* `Snakefile`s are formatted with
[snakefmt](https://github.com/snakemake/snakefmt),
* Python wrapper scripts are formatted with
[black](https://black.readthedocs.io).
* Conda environments use a minimal amount of channels, in recommended
ordering. E.g. for bioconda, use (conda-forge, bioconda, nodefaults, as
conda-forge should have highest priority and defaults channels are
usually not needed because most packages are in conda-forge nowadays).
  • Loading branch information
fgvieira committed May 12, 2023
1 parent d5be001 commit b08081f
Show file tree
Hide file tree
Showing 4 changed files with 16 additions and 4 deletions.
1 change: 1 addition & 0 deletions bio/fastqc/environment.yaml
Expand Up @@ -4,3 +4,4 @@ channels:
- nodefaults
dependencies:
- fastqc =0.12.1
- snakemake-wrapper-utils =0.5.3
1 change: 1 addition & 0 deletions bio/fastqc/meta.yaml
@@ -1,6 +1,7 @@
name: fastqc
description: |
Generate fastq qc statistics using fastqc.
url: https://github.com/s-andrews/FastQC
authors:
- Julian de Ruiter
input:
Expand Down
5 changes: 4 additions & 1 deletion bio/fastqc/test/Snakefile
Expand Up @@ -4,9 +4,12 @@ rule fastqc:
output:
html="qc/fastqc/{sample}.html",
zip="qc/fastqc/{sample}_fastqc.zip" # the suffix _fastqc.zip is necessary for multiqc to find the file. If not using multiqc, you are free to choose an arbitrary filename
params: "--quiet"
params:
extra = "--quiet"
log:
"logs/fastqc/{sample}.log"
threads: 1
resources:
mem_mb = 1024
wrapper:
"master/bio/fastqc"
13 changes: 10 additions & 3 deletions bio/fastqc/wrapper.py
Expand Up @@ -9,10 +9,13 @@
from os import path
import re
from tempfile import TemporaryDirectory

from snakemake.shell import shell
from snakemake_wrapper_utils.snakemake import get_mem

extra = snakemake.params.get("extra", "")
log = snakemake.log_fmt_shell(stdout=True, stderr=True)
# Define memory per thread (https://github.com/s-andrews/FastQC/blob/master/fastqc#L201-L222)
mem_mb = int(get_mem(snakemake, "MiB") / snakemake.threads)


def basename_without_ext(file_path):
Expand All @@ -35,8 +38,12 @@ def basename_without_ext(file_path):
# use the same fastqc dir, we create a temp dir.
with TemporaryDirectory() as tempdir:
shell(
"fastqc {snakemake.params} -t {snakemake.threads} "
"--outdir {tempdir:q} {snakemake.input[0]:q}"
"fastqc"
" --threads {snakemake.threads}"
" --memory {mem_mb}"
" {extra}"
" --outdir {tempdir:q}"
" {snakemake.input[0]:q}"
" {log}"
)

Expand Down

0 comments on commit b08081f

Please sign in to comment.