gtdbtk throws memory allocation error

[ycsong]

Hi Francisco,

I am having some issues with gtdbtk, where I am running into a memory allocation error. I think I know the source of the issue and the solution for it, based on what I read in the gtdbtk forum: Ecogenomics/GTDBTk#267.

Basically, I need to submit the job to a partition in our server that can accommodate the memory requirement. My question is how this might be possible in metaGEM. I see there is a cluster_config.json file, but I am not sure how I would edit this file. I tried adding in a "partition" field in there, but that doesn't seem to be working.

Your help here would be much appreciated. Thank you.

[ycsong]

Hi again,

I resolved the issue by adding the 'partition' parameter to the lines in the metaGEM.sh that executes snakemake & sbatch:

315:    snakemake all -j $njobs -n -k --cluster-config ../config/cluster_config.json -c "sbatch -A {cluster.account} -t {cluster.time} --nodes {cluster.n} --ntasks {cluster.tasks} --cpus-per-task {cluster.n} --partition {cluster.partition} --output {cluster.output}"

428:                [Yy]* ) echo "nohup snakemake all -j $njobs -k --cluster-config ../config/cluster_config.json -c 'sbatch -A {cluster.account} -t {cluster.time} --nodes {cluster.n} --ntasks {cluster.tasks} --partition {cluster.partition} --cpus-per-task {cluster.n} --output {cluster.output}' &"|bash; break;;
447:                [Yy]* ) echo "nohup snakemake all -j $njobs -k --cluster-config ../config/cluster_config.json -c 'sbatch -A {cluster.account} -t {cluster.time} --mem {cluster.mem} --nodes {cluster.n} --ntasks {cluster.tasks} --partition {cluster.partition} --cpus-per-task {cluster.n} --output {cluster.output}' &"|bash; break;;

I also added the partition string back on the cluster_config.json file as well.

[franciscozorrilla]

Hey Young, glad you resolved it! That is exactly what I was going to suggest, unfortunately my cluster is down at the moment so I couldn't double check this.

Regarding the gtdbtk implementation, it currently runs one job per sample. The optimal implementation depends on your dataset, but I would suggest to consider running a single job with all MAGs from your different samples (e.g. #130).

Best,
Francisco