plotFinger fails due to insaficient wall time

[sunta3iouxos]

Hi all,
How can I increase the wall time for plotfinger from 6 to 24hrs?
it constantly fails. and also does not properly shutdowns the commands. running the squeue there is no job, but running htop, I can see the jobs still lingering and have to mannualy kill them.

Best
Theodoros

[katsikora]

Hi Theodoros,

I'm sorry to hear that plot fingerpring fails for you. Why do you think it should take more than 6 hours? Ideally, it should be done within minutes. Perhaps increasing the memory resource would help it run faster on your system?

The zombie jobs are not unusual with anything written in python.

Best wishes,
Katarzyna

[sunta3iouxos]

Hi Katarzyna,
this is the error I get. Notice that the provided memory is: mem_mb=115345, disk_mb=115345

---- This analysis has been done using snakePipes version 2.8.1 ----
Sample sheet found and header is ok!
/scratch/tgeorgom/mamba/snakepipes_devel/lib/python3.11/site-packages/snakePipes/workflows/ChIP-seq/internals.snakefile:145: UserWarning: 16 out of 16 have a matching control
  warnings.warn( str(len(chip_samples_w_ctrl)) + " out of " + str(len(chip_samples)) + " have a matching control ")
Building DAG of jobs...
Using shell: /bin/bash
Provided cores: 24
Rules claiming more threads will be scaled down.
Provided resources: mem_mb=115345, disk_mb=115345
Select jobs to execute...

[Tue Apr 16 09:32:35 2024]
rule plotFingerprint:
    input: filtered_bam/A006200376_218434_S7_L001.filtered.bam, filtered_bam/A006200376_218437_S8_L001.filtered.bam, filtered_bam/A006200376_218439_S9_L001.filtered.bam, filtered_bam/A006200376_218442_S10_L001.filtered.bam, filtered_bam/A006200376_218444_S11_L001.filtered.bam, filtered_bam/A006200376_218446_S12_L001.filtered.bam, filtered_bam/A006200376_218448_S13_L001.filtered.bam, filtered_bam/A006200376_218450_S14_L001.filtered.bam, filtered_bam/A006200376_218452_S15_L001.filtered.bam, filtered_bam/A006200376_218454_S16_L001.filtered.bam, filtered_bam/A006200376_218456_S17_L001.filtered.bam, filtered_bam/A006200376_218458_S18_L001.filtered.bam, filtered_bam/A006200376_218460_S19_L001.filtered.bam, filtered_bam/A006200376_218462_S20_L001.filtered.bam, filtered_bam/A006200376_218464_S21_L001.filtered.bam, filtered_bam/A006200376_218466_S22_L001.filtered.bam, filtered_bam/A006200376_218468_S23_L001.filtered.bam, filtered_bam/A006200376_218470_S24_L001.filtered.bam, filtered_bam/A006200376_218472_S25_L001.filtered.bam, filtered_bam/A006200376_218474_S26_L001.filtered.bam, filtered_bam/A006200376_218434_S7_L001.filtered.bam.bai, filtered_bam/A006200376_218437_S8_L001.filtered.bam.bai, filtered_bam/A006200376_218439_S9_L001.filtered.bam.bai, filtered_bam/A006200376_218442_S10_L001.filtered.bam.bai, filtered_bam/A006200376_218444_S11_L001.filtered.bam.bai, filtered_bam/A006200376_218446_S12_L001.filtered.bam.bai, filtered_bam/A006200376_218448_S13_L001.filtered.bam.bai, filtered_bam/A006200376_218450_S14_L001.filtered.bam.bai, filtered_bam/A006200376_218452_S15_L001.filtered.bam.bai, filtered_bam/A006200376_218454_S16_L001.filtered.bam.bai, filtered_bam/A006200376_218456_S17_L001.filtered.bam.bai, filtered_bam/A006200376_218458_S18_L001.filtered.bam.bai, filtered_bam/A006200376_218460_S19_L001.filtered.bam.bai, filtered_bam/A006200376_218462_S20_L001.filtered.bam.bai, filtered_bam/A006200376_218464_S21_L001.filtered.bam.bai, filtered_bam/A006200376_218466_S22_L001.filtered.bam.bai, filtered_bam/A006200376_218468_S23_L001.filtered.bam.bai, filtered_bam/A006200376_218470_S24_L001.filtered.bam.bai, filtered_bam/A006200376_218472_S25_L001.filtered.bam.bai, filtered_bam/A006200376_218474_S26_L001.filtered.bam.bai
    output: deepTools_ChIP/plotFingerprint/plotFingerprint.metrics.txt
    log: deepTools_ChIP/logs/plotFingerprint.out, deepTools_ChIP/logs/plotFingerprint.err
    jobid: 0
    benchmark: deepTools_ChIP/.benchmark/plotFingerprint.benchmark
    reason: Missing output files: deepTools_ChIP/plotFingerprint/plotFingerprint.metrics.txt, deepTools_ChIP/.benchmark/plotFingerprint.benchmark
    threads: 24
    resources: mem_mb=115345, disk_mb=115345, tmpdir=/scratch/tgeorgom_temp

Activating conda environment: ../../mamba/snakepipes_devel/envs/c21237e8c0954c1620210ed0f7ba2e63_
slurmstepd: error: *** JOB 18773077 ON cheops11804 CANCELLED AT 2024-04-16T15:31:45 DUE TO TIME LIMIT ***

[sunta3iouxos]

there is also this thing that I do not know if it is relevant:

/scratch/tgeorgom/mamba/snakepipes_devel/lib/python3.11/site-packages/snakePipes/workflows/ChIP-seq/internals.snakefile:145: UserWarning: 16 out of 16 have a matching control
  warnings.warn( str(len(chip_samples_w_ctrl)) + " out of " + str(len(chip_samples)) + " have a matching control ")

[katsikora]

345

Hi,

am I reading this correctly, that rule plotFingerprint is given 100Gb memory? That's a lot. I believe, by default, it should get 1Gb.
Did you modify the memory resource yourself?

Best,

Katarzyna

[katsikora]

there is also this thing that I do not know if it is relevant:

/scratch/tgeorgom/mamba/snakepipes_devel/lib/python3.11/site-packages/snakePipes/workflows/ChIP-seq/internals.snakefile:145: UserWarning: 16 out of 16 have a matching control
  warnings.warn( str(len(chip_samples_w_ctrl)) + " out of " + str(len(chip_samples)) + " have a matching control ")

That's not related. It's just a warning message if control samples were found in the chipdict yaml file or not. Saves a lot of headache with some cutNx experiments.

[sunta3iouxos]

345

Hi,

am I reading this correctly, that rule plotFingerprint is given 100Gb memory? That's a lot. I believe, by default, it should get 1Gb. Did you modify the memory resource yourself?

Best,

Katarzyna

regarding the memory, I increased it just in case.
This what I got from another run of the same bam files, BUT in this case I used mm38 instead of mm39.

s       h:m:s   max_rss max_vms max_uss max_pss io_in   io_out  mean_load       cpu_time
3611.4827       1:00:11 4608.45 88808.61        2212.22 2299.83 0.00    0.00    1649.02 357.03
/scratch/tgeorgom/PAV11/bamGRnCm38/deepTools_ChIP/.benchmark/plotFingerprint.benchmark (END)

with the following settings from lib/python3.11/site-packages/snakePipes/shared/cluster.yaml:
plotFingerprint:
memory: 3G
and from lib/python3.11/site-packages/snakePipes/shared/defaults.yaml
max_thread: 24