Segmentation fault with error estimation using -doError

[jcaccavo]

Hi there,

I can't seem to avoid a Segmentation fault when running error estimation using -doError 1.

I've reviewed the numerous ANGSD issues involving segmentation faults. Of these, #561, #491, #479, #426, #394, and #105 seemed to propose the most promising solutions, either in the form of updating the version of ANGSD used, or modifying the -nThreads parameter.

However, updating from version 0.933 to 0.940 did not remove this issue, nor did reducing the number of threads allocated from 40, to 8, to 5.

In one instance I removed entirely the -nThreads parameter; the analysis is running.. but it has been running already for 6 days on our HPC, which normally does not have any memory allocation/computing issues. Notably I've also run many ANGSD analyses to generate beagle, maf, and glf files, using the -nThreads parameter, on sets of 15 - 43 bam files, all without this sort of issue (and it would take on the order of hours, not days, to run these analyses on the same HPC). It is only in trying to run the error estimation with -doError 1 that this problem comes up.

I've been trying this analysis on 3 different datasets

1. noDS: 43 bam files, and 5,044,176 sites
2. noDS otolith: 20 bam files, 9,332,798 sites
3. noDS tissue: 23 bam files, 4,249,159 sites

I am using the following code in ANGSG 0.940 to execute the analysis (e.g. for the noDS tissue dataset):
../angsd -doMajorMinor 2 -doCounts 1 -doError 1 -minSites 4249159 -out no_repeats_depthfilter_noDS_tissue_error -bam ../no_repeats_depthfilter_noDS_tissue_bam.filelist
Or with the -nThreads parameter:
../angsd -doMajorMinor 2 -doCounts 1 -doError 1 -minSites 4249159 -out no_repeats_depthfilter_noDS_tissue_error -bam ../no_repeats_depthfilter_noDS_tissue_bam.filelist -nThreads 5

Some examples of the segmentation fault error in the output are as follows:

	-> Printing at chr: HiC_scaffold_13 pos:9282930 chunknumber 87800 contains 258 sites
	 Triggering error estimation with nsites=5044200
	 Triggering error estimation with nsites=5044243
	 Triggering error estimation with nsites=5044302
	 Triggering error estimation with nsites=5044346
32_angsd_error.sh: line 17: 85340 Segmentation fault      angsd -doMajorMinor 2 -doCounts 1 -doError 1 -minSites 5044176 -out no_repeats_depthfilter_error -bam bam_filelists/no_repeats_depthfilter_bam.filelist -nThreads 5

-> Allocated ~ 620 million nodes to the nodepool, this is not an estimate of the memory usage
	-> Allocated ~ 630 million nodes to the nodepool, this is not an estimate of the memory usage
	-> Allocated ~ 640 million nodes to the nodepool, this is not an estimate of the memory usage
	-> Allocated ~ 650 million nodes to the nodepool, this is not an estimate of the memory usage
	-> Allocated ~ 660 million nodes to the nodepool, this is not an estimate of the memory usage
32_angsd_error_2.sh: line 20: 158141 Segmentation fault      angsd -doMajorMinor 2 -doCounts 1 -doError 1 -minSites 9332798 -out no_repeats_depthfilter_noDS_otolith_error -bam bam_filelists/no_repeats_depthfilter_noDS_otolith_bam.filelist -nThreads 40

-> Printing at chr: HiC_scaffold_13 pos:12846045 chunknumber 97300 contains 251 sites
	 Triggering error estimation with nsites=4249209
	 Triggering error estimation with nsites=4249245
	 Triggering error estimation with nsites=4249322
	 Triggering error estimation with nsites=4249385
32_angsd_error_2.sh: line 23: 34283 Segmentation fault      /srv/public/users/larissasa/programs/angsd/angsd -doMajorMinor 2 -doCounts 1 -doError 1 -minSites 4249159 -out no_repeats_depthfilter_noDS_tissue_error -bam bam_filelists/no_repeats_depthfilter_noDS_tissue_bam.filelist -nThreads 5

I am happy to provide more information or data as needed. Thank you in advance for your help.

Best,
Jilda