munge_sumstats.py taking forever to complete analysis

[chirrie]

Is there a way to speed up munge_sumstats.py. It has been running for the last two weeks and no hopes of it completing the ran. The GWAS files is not so big, so I don't understand why it is taking that long

[Fabian-RY]

I'm having the same issue using the --merge-alleles parameter. In my case, i found out that the never ending step is line 257 in munge_sumstats.py:

        ii = dat.SNP.isin(merge_alleles.SNP)

This is a Pandas method for the Series class. I've been unable to find any more info, but it's been running for days, and i also have no hope for it to finish.

¿Did your dataset finally ended?

[chirrie]

You can try using --chunk size 50000. It helped

…

On Tue, Mar 12, 2024 at 1:50 PM Fabián Robledo Yagüe < ***@***.***> wrote: I'm having the same issue using the --merge-alleles parameter. In my case, i found out that the never ending step is line 257 in munge_sumstats.py: ii = dat.SNP.isin(merge_alleles.SNP) This is a Pandas method for the Series class. I've been unable to find any more info, but it's been running for days, and i also have no hope for it to finish. ¿Did your dataset finally ended? — Reply to this email directly, view it on GitHub <#420 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZTQ2X3HIO3SWEH7AJX3PDYX3TXHAVCNFSM6AAAAABEIXF5P6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOJRGQ3TEMRXHA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- Vivien Chebii, Ph.D. Post doctoral Scientist (Bioinformatics) Sydney Brenner Institute for Molecular Biosciences, Wits University, South Africa Email: ***@***.*** ***@***.***>*; ***@***.*** Mobile: +254 722 712 248 Linkedin: www.linkedin.com/in/viviennechebii/

[Fabian-RY]

Thanks, i'll try that

[Fabian-RY]

Wow, it really worked, it took only seconds to finish with the --chunk 50 000 size.

[chirrie]

Glad to hear it!

…

On Wed, Mar 13, 2024 at 10:50 AM Fabián Robledo Yagüe < ***@***.***> wrote: Wow, it really worked, it took only seconds to finish with the --chunk 50 000 size. — Reply to this email directly, view it on GitHub <#420 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZTQ2XCDXWYS3Y74Z5OTC3YYAHMDAVCNFSM6AAAAABEIXF5P6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOJTHA2DIMJRGA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- Vivien Chebii, Ph.D. Post doctoral Scientist (Bioinformatics) Sydney Brenner Institute for Molecular Biosciences, Wits University, South Africa Email: ***@***.*** ***@***.***>*; ***@***.*** Mobile: +254 722 712 248 Linkedin: www.linkedin.com/in/viviennechebii/