Out of Memory Error when using regexp_extract in an UPDATE statement #11971
Unanswered
CuriousTim
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I have a directory of ~5600 tab-delimited files all of the format
The CONTIG, START, and END columns are identical in all files (different numbers than shown here); only the COUNT column is different across files. I read the files into a database like so
and I get a table of ~2.1B rows.
Then I tried to generate an ID from the
filename
column that is automatically created. So I ranALTER
followed by anUPDATE
.I ran all of this using the R API and the first time I tried, my R session crashed because it ran out of memory (using a 26GB RAM machine). Then I tried setting a memory limit with
SET memory_limit = '16GB';
and got an error after about a minute of running.Is it expected for this kind of UPDATE statement to use so much memory?
Beta Was this translation helpful? Give feedback.
All reactions