Test suite hits 2GB limit on 32bit R-devel #2767
Comments
|
Isn't i386 limited to 2 GB? |
|
@HughParsonage Oh ... yes! Just checked and yes both jobs failed just on 32bit. Thanks! So it's not AppVeyor then. It's just a 32bit / 2GB issue. That leaves us with why there's a gap between the 160MB in memtest chart and 2GB. If the memtest scale is 10x out (does the output of (I guess internal R object headers have increased a little bit in R-devel, causing, on 32-bit, R-release to be just under the 2GB limit and R-devel to be just over. Explaining why it's just R-devel.) |
mattdowle
changed the title
|
Failed build that happened to me was on And there were no suggested packages installed Probably we shouldn't use |
|
Just on that error, although yes the platform is 64bit, the multiarch R CMD check runs tests for both 32bit and 64bit. When I checked your fail, I downloaded the failure.zip artifact and looked inside both i386 and x64 directories. Just the i386 directory contained the main.fail file while x64 directory contained main.out passing ok. Oh ... how about just |
|
I checked locally memory usage with supposedly good tool R 3.4.4R CMD check took 364MB RSS. test.data.table took 351MB RSS. R-devel (2018-04-06)R CMD check took 224MB RSS. test.data.table took 207MB RSS. |
|
Just happened on win-builder ... so, as before, it's ok on 64bit but not 32bit. |
|
have we done something like -- every 500 tests, clean up all the stale test objects laying around (x, y, DT, DT1, DT2, dt, df, ...)? should be minor but might help |
|
Perhaps change:
to
or |
|
@MichaelChirico Jan implemented the memory tracing but it doesn't look like it's using more than 150MB of RAM. It's odd. Yes I thought there would be some large stale objects to @HughParsonage We don't think the nq tests are using the ram, though. Somewhere earlier. But ok might as well turn off on 32bit, no harm ... |
|
My guess, not backed by anything, is that R 3.5.0 has some memory bug in 32bit win (AFAIK this issue is related only to this version/platform/arch). There were significant improvements for memory handling in R 3.5.0, on linux tests reports 207MB while R 3.4.4 took 351MB. |
|
I just ran |
|
@HughParsonage could you occupy most of memory by some other process? leave 1.5-2GB of memory for tests. |
|
With less than 1 GB available, I was able to run it without problems. |
|
@HughParsonage |
|
Yes, sorry just |
|
Trouble is we're flying blind as neither AppVeyor or win-builder tells use what and when uses the ram. Jan said GenomicRanges pulls in a large bunch of dependencies, and we want to remove it anyway, so that would be reasonable to try next then. |
|
Jan's idea is good to inspect the environment at the end of tests.Rraw to see if any large objects are left hanging around. If so, can be |
|
Installing GenomicRanges result installing 11 packages, one of them 17MB. Appveyor does not install GenomicRanges so issues are not related to it. Assuming win builder failure is caused by same issue as appveyor 32bit win r-devel builds. Nice helpers for checking memory usage of objects in workspace: https://stackoverflow.com/questions/1358003/tricks-to-manage-the-available-memory-in-an-r-session |
|
The plot thickens. Just got this from win-builder (with latest master as of now because I was testing my commits turning off large tests on 32bit) ... 1372.* is GenomicRanges |
|
So testing with GenomicRanges make sense... |
|
I may be imagining it, but AppVeyor 32bit seems more stable now after 3-4 runs. It was still failing with R's So that no tests are skipped by AppVeyor. It's possible that there's a |
|
I removed those packages locally (bit64, xts, nanotime and chron) and reran |
|
One final attempt. On Windows 32bit R-devel I did the following : for (i in 1:10) if (!test.data.table()) stop("one or more tests failed")and then after a few iterations I saw it!! And indeed the R process (32bit) is close to 4GB. which is way above what we have observed and were expecting since we know the test suite doesn't use that much. Does anyone else see it on Windows? Need to repeat 10 times like that it seems. Maybe if I repeat 10 times locally it'll hit that locally then I can trace with linux tools. Will try ... Feels to me like a true data.table bug that is triggered by random data in a test causing a pesky branch to be hit. Not actually 32-bit or Windows related. Just showing up there, so far, by chance. This could be the same as #2822, albeit a crash there. |
|
The latest 'memory exhausted' fail occurred on R-devel only, as before. But both 32bit and 64bit this time so it's not 32bit-only. Also, the suggests were loaded this time and their tests ran. So it's not to do with whether bit64, xts or nanotime are installed or not, either. Link to log : https://ci.appveyor.com/project/Rdatatable/data-table/build/1.0.2472/job/17yi0fe0562dht8v Copy and paste of AppVeyor log |
|
Dont't know if related but I just ran into this fail during the join below that has been running fine before. The dt object is around 4gb in memory but when doing this join R memory usage in Activity Monitor increases to 16... my sessioninfo below. dt[
degrees[
dt,
max.int(SunNiva),
on = .(id, KalenderAr < admission_year),
by = .EACHI
],
tmp.SunNiva_achieved_degree := i.V1,
on = .(id, admission_year = KalenderAr)
] |
|
@adamaltmejd Thanks for this -- great. Could you try again with master as of now please; a few memory faults fixed in the last few days. If it still fails, is there any way to generate some dummy data for us so we've got something to run? |
|
Just tried reinstalling with install_github("Rdatatable/data.table", build_vignettes=FALSE), same error. Before doing that however I noticed that also my backup script that was using dplyr to get around some other data.table issues (fread and NUL bytes) also ran into the same memory error on a dplyr operation. Maybe its actually something with my data that has changed that I can't pinpoint, or with R 3.5.0 i guess. Or its related to the data table object already in memory.. Kind of tricky to reproduce, data is confidential. But will try as soon as I have some more time. |
|
@mattdowle I run memtest for windows and got the following (windows uses only # x64 win r-devel
data.table::fread("wget -q https://github.com/Rdatatable/data.table/files/1995905/memtest.csv.zip && unzip -cq memtest.csv.zip"
)[, last_GC_used := data.table::shift(GC_used)
][, GC_used_diff := GC_used-last_GC_used
][order(-GC_used_diff), head(.SD, 10), .SDcols=c("test", "GC_used", "GC_used_diff")]
# test GC_used GC_used_diff
# <num> <num> <num>
# 1: 1848.000 7875.9 441.8
# 2: 1835.000 7405.1 316.2
# 3: 1739.400 6581.1 58.6
# 4: 1739.100 6519.8 49.6
# 5: 1158.000 100.3 48.5
# 6: 1750.800 6668.4 37.4
# 7: 1549.000 2670.7 35.4
# 8: 1542.000 2597.2 35.1
# 9: 1642.001 3661.0 33.8
#10: 1779.800 6960.8 24.2So 1848 and 1835 seems to be contributing the most to @HughParsonage are you able to run memtest on your R-devel to confirm below numbers? just set env var Windows R Under development (unstable) (2018-05-05 r74699)i386x64Windows R version 3.5.0 (2018-04-23)i386x64Linux R version 3.4.4 (2018-03-15) |
|
R dev r74708 Windows 64 bit data.table 1.11.3 2018-05-11 01:26:09 UTC |
|
It seems to be very fast on your machine, but also confirms memory used. |
|
Have I done something wrong? It's a 5 year old laptop that cost about $500... |
|
Not sure. 50 min on my machine was actually R CMD check and installing some deps from source, not just test.data.table. Tests and examples for i386 and x64. How much RAM you have? |
|
8 GB |
|
Just managed to run a version of the script referenced in my post above but with a larger data set on a proper server with 120GB RAM, and it worked, using 80Gb memory at peak. Using R 3.5.0 and CRAN version of datatable (server for confidential data so not allowed to install newer version). |
I've just seen this occur, here :
https://ci.appveyor.com/project/Rdatatable/data-table/build/1.0.2337/job/3rppwcw31ent8g1r
Jan saw it too today here :
#2761 (comment)
So, despite all our efforts, there's still more to do.
The error I saw is similar to Jan's :
That message is R's, here :
https://github.com/wch/r-source/blob/trunk/src/main/memory.c#L2059
and looking at the context there (inside an err_malloc function) suggests AppVeyor instance is declining R's malloc request. (i.e. it doesn't seem like some kind of stack limit inside R itself).
@jangorecki
rm()'d after those tests, then perhaps more memory will be available for the tests at the end. However, as I said above, it's strange that the memtest charts show only 160MB being used! Reducing those 3 tests will be wasted effort if, somehow, the memtest results are not capturing the actual mem usage on AppVeyor.The text was updated successfully, but these errors were encountered: