In R-4.2.0 on windows, package 'stats' in options("defaultPackages") was not found. #151

nhirschey · 2022-04-25T11:11:22Z

Using R version 4.2.0 on Windows, this prior issue with loading stats.dll #127 seems to have resurfaced.

Here, I show that rnorm works with R 4.0.2 but fails with R 4.2.0.

Works with old R version 4.0.2

#r "nuget: R.NET"
open RDotNet

REngine.SetEnvironmentVariables("c:/program files/r/r-4.0.2/bin/x64", "c:/program files/r/r-4.0.2")
let  engine = REngine.GetInstance()
engine.Evaluate("set.seed(1);rnorm(1)").AsNumeric()
// val it: NumericVector = seq [-0.6264538107]

Fails with R version 4.2.0

#r "nuget: R.NET"
open RDotNet

REngine.SetEnvironmentVariables("c:/program files/r/r-4.2.0/bin/x64", "c:/program files/r/r-4.2.0")
let  engine = REngine.GetInstance()
(*
During startup - Warning message:
package 'stats' in options("defaultPackages") was not found 
val engine: REngine
*)
engine.Evaluate("set.seed(1);rnorm(1)").AsNumeric()
(*
Error in rnorm(1) : could not find function "rnorm"
In addition: Warning message:
'memory.limit()' is no longer supported
RDotNet.EvaluationException: Error in rnorm(1) : could not find function "rnorm"

   at RDotNet.REngine.Parse(String statement, StringBuilder incompleteStatement, REnvironment environment)
   at RDotNet.REngine.Defer(String statement, REnvironment environment)+MoveNext()
   at System.Linq.Enumerable.TryGetLast[TSource](IEnumerable`1 source, Boolean& found)
   at System.Linq.Enumerable.LastOrDefault[TSource](IEnumerable`1 source)
   at RDotNet.REngine.Evaluate(String statement, REnvironment environment)
   at <StartupCode$FSI_0005>.$FSI_0005.main@()
Stopped due to error
*)

> engine.Evaluate("library(stats)").AsCharacter()
(*
Error: package or namespace load failed for 'stats' in inDL(x, as.logical(local), as.logical(now), ...):
 unable to load shared object 'C:/Program Files/R/R-4.2.0/library/stats/libs/x64/stats.dll':
  LoadLibrary failure:  The specified module could not be found.
RDotNet.EvaluationException: Error: package or namespace load failed for 'stats' in inDL(x, as.logical(local), as.logical(now), ...):
 unable to load shared object 'C:/Program Files/R/R-4.2.0/library/stats/libs/x64/stats.dll':
  LoadLibrary failure:  The specified module could not be found.

   at RDotNet.REngine.Parse(String statement, StringBuilder incompleteStatement, REnvironment environment)
   at RDotNet.REngine.Defer(String statement, REnvironment environment)+MoveNext()
   at System.Linq.Enumerable.TryGetLast[TSource](IEnumerable`1 source, Boolean& found)
   at System.Linq.Enumerable.LastOrDefault[TSource](IEnumerable`1 source)
   at RDotNet.REngine.Evaluate(String statement, REnvironment environment)
   at <StartupCode$FSI_0006>.$FSI_0006.main@()
Stopped due to error
*)

The text was updated successfully, but these errors were encountered:

lrasmus · 2022-04-29T20:06:09Z

Thanks for reporting @nhirschey! I'm seeing the same thing and will investigate a solution.

lrasmus · 2022-05-03T20:17:15Z

Just some placeholder notes/updates -

Going off some other/older posts, it seems like stats.dll is failing to load because of some dependency not being found. When you load it in dependency walker, you can see R.dll, Rblas.dll, and Rlapack.dll are dependencies, all located (for me) in C:\Program Files\R\R-4.2.0\bin\x64. I've been looking at the way (per #127) that Renviron is loaded, how PATH is being set, etc, because the behavior is like the dependency can't be found on the path when trying to load. Playing around with a suggestion in https://stackoverflow.com/a/48211343/5670646 (NOTE I don't recommend this as a solution, I was just doing these steps to troubleshoot), once Rblas.dll and Rlapack.dll are put in the same spot as stats.dll, that error goes away.

I need to go back to why, even if I explicitly set the folder where those dependency DLLs are found, it's failing to load. There's probably something in the basics of how DLLs are loaded that I'm just forgetting.

lrasmus · 2022-05-11T19:52:23Z

More placeholder notes (sorry) -

Thanks to @nmunozgarcia for noticing the connection in #152. It seems similar - setting PATH explicitly doesn't seem to fix the issue, which is odd because it should. But that's similar to the behavior folks are seeing with R_HOME - they set it in the environment, but R embedded doesn't pick it up. So current theory (which is a good one) would be that PATH updates are similarly not being picked up/recognized. This assumes that the root cause is the PATH needs to be set so it can "find" the core R dlls.

I did test explicitly setting my PATH in .NET code, and I confirmed within R.NET right before it calls R that the environment variable is there as I expected - and it was. So I know PATH wasn't being overridden somewhere after I thought I set it.

Some references:

https://tdhock.github.io/blog/2020/embedded-R/ - Also notes about R_HOME issues
https://github.com/alexgutteridge/rsruby - Another embedded R
https://svn.r-project.org/R/trunk/src/unix/system.c - where the "R home" error occurs

lrasmus · 2022-05-11T20:41:35Z

Not sure this is adding a lot of knowledge, but just confirmation via MSYS2 environment (from Rtools42) that unless the path is set to include the R core DLLs, it'll fail to load stats.dll

$ cd /C/Program\ Files/R/R-4.2.0/library/stats/libs/x64
$ ldd stats.dll
        ntdll.dll => /c/windows/SYSTEM32/ntdll.dll (0x7ff9c4870000)
        KERNEL32.DLL => /c/windows/System32/KERNEL32.DLL (0x7ff9c29d0000)
        KERNELBASE.dll => /c/windows/System32/KERNELBASE.dll (0x7ff9c24c0000)
        msvcrt.dll => /c/windows/System32/msvcrt.dll (0x7ff9c3550000)
        umppc15005.dll => /c/windows/System32/umppc15005.dll (0x22a939c0000)
        umppc15005.dll => /c/windows/System32/umppc15005.dll (0x22a939c0000)
        stats.dll => /c/Program Files/R/R-4.2.0/library/stats/libs/x64/stats.dll (0x7ff99df90000)
        ucrtbase.dll => /c/Windows/System32/ucrtbase.dll (0x7ff9c23c0000)
        R.dll => not found
        Rblas.dll => not found
        Rlapack.dll => not found
        api-ms-win-crt-private-l1-1-0.dll => not found
        api-ms-win-crt-string-l1-1-0.dll => not found
        api-ms-win-crt-environment-l1-1-0.dll => not found
        api-ms-win-crt-time-l1-1-0.dll => not found
        api-ms-win-crt-math-l1-1-0.dll => not found
        api-ms-win-crt-heap-l1-1-0.dll => not found
        api-ms-win-crt-runtime-l1-1-0.dll => not found
        api-ms-win-crt-stdio-l1-1-0.dll => not found


$ echo $PATH
/usr/local/bin:/usr/bin:/bin:/opt/bin:/c/Windows/System32:/c/Windows:/c/Windows/System32/Wbem:/c/Windows/System32/WindowsPowerShell/v1.0/:/c/progra~1/git/cmd:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl


$ export PATH=$PATH:/C/Program\ Files/R/R-4.2.0/bin/x64/

$ echo $PATH
/usr/local/bin:/usr/bin:/bin:/opt/bin:/c/Windows/System32:/c/Windows:/c/Windows/System32/Wbem:/c/Windows/System32/WindowsPowerShell/v1.0/:/c/progra~1/git/cmd:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/C/Program Files/R/R-4.2.0/bin/x64/


$ ldd stats.dll
        ntdll.dll => /c/windows/SYSTEM32/ntdll.dll (0x7ff9c4870000)
        KERNEL32.DLL => /c/windows/System32/KERNEL32.DLL (0x7ff9c29d0000)
        KERNELBASE.dll => /c/windows/System32/KERNELBASE.dll (0x7ff9c24c0000)
        msvcrt.dll => /c/windows/System32/msvcrt.dll (0x7ff9c3550000)
        umppc15005.dll => /c/windows/System32/umppc15005.dll (0x1de77760000)
        umppc15005.dll => /c/windows/System32/umppc15005.dll (0x1de77760000)
        ucrtbase.dll => /c/windows/System32/ucrtbase.dll (0x7ff9c23c0000)
        R.dll => /c/Program Files/R/R-4.2.0/bin/x64/R.dll (0x7ff959eb0000)
        ADVAPI32.dll => /c/windows/System32/ADVAPI32.dll (0x7ff9c3ce0000)
        sechost.dll => /c/windows/System32/sechost.dll (0x7ff9c2930000)
        RPCRT4.dll => /c/windows/System32/RPCRT4.dll (0x7ff9c4700000)
        SHELL32.dll => /c/windows/System32/SHELL32.dll (0x7ff9c3f00000)
        msvcp_win.dll => /c/windows/System32/msvcp_win.dll (0x7ff9c2320000)
        USER32.dll => /c/windows/System32/USER32.dll (0x7ff9c2ff0000)
        win32u.dll => /c/windows/System32/win32u.dll (0x7ff9c1f90000)
        GDI32.dll => /c/windows/System32/GDI32.dll (0x7ff9c3e40000)
        gdi32full.dll => /c/windows/System32/gdi32full.dll (0x7ff9c20b0000)
        Rblas.dll => /c/Program Files/R/R-4.2.0/bin/x64/Rblas.dll (0x7ff9b2260000)
        Rlapack.dll => /c/Program Files/R/R-4.2.0/bin/x64/Rlapack.dll (0x7ff964370000)
        Riconv.dll => /c/Program Files/R/R-4.2.0/bin/x64/Riconv.dll (0x7ff9bde60000)
        Rgraphapp.dll => /c/Program Files/R/R-4.2.0/bin/x64/Rgraphapp.dll (0x7ff9abba0000)
        comdlg32.dll => /c/windows/System32/comdlg32.dll (0x7ff9c3470000)
        combase.dll => /c/windows/System32/combase.dll (0x7ff9c2a90000)
        shcore.dll => /c/windows/System32/shcore.dll (0x7ff9c3d90000)
        SHLWAPI.dll => /c/windows/System32/SHLWAPI.dll (0x7ff9c3720000)
        IMM32.dll => /c/windows/System32/IMM32.dll (0x7ff9c3cb0000)
        VERSION.dll => /c/windows/SYSTEM32/VERSION.dll (0x7ff9b7e40000)
        WINMM.dll => /c/windows/SYSTEM32/WINMM.dll (0x7ff9b73b0000)
        COMCTL32.dll => /c/windows/WinSxS/amd64_microsoft.windows.common-controls_6595b64144ccf1df_5.82.19041.1110_none_792d1c772443f647/COMCTL32.dll (0x7ff9a1800000)
        MSIMG32.dll => /c/windows/SYSTEM32/MSIMG32.dll (0x7ff9b84e0000)

lrasmus · 2022-05-11T21:11:13Z

Further confirmation R's view of PATH differs from the Environment's view of PATH

Environment.SetEnvironmentVariable("PATH", "C:/Program Files/R/R-4.2.0/bin/x64;" + Environment.GetEnvironmentVariable("PATH"));
REngine.SetEnvironmentVariables("c:/program files/r/r-4.2.0/bin/x64", "c:/program files/r/r-4.2.0");
Console.WriteLine(Environment.GetEnvironmentVariable("PATH"));

Produces:

c:/program files/r/r-4.2.0/bin/x64;C:/Program Files/R/R-4.2.0/bin/x64;C:\windows\system32;C:\windows;C:\windows\System32\Wbem;C:\windows\System32\WindowsPowerShell\v1.0\;C:\windows\System32\OpenSSH\;C:\Program Files\R\R-4.2.0\bin\x64;C:\Program Files\SASHome\SASFoundation\9.4\core\sasexe;C:\Program Files\SASHome\SASFoundation\9.4\ets\sasexe;C:\Program Files\SASHome\Secure\ccme4;C:\Program Files\SASHome\x86\Secure\ccme4;C:\Program Files\SASHome\SASFoundation\9.4\core\sasext;C:\Program Files (x86)\Microsoft SQL Server\150\DTS\Binn\;C:\Program Files\Azure Data Studio\bin;C:\Program Files\Microsoft SQL Server\130\Tools\Binn\;C:\Program Files\Microsoft SQL Server\Client SDK\ODBC\170\Tools\Binn\;C:\Program Files\MATLAB\R2021b\bin;C:\Program Files\dotnet\;C:\Program Files\Git\cmd;C:\Users\test\AppData\Local\Programs\Python\Python310\Scripts\;C:\Users\test\AppData\Local\Programs\Python\Python310\;C:\Users\test\AppData\Local\Microsoft\WindowsApps;C:\Program Files\Azure Data Studio\bin;C:\Users\test\.dotnet\tools;

Note that my R directory exists in the PATH. Then we go ahead and load R.

var engine = REngine.GetInstance();

Calling Console.WriteLine(engine.Evaluate("Sys.getenv('PATH')").AsCharacter()[0]); does not have my R directory in the PATH anymore:

C:\rtools42/x86_64-w64-mingw32.static.posix/bin;C:\rtools42/usr/bin;C:\windows\system32;C:\windows;C:\windows\System32\Wbem;C:\windows\System32\WindowsPowerShell\v1.0\;C:\windows\System32\OpenSSH\;C:\Program Files\R\R-4.2.0\bin\x64;C:\Program Files\SASHome\SASFoundation\9.4\core\sasexe;C:\Program Files\SASHome\SASFoundation\9.4\ets\sasexe;C:\Program Files\SASHome\Secure\ccme4;C:\Program Files\SASHome\x86\Secure\ccme4;C:\Program Files\SASHome\SASFoundation\9.4\core\sasext;C:\Program Files (x86)\Microsoft SQL Server\150\DTS\Binn\;C:\Program Files\Azure Data Studio\bin;C:\Program Files\Microsoft SQL Server\130\Tools\Binn\;C:\Program Files\Microsoft SQL Server\Client SDK\ODBC\170\Tools\Binn\;C:\Program Files\MATLAB\R2021b\bin;C:\Program Files\dotnet\;C:\Program Files\Git\cmd;C:\Users\test\AppData\Local\Programs\Python\Python310\Scripts\;C:\Users\test\AppData\Local\Programs\Python\Python310\;C:\Users\test\AppData\Local\Microsoft\WindowsApps;C:\Program Files\Azure Data Studio\bin;C:\Users\test\.dotnet\tools;

Finally, calling Console.WriteLine(Environment.GetEnvironmentVariable("PATH")); still has my R directory in the PATH:

c:/program files/r/r-4.2.0/bin/x64;C:\rtools42/x86_64-w64-mingw32.static.posix/bin;C:\rtools42/usr/bin;C:\windows\system32;C:\windows;C:\windows\System32\Wbem;C:\windows\System32\WindowsPowerShell\v1.0\;C:\windows\System32\OpenSSH\;C:\Program Files\R\R-4.2.0\bin\x64;C:\Program Files\SASHome\SASFoundation\9.4\core\sasexe;C:\Program Files\SASHome\SASFoundation\9.4\ets\sasexe;C:\Program Files\SASHome\Secure\ccme4;C:\Program Files\SASHome\x86\Secure\ccme4;C:\Program Files\SASHome\SASFoundation\9.4\core\sasext;C:\Program Files (x86)\Microsoft SQL Server\150\DTS\Binn\;C:\Program Files\Azure Data Studio\bin;C:\Program Files\Microsoft SQL Server\130\Tools\Binn\;C:\Program Files\Microsoft SQL Server\Client SDK\ODBC\170\Tools\Binn\;C:\Program Files\MATLAB\R2021b\bin;C:\Program Files\dotnet\;C:\Program Files\Git\cmd;C:\Users\test\AppData\Local\Programs\Python\Python310\Scripts\;C:\Users\test\AppData\Local\Programs\Python\Python310\;C:\Users\test\AppData\Local\Microsoft\WindowsApps;C:\Program Files\Azure Data Studio\bin;C:\Users\test\.dotnet\tools;

lrasmus · 2022-05-11T21:25:16Z

This workaround appears to fix the issue. I'm not satisfied yet though, because I want to know why this is happening!!

static void Main(string[] args)
{
    REngine.SetEnvironmentVariables("c:/program files/r/r-4.2.0/bin/x64", "c:/program files/r/r-4.2.0");
    var engine = REngine.GetInstance();

    // Workaround - explicitly include R libs in PATH so R environment can find them.  Not sure why R can't find them when
    // we set this via Environment.SetEnvironmentVariable
    engine.Evaluate("Sys.setenv(PATH = paste(\"C:/Program Files/R/R-4.2.0/bin/x64\", Sys.getenv(\"PATH\"), sep=\";\"))");

    // This now works
    engine.Evaluate("library(stats)");
    engine.Evaluate("set.seed(1);rnorm(1)").AsNumeric();
}

nhirschey · 2022-05-12T00:15:00Z

Thanks for the workaround, that's really nice! Regarding where it's coming from, here's a long shot: this stack overflow comment talks about some ways that R PATH being set on startup.

I searched the referenced folder, C:\Program Files\R\R-4.2.0\etc, for PATH and I can see the path being set in the Rcmd_environ file. Is rdotnet calling Rcmd?

C:\Program Files\R\R-4.2.0\etc> ls * | select-string "PATH"

Rcmd_environ:38:R_RTOOLS42_PATH="${RTOOLS42_HOME:-c:/rtools42}/x86_64-w64-mingw32.static.posix/bin;${RTOOLS42_HOME:-c:/rtools42}/usr/bin"
Rcmd_environ:39:#   trailing slash to protect against trailing backslash in PATH
Rcmd_environ:40:PATH="${R_CUSTOM_TOOLS_PATH:-${R_RTOOLS42_PATH}};${PATH}/"
Rprofile.site:12:# .Library.site <- file.path(chartr("\\", "/", R.home()), "site-library")

lrasmus · 2022-05-13T15:01:59Z

Awesome find @nhirschey!!! Thanks so much. rdotnet doesn't itself explicitly call Rcmd_environ. I tried editing that file to update the PATH, and it didn't appear to pick up my changed path. Even going so far as to remove that file entirely didn't seem to make a difference for when rdotnet runs. I think that is only used for R CMD invocation.

But along those lines, I think the PATH changes are being made in C:\Program Files\R\R-4.2.0\library\base\R\Rprofile. At the bottom of that file, there is code to update the path with RTools. I can also edit it as such and have it work:

local({
    setRtools42Path <- 0
     setRtools42Path <- 1

    if (setRtools42Path) {
        rthome <- Sys.getenv("RTOOLS42_HOME", "c:/rtools42")
        rtpath <- paste0(rthome, "/x86_64-w64-mingw32.static.posix/bin;",
                         rthome, "/usr/bin")
        path <- Sys.getenv("R_CUSTOM_TOOLS_PATH", rtpath)
	## ADDED rbin
	rbin <- paste0(Sys.getenv("R_HOME"), "/bin/x64")
        Sys.setenv(R_RTOOLS42_PATH = rtpath)
	## UPDATED
        Sys.setenv(PATH = paste0(rbin, ";", path, ";", Sys.getenv("PATH")))
    }
})

lrasmus · 2022-05-13T15:24:00Z

I'm still confused why setting the PATH environment variable in the C# code had no effect. I know that loaded processes inherit a copy of their parent's environment, so if R.dll was loaded and THEN we update PATH, it makes sense that R.dll wouldn't know about those changes. But, it looks like R.dll is loaded AFTER I was updating PATH. I verified this via ListDLLs by setting a breakpoint in my C# test app before any rdotnet methods are called. I run listdlls TestApp.exe and confirm that R.dll isn't loaded. Then I continue on to initialize the R engine, and verify using listdlls TestApp.exe that R.dll is now loaded.

lrasmus · 2022-05-13T15:29:06Z

Workarounds that are available:

Add C:\Program Files\R\R-4.2.0\bin\x64 to the PATH environment variable using the standard Windows System Properties > Environment Variables
Update C:\Program Files\R\R-4.2.0\library\base\R\Rprofile (per earlier comment)
In calling application, immediately call engine.Evaluate("Sys.setenv(PATH = paste(\"C:/Program Files/R/R-4.2.0/bin/x64\", Sys.getenv(\"PATH\"), sep=\";\"))"); after initializing REngine (per earlier comment)

gkapellmann · 2023-02-24T00:11:13Z

Just to add some info, I am using R 4.2.2 and workaround 3 didnt work, but workaround 2 worked like a charm.

robreno · 2023-03-07T05:55:18Z

Just to add some info, I am using R 4.2.2 and workaround 3 didnt work, but workaround 2 worked like a charm.

Please, how did you setup your F# project and get it to work. I am having no luck.

nmunozgarcia mentioned this issue May 10, 2022

R HOME Environment variable is not read or used in Linux (and maybe macOS) #152

Open

lrasmus mentioned this issue May 11, 2022

R 4.2 not working StatTag/StatTag#20

Open

nhirschey mentioned this issue Jun 21, 2022

[Windows only] RProvider doesn't work with R > 4.0.2 fslaborg/RProvider#224

Open

nhirschey mentioned this issue Dec 1, 2022

HELP : package or namespace load failed for 'stats' in inDL(x, as.logical(local), as.logical(now), ...): #153

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

In R-4.2.0 on windows, package 'stats' in options("defaultPackages") was not found. #151

In R-4.2.0 on windows, package 'stats' in options("defaultPackages") was not found. #151

nhirschey commented Apr 25, 2022

lrasmus commented Apr 29, 2022

lrasmus commented May 3, 2022

lrasmus commented May 11, 2022

lrasmus commented May 11, 2022

lrasmus commented May 11, 2022

lrasmus commented May 11, 2022

nhirschey commented May 12, 2022

lrasmus commented May 13, 2022

lrasmus commented May 13, 2022 •

edited

lrasmus commented May 13, 2022

gkapellmann commented Feb 24, 2023

robreno commented Mar 7, 2023

In R-4.2.0 on windows, package 'stats' in options("defaultPackages") was not found. #151

In R-4.2.0 on windows, package 'stats' in options("defaultPackages") was not found. #151

Comments

nhirschey commented Apr 25, 2022

Works with old R version 4.0.2

Fails with R version 4.2.0

lrasmus commented Apr 29, 2022

lrasmus commented May 3, 2022

lrasmus commented May 11, 2022

lrasmus commented May 11, 2022

lrasmus commented May 11, 2022

lrasmus commented May 11, 2022

nhirschey commented May 12, 2022

lrasmus commented May 13, 2022

lrasmus commented May 13, 2022 • edited

lrasmus commented May 13, 2022

gkapellmann commented Feb 24, 2023

robreno commented Mar 7, 2023

lrasmus commented May 13, 2022 •

edited