packages that use modules with classes don't work unless Rcpp is attached

[s-u]

Packages that use Rcpp modules which expose C++ classes work only if Rcpp is explicitly present on the search path. This has been described, but mis-identified in #118.

To replicate, create a skeleton package via Rcpp::Rcpp.package.skeleton(module=TRUE), build, install and run:

> library(anRpackage)
Warning message:
 class "C++Object" is defined (with package slot ‘Rcpp’) but no metadata object found to revise subclass information---not exported?  Making a copy in package ‘anRpackage’ 
> World$new()
Error in .Object$initialize(...) : 
  could not find function "cpp_object_initializer"

Note that the skeleton uses Imports: Rcpp which is crucial here. If Rcpp is explicitly attached, it works:

> library(Rcpp)
> library(anRpackage)
> World$new()
C++ object <0x1136d40> of class 'World' <0xa6c8b0>

This turns out to be a major problem for the following reason: A package can try to work around this problem by using Depends: Rcpp, but that only applies if the package is also attached itself. This means that the "I don't work if not attached" property becomes contagious in that it bubbles through the package dependency chain - if a package has ever even one ancestor in the dependency chain that is a package with Rcpp module then it itself becomes unusable unless Rcpp attached - so in theory all packages must depend on Rcpp - whether they use it or not. Even if desirable ;), it's not practical, unfortunately.

[eddelbuettel]

Thanks for the detailed analysis. But I am wondering how this fools R CMD check for packages that use Imports: and not Depends:. If I understand you correctly these should now work. Yet they do (eg RcppCNPy). Any idea?

[eddelbuettel]

I guess the regression test does not use module=TRUE :-/

[s-u]

It doesn't if the package actually tests module calls. In the template example replace

\examples{
\dontrun{
rcpp_hello_world()
}
}

with

\examples{
rcpp_hello_world()
World$new()
}

and you'll get

* checking examples ... ERROR
Running examples in ‘anRpackage-Ex.R’ failed
The error most likely occurred in:

> ### Name: rcpp_hello_world
> ### Title: Simple function using Rcpp
> ### Aliases: rcpp_hello_world
> 
> ### ** Examples
> 
> rcpp_hello_world()
[[1]]
[1] "foo" "bar"

[[2]]
[1] 0 1

> World$new()
Error in .Object$initialize(...) : 
  could not find function "cpp_object_initializer"
Calls: <Anonymous> ... <Anonymous> -> initialize -> initialize -> <Anonymous>
Execution halted

As for RcppCNPy, AFAIK it doesn't wrap a class, the module only exports functions - that may be the triggering point. I'm not an Rcpp guru, so I don't know enough about the internals, but the C++Object wrapper class may have something to do with it - i.e. the issue may exist only when a C++ class is reflected to R in the module.

[eddelbuettel]

If we add one more line to NAMESPACE, and (as usual) load the methods package, things are better:

edd@max:/tmp$ Rscript -e 'print(search()); library(methods); library(anRpackage); World$new()'
[1] ".GlobalEnv"        "package:stats"     "package:graphics"  "package:grDevices" "package:utils"     "package:datasets"  "Autoloads"
[8] "package:base"
Warning message:
class "C++Object" is defined (with package slot ‘Rcpp’) but no metadata object found to revise subclass information---not exported?  Making a copy in package ‘anRpackage’
C++ object <0x1da3500> of class 'World' <0x210e530>
edd@max:/tmp$ 

Given that there was an explicit warning, I just added this one:

edd@max:/tmp$ tail -1 anRpackage/NAMESPACE 
importFrom(Rcpp, evalCpp, cpp_object_initializer)
edd@max:/tmp$ 

[s-u]

Ok, thanks, importing cpp_object_initializer (explicitly or implicitly) makes this go away for that package.

Unfortunately, I suspect there is something deeper involved - when there is one extra level of dependency, the error returns (that is the case for my original problem that triggered me to look into this where I have gitgist imports guitar imports Rcpp and class calls triggered from gitgists will fail). I suspect that it has to do with this warning:

Warning message:
class "C++Object" is defined (with package slot ‘Rcpp’) but no metadata object found to revise subclass information---not exported?  Making a copy in package ‘.GlobalEnv’ 

because it means the scope of the class may the wrong and it will be only looking the the search path instead of using the package as it should. I'll try to cook up a minimal example...

[eddelbuettel]

Fair enough re RcppCNPy, but RcppBDT is a better, more complete example with classes.

The whole issue may be related to the 'no linking' init step, and rather hard/impossible to debug.

[s-u]

I had a peek at the init and I think it's possible to trace, but will take substantial effort to do it - this may be related to the delayed module registration or some such. I don't have enough time to dig deeper right now, but I need to get this fixed before our next RCloud release so I'll put it on my ToDo list and keep you posted.

[eddelbuettel]

That would be very, very welcome. Much appreciated.

[s-u]

I can confirm that this is essentially a scoping issue. As far as I can tell (and this is to be taken with some grain of salt since the Rcpp module code contains a lot of voodoo that I cannot claim to fully understand) the problem is how modules are registered - since they are populated in the package's namespace by hand, the S4 class metadata is missing in the package's namespace (see ls(asNamespace(..), all=TRUE) - . This is what the warning above is about - normally, the class metadata would be updated in the package's namespace, but since it's missing S4 falls back to updating in the the workspace (see ls(all=TRUE)) - a bad idea here. This means that instead of package namespace, the global environment is used for the S4 metadata - which only works if everything is attached since this bypasses the regular namespace chain. This is what you get for using S4 ;). The proper fix would be to make sure the metadata is populated in the package namespace when loading the modules.

[eddelbuettel]

Thanks for the follow-up and analysis. Modules code is indeed close to voodoo. So maybe this is an issue to be documented ("you currently need to attach Rcpp when using modules with classes") and to be tabled for a potential future improvement?

[s-u]

Well, the problem with "you currently need to attach Rcpp when using modules with classes" is that it is transitive, so it's more like "if you ever import an Rcpp package with modules then you and anyone using your package has to attach Rcpp" - which is not realistic. Note that the import issue is just a symptom of the deeper issue of defining classes in the wrong place - so I'd still maintain this is a bug that needs fixing. The issue is finding whoever knows the module registration voodoo so it can be fixed :)

[ha0ye]

FWIW, I resolved the warning in my own package by making documentation for each of the S4 classes associated with each Rcpp module. A bit of a hack, but it removes the need to attach Rcpp.

[eddelbuettel]

Ok, For reference, do you have a link or example?

[ha0ye]

Yep, see https://github.com/ha0ye/rEDM