Many packages are broken on Mac with CRAN version of Rcpp -- unless 10.11 SDK is used as on CRAN

[wch]

The CRAN version of httpuv will not build using the CRAN version of Rcpp, on Mac. Simple example:

install.packages("Rcpp")
install.packages("httpuv", type = "source")

(For more info, see rstudio/httpuv#260 (comment) and other comments in that issue.)

In that issue, I and others use the Mac system toolchain, but @kevinushey has tested with the CRAN recommended toolchain and ecountered the same problems.

I've learned that several other packages have similar problems: http://lists.r-forge.r-project.org/pipermail/rcpp-devel/2020-March/010412.html

In that discussion thread, it is claimed that these packages pass R CMD check on CRAN: http://lists.r-forge.r-project.org/pipermail/rcpp-devel/2020-March/010414.html

However, @jeroen has informed me that (A) CRAN does not have the resources to re-check downstream dependencies of new packages on Mac, and (B), it does not re-build the downstream dependencies either.

I don't know how I could confirm (A) from the public information that CRAN provides, but it is simple to confirm that (B) is true. These are the dates in https://www.r-project.org/nosvn/R.check/r-release-osx-x86_64/ :

• Rcpp_1.0.4.tgz: 2020-03-18
• httpuv_1.5.2.tgz: 2019-09-12
• dplyr_0.8.5.tgz: 2020-03-08

These dates show that Rcpp's downstream dependencies were NOT rebuilt on Mac.

So from all this, I feel fairly confident that the CRAN version of Rcpp does actually break its downstream dependencies on Mac. I know that the issue is fixed in the development version of Rcpp, and I think that this is very strong reason to release a new version of Rcpp to CRAN.

[coatless]

@wch agreed to the need to bump Rcpp to address downstream failures on macOS. I think the fix release was being held because new issues were being encountered after the 1.0.4 release. (c.f. R 3.3.x issue)

[wch]

@coatless Based on this comment, my understanding is that there is no plan for a patch release: hadley/purrrlyr#11 (comment)

[eddelbuettel]

I have been in communication with CRAN about this, and they

1. do not see breakage on macOS (on several of their sites / installations)
2. do not see a need for a patch release
   .

I actually contacted CRAN right after the first incremental dev release after 1.0.4 with a heads-up note --- but as everything appears to be fine on their end based on every communication I had with them.

That may of course change. Maybe, as you say, Simon is behind and if there is genuine breakage they too will see it and then request a new version I will most likely comply with any request of theirs.

[eddelbuettel]

This issue, I think, also fundamentally the same as #1046 which was closed by #1047.

[jjallaire]

I think if we know for sure that downstream dependencies are broken and we know that CRAN may not detect these breakages for some time it would be good to proactively release a patch.

[eddelbuettel]

Sure -- and in fact we have made four such patch release to the drat. Running the install.packages() command gets you the fourth, the other three are still in the same directory.

Details

edd@rob:~$ Rscript -e 'install.packages("Rcpp", repos="https://rcppcore.github.io/drat")'
Installing package into ‘/usr/local/lib/R/site-library’
(as ‘lib’ is unspecified)
trying URL 'https://rcppcore.github.io/drat/src/contrib/Rcpp_1.0.4.4.tar.gz'
Content type 'application/gzip' length 2710270 bytes (2.6 MB)
==================================================
downloaded 2.6 MB

* installing *source* package ‘Rcpp’ ...
** using staged installation
** libs
ccache g++ -I"/usr/share/R/include" -DNDEBUG -I../inst/include/    -fpic  -g -O3 -Wall -pipe -pedantic  -c api.cpp -o api.o
ccache g++ -I"/usr/share/R/include" -DNDEBUG -I../inst/include/    -fpic  -g -O3 -Wall -pipe -pedantic  -c attributes.cpp -o attributes.o
ccache g++ -I"/usr/share/R/include" -DNDEBUG -I../inst/include/    -fpic  -g -O3 -Wall -pipe -pedantic  -c barrier.cpp -o barrier.o
ccache g++ -I"/usr/share/R/include" -DNDEBUG -I../inst/include/    -fpic  -g -O3 -Wall -pipe -pedantic  -c date.cpp -o date.o
ccache g++ -I"/usr/share/R/include" -DNDEBUG -I../inst/include/    -fpic  -g -O3 -Wall -pipe -pedantic  -c module.cpp -o module.o
ccache g++ -I"/usr/share/R/include" -DNDEBUG -I../inst/include/    -fpic  -g -O3 -Wall -pipe -pedantic  -c rcpp_init.cpp -o rcpp_init.o
ccache g++ -Wl,-S -shared -L/usr/lib/R/lib -Wl,-Bsymbolic-functions -Wl,-z,relro -o Rcpp.so api.o attributes.o barrier.o date.o module.o rcpp_init.o -L/usr/lib/R/lib -lR
installing to /usr/local/lib/R/site-library/00LOCK-Rcpp/00new/Rcpp/libs
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
** checking absolute paths in shared objects and dynamic libraries
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (Rcpp)

The downloaded source packages are in
        ‘/tmp/RtmpIvgvlX/downloaded_packages’
edd@rob:~$ 

But I still defer to CRAN for the need of a CRAN release. So far they have not spoken up in need of one --- but in fact confirmed that everything looks ok to them.

Now, there could be a bias if in fact all macOS work is outsourced to the macOS maintainer. But we know that some of them (i.e. R Core, not just CRAN) develop on macOS too. I think it would bubble up.

[jjallaire]

Okay, but if we have better radar right now and we know that lots of R users are stuck unable to install packages I think we should be proactive. It's not about whether we defer to CRAN as I think CRAN is often notified proactively by maintainers about the need for a critical release before they manage to detect it. I think they appreciate the heads up so long as the fixes aren't trivial/spurious in terms of users affected and that they don't result from carelessness on the part of the maintainer.

Maybe the discussion then is really about the magnitude of the breakage (i.e. how many / what percent of Mac packages/users are affected). I don't have a good sense of that but perhaps others do.

[eddelbuettel]

What is preventing these users from issueing the one-line command below?

install.packages("Rcpp", repos="https://rcppcore.github.io/drat")

If the answer is "they cannot (generally) compile from source", could someone please provide me with a downloadable .tgz for the platform to support binary installations.

[jjallaire]

They have to know to issue that command in the first place. If an analyst in the wild gets a big stream of compilation errors they often are totally mystified and aren't even aware of the possibility that they could install patched versions from source (let alone where to get them, esp. if the error is from a downstream dependency of the package that failed).

[wch]

One more possible issue: If packages can't compile against CRAN Rcpp on Mac, then that also causes problems for releasing a new version of these downstream dependencies.

[eddelbuettel]

Yes. One of the first things I tried when the initial bug report came in was a rhub check on macOS for an Rcpp dependee -- it built fine. (May have been a cache artefact.) But my understanding was all this time that it a) it is not widespread and b) easy to fix via the drat. I will circle back with Vienna and mention this discussion. Thank you all for the input so far.

[langmm]

I just wanted to add that I am also experiencing this issue and would request a bump. While the temp fix to install from the drat works, it complicates both the CI builds for downstream dependencies (temporary patch that will need to be removed later, esp. for conda based builds) and the installation process for those who are new to programming/R and do not know to check Github issues, particularly in the case where Rcpp is installed as a dependency.

[coatless]

Sidenote: @langmm woah, NCSA is deploying Rcpp in production? This is awesome!

[jeroen]

It depends on which version of MacOS and xcode that you run. I have started seeing a lot of problems on MacOS 10.14 + xcode 11.3.

Here is an example log file: https://travis-ci.org/github/autobrew/autobrew.github.io/jobs/666857475 (search for: unknown type name 'uuid_t'; errors).

You can reproduce this problem on travis by using:

os: osx
osx_image: xcode11.3

This is especially unfortunate because xcode11.3 is precisely the version of toolchain that CRAN will be targeting for R 4.0. The difference is that CRAN runs a different version of MacOS than travis, so the problem may go unnoticed on the mac builder.

I think it is important that this is fixed on CRAN asap, even if the CRAN builder configuration is not affected. Many users that build packages from source will be.

[jozimck]

I am also experiencing this issue on MacOS 10.13.