[Error] "cannot open the connection [...] /usr/lib/R/library/TMB/Matrix-version'" by package TMB needed by package mmrm

[adrianolszewski]

Hi,
I tried to download and use the mmrm package, used for modelling longitudinal clinical trials.
I was able to install it, however it failed to load - as follows:

The code:

d <- data.frame(response = c(rnorm(100), rnorm(100, mean=1), rnorm(100, mean=2)),
  time = factor(rep(c("T1", "T2", "T3"), each=100)),
  ID = factor(1:300))

head(d)

library(mmrm) # lots of packages being downloaded... done

mmrm(response ~ time + cs(time | ID), data=d)

Let's check what's going on deeper:

By the way, the reported version is 0.3.7, but 0.3.11 is on the CRAN. How often the packages are updated in the webr repository? (more/less)

---

The example wouldn't converge to a solution anyway, but this example does (taken from some mmrm issues)

library(dplyr)

set.seed(0228)

d <- bind_rows(expand.grid(visit = 1:20, subject = 1:200, trt = "a") %>%
                   mutate(aval = rbeta(nrow(.), 7, 1)),
               expand.grid(visit = 1:20, subject = 201:400, trt = "b") %>%
                   mutate(aval = rbeta(nrow(.), 4, 1))) %>%
    mutate(visit = factor(visit),
           subject = factor(subject))

unique_visits <- unique(d$visit)

for (i in seq_along(unique_visits)) {
    these_rows <- which(d$visit == unique_visits[i])
    d$aval[these_rows] <- d$aval[these_rows] *
        rbinom(n = length(these_rows), size = 1, prob = 1 / as.numeric(unique_visits[i]))
}

d <- d[d$aval != 0, ]

fit1 <- mmrm(aval ~ trt + ar1(visit | subject), data = d)
summary(fit1)$coefficients

cannot complete due to this problem, yet it completed in https://livecodes.io/?template=r
which seems to be based on WebR?

also this one:

x <- structure(list(PatientId = c("A1AA2", "A1AA2", "A1AA3", "A1AA3", 
"A1AA3", "A1AA4", "A1AA4", "A1AA4", "A1AA5", "A1AA5", "A1AA5", 
"A1AA6", "A1AA6", "A1AA7", "A1AA7", "A1AA7", "A1AA8", "A1AA9", 
"A1AA9", "A1AA9", "A2AA1", "A2AA1", "A2AA2", "A2AA2", "A2AA2", 
"A2AA4", "A2AA4", "A2AA5", "A2AA6", "A2AA8", "A2AA8", "A2AA9", 
"A2A1A", "A2A11", "A2A11", "A2A12", "A2A12", "A2A12", "A2A13", 
"A2A13", "A2A14", "A2A15", "A2A16", "A2A16", "A2A17", "A2A19", 
"A2A19", "A2A2A", "A2A2A", "A2A22", "A2A22", "A2A23", "A2A23", 
"A2A23", "A2A24", "A2A24", "A2A25", "A2A25", "A2A26", "A2A26", 
"A4AA1", "A4AA1", "A4AA1", "A4AA2", "A4AA2", "A6AA1", "A6AA1", 
"A6AA1", "A6AA2", "A6AA2", "A6AA3", "A6AA3", "A6AA3", "A6AA4", 
"A6AA4", "A6AA4", "A6AA5", "A6AA5", "A6AA6", "A6AA6", "A6AA7", 
"A6AA7", "A6AA8", "A6AA8", "A7AA1", "A7AA1", "A8AA1", "A8AA1", 
"A8AA1", "A8AA2", "A8AA2", "A8AA3", "A8AA3", "A8AA3", "A8AA4", 
"A8AA4", "A8AA4", "A8AA5", "A8AA5", "A8AA5", "A8AA6", "A8AA6", 
"A8AA6", "A8AA7", "A8AA8", "A8AA8", "A8AA8", "A9AA2", "A9AA2", 
"A9AA3", "1AAA1", "1AAA1", "1AAA2", "1AAA2", "1AAA2", "1AAA3", 
"1AAA3", "13AA1", "13AA1", "13AA3", "13AA3", "13AA4", "13AA5", 
"13AA5", "13AA5", "16AA1", "16AA1", "16AA1", "16AA2", "16AA2", 
"16AA2", "17AA1", "17AA1", "18AA1", "18AA1"), Diff = c(0.78, 
0.96, 0.08, 0.05, -0.12, 0.01, -0.05, -0.01, 0.65, 0.87, 1.1, 
0.49, 0.18, -0.62, -0.51, -0.42, 0.4, 0.09, -0.01, -0.08, 0.15, 
-0.1, 0.03, 0.04, 0.17, -0.08, -0.08, -0.04, -0.12, 0.19, 0.16, 
-0.15, 0, 0.38, 0.32, 0.71, 0.56, -0.03, 0.3, 0.38, -0.43, 0.4, 
0.38, 0.3, -0.09, 0.07, -0.03, 0.63, 2.03, 0.18, 0.27, 0.18, 
0.31, 0.37, 0.54, 0.42, 0.75, 0.62, 0.23, 0.3, 0.32, 0.21, 0.43, 
0.2, -0.12, 0.18, -0.1, 0.12, 0.14, -0.12, -0.13, 0.37, 0.26, 
0.18, -0.28, 0.1, 1.28, 1.51, 0.26, 0.47, 0.11, -0.26, 0.12, 
0.11, 0.58, 0.62, -0.08, -0.22, 0.29, -0.03, -0.04, 0.78, 0.6, 
1.12, -0.31, -0.04, -0.12, 0.27, 0.38, 0.45, 0.19, 0.17, 0.08, 
-0.12, 0.95, 0.48, 0.97, -0.02, -0.02, 0.29, 0.12, 0.2, 0.36, 
0.39, 0.29, -1.14, -0.6, 0.22, 0.86, 1.53, 0.9, 1.03, -0.47, 
0.32, -0.11, 0.27, 0.3, 0.49, 0.93, 0.92, 0.84, -0.09, -0.29, 
0.08, 0.05), Timepoint = structure(c(1L, 2L, 1L, 2L, 3L, 1L, 
2L, 3L, 1L, 2L, 3L, 1L, 3L, 1L, 2L, 3L, 2L, 1L, 2L, 3L, 1L, 2L, 
1L, 2L, 3L, 1L, 2L, 3L, 1L, 2L, 3L, 3L, 1L, 1L, 3L, 1L, 2L, 3L, 
2L, 3L, 2L, 2L, 1L, 3L, 2L, 1L, 2L, 2L, 3L, 2L, 3L, 1L, 2L, 3L, 
1L, 3L, 2L, 3L, 2L, 3L, 1L, 2L, 3L, 1L, 2L, 1L, 2L, 3L, 1L, 2L, 
1L, 2L, 3L, 1L, 2L, 3L, 1L, 2L, 1L, 2L, 1L, 2L, 1L, 2L, 1L, 2L, 
1L, 2L, 3L, 1L, 2L, 1L, 2L, 3L, 1L, 2L, 3L, 1L, 2L, 3L, 1L, 2L, 
3L, 1L, 1L, 2L, 3L, 1L, 2L, 2L, 1L, 2L, 1L, 2L, 3L, 1L, 2L, 1L, 
3L, 1L, 2L, 1L, 1L, 2L, 3L, 1L, 2L, 3L, 1L, 2L, 3L, 1L, 2L, 1L, 
2L), levels = c("T1", "T2", "T3"), class = "factor")), class = "data.frame", row.names = c(NA, 
-135L), Version = 5L, Date = structure(1698663319.543, tzone = "UTC", class = c("POSIXct", 
"POSIXt")))

library(mmrm)
summary(mmrm(Diff ~ Timepoint + us(Timepoint | PatientId), data = x))

vs.

[georgestagg]

Due to restrictions in the WebAssembly environment, not all R packages work with webR. From what I can tell, it looks like the TMB package requires a system C++ compiler at runtime to work. This would mean building an entire C++ compiler suite that not only outputs Wasm bytecode, but also runs under Wasm.

It is likely that such a compiler will not be available for a long time, so I am going to close this issue as "not planned" for the time being.

[georgestagg]

I checked livecodes.io, the TMB package also fails there, with

Timing stopped at: 0 0 0.007
Error: package or namespace load failed for ‘TMB’:
 .onAttach failed in attachNamespace() for 'TMB', details:
  call: if (nzchar(cxx)) {
  error: argument is of length zero

in the JS error console.

It is possible livecodes.io worked because it's running an older version of mmrm. We'll continue to track getting that package up and running in #391.

[adrianolszewski]

Thank you for the response! I'm posting an issue on the TMB github.
Which version of TMB do you build as a web assembly?

[adrianolszewski]

Hello @georgestagg,
I just got a response from the TMB package maintainer. It's in the linked issue, but let me also cite it here:

I don't know exactly why this is happening, but this might be resolvable by changing the upstream-package-dependency check to be more self-contained along the lines suggested glmmTMB/glmmTMB#1005.

I don't know how the webassembly machinery works (does it take packages from CRAN? From r-universe? From the master/main HEAD on GitHub?), but would be willing to make a pull request along these lines if desired.

Someone who knew more about the internals of webassembly might be able to diagnose the problem better.

Would you be willing to exchange some details on this topic with @bbolker ?

[bbolker]

A couple of comments:

• I am not the maintainer of TMB (that would be Kasper Kristensen)
• while running user-specified models with TMB does require a C++ compiler at runtime, there are several packages built on TMB that use pre-compiled models and so shouldn't be subject to this constraint:

tools::package_dependencies("TMB", reverse = TRUE)
$TMB
 [1] "disaggregation"  "DLMtool"         "dsem"            "ef"             
 [5] "fishmethods"     "FRK"             "gadget3"         "GeoAdjust"      
 [9] "gllvm"           "glmmTMB"         "hmmTMB"          "LocalCop"       
[13] "marked"          "MLZ"             "mmrm"            "phylosem"       
[17] "QFASA"           "RTMB"            "SAMtool"         "sdmTMB"         
[21] "SPAS"            "SpatialGEV"      "ssdtools"        "stelfi"         
[25] "stochvolTMB"     "tmbstan"         "tramME"          "tsdistributions"
[29] "unmarked" 

Of these, TMB, RTMB would need run-time compilation of models; I suspect most of the rest wouldn't.

• I think the specific problem here (inadequate permission to reference a system file) could be overcome fairly easily, but it would be helpful to understand the wasm pipeline: for example, if I make this change on a fork of the repository, would it be fairly easy to test from there?

[bbolker]

Also, FWIW I'm a little puzzled by the if (nzchar(cxx)) error quoted here; at least at present, nzchar doesn't seem to exist anywhere in the TMB code base ... ??

[bbolker]

Quick update here, the TMB maintainer (Kasper Kristensen) has pushed a commit that removes the need to reference an external file, if anyone wants to re-test ...

[georgestagg]

Thanks for clarifying @bbolker.

if I make this change on a fork of the repository, would it be fairly easy to test from there?

Yes, I can rebuild R packages for WebAssembly from forks or branches on GitHub.

while running user-specified models with TMB does require a C++ compiler at runtime, there are several packages built on TMB that use pre-compiled models and so shouldn't be subject to this constraint

Good to know. We can work to get this up and running in webR, with the understanding that it is only this pre-compiled part of the package that can be supported. I've reopened the issue.

the TMB maintainer (Kasper Kristensen) has pushed kaskr/adcomp#392 that removes the need to reference an external file

Great! In truth, this issue was not a showstopper problem, it could be worked around by loading TMB with the webR specific option mount = FALSE:

webr::install("TMB", mount = FALSE)

Setting this to TRUE is more efficient and is now the default, but it means that the package files are read-only on the virtual filesystem, causing the above problems. The change to TMB is a good one, as it means this workaround is no longer required.

---

The other issue with TMB was that it failed to load in another way:

library(TMB)
Building example simple 
Timing stopped at: 0 0 0.007
Error: package or namespace load failed for ‘TMB’:
 .onAttach failed in attachNamespace() for 'TMB', details:
  call: webr_hook_system(command)
  error: The `system()` function is unsupported under Emscripten.

In previous versions of webR, it is this problem that manifested as the if (nzchar(cxx)) error. It comes from the attempt to use the system() command at some point during the load of TMB. In the constrained WebAssembly environment, system() is not supported.

Strangely enough, I just recompiled the latest version of TMB for WebAssembly and this particular issue is now also fixed. I'm not entirely sure when that particular fix occurred, but I'm happy to push this updated version to the webR package repository now.

---

I have now done this, and the latest TMB build now seems to load cleanly in webR 🎉

It would be good if there were some test code I could run before closing the issue as complete.

[bbolker]

There's not really anything you can do with TMB alone that doesn't require a compiler, but you could try loading and running an example one of the downstream packages that comes with precompiled code (mmrm, glmmTMB, etc.) — such as the example in the post that started this thread ...

... which seems to work fine.

[adrianolszewski]

Thank you wholeheartedly(!) both @bbolker and @georgestagg! I'm deeply grateful to both of you for fixing this!

That's the power of open source, that's the power of collaboration! Now I think the issue can be closed.
( A bonus for me: I learned some interesting technical details, so I will be a more aware user of webr in future :P )