Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Changes in Matrix 1.6-2 breaking irlba: function 'as_cholmod_sparse' not provided by package 'Matrix' #70

Open
PeteHaitch opened this issue Nov 9, 2023 · 22 comments
Open

Changes in Matrix 1.6-2 breaking irlba: function 'as_cholmod_sparse' not provided by package 'Matrix' #70

PeteHaitch opened this issue Nov 9, 2023 · 22 comments

Comments

@PeteHaitch
Copy link

Hi @bwlewis,

Changes in Matrix 1.6-2 seem to have broken irlba::irlba() for at least some types of sparse matrices.
I'm afraid I don't understand enough of Matrix and irlba to provide much help at this stage.

Cheers,
Pete

With Matrix 1.6-2

# Example taken from help("sparseMatrix", package = "Matrix").
i <- c(1,3:8); j <- c(2,9,6:10); x <- 7 * (1:7)
A <- Matrix::sparseMatrix(i, j, x = x)

irlba::irlba(A)
#> Warning in irlba::irlba(A): You're computing too large a percentage of total
#> singular values, use a standard svd instead.
#> Error in irlba::irlba(A): function 'as_cholmod_sparse' not provided by package 'Matrix'
Session info 
sessionInfo()
#> R version 4.3.1 (2023-06-16)
#> Platform: x86_64-pc-linux-gnu (64-bit)
#> Running under: CentOS Linux 7 (Core)
#> 
#> Matrix products: default
#> BLAS:   /stornext/System/data/apps/R/R-4.3.1/lib64/R/lib/libRblas.so 
#> LAPACK: /stornext/System/data/apps/R/R-4.3.1/lib64/R/lib/libRlapack.so;  LAPACK version 3.11.0
#> 
#> locale:
#>  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
#>  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
#>  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
#>  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
#>  [9] LC_ADDRESS=C               LC_TELEPHONE=C            
#> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
#> 
#> time zone: Australia/Melbourne
#> tzcode source: system (glibc)
#> 
#> attached base packages:
#> [1] stats     graphics  grDevices utils     datasets  methods   base     
#> 
#> loaded via a namespace (and not attached):
#>  [1] digest_0.6.33     fastmap_1.1.1     Matrix_1.6-2      xfun_0.41        
#>  [5] lattice_0.21-8    glue_1.6.2        knitr_1.45        htmltools_0.5.7  
#>  [9] rmarkdown_2.25    lifecycle_1.0.4   cli_3.6.1         grid_4.3.1       
#> [13] reprex_2.0.2      withr_2.5.2       compiler_4.3.1    rstudioapi_0.15.0
#> [17] tools_4.3.1       irlba_2.3.5.1     evaluate_0.23     yaml_2.3.7       
#> [21] rlang_1.1.2       fs_1.6.3

With Matrix 1.6-1

# Example taken from help("sparseMatrix", package = "Matrix").
i <- c(1,3:8); j <- c(2,9,6:10); x <- 7 * (1:7)
A <- Matrix::sparseMatrix(i, j, x = x)

irlba::irlba(A)
#> Warning in irlba::irlba(A): You're computing too large a percentage of total
#> singular values, use a standard svd instead.
#> $d
#> [1] 49.00000 44.27189 35.00000 28.00000 21.00000
#> 
#> $u
#>               [,1]          [,2]          [,3]          [,4]          [,5]
#> [1,]  1.179612e-16 -6.418477e-17 -1.942890e-16  6.938894e-17  1.110223e-16
#> [2,]  0.000000e+00  0.000000e+00  0.000000e+00  0.000000e+00  0.000000e+00
#> [3,]  3.105155e-16  3.162278e-01 -6.245005e-17  1.110223e-16 -3.816392e-17
#> [4,] -1.734723e-16 -2.428613e-17 -3.885781e-16  5.273559e-16  1.000000e+00
#> [5,] -1.942890e-16 -2.602085e-16  9.436896e-16  1.000000e+00 -3.608225e-16
#> [6,] -2.498002e-16 -2.012279e-16 -1.000000e+00  1.054712e-15  2.775558e-17
#> [7,]  9.419548e-16  9.486833e-01 -1.318390e-16  2.914335e-16 -1.734723e-17
#> [8,]  1.000000e+00 -1.334002e-15 -3.885781e-16 -3.053113e-16  8.326673e-17
#> 
#> $v
#>                [,1]          [,2]          [,3]          [,4]          [,5]
#>  [1,] -6.938894e-17 -1.474515e-16 -1.665335e-16 -1.387779e-16  0.000000e+00
#>  [2,] -9.714451e-17 -9.020562e-17  2.220446e-16  5.551115e-17  1.665335e-16
#>  [3,]  6.071532e-18  1.192622e-17  2.081668e-17  1.040834e-17  1.734723e-18
#>  [4,]  1.127570e-17  1.474515e-17  3.122502e-17  2.775558e-17 -1.040834e-17
#>  [5,] -1.994932e-17 -4.770490e-17 -4.857226e-17 -1.734723e-17 -1.040834e-17
#>  [6,] -7.632783e-17  5.898060e-17 -3.330669e-16  6.938894e-16  1.000000e+00
#>  [7,] -8.326673e-17 -5.898060e-17  1.249001e-15  1.000000e+00 -3.053113e-16
#>  [8,] -3.816392e-16 -1.734723e-16 -1.000000e+00  9.714451e-16  2.775558e-17
#>  [9,]  1.065120e-15  1.000000e+00 -1.474515e-16  2.029626e-16 -1.908196e-16
#> [10,]  1.000000e+00 -1.234689e-15 -5.134781e-16  2.012279e-16  2.463307e-16
#> 
#> $iter
#> [1] 1
#> 
#> $mprod
#> [1] 16
Session info 
sessionInfo()
#> R version 4.3.1 (2023-06-16)
#> Platform: x86_64-pc-linux-gnu (64-bit)
#> Running under: CentOS Linux 7 (Core)
#> 
#> Matrix products: default
#> BLAS:   /stornext/System/data/apps/R/R-4.3.1/lib64/R/lib/libRblas.so 
#> LAPACK: /stornext/System/data/apps/R/R-4.3.1/lib64/R/lib/libRlapack.so;  LAPACK version 3.11.0
#> 
#> locale:
#>  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
#>  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
#>  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
#>  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
#>  [9] LC_ADDRESS=C               LC_TELEPHONE=C            
#> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
#> 
#> time zone: Australia/Melbourne
#> tzcode source: system (glibc)
#> 
#> attached base packages:
#> [1] stats     graphics  grDevices utils     datasets  methods   base     
#> 
#> loaded via a namespace (and not attached):
#>  [1] digest_0.6.33     fastmap_1.1.1     Matrix_1.6-1      xfun_0.41        
#>  [5] lattice_0.21-8    glue_1.6.2        knitr_1.45        htmltools_0.5.7  
#>  [9] rmarkdown_2.25    lifecycle_1.0.4   cli_3.6.1         grid_4.3.1       
#> [13] reprex_2.0.2      withr_2.5.2       compiler_4.3.1    rstudioapi_0.15.0
#> [17] tools_4.3.1       irlba_2.3.5.1     evaluate_0.23     yaml_2.3.7       
#> [21] rlang_1.1.2       fs_1.6.3
@PeteHaitch PeteHaitch changed the title Changes in **Matrix** 1.6-2 breaking **irlba**: function 'as_cholmod_sparse' not provided by package 'Matrix' Changes in Matrix 1.6-2 breaking irlba: function 'as_cholmod_sparse' not provided by package 'Matrix' Nov 9, 2023
@Michael-Geuenich Michael-Geuenich mentioned this issue Nov 9, 2023
Open
@frac2738 frac2738 mentioned this issue Nov 9, 2023
Open
@bwlewis
Copy link
Owner

bwlewis commented Nov 9, 2023 via email

@PeteHaitch
Copy link
Author

PeteHaitch commented Nov 9, 2023 •
edited

Re-installing irlba from source after upgrading to Matrix v1.6-2 seems to have resolved the error.

# Example taken from help("sparseMatrix", package = "Matrix").
i <- c(1,3:8); j <- c(2,9,6:10); x <- 7 * (1:7)
A <- Matrix::sparseMatrix(i, j, x = x)

irlba::irlba(A)
#> Warning in irlba::irlba(A): You're computing too large a percentage of total
#> singular values, use a standard svd instead.
#> $d
#> [1] 49.00000 44.27189 35.00000 28.00000 21.00000
#> 
#> $u
#>               [,1]          [,2]          [,3]          [,4]          [,5]
#> [1,] -4.076600e-17 -1.873501e-16 -1.387779e-17 -2.775558e-17  1.387779e-16
#> [2,]  0.000000e+00  0.000000e+00  0.000000e+00  0.000000e+00  0.000000e+00
#> [3,]  8.023096e-18 -3.162278e-01 -1.942890e-16 -1.873501e-16  1.040834e-16
#> [4,]  3.469447e-18 -1.665335e-16  7.216450e-16  1.426637e-14 -1.000000e+00
#> [5,] -3.122502e-17 -1.249001e-16 -4.440892e-16  1.000000e+00  1.459943e-14
#> [6,]  1.301043e-16  5.689893e-16 -1.000000e+00 -4.440892e-16 -8.881784e-16
#> [7,] -6.938894e-18 -9.486833e-01 -4.857226e-16 -5.273559e-16  8.326673e-17
#> [8,]  1.000000e+00 -1.656661e-16  1.578598e-16 -2.081668e-17 -5.551115e-17
#> 
#> $v
#>                [,1]          [,2]          [,3]          [,4]          [,5]
#>  [1,] -7.047314e-18  1.214306e-17  3.469447e-18  6.938894e-18  6.938894e-18
#>  [2,]  2.081668e-17 -9.020562e-17  3.053113e-16  1.110223e-16  1.110223e-16
#>  [3,]  8.890458e-18 -1.214306e-17  1.387779e-17 -3.469447e-18 -5.551115e-17
#>  [4,] -4.250073e-17  5.204170e-17 -3.469447e-17  6.938894e-18  0.000000e+00
#>  [5,] -2.222614e-17  3.122502e-17 -1.734723e-17 -3.469447e-18  1.387779e-17
#>  [6,] -3.729655e-17 -5.551115e-17  1.054712e-15  1.915135e-14 -1.000000e+00
#>  [7,] -1.175275e-16 -9.714451e-17 -6.522560e-16  1.000000e+00  1.948441e-14
#>  [8,] -2.211772e-17  3.920475e-16 -1.000000e+00 -7.563394e-16 -1.415534e-15
#>  [9,]  3.642919e-17 -1.000000e+00 -8.187895e-16 -3.677614e-16  2.012279e-16
#> [10,]  1.000000e+00 -5.204170e-18  1.600282e-16  5.854692e-17 -8.847090e-17
#> 
#> $iter
#> [1] 1
#> 
#> $mprod
#> [1] 16
Session info 
sessionInfo()
#> R version 4.3.1 (2023-06-16)
#> Platform: x86_64-pc-linux-gnu (64-bit)
#> Running under: CentOS Linux 7 (Core)
#> 
#> Matrix products: default
#> BLAS:   /stornext/System/data/apps/R/R-4.3.1/lib64/R/lib/libRblas.so 
#> LAPACK: /stornext/System/data/apps/R/R-4.3.1/lib64/R/lib/libRlapack.so;  LAPACK version 3.11.0
#> 
#> locale:
#>  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
#>  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
#>  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
#>  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
#>  [9] LC_ADDRESS=C               LC_TELEPHONE=C            
#> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
#> 
#> time zone: Australia/Melbourne
#> tzcode source: system (glibc)
#> 
#> attached base packages:
#> [1] stats     graphics  grDevices utils     datasets  methods   base     
#> 
#> loaded via a namespace (and not attached):
#>  [1] digest_0.6.33     fastmap_1.1.1     Matrix_1.6-2      xfun_0.41        
#>  [5] lattice_0.22-5    glue_1.6.2        knitr_1.45        htmltools_0.5.7  
#>  [9] rmarkdown_2.25    lifecycle_1.0.4   cli_3.6.1         grid_4.3.1       
#> [13] reprex_2.0.2      withr_2.5.2       compiler_4.3.1    rstudioapi_0.15.0
#> [17] tools_4.3.1       irlba_2.3.5.1     evaluate_0.23     yaml_2.3.7       
#> [21] rlang_1.1.2       fs_1.6.3

I never fully understand when packages have to be re-built/re-installed when dependencies like Matrix update.
But if I had to guess, perhaps these remaps (https://github.com/cran/Matrix/blob/323f4d9b0722ba6f61682874a8d92ebdbf93d8fa/inst/include/Matrix/remap.h) are picked up when irlba is re-installed from source.

@bwlewis
Copy link
Owner

bwlewis commented Nov 9, 2023 via email

@PeteHaitch
Copy link
Author

And a bit more info now posted by the Matrix developers: https://stat.ethz.ch/pipermail/r-package-devel/2023q4/010051.html

@hpages
Copy link

hpages commented Nov 16, 2023 •
edited

Thanks @PeteHaitch.

@bwlewis Note that it's enough for you to bump irlba version and resubmit to CRAN. The new version will then propagate to the user machine which will trigger a reinstallation.

The problem affects any user who installed irlba before Matrix 1.6-2 got into the wild and landed on their machine. Unfortunately the info posted by the Matrix developers won't help them, and it's not realistic to expect that the thousands of users in that situation reinstall irlba manually. Most users will see the error in a context involving many packages and won't have a clue what package is actually the culprit. This is the case for Bioconductor users where 40+ Bioconductor packages are affected indirectly via irlba.

However, if a new version of irlba shows up on CRAN, then the problem will be solved for all the users and system admins that run update.packages() on a regular basis.

Thanks,
H.

@hpages hpages mentioned this issue Nov 17, 2023
Closed
@cyang-2014 cyang-2014 mentioned this issue Nov 18, 2023
Open
@kvittingseerup
Copy link

Is there any update on how to solve this problem?

@alanocallaghan alanocallaghan mentioned this issue Nov 21, 2023
Closed
@lambdamoses lambdamoses mentioned this issue Nov 22, 2023
Closed
@plger plger mentioned this issue Nov 24, 2023
Open
@sfplab
Copy link

sfplab commented Nov 26, 2023

I tried reinstalling irlba, but the issue persists with Matrix 1.6-3. Major issue for anyone running UMAP.

@PeteHaitch
Copy link
Author

@sfplab Did you re-install both from source?
This is what worked for me:

install.packages("Matrix", type = "source")
install.packages("irlba", type = "source")

@jlmelville jlmelville mentioned this issue Nov 27, 2023
Open
@sfplab
Copy link

sfplab commented Nov 27, 2023 •
edited

Hi @PeteHaitch - I hadn't installed from source, because it threw an error on my M1:

ld: warning: search path '/opt/R/arm64/gfortran/lib/gcc/aarch64-apple-darwin20.2.0/11.0.0' not found
ld: warning: search path '/opt/R/arm64/gfortran/lib' not found
ld: library 'gfortran' not found

After going down some rabbit holes, I finally found this page on CRAN - downloading the gnu fortran package at the top of the page solved my issues installing Matrix and irlba from source:
https://cran.r-project.org/bin/macosx/tools/

Leaving this info here in case anyone else has issues with fortran-dependent source install in R. And yes, RunUMAP calls to Matrix & irlba now work as expected. Thank you, @PeteHaitch!

@PeteHaitch
Copy link
Author

I use an M1 and I know that there's a lot of conflicting advice online re macOS, arm64 and R.
I try to follow the advice of Simon Urbanek (R/CRAN's macOS maintainer) because that's what's officially supported and works well for me.

@kvittingseerup
Copy link

I have an R package that depends on irlba working - is there any solution on the horizon that does not require all my users to install from source?

@bwlewis
Copy link
Owner

bwlewis commented Nov 28, 2023 via email

@SunShiFu86 SunShiFu86 mentioned this issue Dec 5, 2023
Open
@mluzvercesi mluzvercesi mentioned this issue Dec 12, 2023
Closed
@zijunmeng
Copy link

reinstall Matrix version = 1.6_1.1 can solve this problem.

@EmilHvitfeldt EmilHvitfeldt mentioned this issue Jan 10, 2024
Merged
@lassedochreden lassedochreden mentioned this issue Jan 12, 2024
Closed
@qingmemng
Copy link

qingmemng commented Jan 16, 2024 •
edited

Is there any solutions now ? I meet the same problem and it does not work even I changed Matrix version changes to 1.6-1.1.
seurat v 4.4.0 seuratobject v4.1.4
The below is the code.

scRNA1 <- RunUMAP(scRNA1, reduction = "pca", dims = 1:20)

Warning: The default method for RunUMAP has changed from calling Python UMAP via reticulate to the R-native UWOT using the cosine metric
To use Python UMAP via reticulate, set umap.method to 'umap-learn' and metric to 'correlation'
This message will be shown once per session
09:40:18 UMAP embedding parameters a = 0.9922 b = 1.112
09:40:18 Read 12279 rows and found 20 numeric columns
09:40:18 Using Annoy for neighbor search, n_neighbors = 30
09:40:18 Building Annoy index with metric = cosine, n_trees = 50
0% 10 20 30 40 50 60 70 80 90 100%
[----|----|----|----|----|----|----|----|----|----|
**************************************************|
09:40:19 Writing NN index file to temp file C:\Users\Admini\AppData\Local\Temp\RtmpiCBaL8\filed0c4d6a7319
09:40:19 Searching Annoy index using 12 threads, search_k = 3000
09:40:19 Annoy recall = 100%
09:40:20 Commencing smooth kNN distance calibration using 12 threads with target n_neighbors = 30
09:40:20 Initializing from normalized Laplacian + noise (using irlba)
Error in irlba::irlba(L, nv = n, nu = 0, maxit = iters) :
function 'sexp_as_cholmod_sparse' not provided by package 'Matrix'

@hpages
Copy link

hpages commented Jan 16, 2024

@bwlewis Any ETA for a new version of irlba on CRAN? Just to be clear, you don't need to make any change to the package, only bump its version and resubmit to CRAN. Thanks!

@liuyifang
Copy link

I am using mac M3, after install gfortran-12.2-universal.pkg
then I can install

install.packages("Matrix", type = "source")
install.packages("irlba", type = "source")

@sreichl sreichl mentioned this issue Jan 21, 2024
Closed
@jiajic jiajic mentioned this issue Feb 7, 2024
Closed
@rfriedman22 rfriedman22 mentioned this issue Feb 9, 2024
Open
@HaidongDeng
Copy link

I am using mac M3, after install gfortran-12.2-universal.pkg then I can install

install.packages("Matrix", type = "source")
install.packages("irlba", type = "source")

I also met the same problem and your method solved it, thanks!

@nigiord nigiord mentioned this issue Feb 27, 2024
Open
@evanbiederstedt evanbiederstedt mentioned this issue Mar 9, 2024
Closed
@shnsk1995
Copy link

I have tried installing Matrix and irlba packages from source after installing gfortran-12.2-universal.pkg as suggested. However, I get the following error while trying to install Matrix package. Here is the screenshot. Can anyone help please?

image

@shnsk1995
Copy link

Installing gfortran-ARM-13.2-Sonoma.dmg for Sonoma 14 macOS made it possible for me to install the Matrix and irlba packages from source and this in turn solved the error "function 'as_cholmod_sparse' not provided by package 'Matrix'".

I have tried installing Matrix and irlba packages from source after installing gfortran-12.2-universal.pkg as suggested. However, I get the following error while trying to install Matrix package. Here is the screenshot. Can anyone help please?

image

Thank you all!

@shwetha141 shwetha141 mentioned this issue Apr 11, 2024
Closed
@lgeistlinger lgeistlinger mentioned this issue Apr 18, 2024
Closed
@hpages
Copy link

hpages commented Apr 19, 2024

@VitorAguiar Your comment is very misleading! Nobody said that you had to install Matrix v1.6_1.1 to fix the "as_cholmod_sparse" issue. If your version of Matrix is >= 1.6.2, all you need to do to fix this issue is reinstall irlba from source. And that does solve the problem.

Also why would you install an archived version of Matrix? Current version is 1.6-5 and this is what everybody uses so it should work with Seurat. The fact that Seurat v4.1.1 required at least v1.6-3 doesn't mean that you must install this exact version.

Anyways it's unfortunate that 5 months have passed and irlba version still didn't get bumped (with resubmission to CRAN) as this would have solved the problem once for all, and by doing so avoided a lot of confusion and frustration.

@VitorAguiar
Copy link

VitorAguiar commented Apr 19, 2024 •
edited

@hpages I was just sharing what worked for me. Your suggestion to reinstall irlba from source and current version of Matrix did not solve the problem for me. That's why I had to install an archived version of Matrix, one that was new enough to work with Seurat. Since you feel that my comment was misleading, I am deleting it.

@hpages
Copy link

hpages commented Apr 19, 2024 •
edited

@VitorAguiar The breaking change in Matrix happened when they went from version 1.6.1 to 1.6.2. So why installing Matrix 1.6.5 (current version) vs 1.6-3 would make any difference as far as the "as_cholmod_sparse" issue is concerned? Therefore when you say that

install.packages("Matrix", type = "source")
install.packages("irlba", type = "source")

didn't fix the problem for you, but that specifically installing Matrix 1.6.3 did solve it, I suspect that something else was going on in your case. And since we don't have the full story (i.e. output of commands + sessionInfo()) I guess we'll never know.

In any case it is very very unlikely that installing Matrix 1.6.3 instead of the current Matrix was necessary. Just wanted to clarify this in case other people seeing your comment might think this is a good idea .

@eds35016 eds35016 mentioned this issue Jun 5, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests