IntegrateData

[JLi-github]

Hi,

I have a question about IntegrateData: integrate <- IntegrateData(anchorset = anchors, dims = 1:50, features.to.integrate = all_genes)

I got an error: Integrating data
Merging dataset 15 into 23 4
Extracting anchors for merged samples
Finding integration vectors
Finding integration vector weights
0% 10 20 30 40 50 60 70 80 90 100%
[----|----|----|----|----|----|----|----|----|----|
**************************************************|
Integrating data
Merging dataset 24 20 1 into 19 10 3
Extracting anchors for merged samples
Finding integration vectors
Error in .T2C(newTMat(i = c(ij1[, 1], ij2[, 1]), j = c(ij1[, 2], ij2[, :
'Calloc' could not allocate memory (298833518 of 8 bytes)

[cailing20]

Similar problems here:
One run:

Integrating data
Merging dataset 2 17 18 5 3 13 1 into 4 19 16 14 15 6 8 7
Extracting anchors for merged samples
Finding integration vectors
Error in subCsp_rows(x, i, drop = drop) : 
  Cholmod error 'problem too large' at file ../Core/cholmod_sparse.c, line 89
Calls: IntegrateData ... callGeneric -> eval -> eval -> [ -> [ -> subCsp_rows
Execution halted

another run:

Integrating data
Merging dataset 3 13 1 into 2 17 18 5
Extracting anchors for merged samples
Finding integration vectors
Error in .T2C(newTMat(i = c(ij1[, 1], ij2[, 1]), j = c(ij1[, 2], ij2[,  : 
  unable to coerce from TsparseMatrix to [CR]sparseMatrixwhen length of 'i' slot exceeds 2^31-1
Calls: IntegrateData ... FindIntegrationMatrix -> - -> - -> .Arith.Csparse -> .T2C
Execution halted

[jeraldnoble]

I have the same sparse matrix error. I do not encounter the error when using Seurat 5, but that's a workaround instead of a solution. I haven't seen the sparse matrix error in any of the Issues threads for some reason.

[cailing20]

Thank you for the suggestion! I tried Seurat 5, but still couldn't get the step finished.
Got the following error for CCA integration:

[----|----|----|----|----|----|----|----|----|----|
**************************************************|
Integrating data
Merging dataset 4 8 into 1 5 2 9 3 7
Extracting anchors for merged samples
Finding integration vectors
Error in subCsp_rows(x, i, drop = drop) : 
  Cholmod error 'problem too large' at file ../Core/cholmod_sparse.c, line 89
Calls: IntegrateData ... callGeneric -> eval -> eval -> [ -> [ -> subCsp_rows
Execution halted

And the following error for RPCA:

[----|----|----|----|----|----|----|----|----|----|
**************************************************|
Integrating data
Merging dataset 7 into 1 3 2
Extracting anchors for merged samples
Finding integration vectors
Error in .T2C(newTMat(i = c(ij1[, 1], ij2[, 1]), j = c(ij1[, 2], ij2[,  : 
  unable to coerce from TsparseMatrix to [CR]sparseMatrixwhen length of 'i' slot exceeds 2^31-1
Calls: IntegrateData ... FindIntegrationMatrix -> - -> - -> .Arith.Csparse -> .T2C
Execution halted

I guess I'll try Scanpy and Harmony next...

[BenjaminDEMAILLE]

Same

[BenjaminDEMAILLE]

try preserve.order = T

[alyenkin]

@BenjaminDEMAILLE did changing that parameter work for you?

[BenjaminDEMAILLE]

Hi I got :

error in .T2C(newTMat(i = c(ij1[, 1], ij2[, 1]), j = c(ij1[, 2], ij2[,  : 
  unable to coerce from TsparseMatrix to [CR]sparseMatrixwhen length of 'i' slot exceeds 2^31-1

How can I solve @satijalab / @mojaveazure ?

[MikedeKokkie]

any fix for this yet?

[hd4git]

Hi

Did anyone manage to fix this ? I got the same error as well after running for 2 days.

Finding integration vectors
Error in .T2C(newTMat(i = c(ij1[, 1], ij2[, 1]), j = c(ij1[, 2], ij2[,  : 
  unable to coerce from TsparseMatrix to [CR]sparseMatrixwhen length of 'i' slot exceeds 2^31-1
Calls: IntegrateData ... FindIntegrationMatrix -> - -> - -> .Arith.Csparse -> .T2C
In addition: Warning messages:
1: Attempting to merge an SCTAssay with another Assay type 
Converting all to standard Assay objects. 
2: Attempting to merge an SCTAssay with another Assay type 
Converting all to standard Assay objects. 
Execution halted

https://github.com/satijalab / @mojaveazure could you please indicate the source of error or recommend some solution to this.

Thanks !!

[sbombin]

I have the same problem. I've tried integration with CCA and RPCA, and with SCT and LogNormalize workflows. However, all methods output same error:

Extracting anchors for merged samples
Finding integration vectors
Error in .T2C(newTMat(i = c(ij1[, 1], ij2[, 1]), j = c(ij1[, 2], ij2[, :
unable to coerce from TsparseMatrix to [CR]sparseMatrixwhen length of 'i' slot exceeds 2^31-1
Calls: IntegrateData ... FindIntegrationMatrix -> - -> - -> .Arith.Csparse -> .T2C
Execution halted

[ahdee]

Hey all, the solution is that 4.3 is messing things up. Just upgrade to version 5 which is just 4.999 and it works again.

[hdhimanICL]

I tried with version 5 as well. It gives the same error.

[sbombin]

If you have more than ~7 samples, it will probably not work without the 'reference' option. I have 20 samples that I needed to integrate, and the maximum number of sample for the reference is around 7. Hope, it will help you too

[hd4git]

Thanks for your reply. But I am confused where do you mean to use the reference option ?
IntegrateData function doesn't seem to use reference option
https://satijalab.org/seurat/reference/integratedata

[hd4git]

Just found this https://satijalab.org/seurat/articles/integration_large_datasets.html which mentions the use of reference for FindIntegrationAnchors
I was using scTransform based analysis earlier. Will try this pipeline now.
Hopefully it works this time.

Thanks again for your suggestion

[philoel]

I'm getting a 404 returned when I try to open the link to integration_large_datasets.html ; could you share how you changed your code to make it work (if it worked)?

[jeraldnoble]

This might be the link now https://satijalab.org/seurat/articles/parsebio_sketch_integration

[anemartinezlarrinaga2898]

Hello Im having the same error did anyone now how to solved it?

[yamihn]

same error

[lio123-bit]

I had the same error.
When I used the "rpca", the code worked.

scRNA.anchors <- FindIntegrationAnchors(object.list = scRNAlist,anchor.features = features, reduction="rpca", dims = 1:30)
scRNA1 <- IntegrateData(anchorset = scRNA.anchors,k.weight=59, dims = 1:30)

[sc01619]

Having the same error:

Integrating data
Merging dataset 20 21 into 18 19
Extracting anchors for merged samples
Finding integration vectors
Error in .M2C(newTMat(i = c(ij1[, 1], ij2[, 1]), j = c(ij1[, 2], ij2[, :
unable to aggregate TsparseMatrix with 'i' and 'j' slots of length exceeding 2^31-1

Has anyone found a solution?

[GrigorisGeor]

Hi, we came across the same error, and it doesn't allow you to integrate the data after running the FindAnchors function if your data is too big. To solve the problem try using the functions of the newest version of seurat v5.0.
In our analysis, we split our samples into healthy and disease states and created a separate object for each category.
Then by using merge we combined the Seurat objects and proceeded with the pipeline available in Seurat for the integration.
It worked just fine for CCAIntegration and Harmony that we use.