New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[GraphBolt][Bug] SEGV when preprocessing OnDiskDataset
#7364
Comments
could you make sure the |
Yes, the node ids in the edge file are consecutive from 0 to |
how large is your dataset? And could you try to comment out below line? |
num_nodes = 131814559 and num_edges = 5507679822. |
oh, it's a large graph with more than 5B edges. what's your instance for running this? how much is then RAM? |
I'm running on an aliyun server with over 700GB RAM |
@yxy235 could you try to reproduce this error on |
OK |
I have tried to reproduce this, but I didn't get any errors with a random same-size graph. |
@yxy235 Could you try using this data? https://mega.nz/folder/OWBwEQQL#nfkbhC35N4aLavIpCS2Cig (the sha256 is of the decompressed |
OK. I have reproduced the error, I'm trying to debug now. |
@easypickings Could you try to change the dtype of your edge.npy to |
TBD: dgl/src/array/cpu/spmat_op_impl_coo.cc Line 749 in f0213d2
dgl/src/array/cpu/spmat_op_impl_coo.cc Line 538 in f0213d2
We should determine dtype of csr through coo.row->shape[0] rather than coo.row->dtype. If shape is bigger than MAX_INT32 and no matter coo.row->dtype is int32 or int64, we should use int64. |
🐛 Bug
To Reproduce
When trying to construct a
OnDiskDataset
with the UK-Union graph, I get segmentation fault during preprocessing. The error message is eithermunmap_chunk(): invalid pointer
ordouble free or corruption (out)
. I further locate the error comes from the following line:dgl/python/dgl/graphbolt/impl/ondisk_dataset.py
Line 97 in 1547bd9
Steps to reproduce the behavior:
execute the code:
Expected behavior
Environment
conda
,pip
, source): pipAdditional context
The text was updated successfully, but these errors were encountered: